www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - CI seems to have regressed?

reply Manu <turkeyman gmail.com> writes:
So since last time I was pushing to DMD regularly (several years ago), it
seems like the CI time has inflated enormously.
Maybe this happened slowly and nobody noticed, but I would say it's
night-and-day difference since last time I checked in.
There's like 50 build jobs. Of the main ones, the nix-ey ones take 10-15m,
the mac ones take like 25m, and the windows ones are 30 minutes and
counting...

I wonder if any effort has been made to improve this?
Just a cursory look, I noticed some low-hanging fruit;
1. Some build machines don't seem to be in an operational state by default,
and I noticed apt-get or other package managers fetching at the start of
the build to get the build environment in order, which took over a minute
each time; so that could be an instant ~10% improvement by updating the
images?
2. The DMD unit tests are most of it. Is there any reason they can't be
batched?

The 'compilable' tests could probably be built all-at-once to some extent?
There's about 1.5 thousand of those... it'd probably be almost instant if
they were batched.
The 'runnable' ones could conceivably be batched too, but they all seem to
have separate main() functions rather than using unittests... if there's
some way to sequence the entrypoints, that's another thousand or so, and
they're probably the slowest ones, since they also link and execute each
one.
There's another almost 2000 'fail_compilation' tests, and they seem to move
through slightly faster (I guess they don't need to link), but that's still
a lot.

Has anyone poked at this? Are there known inhibitors?
Mar 10
parent reply "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 10/03/2025 11:11 PM, Manu wrote:

 Has anyone poked at this? Are there known inhibitors?


A lot of the tests have switches applied to them.

They are not as simple as it initially appears.

Also each test may have a main function.

I'm sure there are some low hanging fruit, and a new test runner may be 
able to take advantage of that. But I suspect that this isn't an easy task.
Mar 10
parent reply Manu <turkeyman gmail.com> writes:
On Mon, 10 Mar 2025 at 20:36, Richard (Rikki) Andrew Cattermole via
Digitalmars-d <digitalmars-d puremagic.com> wrote:


 On 10/03/2025 11:11 PM, Manu wrote:

 Has anyone poked at this? Are there known inhibitors?


 A lot of the tests have switches applied to them.


Yup, I saw that. They need to be batched by switch group I guess...

They are not as simple as it initially appears.

 Also each test may have a main function.


Obviously that design doesn't scale.
Maybe put a compiler hack which takes multiple main() functions from
separate modules and queues them to run sequentially... that's probably
easier than re-engineering thousands of test files?



 I'm sure there are some low hanging fruit, and a new test runner may be
 able to take advantage of that. But I suspect that this isn't an easy task.



This appears to be a whole new test runner already? It doesn't look
familiar at all to me since last time I was pushing to DMD. (like ~2020?)

But yeah, nar, it clearly needs some work. I reckon it's literally a
maximum priority issue; I can say that I'm not motivated to commit to DMD
like this. 'ain't no one got time for that! Am I the only one, or is it
repellant to other potential contributors?
I mean, like, you can't meaningfully focus on the work you're doing,
because you make a tweak and then twiddle your thumbs for way too long to
find out if you made a mistake... I've long since moved on to doing
something else in that timeframe.
Like, Nick asked me to add back-ticks to a comment... an hour later, he's
long gone off and doing something else... as you would expect!
We could build and run Overwatch tests in a quarter of the time (an
enormous project!), and I thought that was bad then! :P ... maybe DLF have
resources to direct at this, and/or I wonder if it's an obvious GSOC
project?
If it were my project, I would treat any resources directed to this as an
investment.
Mar 10
parent "Richard (Rikki) Andrew Cattermole" <richard cattermole.co.nz> writes:
On 10/03/2025 11:59 PM, Manu wrote:

 On Mon, 10 Mar 2025 at 20:36, Richard (Rikki) Andrew Cattermole via 
 Digitalmars-d <digitalmars-d puremagic.com <mailto:digitalmars- 
 d puremagic.com>> wrote:
 
     On 10/03/2025 11:11 PM, Manu wrote:
      > Has anyone poked at this? Are there known inhibitors?
 
     A lot of the tests have switches applied to them.
 
 
 Yup, I saw that. They need to be batched by switch group I guess...
 
     They are not as simple as it initially appears.
 
     Also each test may have a main function.
 
 
 Obviously that design doesn't scale.
 Maybe put a compiler hack which takes multiple main() functions from 
 separate modules and queues them to run sequentially... that's probably 
 easier than re-engineering thousands of test files?


s/void main()/shared static this()/

A pretty straight forward regex to apply with the help of an IDE like 
IntelliJ.


     I'm sure there are some low hanging fruit, and a new test runner may be
     able to take advantage of that. But I suspect that this isn't an
     easy task.
 
 
 This appears to be a whole new test runner already? It doesn't look 
 familiar at all to me since last time I was pushing to DMD. (like ~2020?)


Not that I am aware of.

My general feeling is we need our own runners.

I.e. the GH runners appear to be unreliable when the US is awake, but 
seem to work other times.

One of the unreliability issues I've recently had is related to this 
make file: 
https://github.com/dlang/dmd/blob/6c04f52a39dc498eca1a05f2f9b9837e45cb5e61/druntime/test/shared/Makefile

I suspect the access denied error on 64.obj occurred due to parallelism 
where paths didn't account for it. The dmd testsuite had a similar issue 
that I fixed around a year ago. But I don't know enough about make to 
verify if that is the case here, let alone fix it.
Mar 10