digitalmars.D - CI seems to have regressed?
- Manu (28/28) Mar 10 So since last time I was pushing to DMD regularly (several years ago), i...
- Richard (Rikki) Andrew Cattermole (6/7) Mar 10 A lot of the tests have switches applied to them.
- Manu (26/32) Mar 10 Yup, I saw that. They need to be batched by switch group I guess...
- Richard (Rikki) Andrew Cattermole (15/43) Mar 10 s/void main()/shared static this()/
So since last time I was pushing to DMD regularly (several years ago), it seems like the CI time has inflated enormously. Maybe this happened slowly and nobody noticed, but I would say it's night-and-day difference since last time I checked in. There's like 50 build jobs. Of the main ones, the nix-ey ones take 10-15m, the mac ones take like 25m, and the windows ones are 30 minutes and counting... I wonder if any effort has been made to improve this? Just a cursory look, I noticed some low-hanging fruit; 1. Some build machines don't seem to be in an operational state by default, and I noticed apt-get or other package managers fetching at the start of the build to get the build environment in order, which took over a minute each time; so that could be an instant ~10% improvement by updating the images? 2. The DMD unit tests are most of it. Is there any reason they can't be batched? The 'compilable' tests could probably be built all-at-once to some extent? There's about 1.5 thousand of those... it'd probably be almost instant if they were batched. The 'runnable' ones could conceivably be batched too, but they all seem to have separate main() functions rather than using unittests... if there's some way to sequence the entrypoints, that's another thousand or so, and they're probably the slowest ones, since they also link and execute each one. There's another almost 2000 'fail_compilation' tests, and they seem to move through slightly faster (I guess they don't need to link), but that's still a lot. Has anyone poked at this? Are there known inhibitors?
Mar 10
On 10/03/2025 11:11 PM, Manu wrote:Has anyone poked at this? Are there known inhibitors?A lot of the tests have switches applied to them. They are not as simple as it initially appears. Also each test may have a main function. I'm sure there are some low hanging fruit, and a new test runner may be able to take advantage of that. But I suspect that this isn't an easy task.
Mar 10
On Mon, 10 Mar 2025 at 20:36, Richard (Rikki) Andrew Cattermole via Digitalmars-d <digitalmars-d puremagic.com> wrote:On 10/03/2025 11:11 PM, Manu wrote:Yup, I saw that. They need to be batched by switch group I guess... They are not as simple as it initially appears.Has anyone poked at this? Are there known inhibitors?A lot of the tests have switches applied to them.Also each test may have a main function.Obviously that design doesn't scale. Maybe put a compiler hack which takes multiple main() functions from separate modules and queues them to run sequentially... that's probably easier than re-engineering thousands of test files?I'm sure there are some low hanging fruit, and a new test runner may be able to take advantage of that. But I suspect that this isn't an easy task.This appears to be a whole new test runner already? It doesn't look familiar at all to me since last time I was pushing to DMD. (like ~2020?) But yeah, nar, it clearly needs some work. I reckon it's literally a maximum priority issue; I can say that I'm not motivated to commit to DMD like this. 'ain't no one got time for that! Am I the only one, or is it repellant to other potential contributors? I mean, like, you can't meaningfully focus on the work you're doing, because you make a tweak and then twiddle your thumbs for way too long to find out if you made a mistake... I've long since moved on to doing something else in that timeframe. Like, Nick asked me to add back-ticks to a comment... an hour later, he's long gone off and doing something else... as you would expect! We could build and run Overwatch tests in a quarter of the time (an enormous project!), and I thought that was bad then! :P ... maybe DLF have resources to direct at this, and/or I wonder if it's an obvious GSOC project? If it were my project, I would treat any resources directed to this as an investment.
Mar 10
On 10/03/2025 11:59 PM, Manu wrote:On Mon, 10 Mar 2025 at 20:36, Richard (Rikki) Andrew Cattermole via Digitalmars-d <digitalmars-d puremagic.com <mailto:digitalmars- d puremagic.com>> wrote: On 10/03/2025 11:11 PM, Manu wrote: > Has anyone poked at this? Are there known inhibitors? A lot of the tests have switches applied to them. Yup, I saw that. They need to be batched by switch group I guess... They are not as simple as it initially appears. Also each test may have a main function. Obviously that design doesn't scale. Maybe put a compiler hack which takes multiple main() functions from separate modules and queues them to run sequentially... that's probably easier than re-engineering thousands of test files?s/void main()/shared static this()/ A pretty straight forward regex to apply with the help of an IDE like IntelliJ.I'm sure there are some low hanging fruit, and a new test runner may be able to take advantage of that. But I suspect that this isn't an easy task. This appears to be a whole new test runner already? It doesn't look familiar at all to me since last time I was pushing to DMD. (like ~2020?)Not that I am aware of. My general feeling is we need our own runners. I.e. the GH runners appear to be unreliable when the US is awake, but seem to work other times. One of the unreliability issues I've recently had is related to this make file: https://github.com/dlang/dmd/blob/6c04f52a39dc498eca1a05f2f9b9837e45cb5e61/druntime/test/shared/Makefile I suspect the access denied error on 64.obj occurred due to parallelism where paths didn't account for it. The dmd testsuite had a similar issue that I fixed around a year ago. But I don't know enough about make to verify if that is the case here, let alone fix it.
Mar 10