www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - CI buildbots

reply Manu <turkeyman gmail.com> writes:
This CI situation with the DMD/druntime repos is not okay.
It takes ages... **hours** sometimes, for CI to complete.
It's all this 'auto-tester' one, which seems to lock up on the last few tests.

This makes DMD is a rather unenjoyable project to contribute to.
I had a sudden burst of inspiration, but it's very rapidly wearing off.
May 20 2018
next sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:

 This CI situation with the DMD/druntime repos is not okay.
 It takes ages... **hours** sometimes, for CI to complete.
 It's all this 'auto-tester' one, which seems to lock up on the 
 last few tests.

 This makes DMD is a rather unenjoyable project to contribute to.
 I had a sudden burst of inspiration, but it's very rapidly 
 wearing off.


It might take hours for CI to complete, but it can take weeks or 
months for someone to review your code...so the CI time doesn't 
really seem to matter for myself.  That is unless you're trying 
to use the CI in your modify/test development cycle.  However, 
that's should be solvable by testing locally in most cases.
May 21 2018
parent reply Manu <turkeyman gmail.com> writes:
On 21 May 2018 at 09:22, Jonathan Marler via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:

 This CI situation with the DMD/druntime repos is not okay.
 It takes ages... **hours** sometimes, for CI to complete.
 It's all this 'auto-tester' one, which seems to lock up on the last few
 tests.

 This makes DMD is a rather unenjoyable project to contribute to.
 I had a sudden burst of inspiration, but it's very rapidly wearing off.



 It might take hours for CI to complete, but it can take weeks or months for
 someone to review your code...so the CI time doesn't really seem to matter
 for myself.  That is unless you're trying to use the CI in your modify/test
 development cycle.  However, that's should be solvable by testing locally in
 most cases.


I use CI to test the platforms I don't build locally. That's natural
for cross-platform development.
May 21 2018
parent Jonathan Marler <johnnymarler gmail.com> writes:
On Monday, 21 May 2018 at 22:21:34 UTC, Manu wrote:

 On 21 May 2018 at 09:22, Jonathan Marler via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:

 On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:

 This CI situation with the DMD/druntime repos is not okay.
 It takes ages... **hours** sometimes, for CI to complete.
 It's all this 'auto-tester' one, which seems to lock up on 
 the last few
 tests.

 This makes DMD is a rather unenjoyable project to contribute 
 to.
 I had a sudden burst of inspiration, but it's very rapidly 
 wearing off.



 It might take hours for CI to complete, but it can take weeks 
 or months for someone to review your code...so the CI time 
 doesn't really seem to matter for myself.  That is unless 
 you're trying to use the CI in your modify/test development 
 cycle.  However, that's should be solvable by testing locally 
 in most cases.


 I use CI to test the platforms I don't build locally. That's 
 natural for cross-platform development.


Ah I see.  Well for me it seemed worth the effort to setup at 
least a windows and linux machine which covers most testing.  The 
worst was when we were having intermittent seg faults on 32-bit 
OSx...I eventually borrowed my girlfriends macbook to reproduce 
and debug that one...ugh that was a pain.  In any case I'd 
recommend having one posix and one windows platform.  There's 
always VMs if you need em :)

But to address your original concern, I'm not sure that 
decreasing the testing required to integrate changes is 
necessarily a net positive.  If you can decrease test time while 
maintaining the same coverage then by all means, let's do that!  
But I think in general you have to find a balance between the two.

Since you are using CI for your modify/build/test development 
cycle, one idea would be to define some sort of interface that 
the CI's could use to limit what they are testing for a 
particular PR.  You could have it search through the comments for 
something like "limit test to dmd runnable" or something like 
that.
May 22 2018
prev sibling parent reply Seb <seb wilzba.ch> writes:
On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:

 This CI situation with the DMD/druntime repos is not okay.
 It takes ages... **hours** sometimes, for CI to complete.
 It's all this 'auto-tester' one, which seems to lock up on the 
 last few tests.

 This makes DMD is a rather unenjoyable project to contribute to.
 I had a sudden burst of inspiration, but it's very rapidly 
 wearing off.


As mentioned on GitHub, running the compile+fail testsuite 
locally takes 10 seconds.
Typically a PR doesn't touch much cross-platform stuff and if it 
does, you should get a negative reply pretty early from any CI.
If you use CIs to detect merge conflicts, they will also occur 
locally.

As explained on GitHub auto-tester gives priority to PRs with the 
auto-merge label and will constantly invalidate old builds 
whenever something got merged, so it typically never completes 
for a PR.
OTOH if your PR has been approved, it will have priority access 
and normally it will be merged quite quickly.

That being said, if you have ideas on how to improve the 
ecosystem, please let us/me know (except for adding new machines 
to the auto-tester - that's something that seems to be out of our 
reach).
May 23 2018
parent reply Manu <turkeyman gmail.com> writes:
On 23 May 2018 at 07:09, Seb via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 21 May 2018 at 04:46:15 UTC, Manu wrote:

 This CI situation with the DMD/druntime repos is not okay.
 It takes ages... **hours** sometimes, for CI to complete.
 It's all this 'auto-tester' one, which seems to lock up on the last few
 tests.

 This makes DMD is a rather unenjoyable project to contribute to.
 I had a sudden burst of inspiration, but it's very rapidly wearing off.



 As mentioned on GitHub, running the compile+fail testsuite locally takes 10
 seconds.
 Typically a PR doesn't touch much cross-platform stuff and if it does, you
 should get a negative reply pretty early from any CI.
 If you use CIs to detect merge conflicts, they will also occur locally.

 As explained on GitHub auto-tester gives priority to PRs with the auto-merge
 label and will constantly invalidate old builds whenever something got
 merged, so it typically never completes for a PR.
 OTOH if your PR has been approved, it will have priority access and normally
 it will be merged quite quickly.

 That being said, if you have ideas on how to improve the ecosystem, please
 let us/me know (except for adding new machines to the auto-tester - that's
 something that seems to be out of our reach).


I'm not suggesting adding new machines... I'm suggesting removing the
ones that take ~50 minutes.
I think they're a let loss for the pipeline. A <10 minute machine
could finish up 4 other jobs AND finish mine in the same amount of
time. It practise it's likely to get to mine a lot sooner.
A single machine in the pipeline that takes 50 minutes makes the
pipeline take at least 50 minutes.

Latency > throughput, the pipeline would be better without those 4
machines. (win-farm-1, win-farm-2, bellevue, inglebrook)

I guess the flow I observed on the weekend, where it took me 3 days to
get my batch of PR's merged, is that people reviewed it once it was
already green... so that's at least 50 minutes at the end of my actual
work, THEN they ask me to add `package` or `const` somewhere, so I do,
and that's another hour before they have another look to confirm the
tweak, and then they probably got bored of reviewing PR's in the last
2-3 hours, and went to bed, or to work, or to get lunch or something,
so they reappear the next day.
So in practise, adding 2-3 hours to the cycles adds 24 hours to the
cycle. The latency was the problem for all of my 5-6 patches, not the
throughput.

Maybe those 4 machines could be assigned a rule, where they're only
assigned jobs to re-test PR's with updated DMD's or whatever if the PR
has been silent for >24 hours or something... so they are assigned to
low-frequency jobs, and never assigned to fresh jobs?
May 23 2018
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 5/23/18 4:51 PM, Manu wrote:

 I'm not suggesting adding new machines... I'm suggesting removing the
 ones that take ~50 minutes.


I think this thought has merit. Or at least delegate them to only test 
older PRs.


 I guess the flow I observed on the weekend, where it took me 3 days to
 get my batch of PR's merged, is that people reviewed it once it was
 already green... so that's at least 50 minutes at the end of my actual
 work, THEN they ask me to add `package` or `const` somewhere, so I do,


I don't think everyone does this, and I'd be surprised at any who do, 
with the caveat that if a PR is for a specific platform, they may wait 
to ensure the PR passes that platform before approving.

My workflow is to FIRST look at the PR and see if it looks good, then if 
needed, take a look at the auto tester. Hell, I often times approve a PR 
with an obvious syntax error that I didn't notice, only to see the auto 
tester fail later.

So it was probably a coincidence that they asked for the changes after 
the tests were complete.


 and that's another hour before they have another look to confirm the
 tweak, and then they probably got bored of reviewing PR's in the last
 2-3 hours, and went to bed, or to work, or to get lunch or something,
 so they reappear the next day.
 So in practise, adding 2-3 hours to the cycles adds 24 hours to the
 cycle. The latency was the problem for all of my 5-6 patches, not the
 throughput.


I'm going to level with you, not everyone is on Manu's schedule to get 
things done this weekend. Sorry. This is open-source/volunteer workflow. 
I would be more likely to believe it's just people are going on their 
own schedule when they can donate their time. I submitted a PR at the 
hackathon, and it was approved as I was flying home. Still not merged 
(2+ weeks later). I don't think it has anything to do with the tester.

If these were DMD PRs, you are in for latency that has nothing to do 
with test machines -- there just aren't a lot of people who are 
qualified to or feel comfortable with reviewing PRs.

And ping people who have reviewed, or who you think might be inclined to 
merge. Sqeaky wheels and all that.


 Maybe those 4 machines could be assigned a rule, where they're only
 assigned jobs to re-test PR's with updated DMD's or whatever if the PR
 has been silent for >24 hours or something... so they are assigned to
 low-frequency jobs, and never assigned to fresh jobs? 


Yes, I think this idea makes sense. It's really up to Brad though.

-Steve
May 23 2018