www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Adding more projects to the Project Tester

reply Seb <seb wilzba.ch> writes:
So learning from the recent Vibe.d regression fiasco (we 
temporarily disabled a subconfiguration in Vibe.d and promptly 
got a regression in 2.081), I think we should try to add more 
projects to the Project Tester.

The current list is here:
https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443

Any suggestions?

Why should I add my project to the Project Tester?
--------------------------------------------------

Once a project is added to the Project Tester, DMD can't regress 
on it anymore as for every PR at dmd, druntime, phobos, tools and 
dub the testsuite of the added projects are run.

How does the Project Tester work?
---------------------------------

- By default, it will run the same commands as Travis would do. 
Although, if necessary, custom commands can be used too.
- It will checkout the latest, stable git tag

Requirements
------------

- moderately popular or was prone to regressions in the past
- rather easy to build (i.e. you don't need to download and 
recompile clang)
- no flaky testsuite (random errors in the testsuite due to 
network connectivity shouldn't happen. Though there's 
`DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of 
your testsuite)
- reachable author or development (if there's ever a case where 
we need to push changes via a trivial PR to the repo, it 
shouldn't sit in the queue for weeks)
Jul 05 2018
next sibling parent reply Joakim <dlang joakim.fea.st> writes:
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:

 So learning from the recent Vibe.d regression fiasco (we 
 temporarily disabled a subconfiguration in Vibe.d and promptly 
 got a regression in 2.081), I think we should try to add more 
 projects to the Project Tester.

 The current list is here:
 https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443

 Any suggestions?

 Why should I add my project to the Project Tester?
 --------------------------------------------------

 Once a project is added to the Project Tester, DMD can't 
 regress on it anymore as for every PR at dmd, druntime, phobos, 
 tools and dub the testsuite of the added projects are run.

 How does the Project Tester work?
 ---------------------------------

 - By default, it will run the same commands as Travis would do. 
 Although, if necessary, custom commands can be used too.
 - It will checkout the latest, stable git tag

 Requirements
 ------------

 - moderately popular or was prone to regressions in the past
 - rather easy to build (i.e. you don't need to download and 
 recompile clang)
 - no flaky testsuite (random errors in the testsuite due to 
 network connectivity shouldn't happen. Though there's 
 `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts 
 of your testsuite)
 - reachable author or development (if there's ever a case where 
 we need to push changes via a trivial PR to the repo, it 
 shouldn't sit in the queue for weeks)


The LDC compiler? kinke recently had an issue because of all the 
C++ integration changes upstream:

https://github.com/ldc-developers/ldc/pull/2752#issuecomment-398897813

As perhaps the largest consumer of extern(C++), it may make sense 
to add it for all the C++ work being done. It would require the 
llvm package be pre-installed in the test environ.

List of current projects looks great, was tough to think of 
anything to add.
Jul 05 2018
next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Friday, 6 July 2018 at 05:02:56 UTC, Joakim wrote:

 The LDC compiler? kinke recently had an issue because of all 
 the C++ integration changes upstream:

 https://github.com/ldc-developers/ldc/pull/2752#issuecomment-398897813

 As perhaps the largest consumer of extern(C++), it may make 
 sense to add it for all the C++ work being done. It would 
 require the llvm package be pre-installed in the test environ.

 List of current projects looks great, was tough to think of 
 anything to add.


Thats a great idea! Especially since there is a whole lot more 
extern(C++) stuff coming in the next DMD release.
Jul 06 2018
prev sibling parent Seb <seb wilzba.ch> writes:
On Friday, 6 July 2018 at 05:02:56 UTC, Joakim wrote:

 The LDC compiler? kinke recently had an issue because of all 
 the C++ integration changes upstream:

 https://github.com/ldc-developers/ldc/pull/2752#issuecomment-398897813

 As perhaps the largest consumer of extern(C++), it may make 
 sense to add it for all the C++ work being done. It would 
 require the llvm package be pre-installed in the test environ.

 List of current projects looks great, was tough to think of 
 anything to add.


Good idea. Not sure how easy/hard it is, but let's give it a try: 
https://github.com/dlang/ci/pull/228
Jul 08 2018
prev sibling next sibling parent reply tide <tide tide.tide> writes:
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:

 So learning from the recent Vibe.d regression fiasco (we 
 temporarily disabled a subconfiguration in Vibe.d and promptly 
 got a regression in 2.081), I think we should try to add more 
 projects to the Project Tester.

 The current list is here:
 https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443

 Any suggestions?

 Why should I add my project to the Project Tester?
 --------------------------------------------------

 Once a project is added to the Project Tester, DMD can't 
 regress on it anymore as for every PR at dmd, druntime, phobos, 
 tools and dub the testsuite of the added projects are run.

 How does the Project Tester work?
 ---------------------------------

 - By default, it will run the same commands as Travis would do. 
 Although, if necessary, custom commands can be used too.
 - It will checkout the latest, stable git tag

 Requirements
 ------------

 - moderately popular or was prone to regressions in the past
 - rather easy to build (i.e. you don't need to download and 
 recompile clang)
 - no flaky testsuite (random errors in the testsuite due to 
 network connectivity shouldn't happen. Though there's 
 `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts 
 of your testsuite)
 - reachable author or development (if there's ever a case where 
 we need to push changes via a trivial PR to the repo, it 
 shouldn't sit in the queue for weeks)


Include Windows as part of the testing done.
Jul 06 2018
parent Seb <seb wilzba.ch> writes:
On Friday, 6 July 2018 at 21:25:28 UTC, tide wrote:

 On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:

 So learning from the recent Vibe.d regression fiasco (we 
 temporarily disabled a subconfiguration in Vibe.d and promptly 
 got a regression in 2.081), I think we should try to add more 
 projects to the Project Tester.

 The current list is here:
 https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443

 Any suggestions?

 Why should I add my project to the Project Tester?
 --------------------------------------------------

 Once a project is added to the Project Tester, DMD can't 
 regress on it anymore as for every PR at dmd, druntime, 
 phobos, tools and dub the testsuite of the added projects are 
 run.

 How does the Project Tester work?
 ---------------------------------

 - By default, it will run the same commands as Travis would 
 do. Although, if necessary, custom commands can be used too.
 - It will checkout the latest, stable git tag

 Requirements
 ------------

 - moderately popular or was prone to regressions in the past
 - rather easy to build (i.e. you don't need to download and 
 recompile clang)
 - no flaky testsuite (random errors in the testsuite due to 
 network connectivity shouldn't happen. Though there's 
 `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts 
 of your testsuite)
 - reachable author or development (if there's ever a case 
 where we need to push changes via a trivial PR to the repo, it 
 shouldn't sit in the queue for weeks)


 Include Windows as part of the testing done.


Just to avoid confusion, testing of DMD/Druntime/Phobos/etc. is 
done on Windows:

https://auto-tester.puremagic.com/platform-history.ghtml?projectid=1&os=Win_32
https://auto-tester.puremagic.com/platform-history.ghtml?projectid=1&os=Win_32_64
https://ci.appveyor.com/project/greenify/dmd
...

In my experience are the huge majority of all regressions 
originating in the D frontend where architecture actually doesn't 
matter so much.

That being said, we are currently investigating to move the 
Project Tester from Jenkins to Buildkite 
(https://github.com/dlang/ci/pull/225) and one advantage of this 
move is that it'll be a lot easier to add new machines to the 
build cloud (so you could then hook up that Windows machine in 
your garage with the Project Tester).
However, AFAICT many of the dub projects do have dependencies 
that are harder to install on Windows, so this might not happen 
in the near future.
Jul 08 2018
prev sibling next sibling parent reply JN <666total wp.pl> writes:
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:

 Why should I add my project to the Project Tester?
 --------------------------------------------------

 Once a project is added to the Project Tester, DMD can't 
 regress on it anymore as for every PR at dmd, druntime, phobos, 
 tools and dub the testsuite of the added projects are run.


By the way, is there any policy for outdated dub packages? There 
are some packages in dub registry, which don't build with the 
latest D version anymore. Should such packages stay in the 
registry? It gives a bit inflated view of packages, seems like 
there are many, but 50% (for example) aren't even usable without 
additional effort.
Jul 06 2018
parent reply Basile B. <b2.temp gmx.com> writes:
On Friday, 6 July 2018 at 21:47:34 UTC, JN wrote:

 On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:

 Why should I add my project to the Project Tester?
 --------------------------------------------------

 Once a project is added to the Project Tester, DMD can't 
 regress on it anymore as for every PR at dmd, druntime, 
 phobos, tools and dub the testsuite of the added projects are 
 run.


 By the way, is there any policy for outdated dub packages?


You just found an idea for the score algorithm.
Jul 06 2018
parent reply Timoses <timosesu gmail.com> writes:
On Friday, 6 July 2018 at 23:56:01 UTC, Basile B. wrote:

 On Friday, 6 July 2018 at 21:47:34 UTC, JN wrote:

 By the way, is there any policy for outdated dub packages?


 You just found an idea for the score algorithm.


Why isn't there something like "compiler compatibility" in a dub 
config file? E.g. currently the vibe.d project lists the compiler 
versions the code is compatible with [1].
Wouldn't a field in dub like

"compilerCompatibility":
{
     "dmd": "2.080.0",
     "ldc": "..."
}

be nice? This could also help code.dlang.org to show 
compatibility of libraries/packages.

The only pain I guess would be to keep such a list up to date.

[1]: https://github.com/vibe-d/vibe.d#support
Jul 07 2018
parent Seb <seb wilzba.ch> writes:
On Saturday, 7 July 2018 at 08:04:47 UTC, Timoses wrote:

 On Friday, 6 July 2018 at 23:56:01 UTC, Basile B. wrote:

 On Friday, 6 July 2018 at 21:47:34 UTC, JN wrote:

 By the way, is there any policy for outdated dub packages?


 You just found an idea for the score algorithm.


 Why isn't there something like "compiler compatibility" in a 
 dub config file? E.g. currently the vibe.d project lists the 
 compiler versions the code is compatible with [1].
 Wouldn't a field in dub like

 "compilerCompatibility":
 {
     "dmd": "2.080.0",
     "ldc": "..."
 }

 be nice? This could also help code.dlang.org to show 
 compatibility of libraries/packages.

 The only pain I guess would be to keep such a list up to date.

 [1]: https://github.com/vibe-d/vibe.d#support


I'm not sure it's a good idea to manually hard-code such a list 
into the dub.json.
However, for every release or so, the dub registry could crawl 
all its packages and run `dub test` on them. Of course, there's 
the chance that a build still fails (e.g. due to missing 
dependencies) even though it would pass locally, but each package 
that passes with the latest registry could receive additional 
points.
Jul 08 2018
prev sibling next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:

 Requirements
 ------------

 - moderately popular or was prone to regressions in the past
 - rather easy to build (i.e. you don't need to download and 
 recompile clang)
 - no flaky testsuite (random errors in the testsuite due to 
 network connectivity shouldn't happen. Though there's 
 `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts 
 of your testsuite)
 - reachable author or development (if there's ever a case where 
 we need to push changes via a trivial PR to the repo, it 
 shouldn't sit in the queue for weeks)


Please include the intel-intrinsics package, its test expose a 
regression on 2.081 that I haven't looked after yet. The goal is 
to provide a _stable and usable_ SIMD interface.
Jul 07 2018
parent Seb <seb wilzba.ch> writes:
On Saturday, 7 July 2018 at 16:29:15 UTC, Guillaume Piolat wrote:

 Please include the intel-intrinsics package, its test expose a 
 regression on 2.081 that I haven't looked after yet. The goal 
 is to provide a _stable and usable_ SIMD interface.


https://github.com/dlang/ci/pull/224
Jul 07 2018
prev sibling next sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Friday, 6 July 2018 at 03:19:44 UTC, Seb wrote:

 So learning from the recent Vibe.d regression fiasco (we 
 temporarily disabled a subconfiguration in Vibe.d and promptly 
 got a regression in 2.081), I think we should try to add more 
 projects to the Project Tester.


A meta-problem is that many package unittest (at least on typical 
CI) don't exercise the DMD optimizer since they don't run with -O 
or -release. Dub has no default options to dub test with 
optimizations so ti's very rarely tested.
Jul 07 2018
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, 5 July 2018 21:19:44 MDT Seb via Digitalmars-d wrote:

 So learning from the recent Vibe.d regression fiasco (we
 temporarily disabled a subconfiguration in Vibe.d and promptly
 got a regression in 2.081), I think we should try to add more
 projects to the Project Tester.

 The current list is here:
 https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy#L443

 Any suggestions?

 Why should I add my project to the Project Tester?
 --------------------------------------------------

 Once a project is added to the Project Tester, DMD can't regress
 on it anymore as for every PR at dmd, druntime, phobos, tools and
 dub the testsuite of the added projects are run.

 How does the Project Tester work?
 ---------------------------------

 - By default, it will run the same commands as Travis would do.
 Although, if necessary, custom commands can be used too.
 - It will checkout the latest, stable git tag

 Requirements
 ------------

 - moderately popular or was prone to regressions in the past
 - rather easy to build (i.e. you don't need to download and
 recompile clang)
 - no flaky testsuite (random errors in the testsuite due to
 network connectivity shouldn't happen. Though there's
 `DETERMINISTIC_HINT=1` set, s.t. you could disable such parts of
 your testsuite)
 - reachable author or development (if there's ever a case where
 we need to push changes via a trivial PR to the repo, it
 shouldn't sit in the queue for weeks)


dxml should probably be on the list. Assuming that this is only on *nix and
not Windows (I have yet to write a test script for Windows), then ./test.sh
will run the tests without -release, and ./testRelease.sh will run the tests
with -release (simply running dub test won't do anything useful, because all
of the unittest blocks are versioned to avoid problems when projects
depending on dxml compile their tests).

- Jonathan M Davis
Jul 08 2018
parent reply Seb <seb wilzba.ch> writes:
On Monday, 9 July 2018 at 05:58:17 UTC, Jonathan M Davis wrote:

 On Thursday, 5 July 2018 21:19:44 MDT Seb via Digitalmars-d 
 wrote:

 [...]


 dxml should probably be on the list.


Yes!
https://github.com/dlang/ci/pull/230


 Assuming that this is only on *nix and not Windows


Yes.


 (I have yet to write a test script for Windows)


This might help: https://github.com/Abscissa/AppVeyor-D


 then ./test.sh will run the tests without -release, and 
 ./testRelease.sh will run the tests with -release (simply 
 running dub test won't do anything useful, because all of the 
 unittest blocks are versioned to avoid problems when projects 
 depending on dxml compile their tests).


The Project Tester will by default look at (if existent) the 
.travis.yml and run the same 'script' commands, so it should do 
the right thing, but I will keep this in mind. Thanks for the 
explanation!
Jul 09 2018
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, 9 July 2018 02:44:31 MDT Seb via Digitalmars-d wrote:

 On Monday, 9 July 2018 at 05:58:17 UTC, Jonathan M Davis wrote:

 then ./test.sh will run the tests without -release, and
 ./testRelease.sh will run the tests with -release (simply
 running dub test won't do anything useful, because all of the
 unittest blocks are versioned to avoid problems when projects
 depending on dxml compile their tests).


 The Project Tester will by default look at (if existent) the
 .travis.yml and run the same 'script' commands, so it should do
 the right thing, but I will keep this in mind. Thanks for the
 explanation!



Ah. Then I will have to figure out what needs to go in .travis.yml and add
it.

- Jonathan M Davis
Jul 09 2018