www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D stability testing framework

reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
Hello all,

Having listened to Andrei and Walter's Q&A and read some of the discussion
surrounding it, an idea occurred to me.

How about leveraging the selection of 3rd-party D code out there to provide a
testing framework for D's stability as a language?

The framework would pull in a specified version of dmd, druntime and Phobos, and
build them; and then using those, attempt to pull in and build a selection of
3rd-party libraries and to run test code those libraries would provide.  It
would then report the number of build failures and runtime/unittest failures and
try to classify them according to whether they are due to a change in the
_language_, a change in the runtime, or a change in Phobos.

The idea would be to get reliable statistics on what breaking changes are
causing what degree of pain (and why) for D users.  Currently we have arguments
over how stable the language is and what kind of breakages are or aren't
acceptable.  With a good enough selection of 3rd-party code, it might be
possible to quantify the prospective impact of a change.

Ideally the test code could be provided simply by providing a list of git
repositories and branches to test.  A practical stumbling block might be
different build systems etc., so it might be necessary to have some kind of
standardized build system the testing framework could expect (e.g. a testbuild.d
script, just as some libraries currently provide a build.d script).

The goal would be to provide mutual benefit for the D language and providers of
public test code -- patches to D, druntime or Phobos could be tested to see the
extent of breakage they cause, providers of test code could get automated early
warning that a change to the D frontend, druntime or Phobos is going to impact
their project (and a framework to test patches designed to cope with those
breaking changes).  It should also be possible to run the testing framework on
one's personal machine (so for example, downstream users could test their code
against latest-off-GitHub versions of dmd, druntime and Phobos without having to
make their code publicly available).

I don't know if this idea is practical in reality -- it might be difficult to
distinguish between breakages caused by changes to D and breakages caused by
other problems such as incorrect testbuild scripts, etc.  It's also cheeky as I
definitely don't have the knowhow or time to do this myself.  But I thought I'd
throw it out there to see if it's an idea worth pursuing by someone.

Best wishes,

    -- Joe
May 23 2013
next sibling parent reply "Dicebot" <m.strashun gmail.com> writes:
Why not make one step further and make it part of current CI 
suite? And add as a separate stat table to automatic pull request 
tester. Simple and useful.
May 23 2013
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 05/23/2013 05:11 PM, Dicebot wrote:

 Why not make one step further and make it part of current CI suite? And add as
a
 separate stat table to automatic pull request tester. Simple and useful.


Obviously it could be incorporated into that -- my concern was that it might
involve too big a test run to operate on a per-patch basis.  It should still be
possible to use as a standalone framework, though, because it would be useful
not just for dmd/druntime/phobos devs to test their work but also for authors of
3rd-party libraries to test their code against latest development version of D,
without publishing their code or sending it to some centralized server.
May 23 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/23/13 10:30 AM, Joseph Rushton Wakeling wrote:

 Hello all,

 Having listened to Andrei and Walter's Q&A and read some of the discussion
 surrounding it, an idea occurred to me.

 How about leveraging the selection of 3rd-party D code out there to provide a
 testing framework for D's stability as a language?


I think that's a great idea. What would be a seed of 1-3 projects? I 
wonder how difficult it would be for Brad to set up some beta testing 
scripts on the build machines.

Andrei
May 23 2013
next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 05/23/2013 07:22 PM, Andrei Alexandrescu wrote:

 I think that's a great idea. What would be a seed of 1-3 projects? I wonder how
 difficult it would be for Brad to set up some beta testing scripts on the build
 machines.


Will happily offer my own code, though I currently only have 1 public D project
and it's probably not extensive enough to be useful.  Time to get that graph
library off the ground ... :-)

But as suggestions, how about vibe.d, GtkD and SciD?  I'd also add QtD except
that as far as I can tell, it's not being maintained right now. :-(
May 23 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/23/13 1:54 PM, Joseph Rushton Wakeling wrote:

 On 05/23/2013 07:22 PM, Andrei Alexandrescu wrote:

 I think that's a great idea. What would be a seed of 1-3 projects? I wonder how
 difficult it would be for Brad to set up some beta testing scripts on the build
 machines.


 Will happily offer my own code, though I currently only have 1 public D project
 and it's probably not extensive enough to be useful.  Time to get that graph
 library off the ground ... :-)

 But as suggestions, how about vibe.d, GtkD and SciD?  I'd also add QtD except
 that as far as I can tell, it's not being maintained right now. :-(


Sounds good. I'll discuss with Walter and Brad.

Andrei
May 23 2013
parent reply Brad Roberts <braddr puremagic.com> writes:
On 5/23/13 11:35 AM, Andrei Alexandrescu wrote:

 On 5/23/13 1:54 PM, Joseph Rushton Wakeling wrote:

 On 05/23/2013 07:22 PM, Andrei Alexandrescu wrote:

 I think that's a great idea. What would be a seed of 1-3 projects? I
 wonder how
 difficult it would be for Brad to set up some beta testing scripts on
 the build
 machines.


 Will happily offer my own code, though I currently only have 1 public
 D project
 and it's probably not extensive enough to be useful.  Time to get that
 graph
 library off the ground ... :-)

 But as suggestions, how about vibe.d, GtkD and SciD?  I'd also add QtD
 except
 that as far as I can tell, it's not being maintained right now. :-(


 Sounds good. I'll discuss with Walter and Brad.

 Andrei


I think this ought to be prototyped independent of the current systems. 
  It's got _some_ commonality.  It's also likely to be a lot of work, 
which I don't have time for.
May 23 2013
parent reply "Dicebot" <m.strashun gmail.com> writes:
On Thursday, 23 May 2013 at 18:45:52 UTC, Brad Roberts wrote:

 I think this ought to be prototyped independent of the current 
 systems.
  It's got _some_ commonality.  It's also likely to be a lot of 
 work, which I don't have time for.


If you can write a short spec on what do you need to launch this 
easily, I am happy to volunteer.
May 24 2013
parent Brad Roberts <braddr puremagic.com> writes:
On 5/24/13 1:02 AM, Dicebot wrote:

 On Thursday, 23 May 2013 at 18:45:52 UTC, Brad Roberts wrote:

 I think this ought to be prototyped independent of the current systems.
  It's got _some_ commonality.  It's also likely to be a lot of work,
 which I don't have time for.


 If you can write a short spec on what do you need to launch this easily,
 I am happy to volunteer.


That's part of the work required.. deciding what to actually do.
May 24 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-05-23 19:54, Joseph Rushton Wakeling wrote:


 Will happily offer my own code, though I currently only have 1 public D project
 and it's probably not extensive enough to be useful.  Time to get that graph
 library off the ground ... :-)

 But as suggestions, how about vibe.d, GtkD and SciD?  I'd also add QtD except
 that as far as I can tell, it's not being maintained right now. :-(


DWT and Tango are pretty large projects.

-- 
/Jacob Carlborg
May 23 2013
next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 05/23/2013 09:06 PM, Jacob Carlborg wrote:

 DWT and Tango are pretty large projects.


I'm not familiar with them and didn't know their maintenance status, otherwise
I'd have suggested both.  I was also wondering about your dvm and/or orbit ... ?
May 23 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-05-23 23:26, Joseph Rushton Wakeling wrote:


 I'm not familiar with them and didn't know their maintenance status, otherwise
 I'd have suggested both.  I was also wondering about your dvm and/or orbit ...
?


Tango and DWT is not maintained that much. Basically just to get them 
compile one the latest versions of DMD. Orbit is not ready yet. DVM is 
written in D1.

-- 
/Jacob Carlborg
May 24 2013
parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 05/24/2013 09:53 AM, Jacob Carlborg wrote:

 Tango and DWT is not maintained that much. Basically just to get them compile
 one the latest versions of DMD.


That's sufficiently maintained for the purposes outlined here, and feedback from
the test framework would be helpful in keeping them up to date in that way.
May 24 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-05-24 12:53, Joseph Rushton Wakeling wrote:


 That's sufficiently maintained for the purposes outlined here, and feedback
from
 the test framework would be helpful in keeping them up to date in that way.


Absolutely.

-- 
/Jacob Carlborg
May 24 2013
prev sibling parent reply "Kiith-Sa" <kiithsacmp gmail.com> writes:
D:YAML might be of use if needed.

There's been no release in recent past (featureset has not 
changed, which I'm still planning for a release), but I've been 
maintaining its compatibility with current DMD for a while and I 
have no intention to stop in forseeable future as I use it in 
pretty much every project I start.

It's somewhere around 10-20kLOC.

https://github.com/kiith-sa/D-YAML


Also, Derelict3 might be a good idea. Derelict has been actively 
maintained for pretty much most of D's history.
May 23 2013
parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 05/23/2013 11:39 PM, Kiith-Sa wrote:

 D:YAML might be of use if needed.
 
 Also, Derelict3 might be a good idea. Derelict has been actively maintained for
 pretty much most of D's history.


Nice thoughts. :-)  Thanks for introducing me to YAML -- I may have some use for
that at some point ...
May 23 2013
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 23 May 2013 at 17:22:31 UTC, Andrei Alexandrescu 
wrote:

 On 5/23/13 10:30 AM, Joseph Rushton Wakeling wrote:

 Hello all,

 Having listened to Andrei and Walter's Q&A and read some of 
 the discussion
 surrounding it, an idea occurred to me.

 How about leveraging the selection of 3rd-party D code out 
 there to provide a
 testing framework for D's stability as a language?


 I think that's a great idea. What would be a seed of 1-3 
 projects? I wonder how difficult it would be for Brad to set up 
 some beta testing scripts on the build machines.


I'd be happy to propose some code. The downside is that it 
require more than 2.5Gb of RAM to build, due to dmd leaking and 
bug in the separate compilation model.
May 23 2013