www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Continuous Integration

reply Jason House <jason.james.house gmail.com> writes:
Does anyone have a setup for doing continuous integration with d?
Mar 25 2008
next sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Jason House wrote:

 Does anyone have a setup for doing continuous integration with d?


I'm going to start on that now, since DUnit (latest svn) can output 
vaguely junit-compatible xml. It'll be using CruiseControl. By tomorrow, 
I'll be able to tell you when you should start bothering me about it in 
earnest.

Of course, this will do you little good if you don't use DUnit or some 
other unittest runner besides the default one, since you just get a 
binary value for whether your tests pass, along with, if you're lucky, a 
stack trace. If you're not lucky, you just get the words "Segmentation 
fault".

Well, okay, you could have a static constructor that outputs the first 
portion of the junit xml file, a couple of templates and functions that 
output more xml for each unittest, and your unittest main would output 
the end of the junit xml, or something like that. But at that point, you 
might as well just use DUnit.
Mar 25 2008
next sibling parent reply Jason House <jason.james.house gmail.com> writes:
Christopher Wright Wrote:


 Jason House wrote:

 Does anyone have a setup for doing continuous integration with d?


 
 I'm going to start on that now, since DUnit (latest svn) can output 
 vaguely junit-compatible xml. It'll be using CruiseControl. By tomorrow, 
 I'll be able to tell you when you should start bothering me about it in 
 earnest.
 
 Of course, this will do you little good if you don't use DUnit or some 
 other unittest runner besides the default one, since you just get a 
 binary value for whether your tests pass, along with, if you're lucky, a 
 stack trace. If you're not lucky, you just get the words "Segmentation 
 fault".
 
 Well, okay, you could have a static constructor that outputs the first 
 portion of the junit xml file, a couple of templates and functions that 
 output more xml for each unittest, and your unittest main would output 
 the end of the junit xml, or something like that. But at that point, you 
 might as well just use DUnit.


I'm sensitive to per-unittest code-writing overhead, but already have some in
my code. If DUnit offers comparable overhead, and enhanced functionality, I'll
switch.
Mar 26 2008
parent reply Christopher Wright <dhasenan gmail.com> writes:
Jason House wrote:

 I'm sensitive to per-unittest code-writing overhead, but already have some in
my code. If DUnit offers comparable overhead, and enhanced functionality, I'll
switch.


DUnit uses test fixture classes, so it probably won't suit you at 
present. I'll look into a method with less overhead.
Mar 26 2008
parent reply Jason House <jason.james.house gmail.com> writes:
Christopher Wright wrote:


 Jason House wrote:

 I'm sensitive to per-unittest code-writing overhead, but already have
 some in my code. If DUnit offers comparable overhead, and enhanced
 functionality, I'll switch.


 
 DUnit uses test fixture classes, so it probably won't suit you at
 present. I'll look into a method with less overhead.


Hmmm... maybe :(

Would this be possible?

unittest{
  mixin DUnitTest!("My test name",
  {
    ...// my code
  });
}


My limit is probably something near:

unittest{
  mixin startTest!("My test name");
  try{
    ... // my code
  catch{
    mixin catchTest;
  }
}


I'd like to stay with the D style of unit tests.  Maybe I should check what
enhancement requests are in there.  It'd be nice to be able to hook in a
unit test handler...
Mar 26 2008
next sibling parent reply Sean Kelly <sean invisibleduck.org> writes:
== Quote from Jason House (jason.james.house gmail.com)'s article

 I'd like to stay with the D style of unit tests.  Maybe I should check what
 enhancement requests are in there.  It'd be nice to be able to hook in a
 unit test handler...


Tango has one.  Look at "moduleUnitTester" here:

http://www.dsource.org/projects/tango/docs/current/tango.core.Runtime.html


Sean
Mar 26 2008
parent reply Jason House <jason.james.house gmail.com> writes:
Sean Kelly wrote:


 == Quote from Jason House (jason.james.house gmail.com)'s article

 I'd like to stay with the D style of unit tests.  Maybe I should check
 what
 enhancement requests are in there.  It'd be nice to be able to hook in a
 unit test handler...


 
 Tango has one.  Look at "moduleUnitTester" here:
 
 http://www.dsource.org/projects/tango/docs/current/tango.core.Runtime.html
 
 
 Sean


See http://d.puremagic.com/issues/show_bug.cgi?id=1952 for the enhancement
request.  Per unit test customization (such as a unit test name) is very
useful to have.  I also don't see a way to tell the program to run all unit
tests (and just report the individual passes/failures)
Mar 26 2008
next sibling parent reply Lars Ivar Igesund <larsivar igesund.net> writes:
Jason House Wrote:


 Sean Kelly wrote:
 

 == Quote from Jason House (jason.james.house gmail.com)'s article

 I'd like to stay with the D style of unit tests.  Maybe I should check
 what
 enhancement requests are in there.  It'd be nice to be able to hook in a
 unit test handler...


 
 Tango has one.  Look at "moduleUnitTester" here:
 
 http://www.dsource.org/projects/tango/docs/current/tango.core.Runtime.html
 
 
 Sean


 
 See http://d.puremagic.com/issues/show_bug.cgi?id=1952 for the enhancement
 request.  Per unit test customization (such as a unit test name) is very
 useful to have.  I also don't see a way to tell the program to run all unit
 tests (and just report the individual passes/failures)


Here is a simple program that uses the above hook  - note that you need to link
one of the stacktrace libs (jive on linux) to actually get stacktraces printed
on exceptions, not just assert line numbers. Also, I think it will be unable to
continue after a segfault.:

module myunittestrunner;

import tango.io.Stdout;
import tango.core.Runtime;

bool tangoUnitTester()
{
    uint count = 0;
    Stdout ("NOTE: This is still fairly rudimentary, and will only report
the").newline;
    Stdout ("    first error per module.").newline;
    foreach ( m; ModuleInfo ) // moduleinfo array
    {
        if ( m.unitTest) {
            Stdout.format ("{}. Executing unittests in '{}' ", count, m.name);
            try {
               m.unitTest();
            }
            catch (Exception e) {
                Stdout(" - Unittest failed.").newline;
                Stdout.format("   File '{}', line '{}'.", e.file,
e.line).newline;
                Stdout.format("     Message is : '{}'", e.msg).newline;
                if (e.info)
                    Stdout.format("     TraceInfo: {}",
e.info.toString).newline;
                continue;
            }
            Stdout(" - Success.").newline;
            count++;
        }
    }
    return true;
}

static this() {
    Runtime.moduleUnitTester( &tangoUnitTester );
}

void main() {}
Mar 27 2008
parent reply Jason House <jason.james.house gmail.com> writes:
Lars Ivar Igesund Wrote:


 Jason House Wrote:
 

 Sean Kelly wrote:
 

 == Quote from Jason House (jason.james.house gmail.com)'s article

 I'd like to stay with the D style of unit tests.  Maybe I should check
 what
 enhancement requests are in there.  It'd be nice to be able to hook in a
 unit test handler...


 
 Tango has one.  Look at "moduleUnitTester" here:
 
 http://www.dsource.org/projects/tango/docs/current/tango.core.Runtime.html
 
 
 Sean


 
 See http://d.puremagic.com/issues/show_bug.cgi?id=1952 for the enhancement
 request.  Per unit test customization (such as a unit test name) is very
 useful to have.  I also don't see a way to tell the program to run all unit
 tests (and just report the individual passes/failures)


 
 Here is a simple program that uses the above hook  - note that you need to
link one of the stacktrace libs (jive on linux) to actually get stacktraces
printed on exceptions, not just assert line numbers. Also, I think it will be
unable to continue after a segfault.:


Hmmm...  There's some black magic in your sample.  What's "ModuleInfo"?  Where
is it documented?  The code looks like it does not run unit tests one at a
time, but I'll wait for real analysis of the code until I know what's going on.
Mar 27 2008
parent reply Sean Kelly <sean invisibleduck.org> writes:
== Quote from Jason House (jason.james.house gmail.com)'s article

 Lars Ivar Igesund Wrote:

 Jason House Wrote:


 Sean Kelly wrote:


 == Quote from Jason House (jason.james.house gmail.com)'s article

 I'd like to stay with the D style of unit tests.  Maybe I should check
 what
 enhancement requests are in there.  It'd be nice to be able to hook in a
 unit test handler...


 Tango has one.  Look at "moduleUnitTester" here:

 http://www.dsource.org/projects/tango/docs/current/tango.core.Runtime.html


 Sean


 See http://d.puremagic.com/issues/show_bug.cgi?id=1952 for the enhancement
 request.  Per unit test customization (such as a unit test name) is very
 useful to have.  I also don't see a way to tell the program to run all unit
 tests (and just report the individual passes/failures)


 Here is a simple program that uses the above hook  - note that you need to
link one of the stacktrace libs (jive on linux) to actually




get stacktraces printed on exceptions, not just assert line numbers. Also, I
think it will be unable to continue after a segfault.:

 Hmmm...  There's some black magic in your sample.  What's "ModuleInfo"?  Where
is it documented?


ModuleInfo is declared in the global object.di in Tango, and it isn't
documented anywhere.  Sorry about that.  I need to look into
documenting what's in object.di.


 The code looks like it does not run unit tests one at a time, but I'll wait
for real analysis of the code until I know what's going on.


Unfortunately, the compiler lumps all the unit tests in a particular module
together.  I would prefer if they were split out into a per-
module list however, and this would have to happen for your enhancement request
to go through.


Sean
Mar 27 2008
parent reply Jason House <jason.james.house gmail.com> writes:
Sean Kelly wrote:


 == Quote from Jason House (jason.james.house gmail.com)'s article

 Lars Ivar Igesund Wrote:

 Jason House Wrote:


 Sean Kelly wrote:


 == Quote from Jason House (jason.james.house gmail.com)'s article

 I'd like to stay with the D style of unit tests.  Maybe I should
 check what
 enhancement requests are in there.  It'd be nice to be able to
 hook in a unit test handler...


 Tango has one.  Look at "moduleUnitTester" here:










http://www.dsource.org/projects/tango/docs/current/tango.core.Runtime.html

 Sean


 See http://d.puremagic.com/issues/show_bug.cgi?id=1952 for the
 enhancement
 request.  Per unit test customization (such as a unit test name) is
 very
 useful to have.  I also don't see a way to tell the program to run
 all unit tests (and just report the individual passes/failures)


 Here is a simple program that uses the above hook  - note that you need
 to link one of the stacktrace libs (jive on linux) to actually




 get stacktraces printed on exceptions, not just assert line numbers. Also,
 I think it will be unable to continue after a segfault.:

 Hmmm...  There's some black magic in your sample.  What's "ModuleInfo"? 
 Where is it documented?


 
 ModuleInfo is declared in the global object.di in Tango, and it isn't
 documented anywhere.  Sorry about that.  I need to look into documenting
 what's in object.di.
 

 The code looks like it does not run unit tests one at a time, but I'll
 wait for real analysis of the code until I know what's going on.


 
 Unfortunately, the compiler lumps all the unit tests in a particular
 module together.  I would prefer if they were split out into a per- module
 list however, and this would have to happen for your enhancement request
 to go through.
 
 
 Sean


Please, please add the extra detail to the request!  How should the compiler
do it?
Mar 27 2008
parent reply Sean Kelly <sean invisibleduck.org> writes:
== Quote from Jason House (jason.james.house gmail.com)'s article

 Sean Kelly wrote:

 == Quote from Jason House (jason.james.house gmail.com)'s article

 Lars Ivar Igesund Wrote:

 Jason House Wrote:


 Sean Kelly wrote:


 == Quote from Jason House (jason.james.house gmail.com)'s article

 I'd like to stay with the D style of unit tests.  Maybe I should
 check what
 enhancement requests are in there.  It'd be nice to be able to
 hook in a unit test handler...


 Tango has one.  Look at "moduleUnitTester" here:










 http://www.dsource.org/projects/tango/docs/current/tango.core.Runtime.html

 Sean


 See http://d.puremagic.com/issues/show_bug.cgi?id=1952 for the
 enhancement
 request.  Per unit test customization (such as a unit test name) is
 very
 useful to have.  I also don't see a way to tell the program to run
 all unit tests (and just report the individual passes/failures)


 Here is a simple program that uses the above hook  - note that you need
 to link one of the stacktrace libs (jive on linux) to actually




 get stacktraces printed on exceptions, not just assert line numbers. Also,
 I think it will be unable to continue after a segfault.:

 Hmmm...  There's some black magic in your sample.  What's "ModuleInfo"?
 Where is it documented?


 ModuleInfo is declared in the global object.di in Tango, and it isn't
 documented anywhere.  Sorry about that.  I need to look into documenting
 what's in object.di.


 The code looks like it does not run unit tests one at a time, but I'll
 wait for real analysis of the code until I know what's going on.


 Unfortunately, the compiler lumps all the unit tests in a particular
 module together.  I would prefer if they were split out into a per- module
 list however, and this would have to happen for your enhancement request
 to go through.


 Sean


 Please, please add the extra detail to the request!  How should the compiler
 do it?


I'd like ModuleInfo to get something vaguely like this:

alias void function() UnitTest;
struct NamedPair { char[] name; UnitTest test; }
NamedPair[] unitTests;

Then executing the unit tests for a module could be done on a per-test basis,
with the names you suggested visible as well.  Of course, this means even more
meta-data per module, but it would make unittest more flexible so...


Sean
Mar 27 2008
parent Jason House <jason.james.house gmail.com> writes:
Sean Kelly wrote:


 == Quote from Jason House (jason.james.house gmail.com)'s article

 Sean Kelly wrote:

 == Quote from Jason House (jason.james.house gmail.com)'s article

 Lars Ivar Igesund Wrote:

 Jason House Wrote:


 Sean Kelly wrote:


 == Quote from Jason House (jason.james.house gmail.com)'s
 article

 I'd like to stay with the D style of unit tests.  Maybe I
 should check what
 enhancement requests are in there.  It'd be nice to be able to
 hook in a unit test handler...


 Tango has one.  Look at "moduleUnitTester" here:














http://www.dsource.org/projects/tango/docs/current/tango.core.Runtime.html

 Sean


 See http://d.puremagic.com/issues/show_bug.cgi?id=1952 for the
 enhancement
 request.  Per unit test customization (such as a unit test name)
 is very
 useful to have.  I also don't see a way to tell the program to run
 all unit tests (and just report the individual passes/failures)


 Here is a simple program that uses the above hook  - note that you
 need to link one of the stacktrace libs (jive on linux) to actually




 get stacktraces printed on exceptions, not just assert line numbers.
 Also, I think it will be unable to continue after a segfault.:

 Hmmm...  There's some black magic in your sample.  What's
 "ModuleInfo"? Where is it documented?


 ModuleInfo is declared in the global object.di in Tango, and it isn't
 documented anywhere.  Sorry about that.  I need to look into
 documenting what's in object.di.


 The code looks like it does not run unit tests one at a time, but I'll
 wait for real analysis of the code until I know what's going on.


 Unfortunately, the compiler lumps all the unit tests in a particular
 module together.  I would prefer if they were split out into a per-
 module list however, and this would have to happen for your enhancement
 request to go through.


 Sean


 Please, please add the extra detail to the request!  How should the
 compiler do it?


 
 I'd like ModuleInfo to get something vaguely like this:
 
 alias void function() UnitTest;
 struct NamedPair { char[] name; UnitTest test; }
 NamedPair[] unitTests;
 
 Then executing the unit tests for a module could be done on a per-test
 basis,
 with the names you suggested visible as well.  Of course, this means even
 more meta-data per module, but it would make unittest more flexible so...
 
 
 Sean


I was hoping those kind of thoughts would go into the feature request ;)
Mar 27 2008
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
== Quote from Jason House (jason.james.house gmail.com)'s article

 Sean Kelly wrote:

 == Quote from Jason House (jason.james.house gmail.com)'s article

 I'd like to stay with the D style of unit tests.  Maybe I should check
 what
 enhancement requests are in there.  It'd be nice to be able to hook in a
 unit test handler...


 Tango has one.  Look at "moduleUnitTester" here:

 http://www.dsource.org/projects/tango/docs/current/tango.core.Runtime.html


 See http://d.puremagic.com/issues/show_bug.cgi?id=1952 for the enhancement
 request.  Per unit test customization (such as a unit test name) is very
 useful to have.  I also don't see a way to tell the program to run all unit
 tests (and just report the individual passes/failures)


Yes, there is no way to name the built-in unit tests.  That would be a nice
feature to have, so long as
the name were optional.  I've been asking for unittest blocks to get a more
function-like syntax anyway,
to match invariant in D 2.0, so this provides a good reason for the syntax.  As
for running all unit tests,
simply have your unit tester perform each unit test in a try block/catch block.
 If an exception is thrown
then the test failed, otherwise it succeeded.


Sean
Mar 27 2008
prev sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Jason House wrote:

 Christopher Wright wrote:
 

 Jason House wrote:

 I'm sensitive to per-unittest code-writing overhead, but already have
 some in my code. If DUnit offers comparable overhead, and enhanced
 functionality, I'll switch.


 DUnit uses test fixture classes, so it probably won't suit you at
 present. I'll look into a method with less overhead.


 
 Hmmm... maybe :(
 
 Would this be possible?
 
 unittest{
   mixin DUnitTest!("My test name",
   {
     ...// my code
   });
 }


Yes. I initially had some issues with alias parameters and delegates, 
but those seem to have disappeared. You wouldn't want to put that in a 
unittest block, though, since it's redundant and would give me, and by 
extension you, less control.

You'd also need to either version out your main function or call 
dunit_main from yours when running tests.
Mar 26 2008
parent reply Jason House <jason.james.house gmail.com> writes:
Christopher Wright wrote:


 Jason House wrote:

 Christopher Wright wrote:
 

 Jason House wrote:

 I'm sensitive to per-unittest code-writing overhead, but already have
 some in my code. If DUnit offers comparable overhead, and enhanced
 functionality, I'll switch.


 DUnit uses test fixture classes, so it probably won't suit you at
 present. I'll look into a method with less overhead.


 
 Hmmm... maybe :(
 
 Would this be possible?
 
 unittest{
   mixin DUnitTest!("My test name",
   {
     ...// my code
   });
 }


 
 Yes. 



Cool :)



 I initially had some issues with alias parameters and delegates, 
 but those seem to have disappeared. You wouldn't want to put that in a
 unittest block, though, since it's redundant and would give me, and by
 extension you, less control.


Huh?  Can you explain?  Would I have less control than DUnit currently
provides or less control thant just a hand made unittest{...}?  In either
case, how severe of a gap are you envisioning?  (So far, D unittest blocks
are my only real unit testing experience.  Work will eventually have me
using NUnit, but that hasn't happened yet)



 You'd also need to either version out your main function or call
 dunit_main from yours when running tests.


I'm totally ok with that!  One-time overhead at a program level isn't too
bad.  Most programs that really use unit tests are moderate or large in
size anyway.  It's the per-test overhead that would really irk me as I code
away.
Mar 27 2008
parent Christopher Wright <dhasenan gmail.com> writes:
Jason House wrote:

 Christopher Wright wrote:
 

 Jason House wrote:

 Christopher Wright wrote:


 Jason House wrote:

 I'm sensitive to per-unittest code-writing overhead, but already have
 some in my code. If DUnit offers comparable overhead, and enhanced
 functionality, I'll switch.


 DUnit uses test fixture classes, so it probably won't suit you at
 present. I'll look into a method with less overhead.


 Hmmm... maybe :(

 Would this be possible?

 unittest{
   mixin DUnitTest!("My test name",
   {
     ...// my code
   });
 }


 Yes. 


 
 
 Cool :)
 
 

 I initially had some issues with alias parameters and delegates, 
 but those seem to have disappeared. You wouldn't want to put that in a
 unittest block, though, since it's redundant and would give me, and by
 extension you, less control.


 
 Huh?  Can you explain?  Would I have less control than DUnit currently
 provides or less control thant just a hand made unittest{...}?  In either
 case, how severe of a gap are you envisioning?  (So far, D unittest blocks
 are my only real unit testing experience.  Work will eventually have me
 using NUnit, but that hasn't happened yet)


Currently DUnit does a lot of work with static constructors, and I'm not 
sure how they interact with unittest blocks.

If DUnit used unittest blocks instead of delegates and static 
constructors, that would be less control, since you'd only get two 
benefits: named tests and continuing testing after one failure. Well, 
three, actually; I could also do some aggregate reports and such, if you 
used a main() that I supplied.

If I add some feature later that lets you specify which tests to run via 
commandline options, those wouldn't be available with a unittest{} 
wrapper. Anything that affects the actual running of the tests and not 
the accounting.


 You'd also need to either version out your main function or call
 dunit_main from yours when running tests.


 
 I'm totally ok with that!  One-time overhead at a program level isn't too
 bad.  Most programs that really use unit tests are moderate or large in
 size anyway.  It's the per-test overhead that would really irk me as I code
 away.


Of course, you can make editor macros to take most of the pain away.
Mar 27 2008
prev sibling parent Christopher Wright <dhasenan gmail.com> writes:
Christopher Wright wrote:

 Jason House wrote:

 Does anyone have a setup for doing continuous integration with d?


 
 I'm going to start on that now, since DUnit (latest svn) can output 
 vaguely junit-compatible xml. It'll be using CruiseControl. By tomorrow, 
 I'll be able to tell you when you should start bothering me about it in 
 earnest.


Okay, assuming you are using dsss, you can simply use:

<schedule interval="60">
     <exec workingdir="projects/${project.name}" 
command="/usr/local/bin/dsss" args="build" timeout="5000" />
</schedule>

If you have a binary that includes your tests, you can change that to:

<schedule interval="60">
     <composite>
         <exec workingdir="projects/${project.name}" 
command="/usr/local/bin/dsss" args="build" timeout="5000" />
         <exec workingdir="projects/${project.name}" command="./binary" 
args="some-args" />
     </composite>
</schedule>

If you don't use <composite>, well, that's entirely allowed, but only 
the first element will be executed.

I don't have xml output for dunit working correctly with cruisecontrol yet.
Mar 26 2008
prev sibling parent Sergey Pashin <spashin viewtier.com> writes:
Jason House Wrote:


 Does anyone have a setup for doing continuous integration with d?


Jason,

You might check out our Parabuild. It is free for open source development. And
it will run any build or test that you can run from the command line.

Sergey Pashin
Mar 26 2008