www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - listing multiple unit test failures

reply Spacen Jasset <spacen yahoo.co.uk> writes:
	I have started some unit tests:

	unittest
	{
		// Check row column indexing is correct
		Matrix	m;
		m[2,0] = 59;
		assert(m.matrix_flat[2]==59, "opIndexAssign failed");
		assert(m.matrix_flat[8]!=59, "opIndexAssign failed");
		
	}

If an assert fails the other assertions in a program do not fire. This 
is inconvient since the "real bug" introduced may be caused by some 
other assertion that has not been evaluated yet.

What I am looking for, therefore is more of a writefln of each failure, 
together with a single assertion to abort the program if any test has 
failed.

If there isn't such a feature then I suppose I need a "Global" class 
which holds a bit flag to be set when an assertion is registered, and 
then in main this class could be made to assert if the flag is set.
Mar 11 2008
next sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Spacen Jasset wrote:

     I have started some unit tests:
 
     unittest
     {
         // Check row column indexing is correct
         Matrix    m;
         m[2,0] = 59;
         assert(m.matrix_flat[2]==59, "opIndexAssign failed");
         assert(m.matrix_flat[8]!=59, "opIndexAssign failed");
        
     }
 
 If an assert fails the other assertions in a program do not fire. This 
 is inconvient since the "real bug" introduced may be caused by some 
 other assertion that has not been evaluated yet.
 
 What I am looking for, therefore is more of a writefln of each failure, 
 together with a single assertion to abort the program if any test has 
 failed.
 
 If there isn't such a feature then I suppose I need a "Global" class 
 which holds a bit flag to be set when an assertion is registered, and 
 then in main this class could be made to assert if the flag is set.


You're looking for DUnit, which hasn't been written yet. I'd like to 
write such a thing, but a natural, xUnit-like syntax is difficult in D.

If you want something that targets D2, that's doable; you could write 
something like:

import dunit.api;
class FooTests : TestFixture
{
    mixin(DunitTest);
    override void setup () {}
    override void teardown () {}
    void testSomething()
    {
       // ...
    }
}

But in D1, there's no way (as far as I know) to iterate over the methods 
for a type.

Well, no, there is the vtbl. But that is a REALLY ugly solution.

So you're left with some syntax like:

class FooTests : TestFixture
{
    mixin(DunitTest);
    override void setup () {}
    override void teardown () {}
    this ()
    {
       tests["test for something"] =
       {
          // blah
       }
       tests["test for something else"] =
       {
          // blah
       }
    }
}

I guess that isn't so ugly. I can hack something together tonight.
Mar 11 2008
parent Christopher Wright <dhasenan gmail.com> writes:
Christopher Wright wrote:

 I guess that isn't so ugly. I can hack something together tonight.


That was actually quite easy. I'll add it to Felt and post a link in a 
few minutes.
Mar 11 2008
prev sibling next sibling parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Spacen Jasset wrote:

     I have started some unit tests:
 
     unittest
     {
         // Check row column indexing is correct
         Matrix    m;
         m[2,0] = 59;
         assert(m.matrix_flat[2]==59, "opIndexAssign failed");
         assert(m.matrix_flat[8]!=59, "opIndexAssign failed");
        
     }
 
 If an assert fails the other assertions in a program do not fire. This 
 is inconvient since the "real bug" introduced may be caused by some 
 other assertion that has not been evaluated yet.
 
 What I am looking for, therefore is more of a writefln of each failure, 
 together with a single assertion to abort the program if any test has 
 failed.
 
 If there isn't such a feature then I suppose I need a "Global" class 
 which holds a bit flag to be set when an assertion is registered, and 
 then in main this class could be made to assert if the flag is set.


Not to knock on Mr. Wright's solution, which is probably excellent and 
more directly addressable to the problem at hand, but I'm working on a 
solution (designed to use IPC to be used from other applications, etc.) 
that does just this and a lot more. Check:
http://www.dsource.org/projects/descent/browser/trunk/descent.unittest/flute/src/org/dsource/descent/unittests/flute.d

And the docs at:
http://svn.dsource.org/projects/descent/trunk/descent.unittest/flute/doc/flute.html
Mar 11 2008
next sibling parent Robert Fraser <fraserofthenight gmail.com> writes:
Robert Fraser wrote:

 Spacen Jasset wrote:

     I have started some unit tests:

     unittest
     {
         // Check row column indexing is correct
         Matrix    m;
         m[2,0] = 59;
         assert(m.matrix_flat[2]==59, "opIndexAssign failed");
         assert(m.matrix_flat[8]!=59, "opIndexAssign failed");
            }

 If an assert fails the other assertions in a program do not fire. This 
 is inconvient since the "real bug" introduced may be caused by some 
 other assertion that has not been evaluated yet.

 What I am looking for, therefore is more of a writefln of each 
 failure, together with a single assertion to abort the program if any 
 test has failed.

 If there isn't such a feature then I suppose I need a "Global" class 
 which holds a bit flag to be set when an assertion is registered, and 
 then in main this class could be made to assert if the flag is set.


 
 Not to knock on Mr. Wright's solution, which is probably excellent and 
 more directly addressable to the problem at hand, but I'm working on a 
 solution (designed to use IPC to be used from other applications, etc.) 
 that does just this and a lot more. Check:
 http://www.dsource.org/projects/descent/browser/trunk/descent.unittest/flute/src/org/dsource/desce
t/unittests/flute.d 
 
 
 And the docs at:
 http://svn.dsource.org/projects/descent/trunk/descent.unittest/
lute/doc/flute.html 
 


I should mention flute is in-progress, so unless _really_ don't want to 
change your unittest { } blocks into DUnit tests, use DUnit instead.
Mar 11 2008
prev sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
Robert Fraser wrote:

 Not to knock on Mr. Wright's solution, which is probably excellent and 
 more directly addressable to the problem at hand, but I'm working on a 
 solution (designed to use IPC to be used from other applications, etc.) 
 that does just this and a lot more. Check:
 http://www.dsource.org/projects/descent/browser/trunk/descent.unittest/flute/src/org/dsource/desce
t/unittests/flute.d 
 
 
 And the docs at:
 http://svn.dsource.org/projects/descent/trunk/descent.unittest/
lute/doc/flute.html 
 


Flute seems like a general solution that can easily be adapted to DUnit, 
and that support would increase the popularity of DUnit. But can I use 
Flute today?

Flute plus standard D unittests will be a smaller commitment than DUnit, 
and will require somewhat less from programmers. But there's value in 
requiring organization and separation for your tests, I think. For one, 
it's easier to abstract out shared setup code, and be certain that that 
won't be included in your release builds.

Your trick for naming unittests is also scary. It works, though, I'll 
give it that. Against all odds, and without having to run any of the tests.
Mar 12 2008
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Christopher Wright wrote:

 Robert Fraser wrote:

 Not to knock on Mr. Wright's solution, which is probably excellent and 
 more directly addressable to the problem at hand, but I'm working on a 
 solution (designed to use IPC to be used from other applications, 
 etc.) that does just this and a lot more. Check:
 http://www.dsource.org/projects/descent/browser/trunk/descent.unittest/flute/src/org/dsource/desce
t/unittests/flute.d 


 And the docs at:
 http://svn.dsource.org/projects/descent/trunk/descent.unittest/
lute/doc/flute.html 


 
 Flute seems like a general solution that can easily be adapted to DUnit, 
 and that support would increase the popularity of DUnit. But can I use 
 Flute today?


Not unless you're on Windows/Phobos/D1 (I haven't gotten around to 
updating it for supporting other configs, since it's not officially 
being released until I get it integrated into Descent, which is a much 
larger project than Flute itself).


 Flute plus standard D unittests will be a smaller commitment than DUnit, 
 and will require somewhat less from programmers. But there's value in 
 requiring organization and separation for your tests, I think. For one, 
 it's easier to abstract out shared setup code, and be certain that that 
 won't be included in your release builds.


Strongly agreed. xUnit is a much better solution for large-scale 
development, but sticking mini unit tests in your code is hlpful during 
development phases.


 Your trick for naming unittests is also scary. It works, though, I'll 
 give it that. Against all odds, and without having to run any of the tests.


Indeed, indeed. It would be very helpful if D allowed unittest(name) { } 
syntax.

Flute is mainly designed to be integrated into frontends, in particular 
Descent and CruiseControl. It would be brilliant if I could get DUnit 
tests in there, too, sicne as D moves towards large enterprise-class 
solutions, stronger unit testing support is certainly needed. Would it 
be okay if I added DUnit support alongside Flute support to Descent and 
to my Ant/CruiseControl plugin when I get around to doing that?
Mar 12 2008
parent reply Christopher Wright <dhasenan gmail.com> writes:
== Quote from Robert Fraser (fraserofthenight gmail.com)'s article

 Flute is mainly designed to be integrated into frontends, in particular
 Descent and CruiseControl. It would be brilliant if I could get DUnit
 tests in there, too, sicne as D moves towards large enterprise-class
 solutions, stronger unit testing support is certainly needed. Would it
 be okay if I added DUnit support alongside Flute support to Descent and
 to my Ant/CruiseControl plugin when I get around to doing that?


Have you started such a plugin? If so, I'd love to help out. If not, any
suggestions on where to get started?

I'd be quite thrilled to have DUnit support in Descent, as well. Let me know if
there's anything I can do to make this simpler.
Mar 12 2008
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Christopher Wright wrote:

 Have you started such a plugin? If so, I'd love to help out. If not, any
 suggestions on where to get started?


Nope, it's on my *long* TODO list. I'm not even sure how to go about 
making it a CC plugin, I'd have to check how the JUnit plugin works. It 
might end up being an Ant task with some sort of listener/presentation 
component in CC.
Mar 13 2008
parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Robert Fraser wrote:

 Christopher Wright wrote:

 Have you started such a plugin? If so, I'd love to help out. If not, any
 suggestions on where to get started?


 
 Nope, it's on my *long* TODO list. I'm not even sure how to go about 
 making it a CC plugin, I'd have to check how the JUnit plugin works. It 
 might end up being an Ant task with some sort of listener/presentation 
 component in CC.


I took a look into this & it appears that the best solution is to create 
a tool that outputs unit test result XML in the same format as JUnit. CC 
can just pick that up & display it as JUnit tests. For full CC 
integration, adding another builder (such as a DSSS builder) would be a 
good idea, though.

That said, it would be a cool idea to have a unified, configurable D 
"testing framework" that could run both DUnit tests and built-in 
unittest { } blocks with a plugable reporter interface for output (i.e. 
a JUnit-compatible XML for CC plugability, a pretty Web 2.0-ish page for 
Zen staring sessions, etc.)
Mar 13 2008
parent Christopher Wright <dhasenan gmail.com> writes:
Robert Fraser wrote:

 Robert Fraser wrote:

 Christopher Wright wrote:

 Have you started such a plugin? If so, I'd love to help out. If not, any
 suggestions on where to get started?


 Nope, it's on my *long* TODO list. I'm not even sure how to go about 
 making it a CC plugin, I'd have to check how the JUnit plugin works. 
 It might end up being an Ant task with some sort of 
 listener/presentation component in CC.


 
 I took a look into this & it appears that the best solution is to create 
 a tool that outputs unit test result XML in the same format as JUnit. CC 
 can just pick that up & display it as JUnit tests. For full CC 
 integration, adding another builder (such as a DSSS builder) would be a 
 good idea, though.


I came to the same conclusion, though I didn't find an adequate 
description of the JUnit schema. I'll have to snag some output instead, 
I guess.


 That said, it would be a cool idea to have a unified, configurable D 
 "testing framework" that could run both DUnit tests and built-in 
 unittest { } blocks with a plugable reporter interface for output (i.e. 
 a JUnit-compatible XML for CC plugability, a pretty Web 2.0-ish page for 
 Zen staring sessions, etc.)


Yep, but that's too much work for me to do.
Mar 13 2008
prev sibling parent Robert Fraser <fraserofthenight gmail.com> writes:
Christopher Wright Wrote: 

 That said, it would be a cool idea to have a unified, configurable D 
 "testing framework" that could run both DUnit tests and built-in 
 unittest { } blocks with a plugable reporter interface for output (i.e. 
 a JUnit-compatible XML for CC plugability, a pretty Web 2.0-ish page for 
 Zen staring sessions, etc.)


 
 Yep, but that's too much work for me to do.


I was thinking of doing it myself some time (in the far future). The idea is to
have a pluggable testing solution for different types of tests so it would be
relatively easy to add them into Descent, but with a flexible-enough system,
the bootstrapper could be used for a lot of other things, too. But, indeed,
it's a lot of work.
Mar 15 2008