www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Counting passed/failed unit tests

reply =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
Hi,

Is there any way to count the amount of passed/failed unit tests in a 
module or a program as a whole? From what I can see, druntime simply 
invokes a function on each module which triggers all actual tests. If 
this is all the information that is available, I guess it's impossible, 
but I'm _very_ open to suggestions/ideas here...

Thanks!

- Alex
Oct 24 2011
next sibling parent reply Justin Whear <justin economicmodeling.com> writes:
core.runtime allows a user-supplied unittest runner. Never used it, but 
check out 
http://www.digitalmars.com/d/2.0/phobos/core_runtime.html#moduleUnitTester

Justin


Alex Rønne Petersen wrote:


 Hi,
 
 Is there any way to count the amount of passed/failed unit tests in a
 module or a program as a whole? From what I can see, druntime simply
 invokes a function on each module which triggers all actual tests. If
 this is all the information that is available, I guess it's impossible,
 but I'm _very_ open to suggestions/ideas here...
 
 Thanks!
 
 - Alex
Oct 24 2011
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I'm not sure why it just stops after the first failing unittest
though. What is the point of that 'failed' counter?
Oct 24 2011
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
Ok I can see why, ModuleInfo.unitTest just returns a pointer to a
function that calls all unittests on its own. That's not very
flexible.
Oct 24 2011
parent =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 24-10-2011 20:27, Andrej Mitrovic wrote:

 Ok I can see why, ModuleInfo.unitTest just returns a pointer to a
 function that calls all unittests on its own. That's not very
 flexible.


Exactly. I was hoping there would be some obscure workaround, but it 
doesn't seem like luck is on my side here. :(

- Alex
Oct 24 2011
prev sibling next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:

 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?


It's a long standing issue that when one unit test fails within a module, no 
more within that module are run (though fortunately, a while back it was fixed 
so that other modules' unit tests will still run). As I recall, there had to 
be a change to the compiler to fix it, but I don't known/remember the details. 
Certainly, the issue still stands.

- Jonathan M Davis
Oct 24 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-24 22:08, Jonathan M Davis wrote:

 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:

 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?


 It's a long standing issue that when one unit test fails within a module, no
 more within that module are run (though fortunately, a while back it was fixed
 so that other modules' unit tests will still run). As I recall, there had to
 be a change to the compiler to fix it, but I don't known/remember the details.
 Certainly, the issue still stands.

 - Jonathan M Davis


A workaround is to catch AssertErrors, hook it up with some library code 
and you get a minimal unit test framework: 
https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTester.d

Example of usage:

https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d

-- 
/Jacob Carlborg
Oct 25 2011
parent reply David Gileadi <gileadis NSPMgmail.com> writes:
On 10/25/11 4:04 AM, Jacob Carlborg wrote:

 On 2011-10-24 22:08, Jonathan M Davis wrote:

 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:

 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?


 It's a long standing issue that when one unit test fails within a
 module, no
 more within that module are run (though fortunately, a while back it
 was fixed
 so that other modules' unit tests will still run). As I recall, there
 had to
 be a change to the compiler to fix it, but I don't known/remember the
 details.
 Certainly, the issue still stands.

 - Jonathan M Davis


 A workaround is to catch AssertErrors, hook it up with some library code
 and you get a minimal unit test framework:
 https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTester.d


 Example of usage:

 https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d


As an argument for continuing to run tests after one fails, I'm taking a 
TDD class and the instructor asserted that for unit tests you should 
generally only have one or two assertions per test method.  His 
reasoning is that when something breaks you immediately know the extent 
of your breakage by counting the number of failed methods.  This 
argument is pretty convincing to me.
Oct 26 2011
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, October 26, 2011 08:40 David Gileadi wrote:

 On 10/25/11 4:04 AM, Jacob Carlborg wrote:

 On 2011-10-24 22:08, Jonathan M Davis wrote:

 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:

 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?


 
 It's a long standing issue that when one unit test fails within a
 module, no
 more within that module are run (though fortunately, a while back it
 was fixed
 so that other modules' unit tests will still run). As I recall, there
 had to
 be a change to the compiler to fix it, but I don't known/remember the
 details.
 Certainly, the issue still stands.
 
 - Jonathan M Davis


 
 A workaround is to catch AssertErrors, hook it up with some library code
 and you get a minimal unit test framework:
 https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTest
 er.d
 
 
 Example of usage:
 
 https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d


 
 As an argument for continuing to run tests after one fails, I'm taking a
 TDD class and the instructor asserted that for unit tests you should
 generally only have one or two assertions per test method. His
 reasoning is that when something breaks you immediately know the extent
 of your breakage by counting the number of failed methods. This
 argument is pretty convincing to me.


The value of that would depend on what exactly the tests are doing - 
particularly if tests require some setup - but I could see how it would be 
valuable to essentially list all of the failed assertions rather than just 
failed unittest blocks. However, I don't think that I'd ever do it that way 
simply because the code clutter would be far too great. Even if your unit 
tests consist simply of a bunch of one-liners which are just assertions 
without any setup, having a unittest block per assertion is going to create a 
lot of extra code, which will seriously clutter the file. Now, you have the 
choice to do it either way, which is great, but I don't think that I'd want to 
be leaning towards one assertion per unittest block. I'd generally favor one 
unittest block per function, or at least group of related tests.

- Jonathan M Davis
Oct 26 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-26 17:40, David Gileadi wrote:

 On 10/25/11 4:04 AM, Jacob Carlborg wrote:

 On 2011-10-24 22:08, Jonathan M Davis wrote:

 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:

 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?


 It's a long standing issue that when one unit test fails within a
 module, no
 more within that module are run (though fortunately, a while back it
 was fixed
 so that other modules' unit tests will still run). As I recall, there
 had to
 be a change to the compiler to fix it, but I don't known/remember the
 details.
 Certainly, the issue still stands.

 - Jonathan M Davis


 A workaround is to catch AssertErrors, hook it up with some library code
 and you get a minimal unit test framework:
 https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitTester.d



 Example of usage:

 https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d


 As an argument for continuing to run tests after one fails, I'm taking a
 TDD class and the instructor asserted that for unit tests you should
 generally only have one or two assertions per test method. His reasoning
 is that when something breaks you immediately know the extent of your
 breakage by counting the number of failed methods. This argument is
 pretty convincing to me.


Well, in my library, if an assert error is thrown in a block (passed to 
the "it" method), the whole block is canceled and it will continue with 
the next block. So it's up to the user how the asserts should be laid out.

-- 
/Jacob Carlborg
Oct 26 2011
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:

 On 2011-10-26 17:40, David Gileadi wrote:

 On 10/25/11 4:04 AM, Jacob Carlborg wrote:

 On 2011-10-24 22:08, Jonathan M Davis wrote:

 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:

 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?


 
 It's a long standing issue that when one unit test fails within a
 module, no
 more within that module are run (though fortunately, a while back it
 was fixed
 so that other modules' unit tests will still run). As I recall,
 there
 had to
 be a change to the compiler to fix it, but I don't known/remember
 the
 details.
 Certainly, the issue still stands.
 
 - Jonathan M Davis


 
 A workaround is to catch AssertErrors, hook it up with some library
 code
 and you get a minimal unit test framework:
 https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitT
 ester.d
 
 
 
 Example of usage:
 
 https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d


 
 As an argument for continuing to run tests after one fails, I'm taking a
 TDD class and the instructor asserted that for unit tests you should
 generally only have one or two assertions per test method. His reasoning
 is that when something breaks you immediately know the extent of your
 breakage by counting the number of failed methods. This argument is
 pretty convincing to me.


 
 Well, in my library, if an assert error is thrown in a block (passed to
 the "it" method), the whole block is canceled and it will continue with
 the next block. So it's up to the user how the asserts should be laid out.


It would be disastrous IMHO to continue to run a unittest block after an 
assert failed - at least in the general case. Too often further assertions 
relied on the state assured by the one that failed, so further failures just 
confuse things and give you too much data to have to sift through. As it 
stands with the built-in unit testing faciliities, you can put each assertion 
in its own unittest block if you really want each assertion to run on its own 
(though until it's fixed so that further unittest blocks within the module run 
after the first failure in that module, it wouldn't do you any good).

- Jonathan M Davis
Oct 26 2011
next sibling parent David Gileadi <gileadis NSPMgmail.com> writes:
On 10/26/11 3:45 PM, Jonathan M Davis wrote:

 On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:

 On 2011-10-26 17:40, David Gileadi wrote:

 On 10/25/11 4:04 AM, Jacob Carlborg wrote:

 On 2011-10-24 22:08, Jonathan M Davis wrote:

 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:

 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?


 It's a long standing issue that when one unit test fails within a
 module, no
 more within that module are run (though fortunately, a while back it
 was fixed
 so that other modules' unit tests will still run). As I recall,
 there
 had to
 be a change to the compiler to fix it, but I don't known/remember
 the
 details.
 Certainly, the issue still stands.

 - Jonathan M Davis


 A workaround is to catch AssertErrors, hook it up with some library
 code
 and you get a minimal unit test framework:
 https://github.com/jacob-carlborg/orange/blob/master/orange/test/UnitT
 ester.d



 Example of usage:

 https://github.com/jacob-carlborg/orange/blob/master/tests/Object.d


 As an argument for continuing to run tests after one fails, I'm taking a
 TDD class and the instructor asserted that for unit tests you should
 generally only have one or two assertions per test method. His reasoning
 is that when something breaks you immediately know the extent of your
 breakage by counting the number of failed methods. This argument is
 pretty convincing to me.


 Well, in my library, if an assert error is thrown in a block (passed to
 the "it" method), the whole block is canceled and it will continue with
 the next block. So it's up to the user how the asserts should be laid out.


 It would be disastrous IMHO to continue to run a unittest block after an
 assert failed - at least in the general case. Too often further assertions
 relied on the state assured by the one that failed, so further failures just
 confuse things and give you too much data to have to sift through. As it
 stands with the built-in unit testing faciliities, you can put each assertion
 in its own unittest block if you really want each assertion to run on its own
 (though until it's fixed so that further unittest blocks within the module run
 after the first failure in that module, it wouldn't do you any good).

 - Jonathan M Davis


Agreed--the (rather obscure, sorry) point of my message was that having 
further unittest blocks run within the module even after one fails is a 
good idea.  Jacob's test frameworks sounds like a good effort in that 
direction, but built-in would be even better.
Oct 26 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-26 21:45, Jonathan M Davis wrote:

 On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:

 Well, in my library, if an assert error is thrown in a block (passed to
 the "it" method), the whole block is canceled and it will continue with
 the next block. So it's up to the user how the asserts should be laid out.


 It would be disastrous IMHO to continue to run a unittest block after an
 assert failed - at least in the general case. Too often further assertions
 relied on the state assured by the one that failed, so further failures just
 confuse things and give you too much data to have to sift through. As it
 stands with the built-in unit testing faciliities, you can put each assertion
 in its own unittest block if you really want each assertion to run on its own
 (though until it's fixed so that further unittest blocks within the module run
 after the first failure in that module, it wouldn't do you any good).

 - Jonathan M Davis


My solution works now and if you want the whole unit test block to end 
when an assert error is thrown that is possible too. Just call the "it" 
method with a block once in a unit test block.

You can never convince me that the other solution is better as long as 
it doesn't work.

-- 
/Jacob Carlborg
Oct 26 2011
next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, October 27, 2011 08:54:12 Jacob Carlborg wrote:

 On 2011-10-26 21:45, Jonathan M Davis wrote:

 On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:

 Well, in my library, if an assert error is thrown in a block (passed
 to
 the "it" method), the whole block is canceled and it will continue
 with
 the next block. So it's up to the user how the asserts should be laid
 out.> 


 It would be disastrous IMHO to continue to run a unittest block after an
 assert failed - at least in the general case. Too often further
 assertions relied on the state assured by the one that failed, so
 further failures just confuse things and give you too much data to have
 to sift through. As it stands with the built-in unit testing
 faciliities, you can put each assertion in its own unittest block if
 you really want each assertion to run on its own (though until it's
 fixed so that further unittest blocks within the module run after the
 first failure in that module, it wouldn't do you any good).
 
 - Jonathan M Davis


 
 My solution works now and if you want the whole unit test block to end
 when an assert error is thrown that is possible too. Just call the "it"
 method with a block once in a unit test block.
 
 You can never convince me that the other solution is better as long as
 it doesn't work.


The current solution works just fine if you want a unittest block to terminate 
after a single assertion failure (as it should IMHO). It's just the running of 
subsequent unittest blocks within a module which doesn't work.
And while I'd very much like it to work, I don't personally find it to be all 
that much of a problem.

- Jonathan M Davis
Oct 27 2011
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, October 27, 2011 08:54:12 Jacob Carlborg wrote:

 On 2011-10-26 21:45, Jonathan M Davis wrote:

 On Wednesday, October 26, 2011 21:28:10 Jacob Carlborg wrote:

 Well, in my library, if an assert error is thrown in a block (passed
 to
 the "it" method), the whole block is canceled and it will continue
 with
 the next block. So it's up to the user how the asserts should be laid
 out.> 


 It would be disastrous IMHO to continue to run a unittest block after an
 assert failed - at least in the general case. Too often further
 assertions relied on the state assured by the one that failed, so
 further failures just confuse things and give you too much data to have
 to sift through. As it stands with the built-in unit testing
 faciliities, you can put each assertion in its own unittest block if
 you really want each assertion to run on its own (though until it's
 fixed so that further unittest blocks within the module run after the
 first failure in that module, it wouldn't do you any good).
 
 - Jonathan M Davis


 
 My solution works now and if you want the whole unit test block to end
 when an assert error is thrown that is possible too. Just call the "it"
 method with a block once in a unit test block.
 
 You can never convince me that the other solution is better as long as
 it doesn't work.


The current solution works just fine if you want a unittest block to terminate 
after a single assertion failure (as it should IMHO). It's just the running of 
subsequent unittest blocks within a module which doesn't work.
And while I'd very much like it to work, I don't personally find it to be all 
that much of a problem.

- Jonathan M Davis
Oct 27 2011
prev sibling parent "Martin Nowak" <dawg dawgfoto.de> writes:
On Mon, 24 Oct 2011 22:08:50 +0200, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:


 On Monday, October 24, 2011 11:23 Andrej Mitrovic wrote:

 I'm not sure why it just stops after the first failing unittest
 though. What is the point of that 'failed' counter?


 It's a long standing issue that when one unit test fails within a  
 module, no
 more within that module are run (though fortunately, a while back it was  
 fixed
 so that other modules' unit tests will still run). As I recall, there  
 had to
 be a change to the compiler to fix it, but I don't known/remember the  
 details.
 Certainly, the issue still stands.

 - Jonathan M Davis


It's also annoying that the runtime quits with a non-zero exit code
when you return false from your unit tester.
Somehow the boolean return became a flag for fail/success rather than for
program continuation.

martin
Oct 24 2011