www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 2749] New: Make unittests named and nestable

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2749

           Summary: Make unittests named and nestable
           Product: D
           Version: 1.041
          Platform: All
               URL: http://www.digitalmars.com/d/archives/digitalmars/D/Impr
                    oving_unit_tests_79469.html
        OS/Version: All
            Status: NEW
          Keywords: diagnostic, spec
          Severity: enhancement
          Priority: P2
         Component: DMD
        AssignedTo: bugzilla digitalmars.com
        ReportedBy: gide nwawudu.com


The current unit tests lack some functionality. 
- The current unit tests are not named.
- There is no output that specifically indicates that the tests were run.
- A single failing test will prevent all other tests from running.
- There is no indication of which test failed.
- There is no way to only run a subset of tests.

Changing the language to allow nested and named unittest would help. 
The nesting would allow for the only the tests within the scope to fail and
subsequent tests would run. The naming could be used to provide output and to
select which tests are run.

Example of unittest
===================

unittest ("XML") {
    unittest("elements") {
        assert(isValidXml("<aaa />"));
        assert(isValidXml("<aaa/>"));
        assert(isValidXml("<aaa></aaa>"));
        ...
        unittest("case unmatched") assert(!isValidXml("<AaA></aaa>"));
        unittest("no closing") assert(!isValidXml("<aaa>"));
        ...
    }

    unittest("attributes") {
        assert(isValidXml("<aaa abc='x'/>"));
        assert(isValidXml("<aaa abc="\x\"/>"));
        assert(isValidXml("<aaa abc=\"x\" def=\"y\"/>"));
        ...
        unittest("unquoted") assert(!isValidXml("<aaa abc=x />"));
        unittest("multi attr") assert(!isValidXml("<aaa abc='x' abc='y'/>"));
        ...
    }

    unittest("encoding") {
        assert(encode("hello") is "hello");
        assert(encode("a > b") == "a &gt; b");
        ...
    }       
}


--
Mar 19 2009
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2749







The functionality you are asking for can be built on top of the existing
unittests.

If you are expecting that unittest will be something like NUnit then you will
be disappointed. As they are set up, they are best run as part of normal
program execution during development.


--
Mar 19 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2749








 The functionality you are asking for can be built on top of the existing
 unittests.


No. All unittests in one module are combined to create one function, and there
is no way to provide a name for them.

However, it is possible to change the runtime to continue running unittests
after one fails, printing out the name of the modules with failed tests, and
offer some useful report at the end.


--
Mar 19 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2749






OK so it takes a little boilerplate.

unitests
{
   static const name = "Foo";

   if(!RunTestByName(name)) return;
   try
   {

   /// tests

   }
   catch(Excption e)
   {
      ProcessUnittestError(name,e);
   }
}

Or to be even cleaner

unittest
{
   RunTests!(Foo, Bar, Baz)();
}

void Foo() { /* tests */ }
void Bar() { /* tests */ }
void Baz() { /* tests */ }


--
Mar 19 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2749








 OK so it takes a little boilerplate.


You might as well say, you can write your unittests in main() and use version
blocks to determine whether to run them. It only takes a bit more boilerplate.


--
Mar 19 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2749


smjg iname.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |smjg iname.com







 The current unit tests lack some functionality. 
 - The current unit tests are not named.
 - There is no output that specifically indicates that the tests were run.


I often put writefln statements in my unittests, which seem to solve that
problem.


 - A single failing test will prevent all other tests from running.
 - There is no indication of which test failed.


If, as in your example, all assert statements are within the unittest code,
then how is the line number in the assert error message not an indication?

If you're calling functions to do the asserts, and so you can't tell what in
the unittest triggered it, then the aforementioned writefln statements are one
way around it.


 unittest ("XML") {


Why make the unittest names strings, rather than identifiers?

So if I'm understanding correctly:
- Each unittest block would have a pass/fail condition.
- When an assert is thrown, the immediately containing unittest fails and exits
immediately.
- If one unittest fails, the next will be run, but any unittest it's nested
within would fail.

Have I got this right?  Moreover, would it be legal for statements to follow
nested unittests within a unittest?  How would these be handled?


--
Mar 21 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2749










 The current unit tests lack some functionality. 
 - The current unit tests are not named.
 - There is no output that specifically indicates that the tests were run.


 
 I often put writefln statements in my unittests, which seem to solve that
 problem.
 


Adding writefln is ok, but it's an implicit way of documenting the unittest.


 - A single failing test will prevent all other tests from running.
 - There is no indication of which test failed.


 
 If, as in your example, all assert statements are within the unittest code,
 then how is the line number in the assert error message not an indication?
 
 If you're calling functions to do the asserts, and so you can't tell what in
 the unittest triggered it, then the aforementioned writefln statements are one
 way around it.
 

 unittest ("XML") {


 
 Why make the unittest names strings, rather than identifiers?


I agree that finding the failing assertion is not difficult, the difficultly is
working out what the failing test is meant to be doing. The reason for naming
and nesting, is that it groups related tests together and gives a description
to them, this is the reason why strings are preferred to identifiers. 


 So if I'm understanding correctly:
 - Each unittest block would have a pass/fail condition.


Yes


 - When an assert is thrown, the immediately containing unittest fails and exits
 immediately.


Yes


 - If one unittest fails, the next will be run, but any unittest it's nested
 within would fail.


Any unittest it nested would not be executed, so they don't fail they are
skipped.


 ... Moreover, would it be legal for statements to follow
 nested unittests within a unittest?  How would these be handled?


No reason to prevent statements between unittests, if a statement throws an
exception that will be treated as a failure of that unittest.


--
Mar 22 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2749








 - If one unittest fails, the next will be run, but any unittest it's nested
 within would fail.


 
 Any unittest it nested would not be executed, so they don't fail they are
 skipped.


I said "unittest it's nested within" not "unittest it nested".  Now try again.


--
Mar 22 2009
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2749










 - If one unittest fails, the next will be run, but any unittest it's nested
 within would fail.


 
 Any unittest it nested would not be executed, so they don't fail they are
 skipped.


 
 I said "unittest it's nested within" not "unittest it nested".  Now try again.
 


If a nested test fails it also fails the parent unittest(s).


--
Mar 26 2009
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=2749


Hoenir <mrmocool gmx.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mrmocool gmx.de



If this is integrated into the language it shouldn't require quotes though.
It should be consistent with version() and debug() conditions accepting
integers and identifiers.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jan 17 2010