digitalmars.D - Suggestion (ping Walter): Improve unit testing.
- Gregor Richards (117/117) Apr 21 2007 There are a number of improvements I'd like to see in D's unit testing
- David B. Held (22/33) Apr 21 2007 Yeah, these are good points. There are a few things that you want from
- Derek Parnell (8/15) Apr 21 2007 I find myself often writing a unittest that will deliberately and
- Deewiant (5/18) Apr 21 2007 Ditto. Whenever I add a new test, I test the test by adding "assert (fal...
- Tom (5/144) Apr 21 2007 I agree. Unit testing needs a lot of improvement to be really useful.
- Ary Manzana (14/153) Apr 21 2007 I also very much agree.
- Chris Nicholson-Sauls (96/114) Apr 21 2007 Ary, I was just about to suggest that. And i would also like very much ...
- Dan (5/5) Apr 21 2007 Interesting... I just wrote the first iteration of unittest for my Walnu...
- Jason House (6/20) Apr 22 2007 This method of output would also implicitly require unit tests are run
- Gregor Richards (6/6) May 12 2007 I hate to do this, but ... BUMP!
- BCS (3/15) May 13 2007 you might be abel to hack #1 in your self by rebuilding phobos with a do...
- Gregor Richards (4/8) May 13 2007 Please note that a patch to solve both problems is included in my
- Greg Weber (3/12) May 14 2007 I am new to D and have been using it a lot over the last month. One of ...
- Bill Baxter (4/19) May 14 2007 I agree too. I was baffled when I first started getting linker errors
- Walter Bright (3/4) May 14 2007 There's too much else going on that needs to get done asap. The
There are a number of improvements I'd like to see in D's unit testing
framework. All of them are standard library changes, and I've made them
work in Tango (as an external patch). If they were added to both Phobos
and Tango, DSSS could easily run unit tests, which would improve its
test suite.
Here are two major problems I see with unit tests right now:
1) You can't run only unit tests, you are forced to run main() as well.
2) Unit testing doesn't actually output anything useful, it just works
under the no-news-is-good-news principle.
Each problem has a fairly easy solution in the core library:
1) Adding a special option, handled by the core library's main function,
which will cause only unit tests to run. The changes I made in Tango:
Index: lib/compiler/gdc/dgccmain2.d
===================================================================
--- lib/compiler/gdc/dgccmain2.d (revision 2100)
+++ lib/compiler/gdc/dgccmain2.d (working copy)
-35,7 +35,7
extern (C) void _minit();
extern (C) void _moduleCtor();
extern (C) void _moduleDtor();
-extern (C) void _moduleUnitTests();
+extern (C) void _moduleUnitTests(int);
/***********************************
* These functions must be defined for any D program linked
-108,6 +108,7
{
char[][] args;
int result;
+ int testOnly = 0;
version (GC_Use_Stack_Guess)
{
-137,11 +138,17
}
args = am[0 .. argc];
}
+
+ // check for --d-unittests-only
+ foreach (arg; args) {
+ if (arg == "--d-unittests-only")
+ testOnly = 1;
+ }
void run()
{
_moduleCtor();
- _moduleUnitTests();
+ _moduleUnitTests(testOnly);
result = main_func(args);
isHalting = true;
_moduleDtor();
As you can see, if --d-unittests-only is supplied, it passes a 1 into
the (now modified) _moduleUnitTests function, which will cause it to
exit when finished testing.
2) The _moduleUnitTests function can be improved to provide useful
output. The output I've implemented for Tango is:
Failure in test for module 'test':
tango.core.Exception.AssertException on test.d(7): Assertion failure
TESTS: 1 PASSED: 0 FAILED: 1
The patch to make this work was fairly simple as well:
Index: lib/compiler/gdc/genobj.d
===================================================================
--- lib/compiler/gdc/genobj.d (revision 2100)
+++ lib/compiler/gdc/genobj.d (working copy)
-42,7 +42,8
import tango.stdc.string; // : memcmp, memcpy;
import tango.stdc.stdlib; // : calloc, realloc, free;
import util.string;
- debug(PRINTF) import tango.stdc.stdio; // : printf;
+ //debug(PRINTF) import tango.stdc.stdio; // : printf;
+ extern (C) int printf(char*, ...);
extern (C) void onOutOfMemoryError();
extern (C) Object _d_newclass(ClassInfo ci);
-1054,11 +1055,13
}
/**
- * Run unit tests.
+ * Run unit tests. If testOnly is true, output results and quit.
*/
-extern (C) void _moduleUnitTests()
+extern (C) void _moduleUnitTests(int testOnly)
{
+ int testCount, testFail;
+ testCount = testFail = 0;
debug(PRINTF) printf("_moduleUnitTests()\n");
for (uint i = 0; i < _moduleinfo_array.length; i++)
{
-1070,9 +1073,26
debug(PRINTF) printf("\tmodule[%d] = '%.*s'\n", i, m.name);
if (m.unitTest)
{
- (*m.unitTest)();
+ testCount++;
+ try {
+ (*m.unitTest)();
+ } catch (Exception e) {
+ printf("Failure in test for module '%.*s':\n", m.name);
+ printf(" %.*s on %.*s(%d): %.*s\n",
+ e.classinfo.name, e.file, e.line, e.msg);
+ testFail++;
+ }
}
}
+ if (testOnly != 0 || testFail > 0)
+ {
+ printf("TESTS: %d PASSED: %d FAILED: %d\n",
+ testCount, testCount - testFail, testFail);
+ if (testFail == 0)
+ exit(0);
+ else
+ exit(1);
+ }
}
These additions are fairly minor, and would make unit testing
immeasurably better than it is right now. The sections I modified in
Tango are mostly the same as the Phobos versions, so the patch should be
almost exactly the same as mine above.
Comments?
- Gregor Richards
Apr 21 2007
Gregor Richards wrote:There are a number of improvements I'd like to see in D's unit testing framework. All of them are standard library changes, and I've made them work in Tango (as an external patch). If they were added to both Phobos and Tango, DSSS could easily run unit tests, which would improve its test suite. Here are two major problems I see with unit tests right now: 1) You can't run only unit tests, you are forced to run main() as well. 2) Unit testing doesn't actually output anything useful, it just works under the no-news-is-good-news principle. [...]Yeah, these are good points. There are a few things that you want from unit tests: * Consistent design This is already handled by the unittest mechanism, which is great. However, a unit testing library that offers all the typical primitives found in xUnit would be even greater. This would help a lot in common reporting. * Consistent invocation This means that there is a standard, well-known way to run the tests. Typically 'make test' is a good way. However, 'dmd -unittest foo.d' is also a perfectly good way, as is 'foo.exe --unittest'. At any rate, it doesn't really make sense that unittests are just functions that get executed before main(). I'm pretty sure nobody expects that. * Consistent output This means that someone who has never seen your tests before can tell whether they passed or failed. The problem with the 'no-gnus-is-good-gnus' principle is that it makes it difficult to tell the difference between "the tests all passed" and "the tests didn't run". Having a standard result summary makes it possible for automated tools to parse the results and display them in alternative formats. Dave
Apr 21 2007
On Sat, 21 Apr 2007 01:43:25 -0700, David B. Held wrote:* Consistent output This means that someone who has never seen your tests before can tell whether they passed or failed. The problem with the 'no-gnus-is-good-gnus' principle is that it makes it difficult to tell the difference between "the tests all passed" and "the tests didn't run".I find myself often writing a unittest that will deliberately and definitely fail, just to make sure I'm invoking any of the unittests. -- Derek Parnell Melbourne, Australia "Justice for David Hicks!" skype: derek.j.parnell
Apr 21 2007
Deewiant <deewiant.doesnotlike.spam gmail.com> Derek Parnell wrote:On Sat, 21 Apr 2007 01:43:25 -0700, David B. Held wrote:Ditto. Whenever I add a new test, I test the test by adding "assert (false);" after it to make sure it gets run. -- Remove ".doesnotlike.spam" from the mail address.* Consistent output This means that someone who has never seen your tests before can tell whether they passed or failed. The problem with the 'no-gnus-is-good-gnus' principle is that it makes it difficult to tell the difference between "the tests all passed" and "the tests didn't run".I find myself often writing a unittest that will deliberately and definitely fail, just to make sure I'm invoking any of the unittests.
Apr 21 2007
I agree. Unit testing needs a lot of improvement to be really useful.
--
Tom;
(Tomás Rossi)
Gregor Richards escribió:
There are a number of improvements I'd like to see in D's unit testing
framework. All of them are standard library changes, and I've made them
work in Tango (as an external patch). If they were added to both Phobos
and Tango, DSSS could easily run unit tests, which would improve its
test suite.
Here are two major problems I see with unit tests right now:
1) You can't run only unit tests, you are forced to run main() as well.
2) Unit testing doesn't actually output anything useful, it just works
under the no-news-is-good-news principle.
Each problem has a fairly easy solution in the core library:
1) Adding a special option, handled by the core library's main function,
which will cause only unit tests to run. The changes I made in Tango:
Index: lib/compiler/gdc/dgccmain2.d
===================================================================
--- lib/compiler/gdc/dgccmain2.d (revision 2100)
+++ lib/compiler/gdc/dgccmain2.d (working copy)
-35,7 +35,7
extern (C) void _minit();
extern (C) void _moduleCtor();
extern (C) void _moduleDtor();
-extern (C) void _moduleUnitTests();
+extern (C) void _moduleUnitTests(int);
/***********************************
* These functions must be defined for any D program linked
-108,6 +108,7
{
char[][] args;
int result;
+ int testOnly = 0;
version (GC_Use_Stack_Guess)
{
-137,11 +138,17
}
args = am[0 .. argc];
}
+
+ // check for --d-unittests-only
+ foreach (arg; args) {
+ if (arg == "--d-unittests-only")
+ testOnly = 1;
+ }
void run()
{
_moduleCtor();
- _moduleUnitTests();
+ _moduleUnitTests(testOnly);
result = main_func(args);
isHalting = true;
_moduleDtor();
As you can see, if --d-unittests-only is supplied, it passes a 1 into
the (now modified) _moduleUnitTests function, which will cause it to
exit when finished testing.
2) The _moduleUnitTests function can be improved to provide useful
output. The output I've implemented for Tango is:
Failure in test for module 'test':
tango.core.Exception.AssertException on test.d(7): Assertion failure
TESTS: 1 PASSED: 0 FAILED: 1
The patch to make this work was fairly simple as well:
Index: lib/compiler/gdc/genobj.d
===================================================================
--- lib/compiler/gdc/genobj.d (revision 2100)
+++ lib/compiler/gdc/genobj.d (working copy)
-42,7 +42,8
import tango.stdc.string; // : memcmp, memcpy;
import tango.stdc.stdlib; // : calloc, realloc, free;
import util.string;
- debug(PRINTF) import tango.stdc.stdio; // : printf;
+ //debug(PRINTF) import tango.stdc.stdio; // : printf;
+ extern (C) int printf(char*, ...);
extern (C) void onOutOfMemoryError();
extern (C) Object _d_newclass(ClassInfo ci);
-1054,11 +1055,13
}
/**
- * Run unit tests.
+ * Run unit tests. If testOnly is true, output results and quit.
*/
-extern (C) void _moduleUnitTests()
+extern (C) void _moduleUnitTests(int testOnly)
{
+ int testCount, testFail;
+ testCount = testFail = 0;
debug(PRINTF) printf("_moduleUnitTests()\n");
for (uint i = 0; i < _moduleinfo_array.length; i++)
{
-1070,9 +1073,26
debug(PRINTF) printf("\tmodule[%d] = '%.*s'\n", i, m.name);
if (m.unitTest)
{
- (*m.unitTest)();
+ testCount++;
+ try {
+ (*m.unitTest)();
+ } catch (Exception e) {
+ printf("Failure in test for module '%.*s':\n", m.name);
+ printf(" %.*s on %.*s(%d): %.*s\n",
+ e.classinfo.name, e.file, e.line, e.msg);
+ testFail++;
+ }
}
}
+ if (testOnly != 0 || testFail > 0)
+ {
+ printf("TESTS: %d PASSED: %d FAILED: %d\n",
+ testCount, testCount - testFail, testFail);
+ if (testFail == 0)
+ exit(0);
+ else
+ exit(1);
+ }
}
These additions are fairly minor, and would make unit testing
immeasurably better than it is right now. The sections I modified in
Tango are mostly the same as the Phobos versions, so the patch should be
almost exactly the same as mine above.
Comments?
- Gregor Richards
Apr 21 2007
I also very much agree.
If a summary is going to be printed for the unittests (i.e. TESTS: 1
PASSED: 0 FAILED: 1), it could be great also to label unittests with a
name. This is more informative that just the file and line where the
assertion failed. Compare "foo.bar.Exception.AssertException on
test.d(7)" to "heap sort makes list sorted".
To to this, you can optionaly pass a string to the unittest:
unittest("heap sort makes list sorted") {
// ...
}
It's backward compatible, and allows future nicer integration with IDEs:
they could show a list of the tests passed and failed, and names are
mandatory in this case.
Gregor Richards escribió:
There are a number of improvements I'd like to see in D's unit testing
framework. All of them are standard library changes, and I've made them
work in Tango (as an external patch). If they were added to both Phobos
and Tango, DSSS could easily run unit tests, which would improve its
test suite.
Here are two major problems I see with unit tests right now:
1) You can't run only unit tests, you are forced to run main() as well.
2) Unit testing doesn't actually output anything useful, it just works
under the no-news-is-good-news principle.
Each problem has a fairly easy solution in the core library:
1) Adding a special option, handled by the core library's main function,
which will cause only unit tests to run. The changes I made in Tango:
Index: lib/compiler/gdc/dgccmain2.d
===================================================================
--- lib/compiler/gdc/dgccmain2.d (revision 2100)
+++ lib/compiler/gdc/dgccmain2.d (working copy)
-35,7 +35,7
extern (C) void _minit();
extern (C) void _moduleCtor();
extern (C) void _moduleDtor();
-extern (C) void _moduleUnitTests();
+extern (C) void _moduleUnitTests(int);
/***********************************
* These functions must be defined for any D program linked
-108,6 +108,7
{
char[][] args;
int result;
+ int testOnly = 0;
version (GC_Use_Stack_Guess)
{
-137,11 +138,17
}
args = am[0 .. argc];
}
+
+ // check for --d-unittests-only
+ foreach (arg; args) {
+ if (arg == "--d-unittests-only")
+ testOnly = 1;
+ }
void run()
{
_moduleCtor();
- _moduleUnitTests();
+ _moduleUnitTests(testOnly);
result = main_func(args);
isHalting = true;
_moduleDtor();
As you can see, if --d-unittests-only is supplied, it passes a 1 into
the (now modified) _moduleUnitTests function, which will cause it to
exit when finished testing.
2) The _moduleUnitTests function can be improved to provide useful
output. The output I've implemented for Tango is:
Failure in test for module 'test':
tango.core.Exception.AssertException on test.d(7): Assertion failure
TESTS: 1 PASSED: 0 FAILED: 1
The patch to make this work was fairly simple as well:
Index: lib/compiler/gdc/genobj.d
===================================================================
--- lib/compiler/gdc/genobj.d (revision 2100)
+++ lib/compiler/gdc/genobj.d (working copy)
-42,7 +42,8
import tango.stdc.string; // : memcmp, memcpy;
import tango.stdc.stdlib; // : calloc, realloc, free;
import util.string;
- debug(PRINTF) import tango.stdc.stdio; // : printf;
+ //debug(PRINTF) import tango.stdc.stdio; // : printf;
+ extern (C) int printf(char*, ...);
extern (C) void onOutOfMemoryError();
extern (C) Object _d_newclass(ClassInfo ci);
-1054,11 +1055,13
}
/**
- * Run unit tests.
+ * Run unit tests. If testOnly is true, output results and quit.
*/
-extern (C) void _moduleUnitTests()
+extern (C) void _moduleUnitTests(int testOnly)
{
+ int testCount, testFail;
+ testCount = testFail = 0;
debug(PRINTF) printf("_moduleUnitTests()\n");
for (uint i = 0; i < _moduleinfo_array.length; i++)
{
-1070,9 +1073,26
debug(PRINTF) printf("\tmodule[%d] = '%.*s'\n", i, m.name);
if (m.unitTest)
{
- (*m.unitTest)();
+ testCount++;
+ try {
+ (*m.unitTest)();
+ } catch (Exception e) {
+ printf("Failure in test for module '%.*s':\n", m.name);
+ printf(" %.*s on %.*s(%d): %.*s\n",
+ e.classinfo.name, e.file, e.line, e.msg);
+ testFail++;
+ }
}
}
+ if (testOnly != 0 || testFail > 0)
+ {
+ printf("TESTS: %d PASSED: %d FAILED: %d\n",
+ testCount, testCount - testFail, testFail);
+ if (testFail == 0)
+ exit(0);
+ else
+ exit(1);
+ }
}
These additions are fairly minor, and would make unit testing
immeasurably better than it is right now. The sections I modified in
Tango are mostly the same as the Phobos versions, so the patch should be
almost exactly the same as mine above.
Comments?
- Gregor Richards
Apr 21 2007
Ary Manzana wrote:
I also very much agree.
If a summary is going to be printed for the unittests (i.e. TESTS: 1
PASSED: 0 FAILED: 1), it could be great also to label unittests with a
name. This is more informative that just the file and line where the
assertion failed. Compare "foo.bar.Exception.AssertException on
test.d(7)" to "heap sort makes list sorted".
To to this, you can optionaly pass a string to the unittest:
unittest("heap sort makes list sorted") {
// ...
}
It's backward compatible, and allows future nicer integration with IDEs:
they could show a list of the tests passed and failed, and names are
mandatory in this case.
Ary, I was just about to suggest that. And i would also like very much to see
these kind
of improvements made. (And Gregor, that's darn slick work.) Reiner Pope and I
tried to
make unittests at least a little more verbal in Cashew's UTest module, but it
ends up
taking over the way you write unittests. Consider this snip from Cashew's
array utils:
--------------------------------------------------
version (Unittest) {
static import UTest = cashew .utils .UTest ;
}
unittest {
UTest.Stdout(""c);
UTest.beginModule("cashew.utils.array");
}
unittest { UTest.beginSection("stand-alone"); }
T[] repeat (T) (T needle, size_t len)
body {
T[] haystack = new T[len];
haystack[] = needle;
return haystack;
}
unittest {
UTest.begin(r" repeat(T [, N])"c);
assert(repeat(3, 3U) == [3, 3, 3]);
UTest.end;
}
unittest {
UTest.endSection;
UTest.beginSection("pseudo-member");
}
bool contains (T) (T[] haystack, T needle)
body {
return haystack.indexOf(needle) != NOT_FOUND;
}
unittest {
UTest.begin(r".contains(T)"c);
int[] foo = [1, 2, 3];
assert( foo.contains(2));
assert(! foo.contains(4));
UTest.end;
}
unittest {
UTest.endSection;
UTest.endModule;
}
--------------------------------------------------
Ack. But the output is nice.
--------------------------------------------------
Unittest: cashew.utils.array: begin
[stand-alone]
, array(...)------------------> Pass
, repeat(T [, N])-------------> Pass
, repeatSub(T[], N)-----------> Pass
, assoc([],[])----------------> Pass
[pseudo-member]
, .defaultLength(N)------------> Pass
, .contains(T)-----------------> Pass
, .diff(T[])-------------------> Pass
, .intersect(A,A)--------------> Pass
, .indexOf(T)------------------> Pass
, .indexOfSub(T[])-------------> Pass
, .rindexOf(T)-----------------> Pass
, .rindexOfSub(T[])------------> Pass
, .remove(T)-------------------> Pass
, .removeAll(T)----------------> Pass
, .drop(N)---------------------> Pass
, .dropIf(Dlg)-----------------> Pass
, .dropRange(N,M)--------------> Pass
, .extract(N)------------------> Pass
, .extractRange(N,M)-----------> Pass
, .shift()---------------------> Pass
, .rshift()--------------------> Pass
, .eat(N)----------------------> Pass
, .reat(N)---------------------> Pass
, .fill (T [, N])--------------> Pass
, .fillSub (T[] [, N])---------> Pass
, .unique()--------------------> Pass
, .rotl(N)---------------------> Pass
, .rotr(N)---------------------> Pass
, .push(...)-------------------> Pass
, .shove(...)------------------> Pass
, .filter(Dlg)-----------------> Pass
, .find(Dlg)-------------------> Pass
, .apply(Dlg)------------------> Pass
, .replace(T,T)----------------> Pass
, .replacePairs(T[T])----------> Pass
, .join(T[][], T[])------------> Pass
, .append(T[], T[]...)---------> Pass
, .split(T[]...)---------------> Pass
, .splitLen(N)-----------------> Pass
, .greedySplitLen(N)-----------> Pass
Unittest: cashew.utils.array: end
--------------------------------------------------
I'd very much like to be able to just toss that thing out some day. :) For
the morbidly
curious, here is UTest:
http://dsource.org/projects/cashew/browser/trunk/cashew/utils/UTest.d
-- Chris Nicholson-Sauls
Apr 21 2007
Interesting... I just wrote the first iteration of unittest for my Walnut 2.x engine, and I found the concept rather intuitive. It's merely a block of code that gets executed. You may use asserts, try/catch, scope(failure), printf, and all the rest to produce a unittest that satisfies your needs. I suppose it leaves need for a standard, but it's not lacking for power. I agree with the "unittests without main()". I never want to run both at the same time, unittest is for development, not for production. Sincerely, Dan
Apr 21 2007
Ary Manzana wrote:
I also very much agree.
If a summary is going to be printed for the unittests (i.e. TESTS: 1
PASSED: 0 FAILED: 1), it could be great also to label unittests with a
name. This is more informative that just the file and line where the
assertion failed. Compare "foo.bar.Exception.AssertException on
test.d(7)" to "heap sort makes list sorted".
To to this, you can optionaly pass a string to the unittest:
unittest("heap sort makes list sorted") {
// ...
}
This method of output would also implicitly require unit tests are run
as if they were in try{} blocks. Currently, a unit test failure makes
the entire program exit (and other unit tests are not run). It'd
probably be good to expand the output a bit more to distinguish which
tests were run even though a unit test in an imported module failed.
Apr 22 2007
I hate to do this, but ... BUMP! It's been three weeks since I posted this, and nobody in any kind of position of authority (*cough* Walter) has responded ... a lot of people have said this would be nifty, and yet it continues to be ignored. Can I at least get a "no, I don't agree"? I hate the halting problem :-( - Gregor Richards
May 12 2007
Reply to Gregor,I hate to do this, but ... BUMP! It's been three weeks since I posted this, and nobody in any kind of position of authority (*cough* Walter) has responded ... a lot of people have said this would be nifty, and yet it continues to be ignored. Can I at least get a "no, I don't agree"? I hate the halting problem :-( - Gregor Richardsdo-nothing main (with a lib tool you might not even have to rebuild it
May 13 2007
BCS wrote:do-nothing main (with a lib tool you might not even have to rebuild itPlease note that a patch to solve both problems is included in my original post. - Gregor Richards
May 13 2007
Gregor Richards Wrote:I hate to do this, but ... BUMP! It's been three weeks since I posted this, and nobody in any kind of position of authority (*cough* Walter) has responded ... a lot of people have said this would be nifty, and yet it continues to be ignored. Can I at least get a "no, I don't agree"? I hate the halting problem :-( - Gregor RichardsI am new to D and have been using it a lot over the last month. One of the things that is nice about D is how things seem to 'make sense'. Unit testing requiring the program to be run was completely unexpected. I think that the unit testing in the source code feature is one of the best things about D, but also probably the feature that is lacking most for a better implementation. Greg Weber
May 14 2007
Greg Weber wrote:Gregor Richards Wrote:I agree too. I was baffled when I first started getting linker errors trying to run unittests on some source code that lacked a main(). --bbI hate to do this, but ... BUMP! It's been three weeks since I posted this, and nobody in any kind of position of authority (*cough* Walter) has responded ... a lot of people have said this would be nifty, and yet it continues to be ignored. Can I at least get a "no, I don't agree"? I hate the halting problem :-( - Gregor RichardsI am new to D and have been using it a lot over the last month. One of the things that is nice about D is how things seem to 'make sense'. Unit testing requiring the program to be run was completely unexpected. I think that the unit testing in the source code feature is one of the best things about D, but also probably the feature that is lacking most for a better implementation. Greg Weber
May 14 2007
Gregor Richards wrote:Can I at least get a "no, I don't agree"? I hate the halting problem :-(There's too much else going on that needs to get done asap. The const/invariant/final is currently sucking all my time.
May 14 2007