digitalmars.D - 'live' testing style
- spir (164/164) Feb 13 2011 Hello,
- Tomek =?ISO-8859-2?Q?Sowi=F1ski?= (21/28) Feb 14 2011 run,=20
- spir (25/40) Feb 14 2011 I have thought at something similar (alse after the related thread about...
- Jonathan M Davis (22/52) Feb 14 2011 The number one reason that I want named unit tests is stack traces. unit...
- Don (7/43) Feb 14 2011 Sounds as though a solution would be to have unittests automatically
- Jonathan M Davis (27/74) Feb 14 2011 I still think that having named unit tests is something that we should d...
- spir (11/74) Feb 15 2011 The only /meaningful/ choice for a programmer, in the general case, is
- Jonathan M Davis (19/96) Feb 15 2011 sts
- Lars T. Kyllingstad (12/32) Feb 15 2011 Works now:
- spir (11/42) Feb 15 2011 Agreed, that's about what I do. The whole point is to have it as a stand...
Hello, Back to the subject of using unittest and assert. I'll try to illustrate a testing style that seems to be rare in the D community, is not properly supported by D builtin tools --while this would require only minimal improvements--, and is imo rather sensible, practicle and efficient. Using an example of a current project of mine (a toy homoiconic language, kind of special Lisp). Below is a copy of part of the parser's test suite. Maybe have a quick look before reading further [Note: I don't mean the example is a model of any sort, not even it's good code. It is just illustration of some testing practice.] The testList func tests the match func for List literal notations. Some comments, in form of question-answer: * Why isn't testList a unittest block? Using named funcs, I can switch on & off specific test suites by (un)commenting their call from the main and unique unittest block. Else, either they all run, or none. During development, I only keep active the test func(s) relative to the feature I'm currently working on. Remedy: named unittests. * Why those write(f)(ln) in unitests? For me unittests are not only regression tests, to be used /after/ development information about what the code actually does, how, and why. I constantly run tests, for feedback, just to check all runs fine, or diagnose what goes wrong. Thus, the first method I write for a type if often toString; I give it a form which tells me all I need about an object; commonly, it's just the notation to re-create it. This, and having test in mind for writing toString, also helps in properly defining the type itself. For diagnosis, this information is of primordial value. And, against common sense maybe, data about what goes right, or seems to, is commonly even more important than data about what goes wrong, to determine the cause(s). (*) Remedy: verbose asserts on request. * What are those '// for diagnose' statement? Since the test func has a form of loop over a test suite, a failing assert would fail (!) to tell which case is in cause, instead miserably spitting out a line number only. To get the proper info, I just need to comment out those few lines of code, which would output for instance (artificial failure case): ... `[nil false true]` --> [nil false true] `["a" "ab" "abc"]` --> ["a" "ab" "abc"] `[[3] [3 2] [3 1 2]]` --> [[3] [3 2] [3 1 2]] *** error *** expected: [[3] [32] [3 1 2]] Remedy: improved assertion failure message form. * Why string'ed results? In this case, like in many others due to various reasons, it's uneasy to write correct result objects; instead I would spend much stupid time in trying to construct by hand portions of ASTs. But I have just what I need already: normal runs of unitests in verbose mode write them out for me. Once I'm convinced, as much as possible, all works fine, I just copy paste result /strings/ into assert statements. Then only I activate said assertions and switch verbose mode off, thus getting a regression test suite for free. Even better, this test suite holds information providing code, ready to use in case of failure. Remedy: assert converts test result to string if needed. Summary: 1. Named unittests allowing test suites in the form of (just an example): unittest test1 { ... } unittest test2 { ... } unittest test3 { ... } unittest { test1; test2; test3; } /Unnamed/ unittests are run with --unittest. Named ones are intended to be called from unnamed ones. Backward compatible change. 2. A variant of assert(), or better a distinct check() statement, in the form of: check(expression, expectation) Separating arguments allows writing out messages like (example formats): * case success (outcome == expectation): <expression> --> <outcome> bringing valuable feedback during development. * case failure *** test failure ********** expression : <expression> outcome : <outcome> expectation : <expectation> *************************** In silent mode, check() writes only in case of failure. If the expected result is a separated argument, it is trivial for check to call to!string on the outcome when it's not a string and the expected result is a string; then only compare. I guess both changes require compiler support (but may very well be wrong). As illustrated by the example, D does not prevent one to use such testing practices; instead, its very nice string and literal features make it far better suited for that than any other static language I know. Still, like Walter, I think it's important for (good) testing practices to be supported by the language, so-to-say officially. 'unittest' itself does not bring any feature: one could trivially have a 'test' control func intended to be (un)commented. But the fact it's a builtin feature makes a decisive difference; if only because it then becomes part of the D community's shared culture. Denis (*) Please don't argue on this point, I have been a maintenance engineer for five years, diagnosing broken machines everyday. === sample part of test code ==================== void testList () { writeln("=== List ==============================================="); List list; LexemeStream lexemes; // test data string[] sources = [ `[ ]` , `[ 3 ]` , `[ 3 1 2 ]` , `[zxy x zx]` , `[nil false true]` , `["a" "ab" "abc"]` , `[[3] [3 2] [3 1 2]]` , `[3 "abc" zxy nil [3 1 2]]` , ]; string[] correctResultStrings = [ `[]` , `[3]` , `[3 1 2]` , `[zxy x zx]` , `[nil false true]` , `["a" "ab" "abc"]` , `[[3] [3 2] [3 1 2]]` , `[3 "abc" zxy nil [3 1 2]]` , ]; // test run foreach (i,source ; sources) { lexemes = claroLexer.lexemes(source); list = cast(List)matchList(lexemes); // output writefln("`%s` --> %s", source,list); // for diagnose if ( to!string(list) != correctResultStrings[i] ) { writefln("*** error *** expected: %s", correctResultStrings[i]); stop(); // core.stdc.stdlib.exit(0) } // regression test assertion assert ( to!string(list) == correctResultStrings[i] ); } writeln(); } ... unittest { testList(); //~ testUnit(); //~ testStatement(); //~ ... } ================================================= === success output in 'verbose' mode ============ `[ ]` --> [] `[ 3 ]` --> [3] `[ 3 1 2 ]` --> [3 1 2] `[zxy x zx]` --> [zxy x zx] `[nil false true]` --> [nil false true] `["a" "ab" "abc"]` --> ["a" "ab" "abc"] `[[3] [3 2] [3 1 2]]` --> [[3] [3 2] [3 1 2]] `[3 "abc" zxy nil [3 1 2]]` --> [3 "abc" zxy nil [3 1 2]] ================================================= -- _________________ vita es estrany spir.wikidot.com
Feb 13 2011
spir napisa=B3:* Why isn't testList a unittest block? =20 Using named funcs, I can switch on & off specific test suites by (un)comm=enting=20their call from the main and unique unittest block. Else, either they all=run,=20or none. During development, I only keep active the test func(s) relative=to=20the feature I'm currently working on. Remedy: named unittests.The interesting thing about named unit tests is that their names aren't int= eresting at all. They are usually dull and forced; testing filterFoo will b= e called testFilterFoo, etc. Their only purpose is to suppress running of u= nrelated tests. Now, there is a seemingly unrelated proposal to include every ddoc'ed unit = test in the preceding declaration as an example. This is great because it i= mplies ownership -- a unit test is 'owned' by the symbol above. Going furth= er, it can also be named after its owner. module ooh; void foo(); unittest { test foo... } Compiling with --unittest=3Dooh.foo runs this unittest only. Nested control= as a bonus: compiling with --unittest=3Dooh runs only the tests in module = ooh. So there you go, named unit tests without naming. --=20 Tomek
Feb 14 2011
On 02/14/2011 09:49 PM, Tomek SowiĆski wrote:spir napisaĆ:I have thought at something similar (alse after the related thread about ddoc), but let it down for various reasons: * Completely implicit. In particular, one cannot learn the feature by reading other people's code; actually, one cannot even guess there is a feature there... * Imposed placement, order. Sometimes, placing the unittest immediately after what it test is not the best. Sometimes, one wants to group part ot all unittests. Sometimes, there are tests relative several funcs / types, etdc. * What about several unittests for a given feature. They would all run, or only the first one, or what? * Naming is a clue for the reader. Esp when there are several tests, or when the test check a particular point. * Possible extension: params. (I thinkin particular at benchmarks, for instance, but there would certainly be numerous other cases, including getting args such as data file names from the command line.) unittest testScan (string filename) { ... } or even unittest testScan (string source, string result) { ... } [Well, actually, i'm not fan of such an extension, prefere to keep things simple, and it's already trivial to factor out a piece of test.] Denis -- _________________ vita es estrany spir.wikidot.com* Why isn't testList a unittest block? Using named funcs, I can switch on& off specific test suites by (un)commenting their call from the main and unique unittest block. Else, either they all run, or none. During development, I only keep active the test func(s) relative to the feature I'm currently working on. Remedy: named unittests.The interesting thing about named unit tests is that their names aren't interesting at all. They are usually dull and forced; testing filterFoo will be called testFilterFoo, etc. Their only purpose is to suppress running of unrelated tests. Now, there is a seemingly unrelated proposal to include every ddoc'ed unit test in the preceding declaration as an example. This is great because it implies ownership -- a unit test is 'owned' by the symbol above. Going further, it can also be named after its owner. module ooh; void foo(); unittest { test foo... } Compiling with --unittest=ooh.foo runs this unittest only. Nested control as a bonus: compiling with --unittest=ooh runs only the tests in module ooh. So there you go, named unit tests without naming.
Feb 14 2011
On Monday, February 14, 2011 12:49:11 Tomek Sowi=F1ski wrote:spir napisa=B3:ol* Why isn't testList a unittest block? =20 Using named funcs, I can switch on & off specific test suites by (un)commenting their call from the main and unique unittest block. Else, either they all run, or none. During development, I only keep active the test func(s) relative to the feature I'm currently working on. Remedy: named unittests.=20 The interesting thing about named unit tests is that their names aren't interesting at all. They are usually dull and forced; testing filterFoo will be called testFilterFoo, etc. Their only purpose is to suppress running of unrelated tests. =20 Now, there is a seemingly unrelated proposal to include every ddoc'ed unit test in the preceding declaration as an example. This is great because it implies ownership -- a unit test is 'owned' by the symbol above. Going further, it can also be named after its owner. =20 module ooh; =20 void foo(); =20 unittest { test foo... } =20 Compiling with --unittest=3Dooh.foo runs this unittest only. Nested contr=as a bonus: compiling with --unittest=3Dooh runs only the tests in module ooh. =20 So there you go, named unit tests without naming.The number one reason that I want named unit tests is stack traces. unittes= t428=20 or whatever a unittest block gets named to is pretty useless. So, while an= =20 assert within the unit test is fine, since it gives a file and line number = which=20 is in the unittest block, exceptions thrown from stuff called by the unit t= est=20 become hard to track down, since you don't know which unit test was being r= un.=20 Named unit tests would fix the problem. I believe that the best proposed sy= ntax=20 for that is unittest(testName) { } That way, something like unittest_testName would end up in the stack trace,= and=20 it would be easy to figure out where exception came from. =2D Jonathan M Davis
Feb 14 2011
Jonathan M Davis wrote:On Monday, February 14, 2011 12:49:11 Tomek Sowiński wrote:spir napisał:* Why isn't testList a unittest block? Using named funcs, I can switch on & off specific test suites by (un)commenting their call from the main and unique unittest block. Else, either they all run, or none. During development, I only keep active the test func(s) relative to the feature I'm currently working on. Remedy: named unittests.The interesting thing about named unit tests is that their names aren't interesting at all. They are usually dull and forced; testing filterFoo will be called testFilterFoo, etc. Their only purpose is to suppress running of unrelated tests.Sounds as though a solution would be to have unittests automatically named as "_unittest" ~ __LINE__ rather than an arbitrary integer. Wouldn't work in the case where there are two unittests on the same line: unittest { assert(true); } unittest { assert(true); } But we can deal with that.Now, there is a seemingly unrelated proposal to include every ddoc'ed unit test in the preceding declaration as an example. This is great because it implies ownership -- a unit test is 'owned' by the symbol above. Going further, it can also be named after its owner. module ooh; void foo(); unittest { test foo... } Compiling with --unittest=ooh.foo runs this unittest only. Nested control as a bonus: compiling with --unittest=ooh runs only the tests in module ooh. So there you go, named unit tests without naming.The number one reason that I want named unit tests is stack traces. unittest428 or whatever a unittest block gets named to is pretty useless. So, while an assert within the unit test is fine, since it gives a file and line number which is in the unittest block, exceptions thrown from stuff called by the unit test become hard to track down, since you don't know which unit test was being run.
Feb 14 2011
On Monday, February 14, 2011 17:26:26 Don wrote:Jonathan M Davis wrote:I still think that having named unit tests is something that we should do a= t=20 some point (and it would be nicely backwards compatible), but naming the un= it=20 test function after the line number that it's on would already be a big=20 improvement. I have no idea how they're named/numbered now. I haven't been able to disce= rn=20 any obvious pattern, and with a module like std.datetime, which has lots of= =20 unittest blocks, even if it were clearly unittest1, unittest2, unittest3, e= tc.=20 it would still be quite hard to figure out which unittest block was running= when=20 the exception threw, since you'd have to count them all. Using __LINE__ would be a big improvement - though unless we figure out som= e way=20 of advertising that, most people probably wouldn't have a clue, and it woul= dn't=20 help them much. It would definitely be a positive change though. And since if someone is bizarre enough to put more than one unittest block = on=20 one line, well then they suffer for what is almost certainly a poor choice.= But=20 it would still narrow it down for them. =2D Jonathan M DavisOn Monday, February 14, 2011 12:49:11 Tomek Sowi=F1ski wrote:=20 Sounds as though a solution would be to have unittests automatically named as "_unittest" ~ __LINE__ rather than an arbitrary integer. Wouldn't work in the case where there are two unittests on the same line: unittest { assert(true); } unittest { assert(true); } But we can deal with that.spir napisa=B3:=20 The number one reason that I want named unit tests is stack traces. unittest428 or whatever a unittest block gets named to is pretty useless. So, while an assert within the unit test is fine, since it gives a file and line number which is in the unittest block, exceptions thrown from stuff called by the unit test become hard to track down, since you don't know which unit test was being run.* Why isn't testList a unittest block? =20 Using named funcs, I can switch on & off specific test suites by (un)commenting their call from the main and unique unittest block. Else, either they all run, or none. During development, I only keep active the test func(s) relative to the feature I'm currently working on. Remedy: named unittests.=20 The interesting thing about named unit tests is that their names aren't interesting at all. They are usually dull and forced; testing filterFoo will be called testFilterFoo, etc. Their only purpose is to suppress running of unrelated tests. =20 =20 Now, there is a seemingly unrelated proposal to include every ddoc'ed unit test in the preceding declaration as an example. This is great because it implies ownership -- a unit test is 'owned' by the symbol above. Going further, it can also be named after its owner. =20 module ooh; =20 void foo(); =20 unittest { test foo... } =20 Compiling with --unittest=3Dooh.foo runs this unittest only. Nested control as a bonus: compiling with --unittest=3Dooh runs only the tests in module ooh. =20 So there you go, named unit tests without naming.
Feb 14 2011
On 02/15/2011 02:42 AM, Jonathan M Davis wrote:On Monday, February 14, 2011 17:26:26 Don wrote:The only /meaningful/ choice for a programmer, in the general case, is programmer-chosen names. Don't you think? One can then even nicely use those names like eg "see example use in unittest ApplicationToArithmetics". If people are happy with naming unittests using ordinals (1 2 3...) or line numbers (in realease), well... let them free, after all it's their choice :-) Denis -- _________________ vita es estrany spir.wikidot.comJonathan M Davis wrote:I still think that having named unit tests is something that we should do at some point (and it would be nicely backwards compatible), but naming the unit test function after the line number that it's on would already be a big improvement. I have no idea how they're named/numbered now. I haven't been able to discern any obvious pattern, and with a module like std.datetime, which has lots of unittest blocks, even if it were clearly unittest1, unittest2, unittest3, etc. it would still be quite hard to figure out which unittest block was running when the exception threw, since you'd have to count them all. Using __LINE__ would be a big improvement - though unless we figure out some way of advertising that, most people probably wouldn't have a clue, and it wouldn't help them much. It would definitely be a positive change though. And since if someone is bizarre enough to put more than one unittest block on one line, well then they suffer for what is almost certainly a poor choice. But it would still narrow it down for them.On Monday, February 14, 2011 12:49:11 Tomek SowiĆski wrote:Sounds as though a solution would be to have unittests automatically named as "_unittest" ~ __LINE__ rather than an arbitrary integer. Wouldn't work in the case where there are two unittests on the same line: unittest { assert(true); } unittest { assert(true); } But we can deal with that.spir napisaĆ:The number one reason that I want named unit tests is stack traces. unittest428 or whatever a unittest block gets named to is pretty useless. So, while an assert within the unit test is fine, since it gives a file and line number which is in the unittest block, exceptions thrown from stuff called by the unit test become hard to track down, since you don't know which unit test was being run.* Why isn't testList a unittest block? Using named funcs, I can switch on& off specific test suites by (un)commenting their call from the main and unique unittest block. Else, either they all run, or none. During development, I only keep active the test func(s) relative to the feature I'm currently working on. Remedy: named unittests.The interesting thing about named unit tests is that their names aren't interesting at all. They are usually dull and forced; testing filterFoo will be called testFilterFoo, etc. Their only purpose is to suppress running of unrelated tests. Now, there is a seemingly unrelated proposal to include every ddoc'ed unit test in the preceding declaration as an example. This is great because it implies ownership -- a unit test is 'owned' by the symbol above. Going further, it can also be named after its owner. module ooh; void foo(); unittest { test foo... } Compiling with --unittest=ooh.foo runs this unittest only. Nested control as a bonus: compiling with --unittest=ooh runs only the tests in module ooh. So there you go, named unit tests without naming.
Feb 15 2011
On Tuesday 15 February 2011 00:46:25 spir wrote:On 02/15/2011 02:42 AM, Jonathan M Davis wrote:ngOn Monday, February 14, 2011 17:26:26 Don wrote:Jonathan M Davis wrote:On Monday, February 14, 2011 12:49:11 Tomek Sowi=C5=84ski wrote:spir napisa=C5=82:* Why isn't testList a unittest block? =20 Using named funcs, I can switch on& off specific test suites by (un)commenting their call from the main and unique unittest block. Else, either they all run, or none. During development, I only keep active the test func(s) relative to the feature I'm currently worki=stson. Remedy: named unittests.=20 The interesting thing about named unit tests is that their names aren't interesting at all. They are usually dull and forced; testing filterFoo will be called testFilterFoo, etc. Their only purpose is to suppress running of unrelated tests. =20 =20 Now, there is a seemingly unrelated proposal to include every ddoc'ed unit test in the preceding declaration as an example. This is great because it implies ownership -- a unit test is 'owned' by the symbol above. Going further, it can also be named after its owner. =20 module ooh; =20 void foo(); =20 unittest { test foo... } =20 Compiling with --unittest=3Dooh.foo runs this unittest only. Nested control as a bonus: compiling with --unittest=3Dooh runs only the te=nsin module ooh. =20 So there you go, named unit tests without naming.=20 The number one reason that I want named unit tests is stack traces. unittest428 or whatever a unittest block gets named to is pretty useless. So, while an assert within the unit test is fine, since it gives a file and line number which is in the unittest block, exceptio=do=20 I still think that having named unit tests is something that we should =thrown from stuff called by the unit test become hard to track down, since you don't know which unit test was being run.=20 Sounds as though a solution would be to have unittests automatically named as "_unittest" ~ __LINE__ rather than an arbitrary integer. Wouldn't work in the case where there are two unittests on the same line: unittest { assert(true); } unittest { assert(true); } But we can deal with that.I'm looking for usable stack traces from unit test blocks. Ideally, that wo= uld=20 be with named unit tests, but that's a language change which may not happen= for=20 a while, if ever. Having the unittest block functions named with __LINE__ b= eing=20 used to generate the numeric part of their name would at least give me the= =20 information that I need to use stack traces from exceptions that went throu= gh a=20 unit test block. So, while named unit tests would be nice, the __LINE__ bit= =20 would be a big help. =2D Jonathan M Davisat some point (and it would be nicely backwards compatible), but naming the unit test function after the line number that it's on would already be a big improvement. =20 I have no idea how they're named/numbered now. I haven't been able to discern any obvious pattern, and with a module like std.datetime, which has lots of unittest blocks, even if it were clearly unittest1, unittest2, unittest3, etc. it would still be quite hard to figure out which unittest block was running when the exception threw, since you'd have to count them all. =20 Using __LINE__ would be a big improvement - though unless we figure out some way of advertising that, most people probably wouldn't have a clue, and it wouldn't help them much. It would definitely be a positive change though. =20 And since if someone is bizarre enough to put more than one unittest block on one line, well then they suffer for what is almost certainly a poor choice. But it would still narrow it down for them.=20 The only /meaningful/ choice for a programmer, in the general case, is programmer-chosen names. Don't you think? One can then even nicely use those names like eg "see example use in unittest ApplicationToArithmetics". If people are happy with naming unittests using ordinals (1 2 3...) or line numbers (in realease), well... let them free, after all it's their choice :-)
Feb 15 2011
On Sun, 13 Feb 2011 23:03:06 +0100, spir wrote:1. Named unittests allowing test suites in the form of (just an example): unittest test1 { ... } unittest test2 { ... } unittest test3 { ... } unittest { test1; test2; test3; } /Unnamed/ unittests are run with --unittest. Named ones are intended to be called from unnamed ones. Backward compatible change.Works now: version(unittest) void test1() { ... } version(unittest) void test2() { ... } version(unittest) void test3() { ... } unittest { test1(); test2(); test3(); } -Lars
Feb 15 2011
On 02/15/2011 10:00 AM, Lars T. Kyllingstad wrote:On Sun, 13 Feb 2011 23:03:06 +0100, spir wrote:Agreed, that's about what I do. The whole point is to have it as a standard feature of the language. You don't need the current 'unittest' feature to write unittests, do you? I doesn't even solve typing (or so few?). But as stated by Walter, having it as a builtin feature makes the difference in D code beeing commonly unit-tested. denis -- _________________ vita es estrany spir.wikidot.com1. Named unittests allowing test suites in the form of (just an example): unittest test1 { ... } unittest test2 { ... } unittest test3 { ... } unittest { test1; test2; test3; } /Unnamed/ unittests are run with --unittest. Named ones are intended to be called from unnamed ones. Backward compatible change.Works now: version(unittest) void test1() { ... } version(unittest) void test2() { ... } version(unittest) void test3() { ... } unittest { test1(); test2(); test3(); }
Feb 15 2011