www.digitalmars.com         C & C++   DMDScript  

D - [PROPOSAL] Modify unit test facility

reply Mike Swieton <mike swieton.net> writes:
As suggested by Ben Hinkle, I have written a fairly formal proposal for
changes to the unit test framework. Attached below. I realize that it's a bit
lengthy, but I tried to be complete. Feedback welcome!

Mike Swieton
__
The most exciting phrase to hear in science, the one that heralds the most
discoveries, is not 'Eureka!' but 'That's funny...'
	- Isaac Asimov

                             Unit Testing in D

                      Mike Swieton - mike swieton.net

Contents

     * [1]Introduction
     * [2]What needs to be addressed?
          + [3]Weak assert
          + [4]All or nothing
          + [5]Lack of isolation
          + [6]No reporting
     * [7]Impact
     * [8]User-land unit test library
     * [9]Conclusion

                                 Introduction

   The D programming language includes the innovative ability to embed
   unit tests directly in the code, without the need for hefty frameworks
   or libraries. However, the version of unit testing built in to D
   suffers from severe limitations. This document identifies several
   shortcomings and suggests solutions.

   This document only addresses unit testing as present in version 0.81
   of the Digital Mars D compiler.

   The solutions I present here are based around this criteria:

     * No additional tools may be used. The user must need only the
       compiler/linker and a text editor, and nothing new.
     * Minimum responsibility of the language and compiler. The compiler
       should implement the bulk of the features, because that would
       probably be difficult to upgrade without language/compiler
       changes.

                          What needs to be addressed?

Weak assert

   The function assert() provides poor output, and is difficult to
   improve upon in user code, because it's not a `real' function, but
   instead is a magic function, generated by the compiler.

  Wanted features

   An assertion should be able to check for many things, such as truth,
   equality, nullness, and generally any state that one might want to
   test for. The only one available right now in D is the test for truth.

   An other feature an assertion should have is messages. When an
   assertion fails, it is useful to section know where the failure
   occurred, what the assertion is testing for, conceptually (``i is a
   valid whole number'', and not ``i [10][IMAGE gif] 0''). Currently,
   only filename and line number are reported.

  Solution

   Make assert() a library function, and remove it from the compiler. It
   would then be possible for developers to overload the assert method,
   or delegate to it.

   This solution has a shortcoming: If a user overrides assert(), it is
   not currently possible to get the filename and line number at the
   call[11]1. Fixing this would likely require changes to the language.

   One such possible change would be the addition of more magic
   parameters in the same vein as std.compiler.currLine, such as
   std.compiler.callLine, which would return the line that called the
   current method.

   The above is a terribly ugly hack which I am almost embarrassed to
   suggest, however, as information such as filename and line number can
   only be provided by the compiler, I see no other way to do it than
   magic variables.

  Summary

     * The compiler-generated assert() is removed.
     * A new method is added to the runtime library: void assert() throws
       AssertError
     * New special compiler variables are added: std.compiler.callLine,
       std.compiler.callFile, possibly more.


All or nothing

   It is not possible to execute a subset of available tests in a single
   binary.

   It is a problem because:

     * A nontrivial application is likely to have a very large number of
       tests, which will take some time to perform.
     * Tests may interact. This should not occur, and it is useful to be
       able to execute a test in isolation to ensure that there is no
       interaction.

  Solution

   Expose the available unit tests to Iceland libraries.

   Each unittest block would be changed. Consider:

unittest TestFoo {
    assert(5 == foo.bar());
}

   The name, TestFoo in this case, would be mandatory. This is the only
   visible change. The second change that would be necessary would be
   behind the scenes. The unit test displayed above would be effectively
   be the same as the following definition:

void TestFoo() {
    assert(5 == foo.bar());
}

   The two are functionally identical, except that the unittest style
   will be included only in -unittest builds.

   The tests are exposed to the programmer through a special associative
   array. The keys will be the fully qualified name of the unit test
   (i.e. ``std.dtl.vector.TestSort''), and the value will be a delegate
   to the the unit test. For example:

// Not in the std namespace because it's binary-specific
module my.unittest;
alias void delegate() unittest;
unittest_t[char[]] tests;

   The list will automagically be populated at compile- or link-time with
   all the tests present in the binary being linked. The compiler need
   not be responsible for any code generation other than than detailed
   above. All other functionality can be easily be added by a
   user-library.

  Summary

     * Unit test block syntax changed to include a name.
     * Each unit test block is treated as a separate function.
     * These functions are exposed to the programmer through a magic
       associative array of delegates.
     * Other functionality is provided in a separate library.

Lack of isolation

   Unit tests in D are not separate units. That is, if I have two
   unittest blocks, and an exception (i.e. failed assertion) is thrown
   from one of them, the other test will not be run. This appears to be
   the case even if the blocks are in differing compilation units.

   This is a problem because:

     * A single error in a very long set of tests prevents other tests
       from being run. This could be very bad if it is, for example, if
       it is an automated test process where full results are desired.
     * It is useful to have skeleton tests fail with an "implement me"
       message to prevent me from forgetting about them.

  Solution

   This problem is fully solved by the above ([12][*]).

No reporting

   It is not possible to listen to test results, meaning that there is no
   way to generate reports of success or failures.

  Solution

   This problem is also fully solved by the above ([13][*]).

                                    Impact

   I am suggesting a change in the operation of a fundamental language
   feature. This will affect most existing D code and necessitate changes
   in both the compiler and standard libraries.

   I recognize that I am requesting a large change, and a lot of work for
   Walter (sorry Walter!). Despite these facts, this change is worth it
   because the problems it solves are severe. The shortcomings present in
   the current unit test facility are so great as to prevent their use in
   some large projects. These changes would fix that.

   One negative impact that this change would have is that a unit test
   build would require some custom code to run each of the tests, report
   results, etc. I do not believe that this is a significant shortcoming
   because unit test builds already require a different build target
   (-unittest builds), and because a generic version of this special code
   could be provided automatically, if desired.

                          User-land unit test library

   Much of the solutions above are designed to allow a separate library
   to be developed to add on features that the language need not provide.
   I envision a framework similar to JUnit, but I will not include a
   specification of such a library here.

                                  Conclusion

   I welcome any and all comments on this. Feel free to email me, Mike
   Swieton, at mike swieton.net. Flames, of course, should be redirected
   to the usual bit-bucket.
     _________________________________________________________________

    Footnotes

   ... call[14]1
          Strictly speaking, it is possible to make a call like
          assertEquals(..., std.compiler.currLine), but I don't consider
          this to be valid. The user should not be required to pass in
          that extra parameter.
     _________________________________________________________________


    2004-03-25
Mar 24 2004
next sibling parent Mike Swieton <mike swieton.net> writes:
Damn it, I'm never using a spell checker again. All those references to
Iceland were meant to be "userland". D'oh!

Mike Swieton
__
If the government wants us to respect the law, they should set a better
example.
Mar 24 2004
prev sibling next sibling parent reply larry cowan <larry_member pathlink.com> writes:
In article <pan.2004.03.25.05.15.18.315731 swieton.net>, Mike Swieton says...

...
   One negative impact that this change would have is that a unit test
   build would require some custom code to run each of the tests, report
   results, etc. I do not believe that this is a significant shortcoming
   because unit test builds already require a different build target
   (-unittest builds), and because a generic version of this special code
   could be provided automatically, if desired.


Good ideas!

I don't see where (in suppressed compilation form) the actual running of the
chosen tests goes.  I'd suggest that only one unittest {} without an identifier
be allowed and if it is not given, no unittests run.  This could be in a
separate module or packed up with main(), [or it could even be a unittest
statement in main()?];  If no other (named) unittests are included, this would
do all of the unittesting for small modules.  A generic version for this would
just run all of the named tests, perhaps tagging each with its ID - e.g.,

Unittest abc: messagecontent
Unittest abc: messagecontent
Unittest xyz: messagecontent

A way to specify fatal, or continue-onward-in-this-named-unittest, or
continue-with-next-named-unittest, handling is needed for the asserts.
Mar 25 2004
parent reply Mike Swieton <mike swieton.net> writes:
On Thu, 25 Mar 2004 15:03:08 +0000, larry cowan wrote:

 Good ideas!


Thanks! I wasn't really too sure myself 8-}


 I don't see where (in suppressed compilation form) the actual running of the
 chosen tests goes.


I really wasn't specifying anything as to that. The idea was that since the
unittest array was present, either: 1) the user links in a different main()
with foreach (unittest_t u, my.unittests) { u(); } code in it, or 2) the
compiler could replace main with a boilerplate main that would do that.


 A way to specify fatal, or continue-onward-in-this-named-unittest, or
 continue-with-next-named-unittest, handling is needed for the asserts.


If assert can be written by users, it can be made to throw different
objects, which the custom test running code could be aware of.

Mike Swieton
__
We must respect the other fellow's religion, but only in the sense and to the
extent that we respect his theory that his wife is beautiful and his children
smart.
	- H. L. Mencken
Mar 25 2004
parent reply larry cowan <larry_member pathlink.com> writes:
In article <pan.2004.03.25.15.49.10.89362 swieton.net>, Mike Swieton says...

On Thu, 25 Mar 2004 15:03:08 +0000, larry cowan wrote:

 Good ideas!


Thanks! I wasn't really too sure myself 8-}


 I don't see where (in suppressed compilation form) the actual running of the
 chosen tests goes.


I really wasn't specifying anything as to that. The idea was that since the
unittest array was present, either: 1) the user links in a different main()
with foreach (unittest_t u, my.unittests) { u(); } code in it, or 2) the
compiler could replace main with a boilerplate main that would do that.


If we want to be able to selectively run tests or have dependencies, the
compiler can't just throw in a boilerplate one, but that could be default -
after that, as it is now, the normal main()code runs unless we have thrown a
fatal error.  At the least, we need a default named unittest or a unnamed one -
if that doesn't exist, then run them all by default.


 A way to specify fatal, or continue-onward-in-this-named-unittest, or
 continue-with-next-named-unittest, handling is needed for the asserts.


If assert can be written by users, it can be made to throw different
objects, which the custom test running code could be aware of.


Agreed.  Though some standardization here would be useful.
Mar 25 2004
parent Mike Swieton <mike swieton.net> writes:
On Thu, 25 Mar 2004 17:51:19 +0000, larry cowan wrote:

 If we want to be able to selectively run tests or have dependencies, the
 compiler can't just throw in a boilerplate one, but that could be
 default - after that, as it is now, the normal main()code runs unless we
 have thrown a fatal error.  At the least, we need a default named
 unittest or a unnamed one - if that doesn't exist, then run them all by
 default.


I agree that a boilerplate test runner is not enough for all cases
(perhaps even most), which is why it must be able to be disabled.


If assert can be written by users, it can be made to throw different
objects, which the custom test running code could be aware of.


 Agreed.  Though some standardization here would be useful.


I had been thinking that the library-end implementation of the unit test
stuff would reside inside Phobos, or be standard in any case.

Mike Swieton
__
I am curious to find out if I am wrong.
	- Ray Tomlinson
Mar 25 2004
prev sibling next sibling parent reply C <dont respond.com> writes:
Good ideas, I see two different issues here , assert , and unittest.

I like the idea of named unittests , maybe u could also have a 'main' =

unittest that would be able to call the others enabling reporting.  I li=
ke =

the array of unittests but that sounds hard to implement ;).

I agree about the assert, but as you pointed out moving this to a librar=
y =

function would require the user to enter file and line number .  Its a =

sticky problem thats for sure, but I agree it should be all or nothing a=
nd =

it stands improving.


    Unit tests in D are not separate units. That is, if I have two
    unittest blocks, and an exception (i.e. failed assertion) is thrown=



    from one of them, the other test will not be run.


I agree , but this would require a different 'lenient' ( spelled right ?=
 ) =

assert.  It would be good to get Walters input on all of this.

On a side note , I hope these posts don't come across as too negative.  =
I =

love D , thats why im pouring as much time as I can afford into it.  I =

want it to succeed ( which I know it will, if it doesnt then ive lost al=
l =

hope for humanity ! ) , and im pretty certain so does everyone else here=
!

Rallying round the flag,
Charles

On Thu, 25 Mar 2004 00:15:18 -0500, Mike Swieton <mike swieton.net> wrot=
e:


 As suggested by Ben Hinkle, I have written a fairly formal proposal fo=


r

 changes to the unit test framework. Attached below. I realize that it'=


s =


 a bit
 lengthy, but I tried to be complete. Feedback welcome!

 Mike Swieton
 __
 The most exciting phrase to hear in science, the one that heralds the =



 most
 discoveries, is not 'Eureka!' but 'That's funny...'
 	- Isaac Asimov

                              Unit Testing in D

                       Mike Swieton - mike swieton.net

 Contents

      * [1]Introduction
      * [2]What needs to be addressed?
           + [3]Weak assert
           + [4]All or nothing
           + [5]Lack of isolation
           + [6]No reporting
      * [7]Impact
      * [8]User-land unit test library
      * [9]Conclusion

                                  Introduction

    The D programming language includes the innovative ability to embed=



    unit tests directly in the code, without the need for hefty framewo=


rks

    or libraries. However, the version of unit testing built in to D
    suffers from severe limitations. This document identifies several
    shortcomings and suggests solutions.

    This document only addresses unit testing as present in version 0.8=


1

    of the Digital Mars D compiler.

    The solutions I present here are based around this criteria:

      * No additional tools may be used. The user must need only the
        compiler/linker and a text editor, and nothing new.
      * Minimum responsibility of the language and compiler. The compil=


er

        should implement the bulk of the features, because that would
        probably be difficult to upgrade without language/compiler
        changes.

                           What needs to be addressed?

 Weak assert

    The function assert() provides poor output, and is difficult to
    improve upon in user code, because it's not a `real' function, but
    instead is a magic function, generated by the compiler.

   Wanted features

    An assertion should be able to check for many things, such as truth=


,

    equality, nullness, and generally any state that one might want to
    test for. The only one available right now in D is the test for tru=


th.

    An other feature an assertion should have is messages. When an
    assertion fails, it is useful to section know where the failure
    occurred, what the assertion is testing for, conceptually (``i is a=



    valid whole number'', and not ``i [10][IMAGE gif] 0''). Currently,
    only filename and line number are reported.

   Solution

    Make assert() a library function, and remove it from the compiler. =


It

    would then be possible for developers to overload the assert method=


,

    or delegate to it.

    This solution has a shortcoming: If a user overrides assert(), it i=


s

    not currently possible to get the filename and line number at the
    call[11]1. Fixing this would likely require changes to the language=


.

    One such possible change would be the addition of more magic
    parameters in the same vein as std.compiler.currLine, such as
    std.compiler.callLine, which would return the line that called the
    current method.

    The above is a terribly ugly hack which I am almost embarrassed to
    suggest, however, as information such as filename and line number c=


an

    only be provided by the compiler, I see no other way to do it than
    magic variables.

   Summary

      * The compiler-generated assert() is removed.
      * A new method is added to the runtime library: void assert() thr=


ows

        AssertError
      * New special compiler variables are added: std.compiler.callLine=


,

        std.compiler.callFile, possibly more.


 All or nothing

    It is not possible to execute a subset of available tests in a sing=


le

    binary.

    It is a problem because:

      * A nontrivial application is likely to have a very large number =


of

        tests, which will take some time to perform.
      * Tests may interact. This should not occur, and it is useful to =


be

        able to execute a test in isolation to ensure that there is no
        interaction.

   Solution

    Expose the available unit tests to Iceland libraries.

    Each unittest block would be changed. Consider:

 unittest TestFoo {
     assert(5 =3D=3D foo.bar());
 }

    The name, TestFoo in this case, would be mandatory. This is the onl=


y

    visible change. The second change that would be necessary would be
    behind the scenes. The unit test displayed above would be effective=


ly

    be the same as the following definition:

 void TestFoo() {
     assert(5 =3D=3D foo.bar());
 }

    The two are functionally identical, except that the unittest style
    will be included only in -unittest builds.

    The tests are exposed to the programmer through a special associati=


ve

    array. The keys will be the fully qualified name of the unit test
    (i.e. ``std.dtl.vector.TestSort''), and the value will be a delegat=


e

    to the the unit test. For example:

 // Not in the std namespace because it's binary-specific
 module my.unittest;
 alias void delegate() unittest;
 unittest_t[char[]] tests;

    The list will automagically be populated at compile- or link-time w=


ith

    all the tests present in the binary being linked. The compiler need=



    not be responsible for any code generation other than than detailed=



    above. All other functionality can be easily be added by a
    user-library.

   Summary

      * Unit test block syntax changed to include a name.
      * Each unit test block is treated as a separate function.
      * These functions are exposed to the programmer through a magic
        associative array of delegates.
      * Other functionality is provided in a separate library.

 Lack of isolation

    Unit tests in D are not separate units. That is, if I have two
    unittest blocks, and an exception (i.e. failed assertion) is thrown=



    from one of them, the other test will not be run. This appears to b=


e

    the case even if the blocks are in differing compilation units.

    This is a problem because:

      * A single error in a very long set of tests prevents other tests=



        from being run. This could be very bad if it is, for example, i=


f

        it is an automated test process where full results are desired.=



      * It is useful to have skeleton tests fail with an "implement me"=



        message to prevent me from forgetting about them.

   Solution

    This problem is fully solved by the above ([12][*]).

 No reporting

    It is not possible to listen to test results, meaning that there is=


 no

    way to generate reports of success or failures.

   Solution

    This problem is also fully solved by the above ([13][*]).

                                     Impact

    I am suggesting a change in the operation of a fundamental language=



    feature. This will affect most existing D code and necessitate chan=


ges

    in both the compiler and standard libraries.

    I recognize that I am requesting a large change, and a lot of work =


for

    Walter (sorry Walter!). Despite these facts, this change is worth i=


t

    because the problems it solves are severe. The shortcomings present=


 in

    the current unit test facility are so great as to prevent their use=


 in

    some large projects. These changes would fix that.

    One negative impact that this change would have is that a unit test=



    build would require some custom code to run each of the tests, repo=


rt

    results, etc. I do not believe that this is a significant shortcomi=


ng

    because unit test builds already require a different build target
    (-unittest builds), and because a generic version of this special c=


ode

    could be provided automatically, if desired.

                           User-land unit test library

    Much of the solutions above are designed to allow a separate librar=


y

    to be developed to add on features that the language need not provi=


de.

    I envision a framework similar to JUnit, but I will not include a
    specification of such a library here.

                                   Conclusion

    I welcome any and all comments on this. Feel free to email me, Mike=



    Swieton, at mike swieton.net. Flames, of course, should be redirect=


ed

    to the usual bit-bucket.
      _________________________________________________________________=



     Footnotes

    ... call[14]1
           Strictly speaking, it is possible to make a call like
           assertEquals(..., std.compiler.currLine), but I don't consid=


er

           this to be valid. The user should not be required to pass in=



           that extra parameter.
      _________________________________________________________________=



     2004-03-25




-- =

D Newsgroup.
Mar 25 2004
parent Mike Swieton <mike swieton.net> writes:
On Thu, 25 Mar 2004 12:18:07 -0800, C wrote:

 Good ideas, I see two different issues here , assert , and unittest.


Very true. Perhaps I should not have bundled them in the same document, but
since unit tests need an assert method, it was in my mind.


 I like the idea of named unittests , maybe u could also have a 'main' 
 unittest that would be able to call the others enabling reporting.  I like 
 the array of unittests but that sounds hard to implement ;).


I dislike the idea of a special unit test because that would be something
there would only be one of. I would like to be able to link in a ton of
libraries with full test suites and not worry about conflicting symbols.

As far as the array goes, I thought it may be difficult to implement as well.
I was just at a loss as to other ways. If you want to expose the tests to the
program, how can you do it in an easier way?

With the test array I was trying to solve the problem of needing a test
registry (used CppUnit?). Java gets around this by using reflection. That
seems harder to implement to me, but I would be perfectly satisfied (moreso
even) with it.


 I agree about the assert, but as you pointed out moving this to a library 
 function would require the user to enter file and line number .  Its a 
 sticky problem thats for sure, but I agree it should be all or nothing and 
 it stands improving.




    Unit tests in D are not separate units. That is, if I have two
    unittest blocks, and an exception (i.e. failed assertion) is thrown
    from one of them, the other test will not be run.


 
 I agree , but this would require a different 'lenient' ( spelled right ? ) 
 assert.  It would be good to get Walters input on all of this.


I'm not sure a special assert is actually necessary. I think it would be
useful for many other reasons, useful enough to warrent a user-customizable
one, but I think that these effects could be acheived with the current one.

The existing assert simply throws an exception if it fails. You wouldn't need
to replace assert if you could replace the code that caught that exception.


 On a side note , I hope these posts don't come across as too negative.  I 
 love D , thats why im pouring as much time as I can afford into it.  I 
 want it to succeed ( which I know it will, if it doesnt then ive lost all 
 hope for humanity ! ) , and im pretty certain so does everyone else here!


Exactly! Same here; it's quickly become my favorite language ;)

Mike Swieton
__
Has this world been so kind to you that you should leave with regret? There
are better things ahead than any we leave behind.
        - C. S. Lewis
Mar 25 2004
prev sibling next sibling parent reply Ilya Minkov <minkov cs.tum.edu> writes:
I would like to throw in a few ideas for discussion. Please note that 
these are to be seen separately from each other because some of them 
don't combine or combinations don't make sense.

* Assert is overrided roughly the same way as a GC. That is, by 
assigning a new assert at the run time rather than at link time, since 
the second poses severe problems for no gain.

* An exception object for assert, which needs to have such a 
constructor, or simply a function, which gets an info-string, line and 
filename. The info-string can be filled by a compiler in one of many 
manners such as:
- null, e.g. for executables, where there is no debugging information or 
source could not be found;
- source line, if compiled with debugging information and source is 
available;
- custom string if supplied in assertion?

* Custom string is not necessary when source is output, because 
assert(qexpression && "Error Description"); works as well, because 
string literal is always true.

* Separate asserts for equality, nonequality, and so on are not 
requiered. Let automatic conversions to boolean do their job. Assignment 
in assert should be forbidden, just to make sure someone doesn't make a 
silly typing mistake. The test for object existance is then 
assert(object) as it is now. In Java, such separate asserts do make 
sense due to other shortcomings, but in D (operator overloads, and so 
on) they don't.

* An assertion function needs not have the boolean parameter at all. It 
should be called by the compiler only if the assertion has failed.

* A simple tool can generate programs for automatic tests from the 
source. This is not a consern of the language, and it's probably better 
done with a tool than with a library. And i don't see why there should 
be a preferance of a library over a tool, if a tool is open and only 
requieres D standard libraries. The one trying to write such a tool can 
make use of compiler sources for fast parsing or port ANTLR to do D 
output and write a (partial?) D grammar for it.

* Assert function in fact need not be changable at all, it is enough for 
it to throw an error-like exception object which would contain all the 
information you may need (source snippet, module, line), and do nothing 
else. Then, in the program top level you can decide what to do with it. 
For example, you can display the message in a pop-up window, or onto a 
console, or log it in a file, from there. The start-up code should 
simply output them to stderr.

* If the user thinks some assertions are getting too heavy for him, he 
should make use of the version statement.

* Heavy changes on the language should not be taken now! The changes, if 
taken now, should be very simple and to prevent breaking code by changes 
requiered further on by 2.0! So consider a "partial" change which would 
give you a way of enhancement in further versions.

-eye
Mar 25 2004
next sibling parent Mike Swieton <mike swieton.net> writes:
On Thu, 25 Mar 2004 20:42:53 +0100, Ilya Minkov wrote:


 I would like to throw in a few ideas for discussion. Please note that 
 these are to be seen separately from each other because some of them 
 don't combine or combinations don't make sense.
 
 * Assert is overrided roughly the same way as a GC. That is, by 
 assigning a new assert at the run time rather than at link time, since 
 the second poses severe problems for no gain.


I think assert should be customizable, but I really don't know the best
way of doing so.


 * An exception object for assert, which needs to have such a 
 constructor, or simply a function, which gets an info-string, line and 
 filename. The info-string can be filled by a compiler in one of many 
 manners such as:
 - null, e.g. for executables, where there is no debugging information or 
 source could not be found;
 - source line, if compiled with debugging information and source is 
 available;
 - custom string if supplied in assertion?


I dislike this info-string trick, because it puts more special-case handling
in the compiler. I think that this is more special-purpose of an application.
(*usable* only by assert, whereas the change I suggested would be *useful*
only for assert. I think the distinction is important.)


 * Custom string is not necessary when source is output, because 
 assert(qexpression && "Error Description"); works as well, because 
 string literal is always true.


It's a hack. Even if it becomes a D idiom, it's still a hack.


 * Separate asserts for equality, nonequality, and so on are not 
 requiered. Let automatic conversions to boolean do their job. Assignment 
 in assert should be forbidden, just to make sure someone doesn't make a 
 silly typing mistake. The test for object existance is then 
 assert(object) as it is now. In Java, such separate asserts do make 
 sense due to other shortcomings, but in D (operator overloads, and so 
 on) they don't.


Not strictly true. Consider that C++ unit test frameworks usually add these
assertions. The primary thing that an assertEquals method gives you is good
output: it automatically constructs an error message with your got and
expected values, so you don't need to do it yourself.


 * An assertion function needs not have the boolean parameter at all. It 
 should be called by the compiler only if the assertion has failed.


An interesting proposal, but I think that adds more magic than is necessary.


 * A simple tool can generate programs for automatic tests from the 
 source. This is not a consern of the language, and it's probably better 
 done with a tool than with a library. And i don't see why there should 
 be a preferance of a library over a tool, if a tool is open and only 
 requieres D standard libraries. The one trying to write such a tool can 
 make use of compiler sources for fast parsing or port ANTLR to do D 
 output and write a (partial?) D grammar for it.


I prefer a library over a tool for several reasons:

	- D's most unique feature is built in unit testing. It shouldn't need a
	  tool beyond the compiler to use the features of the language. A library
	  is a lot closer to the language.
	- A tool adds an extra step and will make builds more complex.


 * Assert function in fact need not be changable at all, it is enough for 
 it to throw an error-like exception object which would contain all the 
 information you may need (source snippet, module, line), and do nothing 
 else. Then, in the program top level you can decide what to do with it. 
 For example, you can display the message in a pop-up window, or onto a 
 console, or log it in a file, from there. The start-up code should 
 simply output them to stderr.


The disadvantage of this approach is that your exception object is always the
same, meaning you may not have a goot way of 


 * If the user thinks some assertions are getting too heavy for him, he 
 should make use of the version statement.


I'm not clear on what you mean by this: could you clarify it? Why would the
assertions be too heavy, and how would conditional compilation help it?


 * Heavy changes on the language should not be taken now! The changes, if 
 taken now, should be very simple and to prevent breaking code by changes 
 requiered further on by 2.0! So consider a "partial" change which would 
 give you a way of enhancement in further versions.


I think the unit test shortcomings in D should be addressed. If Walter wants
to do it in 2.0, fine ;) But I do think it should be addressed.

Mike Swieton
__
You can have peace. Or you can have freedom. Don't ever count on having
both at once.
	- Lazarus Long (Robert A. Heinlein)
Mar 25 2004
prev sibling parent Mark T <Mark_member pathlink.com> writes:
Assignment 
in assert should be forbidden, just to make sure someone doesn't make a 
silly typing mistake. >


good suggestion


* A simple tool can generate programs for automatic tests from the 
source. This is not a consern of the language, and it's probably better 
done with a tool than with a library. And i don't see why there should 
be a preferance of a library over a tool, if a tool is open and only 
requieres D standard libraries. The one trying to write such a tool can 
make use of compiler sources for fast parsing or port ANTLR to do D 
output and write a (partial?) D grammar for it.


agreed
actually this would help generate a test framework, the test cases still have be
written.

The D unittest facility still needs a test driver like JUnit to cover all the
desired test cases for a module since multiple instances of the class under test
might have to be created or a module may only consist of related functions such
as a math module.

Don't mess up the language by overdoing the built-in unittest facility.
Mar 26 2004
prev sibling parent Ben Hinkle <bhinkle4 juno.com> writes:
On Thu, 25 Mar 2004 00:15:18 -0500, Mike Swieton <mike swieton.net>
wrote:


As suggested by Ben Hinkle, I have written a fairly formal proposal for
changes to the unit test framework. Attached below. I realize that it's a bit
lengthy, but I tried to be complete. Feedback welcome!

Mike Swieton


Thanks for posting such a nice document (really!). It helps me focus.
__

The most exciting phrase to hear in science, the one that heralds the most
discoveries, is not 'Eureka!' but 'That's funny...'
	- Isaac Asimov

                             Unit Testing in D

                      Mike Swieton - mike swieton.net

Contents

     * [1]Introduction
     * [2]What needs to be addressed?
          + [3]Weak assert
          + [4]All or nothing
          + [5]Lack of isolation
          + [6]No reporting
     * [7]Impact
     * [8]User-land unit test library
     * [9]Conclusion

                                 Introduction

   The D programming language includes the innovative ability to embed
   unit tests directly in the code, without the need for hefty frameworks
   or libraries. However, the version of unit testing built in to D
   suffers from severe limitations. This document identifies several
   shortcomings and suggests solutions.

   This document only addresses unit testing as present in version 0.81
   of the Digital Mars D compiler.

   The solutions I present here are based around this criteria:

     * No additional tools may be used. The user must need only the
       compiler/linker and a text editor, and nothing new.
     * Minimum responsibility of the language and compiler. The compiler
       should implement the bulk of the features, because that would
       probably be difficult to upgrade without language/compiler
       changes.

                          What needs to be addressed?

Weak assert

   The function assert() provides poor output, and is difficult to
   improve upon in user code, because it's not a `real' function, but
   instead is a magic function, generated by the compiler.

  Wanted features

   An assertion should be able to check for many things, such as truth,
   equality, nullness, and generally any state that one might want to
   test for. The only one available right now in D is the test for truth.

   An other feature an assertion should have is messages. When an
   assertion fails, it is useful to section know where the failure
   occurred, what the assertion is testing for, conceptually (``i is a
   valid whole number'', and not ``i [10][IMAGE gif] 0''). Currently,
   only filename and line number are reported.

  Solution

   Make assert() a library function, and remove it from the compiler. It
   would then be possible for developers to overload the assert method,
   or delegate to it.

   This solution has a shortcoming: If a user overrides assert(), it is
   not currently possible to get the filename and line number at the
   call[11]1. Fixing this would likely require changes to the language.

   One such possible change would be the addition of more magic
   parameters in the same vein as std.compiler.currLine, such as
   std.compiler.callLine, which would return the line that called the
   current method.


Here's another variation on how to get the file+line: 
assert could take a function that gets passed the file and line
as input arguments. So
 assert(expr,fcn)
tests the expr and calls Object fcn(char[] filename, int lineno)
if it fails and throws the result of fcn. 
If the fcn can be a nested function then nice message formatting
can happen. If fcn returns null the default AssertError is thrown.
On a related note I wish the filename and line number in
AssertError weren't private. Read-only would be ok. but private
just is too restrictive.


   The above is a terribly ugly hack which I am almost embarrassed to
   suggest, however, as information such as filename and line number can
   only be provided by the compiler, I see no other way to do it than
   magic variables.

  Summary

     * The compiler-generated assert() is removed.
     * A new method is added to the runtime library: void assert() throws
       AssertError
     * New special compiler variables are added: std.compiler.callLine,
       std.compiler.callFile, possibly more.


All or nothing

   It is not possible to execute a subset of available tests in a single
   binary.

   It is a problem because:

     * A nontrivial application is likely to have a very large number of
       tests, which will take some time to perform.
     * Tests may interact. This should not occur, and it is useful to be
       able to execute a test in isolation to ensure that there is no
       interaction.

  Solution

   Expose the available unit tests to User-land libraries.


The naming seems fine to me but having the compiler maintain a
test suite and the test harness is (IMHO) too much for the 
compiler. A variation is to have the default behavior of
the compiler to ignore the test names (if present) and just
run everything like it does now. However the user should be
able to plug in a custom test harness that gets called
by the compiler each time it wants to run a unittest
and it passes it the unittest name and function. The user
harness can filter whatever tests it wants and/or log the
tests and/or catch error so that if a unittest fails the
program can continue.


   Each unittest block would be changed. Consider:

unittest TestFoo {
    assert(5 == foo.bar());
}

   The name, TestFoo in this case, would be mandatory. This is the only
   visible change. The second change that would be necessary would be
   behind the scenes. The unit test displayed above would be effectively
   be the same as the following definition:

void TestFoo() {
    assert(5 == foo.bar());
}

   The two are functionally identical, except that the unittest style
   will be included only in -unittest builds.

   The tests are exposed to the programmer through a special associative
   array. The keys will be the fully qualified name of the unit test
   (i.e. ``std.dtl.vector.TestSort''), and the value will be a delegate
   to the the unit test. For example:

// Not in the std namespace because it's binary-specific
module my.unittest;
alias void delegate() unittest;
unittest_t[char[]] tests;

   The list will automagically be populated at compile- or link-time with
   all the tests present in the binary being linked. The compiler need
   not be responsible for any code generation other than than detailed
   above. All other functionality can be easily be added by a
   user-library.

  Summary

     * Unit test block syntax changed to include a name.
     * Each unit test block is treated as a separate function.
     * These functions are exposed to the programmer through a magic
       associative array of delegates.
     * Other functionality is provided in a separate library.

Lack of isolation

   Unit tests in D are not separate units. That is, if I have two
   unittest blocks, and an exception (i.e. failed assertion) is thrown
   from one of them, the other test will not be run. This appears to be
   the case even if the blocks are in differing compilation units.

   This is a problem because:

     * A single error in a very long set of tests prevents other tests
       from being run. This could be very bad if it is, for example, if
       it is an automated test process where full results are desired.
     * It is useful to have skeleton tests fail with an "implement me"
       message to prevent me from forgetting about them.

  Solution

   This problem is fully solved by the above ([12][*]).


Yup, would be nice to have a hook here.


No reporting

   It is not possible to listen to test results, meaning that there is no
   way to generate reports of success or failures.

  Solution

   This problem is also fully solved by the above ([13][*]).


Again I agree a hook would be nice.


                                    Impact

   I am suggesting a change in the operation of a fundamental language
   feature. This will affect most existing D code and necessitate changes
   in both the compiler and standard libraries.

   I recognize that I am requesting a large change, and a lot of work for
   Walter (sorry Walter!). Despite these facts, this change is worth it
   because the problems it solves are severe. The shortcomings present in
   the current unit test facility are so great as to prevent their use in
   some large projects. These changes would fix that.

   One negative impact that this change would have is that a unit test
   build would require some custom code to run each of the tests, report
   results, etc. I do not believe that this is a significant shortcoming
   because unit test builds already require a different build target
   (-unittest builds), and because a generic version of this special code
   could be provided automatically, if desired.

                          User-land unit test library

   Much of the solutions above are designed to allow a separate library
   to be developed to add on features that the language need not provide.
   I envision a framework similar to JUnit, but I will not include a
   specification of such a library here.

                                  Conclusion

   I welcome any and all comments on this. Feel free to email me, Mike
   Swieton, at mike swieton.net. Flames, of course, should be redirected
   to the usual bit-bucket.
     _________________________________________________________________

    Footnotes

   ... call[14]1
          Strictly speaking, it is possible to make a call like
          assertEquals(..., std.compiler.currLine), but I don't consider
          this to be valid. The user should not be required to pass in
          that extra parameter.
     _________________________________________________________________


    2004-03-25
Mar 25 2004