www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Unit Threaded - a unit testing library for D

reply "Atila Neves" <atila.neves gmail.com> writes:
My very first D project is now feature-complete as far as I can 
see (alpha) and I'd like to share it with the community:

https://github.com/atilaneves/unit-threaded

There are more details on github but here are the highlights:

1. Automatic registration of unit tests via compile-time 
reflection
2. Unit tests can be classes or functions, the latter for minimal 
boilerplate
3. Support for D's unittest blocks
4. Runs in multiple threads by default, possible to run in one 
thread
5. Manual selection of tests to run at the command-line (runs all 
tests by default)

I've used it myself to test my other D projects (2 other for now) 
to make sure it did what I wanted it to. I basically wrote this 
for myself knowing what I'd want from a unit testing library, but 
I'm sure it can be useful for the general D public.

Feedback is of course more than welcome!

Atila

P.S. I totally expect this to break for large real-world projects 
in its current state. It can test some modules in phobos but had 
problems with others, for instance. My own other D projects are 
tiny and even if I had a large D codebase the tests I'd write 
would reflect the biases that go into this library.
Aug 27 2013
next sibling parent "Meta" <jared771 gmail.com> writes:
On Tuesday, 27 August 2013 at 13:07:02 UTC, Atila Neves wrote:
I cannot tell you how handy this will be. I've got a couple of 
projects that badly need this as is, and I can see myself using 
it for all future projects of any notable size.
Aug 27 2013
prev sibling next sibling parent reply "Tobias Pankrath" <tobias pankrath.net> writes:
"But doesn't D have built-in unittest blocks"? Yes, and they're 
massively useful. Even short scripts can benefit from them with 0 
effort and setup. In fact, I use them to test this library. 
However, for larger projects it lacks some functionality:

     If all tests pass, great. If one fails, it's hard to know why.
     The only tool is assert, and you have to write your own 
assert messages (no assertEqual, assertNull, etc.)
     No possibility to run just one particular test
     Only runs in one thread.


-------------

That's exactly the reason I did something like that, too. 
Especially the first and third reason.

However I use a different approach, which I find easier to 
handle: I'm using the build-in unittest blocks to register 
delegates as tests in a central table. These than get executed by 
a test runner in main().

This way you can register multiple tests using i.e. a loop over 
sample data or something similar. And it's very easy to implement 
and has almost no impact on compile time.
Aug 27 2013
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 8/27/13, Tobias Pankrath <tobias pankrath.net> wrote:
 If all tests pass, great. If one fails, it's hard to know why.
Quoting you but responding to OP: There was a pull I made to make assert print out some more info on failure, but it never passed the test-suite. Pull: https://github.com/D-Programming-Language/dmd/pull/1426
      The only tool is assert, and you have to write your own
 assert messages (no assertEqual, assertNull, etc.)
Currently we have assertThrown/assertNotThrown, but assertEqual and friends are easy to implement and we thought about including them. However it needs more community support before it's included in Phobos (i.e. more people need to want these). I have some of these functions here: https://github.com/AndrejMitrovic/minilib/blob/master/src/minilib/core/test.d
      No possibility to run just one particular test
E.g. test only one unittest block rather than the entire module at once? I'd imagine this will be possible with the new getUnitTests trait in 2.064.
      Only runs in one thread.
getUnitTests should help here, although it's still based on compile-time introspection just like your library. I guess if nothing else, this trait will make your library implementation simpler. :)
Aug 27 2013
next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
 Currently we have assertThrown/assertNotThrown, but assertEqual 
 and
 friends are easy to implement and we thought about including 
 them.
 However it needs more community support before it's included in 
 Phobos
 (i.e. more people need to want these).
They're easy to implement and I did to test the library itself. But it's either reimplement them everytime for each project or format assert messages manually. Neither appealed to me.
      No possibility to run just one particular test
E.g. test only one unittest block rather than the entire module at once?
Yes.
I'd imagine this will be possible with the new getUnitTests 
trait in 2.064.
Hadn't heard of that, I'm going to have to take a look.
 getUnitTests should help here, although it's still based on
 compile-time introspection just like your library. I guess if 
 nothing
 else, this trait will make your library implementation simpler. 
 :)
Yay? :)
Aug 27 2013
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 27 August 2013 at 15:42:28 UTC, Andrej Mitrovic wrote:
 getUnitTests should help here, although it's still based on
 compile-time introspection just like your library. I guess if 
 nothing
 else, this trait will make your library implementation simpler. 
 :)
By the, way, can we currently in 2.064 attach UDA's to unittest blocks? Together with getUnitTest that will allow to have same test code base that acts in both "old-school" mode and gets used by some fancy introspection library.
Aug 27 2013
next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 8/27/13, Dicebot <public dicebot.lv> wrote:
 By the way, can we currently in 2.064 attach UDA's to unittest
 blocks?
Yes. E.g.: ----- alias Seq(T...) = T; struct RunTest { } struct SkipTest { } RunTest unittest { } SkipTest unittest { } void main() { alias Tests = Seq!(__traits(getUnitTests, test)); // http://d.puremagic.com/issues/show_bug.cgi?id=7804 workaround alias Attrib1 = Seq!(__traits(getAttributes, Tests[0]))[0]; // ditto alias Attrib2 = Seq!(__traits(getAttributes, Tests[1]))[0]; static assert(is(Attrib1 == RunTest)); static assert(is(Attrib2 == SkipTest)); } ----- The Seq() nonsense is to avoid running into Issue 7804. Working with attributes is generally very quirky, I had to define a whole set of __traits(compiles) versions of templates just in order to be able to extract various attributes from symbols (an attribute can be a type, an instance, a template declaration, a template instance, a literal, etc...). Here's some of the work I've had to do to work with attributes: Helper functions: https://github.com/AndrejMitrovic/minilib/blob/master/src/minilib/core/attributes.d And I use the attributes to enable things like (nevermind the ironic name "simple" :p): https://github.com/AndrejMitrovic/minilib/blob/master/src/minilib/parser/simple_ini/tests.d
Aug 27 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-08-27 18:59, Dicebot wrote:

 By the, way, can we currently in 2.064 attach UDA's to unittest blocks?
 Together with getUnitTest that will allow to have same test code base
 that acts in both "old-school" mode and gets used by some fancy
 introspection library.
Yes, and unit tests for CTFE: http://forum.dlang.org/thread/ks1brj$1l6c$1 digitalmars.com -- /Jacob Carlborg
Aug 27 2013
prev sibling next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On Tue, 27 Aug 2013 17:42:19 +0200
Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:
 
      The only tool is assert, and you have to write your own
 assert messages (no assertEqual, assertNull, etc.)
Currently we have assertThrown/assertNotThrown, but assertEqual and friends are easy to implement and we thought about including them. However it needs more community support before it's included in Phobos (i.e. more people need to want these).
Someone actually crated all of those and made a pull request to Phobos, but it was rejected because people decided it was better to improve the regular assert to be able to display such information automatically. Predictably, these hypothetical improvements to assert never actually got made (it's been probably around a couple years now), so that's why we've still got nothing like that in the stock DMD/Phobos. I do wish we weren't in the habit of letting "perfect" get in the way of progress.
Aug 27 2013
prev sibling parent reply "linkrope" <linkrope github.com> writes:
On Tuesday, 27 August 2013 at 15:42:28 UTC, Andrej Mitrovic wrote:
 I have some of these functions here:
 https://github.com/AndrejMitrovic/minilib/blob/master/src/minilib/core/test.d
Where is your 'assertOp' from the comments of issue 4653? I recently added it to https://github.com/linkrope/dunit
Aug 27 2013
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 8/28/13, linkrope <linkrope github.com> wrote:
 On Tuesday, 27 August 2013 at 15:42:28 UTC, Andrej Mitrovic wrote:
 I have some of these functions here:
 https://github.com/AndrejMitrovic/minilib/blob/master/src/minilib/core/test.d
Where is your 'assertOp' from the comments of issue 4653? I recently added it to https://github.com/linkrope/dunit
It appears I didn't add it! Thanks for reminding me.
Aug 27 2013
prev sibling parent "Atila Neves" <atila.neves gmail.com> writes:
I haven't had a problem with compile times. Ideally the test code 
should be small and well-separated into modules so I'm not too 
worried about that (the compile-time scanning only happens on the 
test modules).

As for the registration, it was one of my goals to not have to do 
any. You could still conceivably loop at compile-time with a 
foreach on a tuple and generate the required test functions / 
classes. In my experience there's usually no need for that kind 
of thing, but YMMV.


 However I use a different approach, which I find easier to 
 handle: I'm using the build-in unittest blocks to register 
 delegates as tests in a central table. These than get executed 
 by a test runner in main().

 This way you can register multiple tests using i.e. a loop over 
 sample data or something similar. And it's very easy to 
 implement and has almost no impact on compile time.
Aug 27 2013
prev sibling next sibling parent reply Russel Winder <russel winder.org.uk> writes:
On Tue, 2013-08-27 at 15:07 +0200, Atila Neves wrote:
 My very first D project is now feature-complete as far as I can=20
 see (alpha) and I'd like to share it with the community:
=20
 https://github.com/atilaneves/unit-threaded
=20
 There are more details on github but here are the highlights:
The trend in other languages, especially the Java-verse, is to use the same testing framework for unit, integration and some system testing. By using the name "Unit Threaded", and talking of unit testing, you seem to be stating that this framework is no good for integration and system testing, just unit testing. Is this the case? Thanks. --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Aug 27 2013
parent "Atila Neves" <atila.neves gmail.com> writes:
Not at all. In fact, I was playing with using it to do
integration testing at work for our C++ code (it has a C
interface). I guess I was just focussed on unit testing because I
got annoyed at how long our unit tests* take to run, which was
what led me to write the C++11 version to begin with. After your
post I've got half a mind to try and speed up all aspects of
testing at work. Thanks!

I guess I or someone else will just have to try and use the
library for other types of testing and fix anything that might
need to be fixed.

I even mentioned at work that if I started a new company or large
project that I'd use D to write tests for anything with a C
interface. There's less glue to write (even with Swig, getting
the Python bindings to work can be a pain), it compiles lightning
fast and is just as nice as writing, say, Python.

Atila

* Even though part of the reason they take so long to run is that
quite a few of them are unit tests in name only, being
mini-integration tests in practice.

 The trend in other languages, especially the Java-verse, is to 
 use the
 same testing framework for unit, integration and some system 
 testing. By
 using the name "Unit Threaded", and talking of unit testing, 
 you seem to
 be stating that this framework is no good for integration and 
 system
 testing, just unit testing. Is this the case?

 Thanks.
Aug 27 2013
prev sibling parent "Atila Neves" <atila.neves gmail.com> writes:
I finally got around to putting it up in the dub registry.

http://code.dlang.org/packages/unit-threaded


On Tuesday, 27 August 2013 at 13:07:02 UTC, Atila Neves wrote:
 My very first D project is now feature-complete as far as I can 
 see (alpha) and I'd like to share it with the community:

 https://github.com/atilaneves/unit-threaded

 There are more details on github but here are the highlights:

 1. Automatic registration of unit tests via compile-time 
 reflection
 2. Unit tests can be classes or functions, the latter for 
 minimal boilerplate
 3. Support for D's unittest blocks
 4. Runs in multiple threads by default, possible to run in one 
 thread
 5. Manual selection of tests to run at the command-line (runs 
 all tests by default)

 I've used it myself to test my other D projects (2 other for 
 now) to make sure it did what I wanted it to. I basically wrote 
 this for myself knowing what I'd want from a unit testing 
 library, but I'm sure it can be useful for the general D public.

 Feedback is of course more than welcome!

 Atila

 P.S. I totally expect this to break for large real-world 
 projects in its current state. It can test some modules in 
 phobos but had problems with others, for instance. My own other 
 D projects are tiny and even if I had a large D codebase the 
 tests I'd write would reflect the biases that go into this 
 library.
Nov 04 2013