www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - About unittest, test runners and assert libraries

reply Zardoz <luis.panadero gmail.com> writes:
I like to talk about the state of unit testing, test runner & 
frameworks and assert libraries, as I did some little experiments 
recently.

DLang, put a strong emphasis on unit testing. Language & tooling 
support for unit testing (unittest blocks, assert, assert helpers 
on std.exception and AssertError on core.exception, dub test ...)

However, the out of the box test runner embedded on the guts of 
DLang, it's pretty limited. So, many test runner has been write, 
as can bee see on dub package repository. This by itself, isn't 
bad. Having several test runner to choose it's good. Specially 
when one test runner it's specialized on some ways that other 
not.. being faster or being straightforward to use, make more 
easy to integrate with IDE, etc. Some of these test runners, are 
really more a test framework, as include some helper libraries 
for mocking & stubing, fluent asserts, bdd, etc.

And separated of the test runners, we have some auxiliary 
libraries, that implements mocking, fluent asserts, etc. Aiming 
to allow to be used on agnostic way respect the test runner, or 
even on the default test runner that comes with DLang.

So, I like to show what I saw trying the five most popular test 
runners (Unit-threaded, Silly, Trial, DUnit and D-Unit), and a 
little details that puzzles me about the different assert 
libraries.



D-Unit try to be a simple implementation of the xUnit Testing 
Framework. It complete ignores unittest blocks and requires to 
encapsulate the tests on a class, following the same pattern that 
does xUnit/jUnit (but using a mixin, instead of extending from 
some base class). Includes some assert helpers following the 
xUnit/jUnit framework, etc. Even can generate xUnit XML report 
files and differentiated between a failed test (ie. a test case 
where an assertion has failed) from an errored test (ie. a test 
where a uncaught exception has been raised)

However, isn't straightforward to use (dub run -c 
*my-d-unit-config*), and only understand as failed test only when 
it's being using his own assert helper functions. A failed assert 
form any other 3rd party assert library will be marked as an 
error.



DUnit tries to being another full framework with mocking & assert 
helpers.

Sadly, it's dead. The **githup repo it's archived*, and don't see 
any update since early 2020's. Also, isn't very straightforward 
to use (requires some extra config on dub.json, but at least 
could be run with a simple dub test). Other annoying thing, it's 
that the runner iot's pretty limited. Can't list the tests, run a 
single test, or show anything useful if the test run OK. Also, 
the pretty print of a error/fail on a test it's implemented on 
his own assert helpers. Using 3rd party assert library on it, 
would be the same experience that using on the DLang embed test 
runner.



Trial aims to be a very powerful test runner. It allow to use a 
config file to select the reporters (in plural), how discover 
tests, policies, and test runner implementation. Also, aims to be 
more easy to integrate with IDEs. Even the author had a Visual 
Code extension that auto uses Trial to discover tests on a 
project. Sadly this extension it's dead, as the author saw zero 
interest on it.

Using directly Trial, it's really weird. I couldn't find a way to 
allow it to be used as dub test or dub run on the project. 
Instead, you must build the Trial executable (dub run trial does 
the work), and it will find&execute the test in your project. 
Also, on my case, only worked using the master branch. Because 
Trial uses internally dub (compiles dub in it'self) and so, it's 
strongly coupled with dub.

It's really sad about the Visual Code extension, because gives 
the quick&easiest experience to run & debug tests that I saw 
using DLang on a IDE.



Silly tries to being a better test runner that it's simple to 
use, could run test on multiple threads, color output, 
list&filter tests.

To use, it's really the most easy & straightforward of any test 
runner that I try. I simply need to add a dependency to Silly, 
and run dub test. If it's being used on a library, it's 
recommended to add the dependency on a on "unittest" config or 
use it on a sub package or separated dub.json .

In my personal opinion, it's the best simply because it's the 
most easy to use. And don't give headhaches to configure or 
require special stuff to discover tests, or put tests inside 
classes, etc. Gives a quick&short summary, and gives good 
information when a test fails/errors.

However, there it's room for improvement seeing what Trial and 
D-Unit could do. Also looks that the author it's a bit absent. 
Plus, there is some issues on the gitlab repository...



Unit-threaded shares some goodies with Silly. Like his name 
pinpoints, run the tests on multiple threads. Also, aims to being 
a testing framework, including hos own assertions library, 
mocking helpers, sandboxing filesystem, integration testing 
facilities...

However, to discover tests, requires manually generating a file 
to register the tests or add a prebuildcommand to dub.json to 
autogenerate it. Something very weird, when the other tests 
runners can discover the test on the fly. Also, this make more 
problematic run some tests only when a dub configuration it's 
activated.

Another weird thing, it's that expects that 3rd party assert 
libraries, use/extend from his own exception (UnitTestException) 
to fail on a test. If not, it show not useful junk stack trace on 
the output.




And this last details, makes my head to scratch, because I saw 
this on many assert libraries. Every assert library, throws a 
plain Exception (or UnitTestException if Unit-threaded it's added 
as dependency). This make really impossible to differentiate a 
failed test from an errored test (Remember, an test case where an 
assertion fails, its a fail test. An test case where a uncaught 
exception it's raised its an error.)

To me, sound logic that any assert library should use the same 
exception that it's throw by DLang's assert(). This is 
AssertError. And that tests runners should count any AssertError 
(or derived) inside a test case as failed, and any other 
exception as an error. This allows to assert libraries and assert 
runners work better without needing to be coupled to extend from 
some internal Exception implemented by a test runner and to have 
code on assert libraries to handle every test runner on the wild.

So, I like to ask **WHY no body does this**. Perhaps i don't see 
the inconvenient. It's a thing that puzzles me, and I think that 
should be fixed. It's one of many rought edged that have DLang 
and that it's pretty to fix.

I have my little experiment fork/branch of Silly doing exactly 
this, and Pijamas develop branch using AssertError. The result, 
it's that Silly counts correctly failed and errored tests and 
Silly & Pijamas not have any coupling between.

PD :  Sorry for my poor english. Also, I have a strong background 
with Java and JavaScript, so I have tendency to use jUnit, Spock, 
jShould, etc... as some kind of gold standard for how unit 
testing should be.
May 10 2021
next sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Monday, 10 May 2021 at 17:20:54 UTC, Zardoz wrote:

 I like to talk about the state of unit testing, test runner & 
 frameworks and assert libraries, as I did some little 
 experiments recently.

 [...]


Just some comments regarding D-Unit:

You can run it just using `dub test` by using a dub configuration 
`unittest`.

If I am not completely wrong it does not ignore `unittest` 
blocks. It shows the result of the unittest blocks in the output.

You use asserts in your productive logic for program errors, 
therefore it make sense that D-Unit Mark's them as errors. (For 
resource errors on the other hand you use Exceptions).

Kind regards
Andre
May 10 2021
parent reply Zardoz <luis.panadero gmail.com> writes:
On Monday, 10 May 2021 at 17:45:03 UTC, Andre Pany wrote:

 On Monday, 10 May 2021 at 17:20:54 UTC, Zardoz wrote:

 I like to talk about the state of unit testing, test runner & 
 frameworks and assert libraries, as I did some little 
 experiments recently.

 [...]


 Just some comments regarding D-Unit:

 You can run it just using `dub test` by using a dub 
 configuration `unittest`.

 If I am not completely wrong it does not ignore `unittest` 
 blocks. It shows the result of the unittest blocks in the 
 output.

 You use asserts in your productive logic for program errors, 
 therefore it make sense that D-Unit Mark's them as errors. (For 
 resource errors on the other hand you use Exceptions).

 Kind regards
 Andre


It just ignores the unit test blocks, You can try here : 
https://github.com/Zardoz89/pijamas/tree/develop

Launch with dub run --root=tests/d-unit
Also, you can launch with dub run --root=tests/d-unit -c 
fail-tests to enable enforce some failing/errors on the test 
cases.

Also, I try again with dub test . It just executes the dlang 
default unit test runner. Perhaps I have something wrong on 
dub.json
May 12 2021
parent reply Andre Pany <andre s-e-a-p.de> writes:
On Wednesday, 12 May 2021 at 18:52:13 UTC, Zardoz wrote:

 On Monday, 10 May 2021 at 17:45:03 UTC, Andre Pany wrote:

 On Monday, 10 May 2021 at 17:20:54 UTC, Zardoz wrote:

 I like to talk about the state of unit testing, test runner & 
 frameworks and assert libraries, as I did some little 
 experiments recently.

 [...]


 Just some comments regarding D-Unit:

 You can run it just using `dub test` by using a dub 
 configuration `unittest`.

 If I am not completely wrong it does not ignore `unittest` 
 blocks. It shows the result of the unittest blocks in the 
 output.

 You use asserts in your productive logic for program errors, 
 therefore it make sense that D-Unit Mark's them as errors. 
 (For resource errors on the other hand you use Exceptions).

 Kind regards
 Andre


 It just ignores the unit test blocks, You can try here : 
 https://github.com/Zardoz89/pijamas/tree/develop

 Launch with dub run --root=tests/d-unit
 Also, you can launch with dub run --root=tests/d-unit -c 
 fail-tests to enable enforce some failing/errors on the test 
 cases.

 Also, I try again with dub test . It just executes the dlang 
 default unit test runner. Perhaps I have something wrong on 
 dub.json


You need to add to dub.json a configuration with name "unittest". 
Within this configuration you set "mainSourceFile" to "apptest.d" 
and you need to exclude you module containing the productive main 
function.

In apptest.d you add a main function and call the dunit_main 
function.

Dub test will cause the right compiler flags to have unittests 
included into your executable. By just using dub run, your 
executable likely does not have the unittests.

Kind regards
Andre
May 12 2021
parent Zardoz <luis.panadero gmail.com> writes:
On Wednesday, 12 May 2021 at 21:23:08 UTC, Andre Pany wrote:

 On Wednesday, 12 May 2021 at 18:52:13 UTC, Zardoz wrote:

 [...]


 You need to add to dub.json a configuration with name 
 "unittest". Within this configuration you set "mainSourceFile" 
 to "apptest.d" and you need to exclude you module containing 
 the productive main function.

 In apptest.d you add a main function and call the dunit_main 
 function.

 Dub test will cause the right compiler flags to have unittests 
 included into your executable. By just using dub run, your 
 executable likely does not have the unittests.

 Kind regards
 Andre


Thanks! I manage to get it working.
May 12 2021
prev sibling parent reply SealabJaster <sealabjaster gmail.com> writes:
On Monday, 10 May 2021 at 17:20:54 UTC, Zardoz wrote:

 To me, sound logic that any assert library should use the same 
 exception that it's throw by DLang's assert(). This is 
 AssertError. And that tests runners should count any 
 AssertError (or derived) inside a test case as failed, and any 
 other exception as an error. This allows to assert libraries 
 and assert runners work better without needing to be coupled to 
 extend from some internal Exception implemented by a test 
 runner and to have code on assert libraries to handle every 
 test runner on the wild.


I see your point here, but I don't think it's as straightforward.

Imagine a custom array type sort of like this, being tested:

```d
     struct MyArray
     {
         int length;
         string get(int index)
         {
             assert(index < length, "Index out of bounds.");
             return "Hello!";
         }
     }

     unittest
     {
         MyArray array;
         array.length = 2;
         assert(array.get(1) == "Hello!"); // Success
         assert(array.get(2) == "Hello!"); // Assert thrown by 
MyArray.get
     }
```

My point is, would the assert thrown by `MyArray.get` be 
considered a test failure, or a programmer error? The test runner 
wouldn't know since it can't distinguish between the two cases 
with just `AssertError`.

While the Error classes are for unrecoverable errors (programmer 
error/bugs), the Exception classes are for known bad-states that 
can be recovered from. E.g. JSON parsing failure is an Exception, 
not an Error.

So in general, it just feels more natural to catch an Exception 
instead of an Error, since catching an Error is bad practice 
anyway.

If you derive from AssertError, you still need to make your own 
assert functions/throw it manually, like exceptions, so it isn't 
really too much different from just handling exceptions instead.

Just an aside, I use Silly since it's easy to use; unittests can 
still run without it since it only needs a pure-string UDA 
` ("Test name")`, and honestly it makes me wonder why D's builtin 
ability to run unittests is so limited.
May 10 2021
parent Zardoz <luis.panadero gmail.com> writes:
On Monday, 10 May 2021 at 19:52:28 UTC, SealabJaster wrote:

 [...]

 Imagine a custom array type sort of like this, being tested:

 ```d
     struct MyArray
     {
         int length;
         string get(int index)
         {
             assert(index < length, "Index out of bounds.");
             return "Hello!";
         }
     }

     unittest
     {
         MyArray array;
         array.length = 2;
         assert(array.get(1) == "Hello!"); // Success
         assert(array.get(2) == "Hello!"); // Assert thrown by 
 MyArray.get
     }
 ```

 My point is, would the assert thrown by `MyArray.get` be 
 considered a test failure, or a programmer error? The test 
 runner wouldn't know since it can't distinguish between the two 
 cases with just `AssertError`.

 While the Error classes are for unrecoverable errors 
 (programmer error/bugs), the Exception classes are for known 
 bad-states that can be recovered from. E.g. JSON parsing 
 failure is an Exception, not an Error.

 So in general, it just feels more natural to catch an Exception 
 instead of an Error, since catching an Error is bad practice 
 anyway.

 If you derive from AssertError, you still need to make your own 
 assert functions/throw it manually, like exceptions, so it 
 isn't really too much different from just handling exceptions 
 instead.

 [...]


I see. I will do that Pijamas throw a custom Exception that 
extends from the appropriated exception from the test runner.
May 12 2021