www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Low level unit test library in druntime

reply Jacob Carlborg <doob me.com> writes:
I've been thinking lately about unit tests in D. The built-in support is 
a bit lacking. There's been many threads with this topic, even an 
attempt to get unit-threaded (or parts of it) into druntime/Phobos.

I was thinking, instead of trying to come up with a unit test framework 
that will satisfy everyone's wishes, we could create a low level unit 
test library in druntime. There are many styles of unit test frameworks 
available that look very different, but (as far as I know) most of them 
have very similar functionality and behavior. Although, not all of them 
may have all of the functionality.

The unit test library would provide functionality for registering test, 
before and after hooks, indicating test failures, running the tests and 
so on. This low level library can then be used to build different kind 
of unit test frameworks on top of. Be it something simple as what we 
have now in D, JUnit style or something like RSpec. The library would 
not provide functionality for finding/collecting the unit tests, that 
part is very framework dependent.

If this is added do druntime, we would update the existing runner to use 
this new library and (at least as a start) configure it to have the 
exact same behavior as the existing runner

To give a better idea of what I'm talking about, at the bottom is an 
example of how the API of the library could look like. Please don't 
focus on any details in the API, it only acts like any illustration to 
give a better understanding of the above descriptions.

Thoughts? Is this something we would like to have in druntime?

class UnitTester
{
     bool parallel;
     bool stopOnFirstFaliure;
     Formatter formatter;

     void beginTestGroup(string title = null, string file = null,
         size_t line = 0);
     void endTestGroup();

     void registerTest(void delegate() test, string title = null,
         string file = null, size_t line = 0);

     void registerTest(Context)(void delegate(Context) test, Context 
context,
         string title = null, string file = null, size_t line = 0);

     void registerBeforeSuiteCallback(void delegate() callback);
     void registerBeforeAllCallback(void delegate() callback);
     void registerBeforeEachCallback(void delegate() callback);

     void registerAfterSuiteCallback(void delegate() callback);
     void registerAfterAllCallback(void delegate() callback);
     void registerAfterEachCallback(void delegate() callback);

     void testFailed(Expected, Actual)(Expected expected, Actual actual);
     void testPassed();
     void testSkipped(string reason = null);
     void testPending(string reason = null);

     void runAll();
     void runSpecfic(string path, size_t line = 0);
     void runSpecfic(Regex regex, string path = null);
}

-- 
/Jacob Carlborg
Aug 26 2016
next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Friday, 26 August 2016 at 17:13:23 UTC, Jacob Carlborg wrote:

 I've been thinking lately about unit tests in D. The built-in 
 support is a bit lacking. There's been many threads with this 
 topic, even an attempt to get unit-threaded (or parts of it) 
 into druntime/Phobos.

 [...]


I'm obviously very biased but having written a unit testing 
library that I'm happy with and proud of I'd just use that 
instead. I wouldn't be interested in this.

I'd much rather have `assert` be magical or have AST macros to 
make the syntax for writing tests better than what it is now.

Atila
Aug 30 2016
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/30/2016 10:44 AM, Atila Neves wrote:

 I'd much rather have `assert` be magical or have AST macros to make the
 syntax for writing tests better than what it is now.


Same here. BTW I'd like unittests that "must not compile" and unittests 
that "must fail dynamically".

For the former case, the compiler should cooperate:

 incompilable unittest { ... }

fails if it passes compilation. So the compiler must know about that 
attribute.

For the latter case, no change in language is necessary, only in druntime:

 mustfail unittest { ... }

Would love these two.


Andrei
Aug 30 2016
next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 30 August 2016 at 15:45:26 UTC, Andrei Alexandrescu 
wrote:

 On 08/30/2016 10:44 AM, Atila Neves wrote:

 I'd much rather have `assert` be magical or have AST macros to 
 make the
 syntax for writing tests better than what it is now.


 Same here. BTW I'd like unittests that "must not compile" and 
 unittests that "must fail dynamically".

 For the former case, the compiler should cooperate:

  incompilable unittest { ... }
 fails if it passes compilation. So the compiler must know about 
 that attribute.


Right now I think you're right and the compiler needs to know. 
But let me see what I can do about it with the language we have 
now.


 For the latter case, no change in language is necessary, only 
 in druntime:

  mustfail unittest { ... }


As previously mentioned, unit-threaded has this.

Atila
Aug 31 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-08-31 12:06, Atila Neves wrote:


 Right now I think you're right and the compiler needs to know. But let
 me see what I can do about it with the language we have now.


Or using AST macros :)

-- 
/Jacob Carlborg
Aug 31 2016
prev sibling parent Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Tuesday, 30 August 2016 at 15:45:26 UTC, Andrei Alexandrescu 
wrote:

 On 08/30/2016 10:44 AM, Atila Neves wrote:

 I'd much rather have `assert` be magical or have AST macros to 
 make the
 syntax for writing tests better than what it is now.


 Same here. BTW I'd like unittests that "must not compile" and 
 unittests that "must fail dynamically".

 For the former case, the compiler should cooperate:

  incompilable unittest { ... }

 fails if it passes compilation. So the compiler must know about 
 that attribute.


There should be a way to specify the error message (or match it 
against a regex); otherwise the test could fail accidentally for 
totally unrelated reasons, and nobody would notice...
Aug 31 2016
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-08-30 16:44, Atila Neves wrote:


 I'm obviously very biased but having written a unit testing library that
 I'm happy with and proud of I'd just use that instead. I wouldn't be
 interested in this.


The point of this low level library is that a high level unit test 
library, like your unit-threaded, can use the low level library in 
druntime. If this was to be implemented I imagine we could move some 
code from unit-threaded to this low level library. No need for unit test 
frameworks to reimplement the low level stuff that would be very similar.


 I'd much rather have `assert` be magical or have AST macros to make the
 syntax for writing tests better than what it is now.


That would be nice to have.

-- 
/Jacob Carlborg
Aug 30 2016
parent reply Seb <seb wilzba.ch> writes:
On Tuesday, 30 August 2016 at 16:06:21 UTC, Jacob Carlborg wrote:

 I'd much rather have `assert` be magical or have AST macros to 
 make the
 syntax for writing tests better than what it is now.


 That would be nice to have.


There is a bit of work on a closely related topic on the wiki:

https://wiki.dlang.org/DIP83
Aug 30 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-08-30 18:17, Seb wrote:


 There is a bit of work on a closely related topic on the wiki:

 https://wiki.dlang.org/DIP83


Or https://wiki.dlang.org/DIP50 :)

-- 
/Jacob Carlborg
Aug 30 2016
parent reply jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 30 August 2016 at 16:18:30 UTC, Jacob Carlborg wrote:

 On 2016-08-30 18:17, Seb wrote:


 There is a bit of work on a closely related topic on the wiki:

 https://wiki.dlang.org/DIP83


 Or https://wiki.dlang.org/DIP50 :)


Or that unit-threaded's  ShouldFail seems very similar to 
Andrei's  MustFail
Aug 30 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/30/2016 01:12 PM, jmh530 wrote:

 On Tuesday, 30 August 2016 at 16:18:30 UTC, Jacob Carlborg wrote:

 On 2016-08-30 18:17, Seb wrote:


 There is a bit of work on a closely related topic on the wiki:

 https://wiki.dlang.org/DIP83


 Or https://wiki.dlang.org/DIP50 :)


 Or that unit-threaded's  ShouldFail seems very similar to Andrei's
  MustFail


Noice! -- Andrei
Aug 30 2016
prev sibling parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D
Subject: Re: Low level unit test library in druntime
References: <npptbk$2mk0$1 digitalmars.com>
In-Reply-To: <npptbk$2mk0$1 digitalmars.com>

--045N7gBD22mFdiQIhBLaGWpGidgWQUCCL
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Definitely not in druntime. Optional extra test runners in Phobos (or
libraries for higher level test suites) are reasonable but I don't want
any changes to fundamentals of how _unit_ tests are defined and used.



--045N7gBD22mFdiQIhBLaGWpGidgWQUCCL--
Aug 30 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-08-30 16:54, Dicebot wrote:

 Definitely not in druntime. Optional extra test runners in Phobos (or
 libraries for higher level test suites) are reasonable but I don't want
 any changes to fundamentals of how _unit_ tests are defined and used.


You both are kind of missing the point. This would not be a runner in 
the same sense as the existing runners. It would be a low level library 
that a runner can use. If this would to be implemented the existing unit 
test runner in druntime would be updated to use this library. It would 
configure the library to behave exactly like the current runner work.

Since this library does collect any unit tests it's up to the code that 
uses the library to collect the tests. The runner in druntime would 
collect the tests exactly as it does now and those how tests are defined 
and used would not be changed at all.

The reason to put in the druntime is because that's where the existing 
runner is located.

The advantage of this low level library is that:

* Third party unit test library don't need to reinvent the wheel

* All third party libraries using this low level library would be 
compatible and can be combined in the same project

* If we would like to, it would be easy to extend the existing runner, 
like not stopping on the first failure. _Not_ saying that we should

-- 
/Jacob Carlborg
Aug 30 2016
parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D
Subject: Re: Low level unit test library in druntime
References: <npptbk$2mk0$1 digitalmars.com> <nq46mh$24bo$1 digitalmars.com>
 <nq4bir$2c4c$1 digitalmars.com>
In-Reply-To: <nq4bir$2c4c$1 digitalmars.com>

--0vLelUKnSiRSBIIklnFOVHANlnGfIRCcc
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 08/30/2016 07:17 PM, Jacob Carlborg wrote:

 The reason to put in the druntime is because that's where the existing
 runner is located.
=20
 The advantage of this low level library is that:
=20
 * Third party unit test library don't need to reinvent the wheel
=20
 * All third party libraries using this low level library would be
 compatible and can be combined in the same project
=20
 * If we would like to, it would be easy to extend the existing runner,
 like not stopping on the first failure. _Not_ saying that we should


I definitely wouldn't want to use API like you proposed if I was to
write my own test runner. Minimal common ground which would be cool to
have is getting range/array of all unittest blocks together with their
annotations. Anything more than that is optional luxury that specific
test systems may define.

And any usage of classes in what is supposed to be a base ground API is
immediate "no" for me :)


--0vLelUKnSiRSBIIklnFOVHANlnGfIRCcc--
Aug 30 2016
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-08-30 18:31, Dicebot wrote:


 I definitely wouldn't want to use API like you proposed if I was to
 write my own test runner. Minimal common ground which would be cool to
 have is getting range/array of all unittest blocks together with their
 annotations. Anything more than that is optional luxury that specific
 test systems may define.


Basically the only thing that would be different in different frameworks :(


 And any usage of classes in what is supposed to be a base ground API is
 immediate "no" for me :)


Yeah that would be crazy, like threads, fibers, sockets and exceptions ;)

-- 
/Jacob Carlborg
Aug 30 2016
parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D
Subject: Re: Low level unit test library in druntime
References: <npptbk$2mk0$1 digitalmars.com> <nq46mh$24bo$1 digitalmars.com>
 <nq4bir$2c4c$1 digitalmars.com> <nq4cdv$2dbm$1 digitalmars.com>
 <nq4nev$2tmh$1 digitalmars.com>
In-Reply-To: <nq4nev$2tmh$1 digitalmars.com>

--ocJMDtGROO136PTNUaXxKPIqjoGEPLtGN
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 08/30/2016 10:40 PM, Jacob Carlborg wrote:

 On 2016-08-30 18:31, Dicebot wrote:
=20

 I definitely wouldn't want to use API like you proposed if I was to
 write my own test runner. Minimal common ground which would be cool to=





 have is getting range/array of all unittest blocks together with their=





 annotations. Anything more than that is optional luxury that specific
 test systems may define.


=20
 Basically the only thing that would be different in different framework=


s :(

What would be different? List of unit test blocks available in the progra=
m?


 And any usage of classes in what is supposed to be a base ground API i=




s

 immediate "no" for me :)


=20
 Yeah that would be crazy, like threads, fibers, sockets and exceptions =


;)

_forcing_ usage of threads, fibers, sockets and exceptions in runtime
outside of their respective modules would be rather outrageous too.
druntime should reasonably simple to port to systems that may not have
any of those - and still degrade gracefully.


--ocJMDtGROO136PTNUaXxKPIqjoGEPLtGN--
Sep 01 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-01 14:02, Dicebot wrote:


 What would be different? List of unit test blocks available in the program?


How the unit tests are written and collected.

unittest { assert(true); }

class FooTest : Test
{
     void testFoo() { assert(true); }
}

describe("foo", {
     it("does something", {
         assert(true);
     });
});

-- 
/Jacob Carlborg
Sep 02 2016
parent reply Dicebot <public dicebot.lv> writes:
On Friday, 2 September 2016 at 19:26:41 UTC, Jacob Carlborg wrote:

 On 2016-09-01 14:02, Dicebot wrote:


 What would be different? List of unit test blocks available in 
 the program?


 How the unit tests are written and collected.

 unittest { assert(true); }

 class FooTest : Test
 {
     void testFoo() { assert(true); }
 }

 describe("foo", {
     it("does something", {
         assert(true);
     });
 });


Then we perfectly understand each other and I will vote against 
introducing any other way/syntax for writing _unit_ tests in D 
with all my passion.
Sep 02 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-09-02 21:52, Dicebot wrote:


 Then we perfectly understand each other and I will vote against
 introducing any other way/syntax for writing _unit_ tests in D with all
 my passion.


This discussion is just going in circles, meeting adjourned.

-- 
/Jacob Carlborg
Sep 03 2016
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 30 August 2016 at 16:31:56 UTC, Dicebot wrote:

 On 08/30/2016 07:17 PM, Jacob Carlborg wrote:

 The reason to put in the druntime is because that's where the 
 existing runner is located.
 
 The advantage of this low level library is that:
 
 * Third party unit test library don't need to reinvent the 
 wheel
 
 * All third party libraries using this low level library would 
 be compatible and can be combined in the same project
 
 * If we would like to, it would be easy to extend the existing 
 runner, like not stopping on the first failure. _Not_ saying 
 that we should


 I definitely wouldn't want to use API like you proposed if I 
 was to write my own test runner. Minimal common ground which 
 would be cool to have is getting range/array of all unittest 
 blocks together with their annotations. Anything more than that 
 is optional luxury that specific test systems may define.

 And any usage of classes in what is supposed to be a base 
 ground API is immediate "no" for me :)


And never mind that any such low level library would suffer from 
the same problem unit-threaded did until dub fixed it: D can't 
reflect on packages so a program must be written that explicitly 
lists all modules that need to be looked at.

The current situation in druntime with the default runner doesn't 
work either because all unit tests end up being a single function 
pointer from the module.

There's a disconnect between what's possible at runtime with 
ModuleInfo and what's possible at compile-time with 
__traits(getUnittests). Fortunately for me, running unit-threaded 
itself as an executable with dub fixed the problem, but in the 
general case this proposal would suffer from the same problems. 
Unless we force everyone to use dub.

Atila
Aug 31 2016
parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D
Subject: Re: Low level unit test library in druntime
References: <npptbk$2mk0$1 digitalmars.com> <nq46mh$24bo$1 digitalmars.com>
 <nq4bir$2c4c$1 digitalmars.com> <nq4cdv$2dbm$1 digitalmars.com>
 <hvirybitoanaurrmskhi forum.dlang.org>
In-Reply-To: <hvirybitoanaurrmskhi forum.dlang.org>

--xjpBTHDObrEFtuW5O7N4dRVIqfxoX52M5
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 08/31/2016 01:01 PM, Atila Neves wrote:

 And never mind that any such low level library would suffer from the
 same problem unit-threaded did until dub fixed it: D can't reflect on
 packages so a program must be written that explicitly lists all modules=



 that need to be looked at.


I don't even think fixing package reflection would truly help here
because there exist legit D projects that don't transitively import all
modules from `main` and recursive compile-time visiting of all symbols
would miss them.


 There's a disconnect between what's possible at runtime with ModuleInfo=



 and what's possible at compile-time with __traits(getUnittests).
 Fortunately for me, running unit-threaded itself as an executable with
 dub fixed the problem, but in the general case this proposal would
 suffer from the same problems. Unless we force everyone to use dub.


To me it feels like mandatory per-requisite for any test runner changes
in druntime right now is to:

1) make compiler emit runtime info on test block basis instead of per mod=
ule
2) store TypeInfo of unittest block attributes as part of that runtime in=
fo


--xjpBTHDObrEFtuW5O7N4dRVIqfxoX52M5--
Sep 01 2016
parent reply ZombineDev <petar.p.kirov gmail.com> writes:
On Thursday, 1 September 2016 at 12:06:21 UTC, Dicebot wrote:

 On 08/31/2016 01:01 PM, Atila Neves wrote:

 And never mind that any such low level library would suffer 
 from the same problem unit-threaded did until dub fixed it: D 
 can't reflect on packages so a program must be written that 
 explicitly lists all modules that need to be looked at.


 I don't even think fixing package reflection would truly help 
 here because there exist legit D projects that don't 
 transitively import all modules from `main` and recursive 
 compile-time visiting of all symbols would miss them.


 [snip]




Not a problem, since you can do things like this:

// Prints:
// tuple("CSVException", "IncompleteCellException",
// "HeaderMismatchException", "Malformed", "JSONFloatLiteral",
// "JSONOptions", "JSON_TYPE", "JSONValue", "JSONException")
pragma (msg, ModuleTypes!("std.csv", "std.json"));


/**
  * Params:
  *    Takes a sequence of module name strings and returns a 
sequence of all types names
  *    defined in those modules
  */
template ModuleTypes(ModuleNames...)
{
     import std.meta : staticMap, aliasSeqOf;

     alias ModuleTypes = aliasSeqOf!(getTypes());

     string[] getTypes()
     {
         import std.meta : AliasSeq;

         string[] result;

         foreach (fqModuleName; ModuleNames)
         {
             mixin ("import " ~ fqModuleName ~ ";");

             foreach (member; __traits(allMembers, 
mixin(fqModuleName)))
             {
                 static if (mixin("is(" ~ member ~ ")"))
                 {
                     result ~= member;
                 }
             }
         }

         return result;
     }
}
Sep 01 2016
parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D
Subject: Re: Low level unit test library in druntime
References: <npptbk$2mk0$1 digitalmars.com> <nq46mh$24bo$1 digitalmars.com>
 <nq4bir$2c4c$1 digitalmars.com> <nq4cdv$2dbm$1 digitalmars.com>
 <hvirybitoanaurrmskhi forum.dlang.org> <nq95jt$1t8$1 digitalmars.com>
 <cbcuuwfrpfkbvqztmekn forum.dlang.org>
In-Reply-To: <cbcuuwfrpfkbvqztmekn forum.dlang.org>

--JsnRGaMMHa7WN74xgISVmIoqN4SHGrmrq
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 09/01/2016 07:17 PM, ZombineDev wrote:

 On Thursday, 1 September 2016 at 12:06:21 UTC, Dicebot wrote:

 On 08/31/2016 01:01 PM, Atila Neves wrote:

 And never mind that any such low level library would suffer from the
 same problem unit-threaded did until dub fixed it: D can't reflect on=







 packages so a program must be written that explicitly lists all
 modules that need to be looked at.


 I don't even think fixing package reflection would truly help here
 because there exist legit D projects that don't transitively import
 all modules from `main` and recursive compile-time visiting of all
 symbols would miss them.


 [snip]




=20
 Not a problem, since you can do things like this:


It is exactly _THE_ problem. You can't have the imaginary test runner to
reliably find all tests automatically, at least all compiled modules
have to be listed explicitly. This is not good, thus I am inclined to
call extending RTTI an only viable long-term solution.



--JsnRGaMMHa7WN74xgISVmIoqN4SHGrmrq--
Sep 01 2016
parent ZombineDev <petar.p.kirov gmail.com> writes:
On Thursday, 1 September 2016 at 16:38:15 UTC, Dicebot wrote:

 On 09/01/2016 07:17 PM, ZombineDev wrote:

 On Thursday, 1 September 2016 at 12:06:21 UTC, Dicebot wrote:

 On 08/31/2016 01:01 PM, Atila Neves wrote:

 And never mind that any such low level library would suffer 
 from the same problem unit-threaded did until dub fixed it: 
 D can't reflect on packages so a program must be written 
 that explicitly lists all modules that need to be looked at.


 I don't even think fixing package reflection would truly help 
 here because there exist legit D projects that don't 
 transitively import all modules from `main` and recursive 
 compile-time visiting of all symbols would miss them.


 [snip]




 
 Not a problem, since you can do things like this:


 It is exactly _THE_ problem. You can't have the imaginary test 
 runner to reliably find all tests automatically, at least all 
 compiled modules have to be listed explicitly. This is not 
 good, thus I am inclined to call extending RTTI an only viable 
 long-term solution.


Ooh, I thought that by "fixing package reflection" Atila meant 
the ability to get a list of all modules/packages that the 
compiler knows about, assuming an all at once compilation. For 
things like dynamic libraries and incremental compilation, 
there's obviously no other way than RTTI. But for many use-cases 
CT reflection should be enough.
Sep 01 2016