www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - selectively running unittest functions

reply "Daniel Davidson" <nospam spam.com> writes:
I've been through this thread: 
http://forum.dlang.org/post/mailman.1454.1369104411.4724.digitalmars-d puremagic.com

I would like the named unittests, as describe there, but I don't 
think it is in the works (if I'm wrong please let me know). So I 
took an approach outlined here 
(http://www.reddit.com/r/programming/comments/1edih2/dconf_2013_day_1_talk_4_writing_testa
le_code_in/c9zg3ry) 
and created a minimal, simple scheme.

The support file is here: http://pastebin.com/tU9BuS3G

To use it:
- import the support file
- mixin an init routine `mixin UTInit!__MODULE__;` to your module
- Instead of using unittest blocks (or in addition) put code in 
annotated functions.
    ` UT void testFoo() { .... }


To run all tests:
   `rdmd -version=UT --main program`
To filter by module (using regex)
    `rdmd -version=UT --main program -m 'foo.*'`
To filter by test function (using regex)
    `rdmd -version=UT --main program -t 'foo.*'`

I think the approach makes sense, but I am looking any 
comments/criticisms on what may be missing or misguided.

The downsides as I see it:
- it is not standard
- it comes with a version, UT. So you can/should wrap all tests 
in version(UT) to ensure they never make it to production
- all test functions must be at module level. Not sure if there 
are any ramifications?

The pluses:
- filtering of test by module and function name
- it works along side of unittest. It can also work with others 
like specd which is a nice way to describe tests and know when 
things fail the location.


Thanks
Dan
Oct 25 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
I strictly believe any unittest enhancing library must be built 
on top of existing unittest blocks using __traits(getUnittest) 
and be 100% compatible with normal `-unittest` mode
Oct 25 2013
next sibling parent reply "Daniel Davidson" <nospam spam.com> writes:
On Friday, 25 October 2013 at 13:04:03 UTC, Dicebot wrote:

 I strictly believe any unittest enhancing library must be built 
 on top of existing unittest blocks using __traits(getUnittest) 
 and be 100% compatible with normal `-unittest` mode


I don't disagree. What exactly does that mean and what would it 
look like?

I'm not familiar with __traits(getUnittest). Is that something 
that could/should be used in what I'm doing?

What I'm missing, and apparently others in the original thread, 
is a way to run tests selectively. It is difficult to do with 
unittest because they are not named. If there were a way to 
annotate the test and pull them out that way it would be great. 
Can it be done?
Oct 25 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Friday, 25 October 2013 at 13:23:46 UTC, Daniel Davidson wrote:

 What I'm missing, and apparently others in the original thread, 
 is a way to run tests selectively. It is difficult to do with 
 unittest because they are not named. If there were a way to 
 annotate the test and pull them out that way it would be great. 
 Can it be done?


You can completely re-implement default test by using runtime 
hook http://wiki.dlang.org/Runtime_Hooks ("_d_unittest" should do 
AFAIK)

In that runner you can manually get all unittest blocks in the 
program as functions using __traits(getUnittest). One can make 
any custom decisions for running specific unittest blocks based 
on User-Defined Attributes attached to it.

Benefit of this approach is that the very same tests remain 
runnable in traditional out-of-the-box way if you don't use that 
library runner.

I remember Jacob Carlborg doing some experiments in that 
direction but did not track any further progress.
Oct 25 2013
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Friday, 25 October 2013 at 13:30:54 UTC, Dicebot wrote:

 You can completely re-implement default test by using runtime 
 hook http://wiki.dlang.org/Runtime_Hooks ("_d_unittest" should 
 do AFAIK)


*default test runner
Oct 25 2013
prev sibling parent reply "Daniel Davidson" <nospam spam.com> writes:
On Friday, 25 October 2013 at 13:30:54 UTC, Dicebot wrote:

 On Friday, 25 October 2013 at 13:23:46 UTC, Daniel Davidson 
 wrote:

 What I'm missing, and apparently others in the original 
 thread, is a way to run tests selectively. It is difficult to 
 do with unittest because they are not named. If there were a 
 way to annotate the test and pull them out that way it would 
 be great. Can it be done?


 You can completely re-implement default test by using runtime 
 hook http://wiki.dlang.org/Runtime_Hooks ("_d_unittest" should 
 do AFAIK)


Pretty sure this is what I'm doing. Only I did not know you could 
UDA a unittest.


 In that runner you can manually get all unittest blocks in the 
 program as functions using __traits(getUnittest). One can make 
 any custom decisions for running specific unittest blocks based 
 on User-Defined Attributes attached to it.


Any samples of how to use __traits(getUnittest,...). I might be 
able to work as you suggest if I can get this to work.


 Benefit of this approach is that the very same tests remain 
 runnable in traditional out-of-the-box way if you don't use 
 that library runner.

 I remember Jacob Carlborg doing some experiments in that 
 direction but did not track any further progress.
Oct 25 2013
parent reply "Dicebot" <public dicebot.lv> writes:
This will work starting with 2.064:

module a;

import std.stdio;

struct ID
{
     string data;
}

 ID("one")
unittest
{
     writeln("1");
}

 ID("two")
unittest
{
     writeln("2");
}

void main()
{
     import std.typetuple;
     alias tests = TypeTuple!( __traits(getUnitTests, a) );

     foreach (test; tests)
     {
         foreach (uda; __traits(getAttributes, test))
         {
             static if (is(typeof(uda) == ID))
             {
                 if (uda == ID("two"))
                     test();
             }
         }
     }
}

It will print stuff twice though because I didn't suppress native 
unittest runner. Didn't actually dwell deep into runtime hooks so 
don't know what is best way to do it (maybe we need one more 
enhancement for it)
Oct 25 2013
next sibling parent "simendsjo" <simendsjo gmail.com> writes:
On Friday, 25 October 2013 at 14:14:39 UTC, Dicebot wrote:

 This will work starting with 2.064:

 module a;

 import std.stdio;

 struct ID
 {
     string data;
 }

  ID("one")
 unittest
 {
     writeln("1");
 }

  ID("two")
 unittest
 {
     writeln("2");
 }

 void main()
 {
     import std.typetuple;
     alias tests = TypeTuple!( __traits(getUnitTests, a) );

     foreach (test; tests)
     {
         foreach (uda; __traits(getAttributes, test))
         {
             static if (is(typeof(uda) == ID))
             {
                 if (uda == ID("two"))
                     test();
             }
         }
     }
 }

 It will print stuff twice though because I didn't suppress 
 native unittest runner. Didn't actually dwell deep into runtime 
 hooks so don't know what is best way to do it (maybe we need 
 one more enhancement for it)


This is great. I love the unittest blocks and UDAs. This way I 
can still have very easy-entry (as in "just write unittest {}") 
unittests while gaining the power of unittest frameworks.

It's a hell of a lot better than having some class with 
attributes containing methods with attributes, containing 
parameters with attributes :) I'm developing RSI when writing 
tests in other languages. (In Java I develop RSI just declaring 
variables...)
Oct 25 2013
prev sibling parent reply "Daniel Davidson" <nospam spam.com> writes:
On Friday, 25 October 2013 at 14:14:39 UTC, Dicebot wrote:

 This will work starting with 2.064:


Ok. I'll keep pressing. Here is an updated version: 
http://pastebin.com/g6FWsTkr

The idea is to be able to just import ut, annotate as you have 
described and get unit tests run. I want to mixin the equivalent 
of your "main" in each module that just pulls in a module 
constructors to evaluate what tests are there. The code in the 
paste crashes on the call to getUnitests.

If I comment out `alias tests = TypeTuple!(__traits(getUnitTests, 
mod));` then I see similar call to  `alias members = 
TypeTuple!(__traits(allMembers, mod));` work just fine.

Is this the right way to use this? Any pointers on the 
segmentation fault?

Thanks
Dan
Oct 25 2013
parent reply "Daniel Davidson" <nospam spam.com> writes:
On Friday, 25 October 2013 at 16:43:23 UTC, Daniel Davidson wrote:

 On Friday, 25 October 2013 at 14:14:39 UTC, Dicebot wrote:

 This will work starting with 2.064:


 Ok. I'll keep pressing. Here is an updated version: 
 http://pastebin.com/g6FWsTkr

 The idea is to be able to just import ut, annotate as you have 
 described and get unit tests run. I want to mixin the 
 equivalent of your "main" in each module that just pulls in a 
 module constructors to evaluate what tests are there. The code 
 in the paste crashes on the call to getUnitests.

 If I comment out `alias tests = 
 TypeTuple!(__traits(getUnitTests, mod));` then I see similar 
 call to  `alias members = TypeTuple!(__traits(allMembers, 
 mod));` work just fine.

 Is this the right way to use this? Any pointers on the 
 segmentation fault?

 Thanks
 Dan



Ok, binary reduction has shown the seg fault was due to a debug 
import. This code causes a crash for me - just because of the 
getUnittests. If I remove the debug import it works. Since I use 
debug imports in general, I'm not so sure it is worthwhile to 
pursue the named unit test approach at this time.

import std.typetuple;
import std.stdio;
debug import std.datetime;
unittest { writeln("In Test!!"); }
mixin("alias mod = " ~ __MODULE__ ~ ";");
alias tests = TypeTuple!(__traits(getUnitTests, mod));
static this() {
   writeln("Done");
}


Thanks
Dan
Oct 25 2013
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
26-Oct-2013 02:36, Daniel Davidson пишет:

 On Friday, 25 October 2013 at 16:43:23 UTC, Daniel Davidson wrote:

 On Friday, 25 October 2013 at 14:14:39 UTC, Dicebot wrote:

 This will work starting with 2.064:


 Ok. I'll keep pressing. Here is an updated version:
 http://pastebin.com/g6FWsTkr

 The idea is to be able to just import ut, annotate as you have
 described and get unit tests run. I want to mixin the equivalent of
 your "main" in each module that just pulls in a module constructors to
 evaluate what tests are there. The code in the paste crashes on the
 call to getUnitests.

 If I comment out `alias tests = TypeTuple!(__traits(getUnitTests,
 mod));` then I see similar call to  `alias members =
 TypeTuple!(__traits(allMembers, mod));` work just fine.

 Is this the right way to use this? Any pointers on the segmentation
 fault?

 Thanks
 Dan



 Ok, binary reduction has shown the seg fault was due to a debug import.


It's a bug so please file it in bugzilla. Don't let them go unnoticed ;)
http://d.puremagic.com/issues/


 This code causes a crash for me - just because of the getUnittests. If I
 remove the debug import it works. Since I use debug imports in general,
 I'm not so sure it is worthwhile to pursue the named unit test approach
 at this time.

 import std.typetuple;
 import std.stdio;
 debug import std.datetime;
 unittest { writeln("In Test!!"); }
 mixin("alias mod = " ~ __MODULE__ ~ ";");
 alias tests = TypeTuple!(__traits(getUnitTests, mod));
 static this() {
    writeln("Done");
 }


 Thanks
 Dan



-- 
Dmitry Olshansky
Oct 26 2013
parent "Daniel Davidson" <nospam spam.com> writes:
On Saturday, 26 October 2013 at 08:09:26 UTC, Dmitry Olshansky 
wrote:

 26-Oct-2013 02:36, Daniel Davidson пишет:

 On Friday, 25 October 2013 at 16:43:23 UTC, Daniel Davidson 
 wrote:

 On Friday, 25 October 2013 at 14:14:39 UTC, Dicebot wrote:

 This will work starting with 2.064:


 Ok. I'll keep pressing. Here is an updated version:
 http://pastebin.com/g6FWsTkr

 The idea is to be able to just import ut, annotate as you have
 described and get unit tests run. I want to mixin the 
 equivalent of
 your "main" in each module that just pulls in a module 
 constructors to
 evaluate what tests are there. The code in the paste crashes 
 on the
 call to getUnitests.

 If I comment out `alias tests = 
 TypeTuple!(__traits(getUnitTests,
 mod));` then I see similar call to  `alias members =
 TypeTuple!(__traits(allMembers, mod));` work just fine.

 Is this the right way to use this? Any pointers on the 
 segmentation
 fault?

 Thanks
 Dan



 Ok, binary reduction has shown the seg fault was due to a 
 debug import.


 It's a bug so please file it in bugzilla. Don't let them go 
 unnoticed ;)
 http://d.puremagic.com/issues/


Yes - I have.
Oct 26 2013
prev sibling parent reply "Gary Willoughby" <dev nomad.so> writes:
On Friday, 25 October 2013 at 13:04:03 UTC, Dicebot wrote:

 I strictly believe any unittest enhancing library must be built 
 on top of existing unittest blocks using __traits(getUnittest) 
 and be 100% compatible with normal `-unittest` mode


I agree, this should be easy to implement. In my unit testing 
framework i could add support to run only unit tests of one 
particular module but in the future when the above is released it 
would be easy to add it on a per test basis.
Oct 25 2013
parent reply "Gary Willoughby" <dev nomad.so> writes:
For information how to implement your own unit test handler see 
the bottom of the report module here:

https://github.com/nomad-software/dunit

 From there you can selectively run tests based on some outside 
criteria. I think the new trait will open up further 
possibilities too.
Oct 25 2013
parent "Dicebot" <public dicebot.lv> writes:
On Friday, 25 October 2013 at 15:55:11 UTC, Gary Willoughby wrote:

 For information how to implement your own unit test handler see 
 the bottom of the report module here:

 https://github.com/nomad-software/dunit

 From there you can selectively run tests based on some outside 
 criteria. I think the new trait will open up further 
 possibilities too.


After 2.064 is out and good support of attributed test blocks is 
added I really recommend to consider separating core modules that 
support it into Phobos proposal. Will need to convince some 
conservative guys this is actually needed, but I think it is 
doable :P
Oct 25 2013
prev sibling parent "Daniel Davidson" <nospam spam.com> writes:
Here is a working solution: 
https://github.com/patefacio/d-help/blob/master/d-help/opmix/ut.d

Currently it only pulls in unittests at the module level. I'm 
sure it will work on unittests scoped to structs/classes, I just 
need to figure out how to determine if a compile time named 
object is an aggregate, as __traits(getUnitTests, foo) will 
complain if foo is not a module or aggregate.

With a source a.d and b.d like in: 
https://github.com/patefacio/d-help/blob/master/d-help/opmix/examples/ut/
it can pull out by module name and uda test name. It has a -s 
summary option as well.

Here is sample output: http://pastebin.com/fYd7k1Kz

Thanks
Dan
Oct 26 2013