www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Helpers for writing unittests

reply "Idan Arye" <GenericNPC gmail.com> writes:
The "Rant after trying Rust a bit" 
thread(http://forum.dlang.org/thread/ckjukjfkgrguhfhkdhhj forum.dlang.org)
talks mostly about traits, and how people want Rust's traits(or C++'s Concepts)
in D. As a general rule, I believe that taking a feature from one language and
sticking it in another works just as nicely as taking off a leg from one chair
and using it to replace a broken leg from a different firm's chair. You'd be
lucky if they have similar length, and even if you sand the legs to have exact
same length the balance will be off. You might be able to sit on it, but it
won't be as comfortable as a chair that all it's parts fit together.

Instead, it's better to look at the problem the feature solves, 
and how the language without that feature approaches the problem. 
In our case, the problem is that compile-time errors in the 
templates only pop up once the template is instantiated. Rust's 
solution is to use Traits, and Walter made it pretty clear that 
the D way to solve this problem is to instantiate the template in 
a unittest.

As seen in that thread, many people find this solution lacking. 
Writing 100% coverage tests for instantiating templates is a long 
and tiring process. In response, the anti-traits camp accuses 
these people of being lazy.

Well, I say - programmers should be lazy! That's why we are 
programming - because we are lazy and want the computers to do 
our work for us. So the current D solution is not enough - but 
that doesn't mean a feature transplant from Rust is a good 
solution - we need to find a D style solution, that'll fit with 
the rest of the language.


What I'm thinking about is a unittest helper that'll help in 
checking different instantiations of the template. A quick proof 
of concept - http://dpaste.dzfl.pl/8907c3a7d54c - shows how the 
unittest found that foo doesn't work with long and ulong, and 
printed easy-to-understand errors.

Just like IntegerTypes we can have many more lists of types for 
different categories of types we want to test - string types, 
range types etc. With this in Phobos, writing unittests with full 
coverage(compile-time only) will be much easier.

Note that we just want to see that it compiles - we don't want to 
actually run it, because than we'd have to supply templated test 
data and test results to compare with, which is a much harder 
problem. These are compile-time mocks - the problem they solve is 
limited to compilation so they will be able to solve it well and 
elegantly. This does not come instead of tests that actually run 
- a unittest can test compilation on all the relavant types and 
actually run and check the results only for a subset of these 
types.

Thoughts?
Jul 30 2015
next sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, 31 July 2015 at 00:07:43 UTC, Idan Arye wrote:

 Thoughts?


Some unit test helpers for this sort of thing might be nice, but 
I don't think that it really buys us much with this particular 
case. You could just as easily do

unittest
{
     foreach(T; TypeTuple!(ubyte, byte, ushort, short, uint, int, 
ulong, long))
         static assert(is(typeof(foo(T.init)));
}

and the code is basically as long as is with assertCompilesWith - 
shorter even. The above example is longer due to not using an 
alias for the integral types like you did, but if that same alias 
were used, then it becomes

unittest
{
     foreach(T; IntegerTypes)
         static assert(is(typeof(foo(T.init)));
}

which isn't all that different than

unittest
{
     assertCompilesWith!(IntegerTypes, (x) {
         foo(x);
     });
}

but it doesn't require a lambda or creating any helpers, and it 
cuts down on the number of unit testing functions that are 
templated (it can cost a lot in terms of compilation time and 
memory, if you use a lot of templates as helpers for unit tests). 
In addition, in most cases, I don't see much point in simply 
testing that the templated function or type compiles with various 
types. It's generally far better to do something like

foreach(T; IntegerTypes)
{
     // Tests which actually test foo with T rather than just that 
it compiles
}

And in that case, explicitly testing compilation is unnecessary. 
You're already testing far more than that.

So, I applaud the attitude and motivation behind your suggestion, 
but this particular helper doesn't really help IMHO.

- Jonathan M Davis
Jul 30 2015
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Friday, 31 July 2015 at 00:30:23 UTC, Jonathan M Davis wrote:

 On Friday, 31 July 2015 at 00:07:43 UTC, Idan Arye wrote:

 Thoughts?


 Some unit test helpers for this sort of thing might be nice, 
 but I don't think that it really buys us much with this 
 particular case. You could just as easily do

 unittest
 {
     foreach(T; TypeTuple!(ubyte, byte, ushort, short, uint, 
 int, ulong, long))
         static assert(is(typeof(foo(T.init)));
 }

 and the code is basically as long as is with assertCompilesWith 
 - shorter even. The above example is longer due to not using an 
 alias for the integral types like you did, but if that same 
 alias were used, then it becomes

 unittest
 {
     foreach(T; IntegerTypes)
         static assert(is(typeof(foo(T.init)));
 }

 which isn't all that different than

 unittest
 {
     assertCompilesWith!(IntegerTypes, (x) {
         foo(x);
     });
 }


The resulting compilation errors are extremely different. With 
your method we get:

/d391/f994.d(31): Error: static assert (is(typeof(__error))) is 
false

which doesn't tell us what the problem is. With my method we get:

/d433/f500.d(25): Error: cannot implicitly convert expression (x) 
of type ulong to int /d433/f500.d(31): Error: template instance 
f500.foo!ulong error instantiating

And yes, a lot of "static stack trace" after that, but these too 
lines tells us what the problem is and which template parameters 
caused it.

Of course, if you just ran the function inside the foreach loop 
you'd get nice error messages as well - but then you'd have to 
write tests that actually run. Which is easy for numbers, because 
they are all the same type of data with different sizes, but can 
get tricky when you have more complex types, that differ more in 
their behavior.
Jul 30 2015
parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, 31 July 2015 at 00:54:30 UTC, Idan Arye wrote:

 The resulting compilation errors are extremely different. With 
 your method we get:

 /d391/f994.d(31): Error: static assert (is(typeof(__error))) is 
 false

 which doesn't tell us what the problem is. With my method we 
 get:

 /d433/f500.d(25): Error: cannot implicitly convert expression 
 (x) of type ulong to int /d433/f500.d(31): Error: template 
 instance f500.foo!ulong error instantiating

 And yes, a lot of "static stack trace" after that, but these 
 too lines tells us what the problem is and which template 
 parameters caused it.


True enough, but you can simply add T.stringof to the static 
assertion, and you'll know which instantiation is failing. And 
maybe what you're suggesting is worth having, but you can get 
pretty much the same thing easily without creating a template to 
do the test for you.


 Of course, if you just ran the function inside the foreach loop 
 you'd get nice error messages as well - but then you'd have to 
 write tests that actually run. Which is easy for numbers, 
 because they are all the same type of data with different 
 sizes, but can get tricky when you have more complex types, 
 that differ more in their behavior.


Yes. It's generally hard to write tests which work with a variety 
of template instantiations, but the point is that knowing that 
the template instantiates with a particular type really doesn't 
tell you much. If you actually want to be testing the template 
with those types, you should be writing tests with those types 
whether you can templatize the tests or whether each 
instantiation needs to be tested separately. That being the case, 
I don't think that unit tests should normally be testing 
explicitly that a template instantiates with a type. Doing that 
is so insufficient that there isn't much point.

- Jonathan M Davis
Jul 31 2015
prev sibling parent "jmh530" <john.michael.hall gmail.com> writes:
On Friday, 31 July 2015 at 00:07:43 UTC, Idan Arye wrote:

 people want Rust's traits(or C++'s Concepts) in D.


Just to quibble, that wasn't my take-a-way from the thread. My 
take-a-way is that Jonathan Davis is really good at explaining 
things.
Jul 30 2015