www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - unittesting generic functions

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Destroy https://issues.dlang.org/show_bug.cgi?id=13291?

Andrei
Aug 13 2014
next sibling parent reply "Brian Schott" <briancschott gmail.com> writes:
On Thursday, 14 August 2014 at 01:10:54 UTC, Andrei Alexandrescu 
wrote:

 Destroy https://issues.dlang.org/show_bug.cgi?id=13291?

 Andrei


So it'll look like this?

functionBody:
      blockStatement
     | inStatement outStatement? bodyStatement?
     | outStatement inStatement? bodyStatement?
     | unittest bodyStatement
     | unittest inStatement outStatement? bodyStatement
     | unittest outStatement inStatement? bodyStatement
     | bodyStatement
     ;

That is, a unittest always requires a body, and thus can't be 
used in interfaces, correct?
Aug 13 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/13/14, 6:38 PM, Brian Schott wrote:

 On Thursday, 14 August 2014 at 01:10:54 UTC, Andrei Alexandrescu wrote:

 Destroy https://issues.dlang.org/show_bug.cgi?id=13291?

 Andrei


 So it'll look like this?

 functionBody:
       blockStatement
      | inStatement outStatement? bodyStatement?
      | outStatement inStatement? bodyStatement?
      | unittest bodyStatement
      | unittest inStatement outStatement? bodyStatement
      | unittest outStatement inStatement? bodyStatement
      | bodyStatement
      ;

 That is, a unittest always requires a body, and thus can't be used in
 interfaces, correct?


That's about right.

Andrei
Aug 13 2014
prev sibling parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Thursday, 14 August 2014 at 01:38:56 UTC, Brian Schott wrote:

 On Thursday, 14 August 2014 at 01:10:54 UTC, Andrei 
 Alexandrescu wrote:

 Destroy https://issues.dlang.org/show_bug.cgi?id=13291?

 Andrei


 So it'll look like this?

 functionBody:
      blockStatement
     | inStatement outStatement? bodyStatement?
     | outStatement inStatement? bodyStatement?
     | unittest bodyStatement
     | unittest inStatement outStatement? bodyStatement
     | unittest outStatement inStatement? bodyStatement
     | bodyStatement
     ;

 That is, a unittest always requires a body, and thus can't be 
 used in interfaces, correct?


Well, there's nothing to test in abstract functions, is there?

I'm more concerned about the unit test appearing before the code, 
when the usual convention is to put the unit tests after the code 
they are testing...
Aug 13 2014
parent reply "Chris Cain" <zshazz gmail.com> writes:
On Thursday, 14 August 2014 at 02:04:09 UTC, Idan Arye wrote:

 Well, there's nothing to test in abstract functions, is there?

 I'm more concerned about the unit test appearing before the 
 code, when the usual convention is to put the unit tests after 
 the code they are testing...


There is precedence conceptually for putting unittests before 
code (TDD as an example of a technique that suggests unittests 
should be written first). Plus, in some ways it makes sense that 
unittests come first because it shows you how to use the function 
in question. I often look at unittests for examples of how to use 
code, almost like documentation (which also comes before the 
code).

Not to mention the fact that we'd have to come up with new syntax 
otherwise. The following is ambiguous:

     void blah(T)(...)
     in { ... }
     body { ... }
     unittest { ... }

Is the unittest
   1. part of the module and thus should only be run once
or is it
   2. part of the templated function and should be run per 
instantiation?

Current behavior is 1. So it'd be really necessary to come up 
with some new, unambiguous syntax to put the unittest after the 
function. Whereas the current proposal doesn't create an 
ambiguity and doesn't need new, special syntax to perform this 
task.
Aug 13 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/13/2014 7:54 PM, Chris Cain wrote:

 Current behavior is 1. So it'd be really necessary to come up with some new,
 unambiguous syntax to put the unittest after the function. Whereas the current
 proposal doesn't create an ambiguity and doesn't need new, special syntax to
 perform this task.


That's a good and insightful point.
Aug 14 2014
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Aug 13, 2014 at 06:10:54PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:

 Destroy https://issues.dlang.org/show_bug.cgi?id=13291?


[...]

1) How to add attributes to the unittest without them clashing with the
function's attributes?

2) How to parse ddoc comments for the unittest?

3) Where should unittest block stand in relation to in/out contracts?

4) How much can you realistically test on a generic type argument T,
that you can't already cover with concrete types? I'm finding that the
per-instantiation behaviour of unittest blocks inside templated
structs/classes is rarely desired, esp. when you write ddoc unittests
(because you want code examples in the docs to involve concrete types,
not abstract types, otherwise they are of limited use to the reader).
Because of this, I often move unittests outside the aggregate or enclose
them in static if's. This suggests that perhaps per-instantiation
unittests are only of limited utility.


T

-- 
"A one-question geek test. If you get the joke, you're a geek:
Seen on a California license plate on a VW Beetle: 'FEATURE'..." --
Joshua D. Wachs - Natural Intelligence, Inc.
Aug 13 2014
next sibling parent reply Nick Treleaven <ntrel-public yahoo.co.uk> writes:
On 14/08/2014 05:02, H. S. Teoh via Digitalmars-d wrote:

 4) How much can you realistically test on a generic type argument T,
 that you can't already cover with concrete types? I'm finding that the
 per-instantiation behaviour of unittest blocks inside templated
 structs/classes is rarely desired, esp. when you write ddoc unittests
 (because you want code examples in the docs to involve concrete types,
 not abstract types, otherwise they are of limited use to the reader).
 Because of this, I often move unittests outside the aggregate or enclose
 them in static if's.


Yes, which is bad as they can be some distance away from methods, so 
they might not be kept in sync. I think it's not possible to have a 
documented unittest for a method that isn't part of the parent template. 
This is necessary to prevent template instantiation recursion sometimes.

Perhaps we could have new syntax for that, e.g.:

struct S(T)
{
     void foo();

     // force module scope, not part of S
     module unittest()
     {
         S!int s;  // no instantiation recursion
         s.foo();
     }
}
Aug 14 2014
parent Nick Treleaven <ntrel-public yahoo.co.uk> writes:
On 14/08/2014 17:14, Nick Treleaven wrote:

 I think it's not possible to have a documented unittest for a method
 that isn't part of the parent template. This is necessary to prevent
 template instantiation recursion sometimes.


Oops, my example wasn't recursive, I meant where the unittest has some 
local aggregate type defined:

struct S(T)
{
     void foo(){}

     ///
     unittest
     {
         class C {}
         S!C s;	// recursive template expansion
         s.foo();
     }
}

unittest
{
     S!int s;
}
Aug 15 2014
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, 14 August 2014 at 04:04:11 UTC, H. S. Teoh via 
Digitalmars-d wrote:

 On Wed, Aug 13, 2014 at 06:10:54PM -0700, Andrei Alexandrescu 
 via Digitalmars-d wrote:

 Destroy https://issues.dlang.org/show_bug.cgi?id=13291?


 [...]

 1) How to add attributes to the unittest without them clashing 
 with the
 function's attributes?


It probably couldn't have any.


 2) How to parse ddoc comments for the unittest?


I wouldn't expect it to make any sense to use a unittest block 
like that for documentation. The documentation examples are going 
to need specific types. It needs to be possible to actually try 
out the examples.


 3) Where should unittest block stand in relation to in/out 
 contracts?


Does it matter? I don't know what  the current grammar says with 
regards to in and out - maybe it requires that in be before out - 
but I don't see any real reason to care about the order. Why not 
just let them be in any order as long as the body is last?


 4) How much can you realistically test on a generic type 
 argument T,
 that you can't already cover with concrete types? I'm finding 
 that the
 per-instantiation behaviour of unittest blocks inside templated
 structs/classes is rarely desired, esp. when you write ddoc 
 unittests
 (because you want code examples in the docs to involve concrete 
 types,
 not abstract types, otherwise they are of limited use to the 
 reader).
 Because of this, I often move unittests outside the aggregate 
 or enclose
 them in static if's. This suggests that perhaps 
 per-instantiation
 unittests are only of limited utility.


I have _rarely_ found it to be useful to write generic tests 
inside of a template. It's a cute thing to be able to do and 
occasionally useful, but for the most part, I find it to be very 
annoying. For a templated function, it's not that big a deal, but 
for tremplated types, it's a nightmare. You're forced to put all 
of your unittest blocks outside of the struct or class, and you 
can't really use ddoc-ed unittest blocks. I'd actually prefer 
that the unittest blocks _didn't_ get generated for each 
instantation, but I think that that ship has unfortunately 
sailed, and it could be a bit of an implementation problem to 
have them not really be part of the template even though they're 
in it.

But aside from whether it's desirable for unittest blocks to be 
instantiated with the template or not, generating tests that way 
rarely makes much sense in my experience. Simple things like 
constructing the types involved and checking them agains the 
desired result just don't work, because you can't generically 
construct values of a type. Do you want [1, 2, 3, 4, 5], or 
["hello", "world", "goodbye"]? Those types invariably depend on 
what the template was instiated with, making the genericity not 
work for the tests.

I'm not necessarily against adding the proposed feature to the 
language, but it's just shorthand to avoid splitting out the 
template from the function, and I doubt that I'd ever use it.

- Jonathan M Davis
Aug 14 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/13/2014 9:02 PM, H. S. Teoh via Digitalmars-d wrote:

 4) How much can you realistically test on a generic type argument T,
 that you can't already cover with concrete types? I'm finding that the
 per-instantiation behaviour of unittest blocks inside templated
 structs/classes is rarely desired, esp. when you write ddoc unittests
 (because you want code examples in the docs to involve concrete types,
 not abstract types, otherwise they are of limited use to the reader).
 Because of this, I often move unittests outside the aggregate or enclose
 them in static if's. This suggests that perhaps per-instantiation
 unittests are only of limited utility.


Just to expand on this, one of the great advantages of template functions and 
separate unittests is the unittest can instantiate the template function with 
"mocked up" arguments. Instantiating the unittest for every type passed to it 
would defeat that method.
Aug 14 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/14/14, 1:25 PM, Walter Bright wrote:

 On 8/13/2014 9:02 PM, H. S. Teoh via Digitalmars-d wrote:

 4) How much can you realistically test on a generic type argument T,
 that you can't already cover with concrete types? I'm finding that the
 per-instantiation behaviour of unittest blocks inside templated
 structs/classes is rarely desired, esp. when you write ddoc unittests
 (because you want code examples in the docs to involve concrete types,
 not abstract types, otherwise they are of limited use to the reader).
 Because of this, I often move unittests outside the aggregate or enclose
 them in static if's. This suggests that perhaps per-instantiation
 unittests are only of limited utility.


 Just to expand on this, one of the great advantages of template
 functions and separate unittests is the unittest can instantiate the
 template function with "mocked up" arguments. Instantiating the unittest
 for every type passed to it would defeat that method.


Clearly unittests that mock up arguments etc. are a useful device for 
unittesting. But the point of generic unittests is a tad different and 
has to do with a fundamental difference between C++'s and D's approach 
to generics.

In C++ it's entirely acceptable to fail to instantiate a template. In 
fact the C++ idiom SFINAE is literally based on that - "Substitution 
Failure Is Not An Error". There's a bunch of detail to it but bottom 
line it's totally fine for a template to fail to instantiate, both 
technically and socially (e.g. you get syntax errors in the template 
code etc). There's a good amount of dissatisfaction in the C++ community 
about that, which has led to a lot of work on concepts.

In D, it's not acceptable to fail to instantiate; a template should 
either instantiate and work, get filtered out by a template constraint, 
or fail with information by means of a static assert. Random syntax 
errors inside the template are considered poor style.

It follows that once a D template gets instantiated, it's supposed to 
work as expected for the entire range of types it was meant to.


Andrei
Aug 14 2014
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Aug 14, 2014 at 02:34:02PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]

 Clearly unittests that mock up arguments etc. are a useful device for
 unittesting. But the point of generic unittests is a tad different and
 has to do with a fundamental difference between C++'s and D's approach
 to generics.
 
 In C++ it's entirely acceptable to fail to instantiate a template. In
 fact the C++ idiom SFINAE is literally based on that - "Substitution
 Failure Is Not An Error". There's a bunch of detail to it but bottom
 line it's totally fine for a template to fail to instantiate, both
 technically and socially (e.g. you get syntax errors in the template
 code etc). There's a good amount of dissatisfaction in the C++
 community about that, which has led to a lot of work on concepts.
 
 In D, it's not acceptable to fail to instantiate; a template should
 either instantiate and work, get filtered out by a template
 constraint, or fail with information by means of a static assert.
 Random syntax errors inside the template are considered poor style.
 
 It follows that once a D template gets instantiated, it's supposed to
 work as expected for the entire range of types it was meant to.


[...]

How does this relate to writing generic unittests? Since the incoming
types are generic, you can't assume anything about them beyond what the
function itself already assumes, so how would the unittest test anything
beyond what the function already does?

For example, if the function performs x+y on two generic arguments x and
y, how would a generic unittest check whether the result is correct,
since you can't assume anything about the concrete values of x and y?
The only way the unittest can check the result is to see if it equals
x+y, which defeats the purpose because it's tautological with what the
function already does, and therefore proves nothing at all.


T

-- 
"640K ought to be enough" -- Bill G. (allegedly), 1984. "The Internet is not a
primary goal for PC usage" -- Bill G., 1995. "Linux has no impact on
Microsoft's strategy" -- Bill G., 1999.
Aug 14 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/14/14, 3:34 PM, H. S. Teoh via Digitalmars-d wrote:

 How does this relate to writing generic unittests? Since the incoming
 types are generic, you can't assume anything about them beyond what the
 function itself already assumes, so how would the unittest test anything
 beyond what the function already does?


Check the workings of the function via an alternate algorithm for 
example. There's plenty of examples, including unittests inside a 
parameterized struct/class test methods within that class.


 For example, if the function performs x+y on two generic arguments x and
 y, how would a generic unittest check whether the result is correct,
 since you can't assume anything about the concrete values of x and y?
 The only way the unittest can check the result is to see if it equals
 x+y, which defeats the purpose because it's tautological with what the
 function already does, and therefore proves nothing at all.


We could ask the same question about x+y for int in particular: it's a 
primitive so there's not much to test.

This does bring up the interesting point that we need a way to generate 
random values of generic types. 
http://www.haskell.org/haskellwiki/Introduction_to_QuickCheck1 comes to 
mind.


Andrei
Aug 14 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Aug 14, 2014 at 05:49:39PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 8/14/14, 3:34 PM, H. S. Teoh via Digitalmars-d wrote:

How does this relate to writing generic unittests? Since the incoming
types are generic, you can't assume anything about them beyond what
the function itself already assumes, so how would the unittest test
anything beyond what the function already does?


 
 Check the workings of the function via an alternate algorithm for
 example.  There's plenty of examples, including unittests inside a
 parameterized struct/class test methods within that class.


Can you give a concrete example?



For example, if the function performs x+y on two generic arguments x
and y, how would a generic unittest check whether the result is
correct, since you can't assume anything about the concrete values of
x and y?  The only way the unittest can check the result is to see if
it equals x+y, which defeats the purpose because it's tautological
with what the function already does, and therefore proves nothing at
all.


 
 We could ask the same question about x+y for int in particular: it's a
 primitive so there's not much to test.


That's not the point. I used x+y just as an example. You can substitute
that with an arbitrarily complex computation, and the question stands:
how would the unittest determine what's the correct result, without
basically reimplementing the entire function?



 This does bring up the interesting point that we need a way to
 generate random values of generic types.
 http://www.haskell.org/haskellwiki/Introduction_to_QuickCheck1 comes
 to mind.


[...]

I agree that's a cool thing to have, but that still doesn't address the
core issue, that is, given a generic type T, of which you have no
further information about how its concrete values might behave, how do
you write a unittest that checks whether some generic computation
involving T produces the correct result? Even if I hand you some random
instance of T, let's call it x, and you're trying to unittest a function
f, how does the unittest know what the *correct* value of f(x) ought to
be?

Let's use a concrete example. Suppose I'm implementing a shortestPath
algorithm that takes an arbitrary graph type G:

	path!G shortestPath(G)(G graph)
		if (isGraph!G)
	{
		... // fancy algorithm here
		return path(result);
	}

Now let's write a generic unittest:

	unittest
	{
		// Let's test shortestPath on some random instance of G:
		auto g = G.random; // suppose this exists
		auto p = shortestPath(g);

		// OK, now what? How do we know p is the shortest
		// path of the random graph instance g?
		assert(p == ... /* what goes here? */);
	}

The only possibilities I can think of, that work, is to either (1) check
if p == shortestPath(g), which is useless because it proves nothing; or
(2) implement a different shortest path algorithm and check the answer
against that, which suffers from numerous problems:

(a) How do we know this second shortest path algorithm, used by the
unittest, is correct? Why, by unittesting it, of course, with a generic
unittest, and checking the result against ... um... the original
shortestPath implementation?  Even if the answers match, all you've
proven is that the two algorithms are equivalent -- they could *both* be
wrong in the same places (e.g., they could both return an empty path),
but you'd never know that.

(b) There might be multiple shortest paths in g, and the second
algorithm might return a different (but still correct) shortest path.
So just because the answers don't match, doesn't prove that the original
algorithm is wrong.

(c) Maybe the unittest's implementation of shortest path is wrong but
shortestPath is actually correct. How do you tell the difference? Keep
in mind that g is a *random*, *arbitrary* instance of G, and when the
unittest runs we know *nothing* about it other than the fact that it's
some random instance of G.

Basically, I can't think of any sane way to write a generic test that
would give meaningful results for *arbitrary* instances of G, about
which the unittest code has no further information other than it's some
random instance of G.

Due to this, I contend that you need to use *concrete* inputs and check
for *concrete* outputs in order for the unittest to be of any value at
all.  Examples to the contrary are more than welcome -- if you can come
up with one.


T

-- 
Frank disagreement binds closer than feigned agreement.
Aug 14 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/14/14, 7:46 PM, H. S. Teoh via Digitalmars-d wrote:

 On Thu, Aug 14, 2014 at 05:49:39PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:

 On 8/14/14, 3:34 PM, H. S. Teoh via Digitalmars-d wrote:

 How does this relate to writing generic unittests? Since the incoming
 types are generic, you can't assume anything about them beyond what
 the function itself already assumes, so how would the unittest test
 anything beyond what the function already does?


 Check the workings of the function via an alternate algorithm for
 example.  There's plenty of examples, including unittests inside a
 parameterized struct/class test methods within that class.


 Can you give a concrete example?


For all integrals converting some value to a string and back should 
yield the same value.

Accelerated searches in any sorted ranges should provide the same result 
as linear searches.

Sorting should provide sorted results.

...


Andrei
Aug 14 2014
prev sibling next sibling parent "xenon325" <anm programmer.net> writes:
On Friday, 15 August 2014 at 02:48:18 UTC, H. S. Teoh via 
Digitalmars-d wrote:

 how do
 you write a unittest that checks whether some generic 
 computation
 involving T produces the correct result? Even if I hand you 
 some random
 instance of T, let's call it x, and you're trying to unittest a 
 function
 f, how does the unittest know what the *correct* value of f(x) 
 ought to
 be?


You can check some properties like `f(x) < f(x+1)`.


 Let's use a concrete example. Suppose I'm implementing a 
 shortestPath
 algorithm that takes an arbitrary graph type G:

 	path!G shortestPath(G)(G graph)
 		if (isGraph!G)
 	{
 		... // fancy algorithm here
 		return path(result);
 	}

 Now let's write a generic unittest:

 	unittest
 	{
 		// Let's test shortestPath on some random instance of G:
 		auto g = G.random; // suppose this exists
 		auto p = shortestPath(g);

 		// OK, now what? How do we know p is the shortest
 		// path of the random graph instance g?
 		assert(p == ... /* what goes here? */);
 	}

 The only possibilities I can think of, that work, is to either 
 (1) check
 if p == shortestPath(g), which is useless because it proves 
 nothing; or
 (2) implement a different shortest path algorithm and check the 
 answer
 against that, which suffers from numerous problems:

 (a) How do we know this second shortest path algorithm, used by 
 the
 unittest, is correct? Why, by unittesting it, of course, with a 
 generic
 unittest, and checking the result against ... um... the original
 shortestPath implementation?


you can resort to inefficient but trivial algorithms in unit 
tests, e.g. brute force.


 (b) There might be multiple shortest paths in g, and the second
 algorithm might return a different (but still correct) shortest 
 path.
 So just because the answers don't match, doesn't prove that the 
 original
 algorithm is wrong.


still it could be possible to verify some invariants:
assert(fast_path.length == simple_path.length);


So, one can use generic unittests to verify the most, umm... 
generic properties of algorithms and complement them with tests 
for concrete types.
Aug 14 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 15/08/14 04:46, H. S. Teoh via Digitalmars-d wrote:


 I agree that's a cool thing to have, but that still doesn't address the
 core issue, that is, given a generic type T, of which you have no
 further information about how its concrete values might behave, how do
 you write a unittest that checks whether some generic computation
 involving T produces the correct result? Even if I hand you some random
 instance of T, let's call it x, and you're trying to unittest a function
 f, how does the unittest know what the *correct* value of f(x) ought to
 be?

 Let's use a concrete example. Suppose I'm implementing a shortestPath
 algorithm that takes an arbitrary graph type G:

 	path!G shortestPath(G)(G graph)
 		if (isGraph!G)
 	{
 		... // fancy algorithm here
 		return path(result);
 	}

 Now let's write a generic unittest:

 	unittest
 	{
 		// Let's test shortestPath on some random instance of G:
 		auto g = G.random; // suppose this exists
 		auto p = shortestPath(g);

 		// OK, now what? How do we know p is the shortest
 		// path of the random graph instance g?
 		assert(p == ... /* what goes here? */);
 	}


I don't know if this is how Andrei is thinking but it might be useful:

1. Instantiate "shortestPath" with random values of random types and 
assert that the instantiation succeeds

2. Then call "shortestPath" with known values of various types to 
validate that the algorithm is correct

-- 
/Jacob Carlborg
Aug 14 2014
prev sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, 14 August 2014 at 21:34:00 UTC, Andrei Alexandrescu 
wrote:

 It follows that once a D template gets instantiated, it's 
 supposed to work as expected for the entire range of types it 
 was meant to.


Yes, but in my experience, it's rarely the case that you can 
actually test a template generically, because construction is not 
generic, so the values to use are hard to generate generically, 
and then creating the values to test the results against are 
frequently just as hard. Where you can test something 
generically, it's great, but it's rarely practical from what I've 
seen. We should support it regardless (and we do currently by 
separating the template declaration from the function 
declaration), and adding the shorter syntax for it that you're 
suggesting may be a good idea, but I really don't see these types 
of tests as being effective in most cases even though it's 
fantastic when you can write such tests.

Also, you still have to specifically instantiate functions like 
this outside of the unittest block that you're proposing - 
particularly if this is in a library, because in that case, the 
function may not actually be used outside of its tests, and the 
unit tests aren't going to run when someone links in the library. 
And if that's the case, you arguably might as well just write 
tests for each of the individual types.

So, ultimately, while these sort of tests can be good, I really 
don't think that they're effective very often.

- Jonathan M Davis
Aug 14 2014
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Aug 14, 2014 at 10:37:22PM +0000, Jonathan M Davis via Digitalmars-d
wrote:
[...]

 Also, you still have to specifically instantiate functions like this
 outside of the unittest block that you're proposing - particularly if
 this is in a library, because in that case, the function may not
 actually be used outside of its tests, and the unit tests aren't going
 to run when someone links in the library. And if that's the case, you
 arguably might as well just write tests for each of the individual
 types.
 
 So, ultimately, while these sort of tests can be good, I really don't
 think that they're effective very often.


[...]

Yeah, when I first read Andrei's proposal, I actually quite liked it...
until I started thinking about use cases where it would be handy, and
then I suddenly realized that I can't think of any! All *useful*
unittests I can think of involve the use of concrete values of concrete
types. Having only .init guaranteed to exist for a generic type T (and
sometimes not even that if you have a Voldemort) greatly cripples the
utility of the unittest.

So I'd love to see an actual, non-trivial, motivating example of where
this kind of per-instantiation unittest is actually useful. That may yet
change my mind about this issue.  ;-)


T

-- 
The richest man is not he who has the most, but he who needs the least.
Aug 14 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/14/14, 3:46 PM, H. S. Teoh via Digitalmars-d wrote:

 Having only .init guaranteed to exist for a generic type T (and
 sometimes not even that if you have a Voldemort) greatly cripples the
 utility of the unittest.


I'm thinking of T.random as a sort of generator of random correct 
objects of type T...

Andrei
Aug 14 2014
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Friday, 15 August 2014 at 00:52:35 UTC, Andrei Alexandrescu 
wrote:

 On 8/14/14, 3:46 PM, H. S. Teoh via Digitalmars-d wrote:

 Having only .init guaranteed to exist for a generic type T (and
 sometimes not even that if you have a Voldemort) greatly 
 cripples the
 utility of the unittest.


 I'm thinking of T.random as a sort of generator of random 
 correct objects of type T...

 Andrei


So now every type will need to declare a way for randomly 
instances of it?

At any rate, random is bad for unit testing. Unit tests need to 
be deterministic, because when a unit test fails you want to 
debug it to see what's wrong. If the unit test generates some 
random data, in your debug it'll probably generate different data 
than the one it generated when the test failed, and the problem 
may or may not repeat.

Of course, you can always use a random function with a predefined 
seed to solve that, but there are more problems yet. Unit tests 
usually operate by setting up the data, calling a function, and 
testing it's result and/or side-effects. If you set up the data 
by hand, you know what the results and side-effects are supposed 
to be.

For example, let's take a look at the example unit test for 
CRC(http://dlang.org/phobos/std_digest_crc.html#CRC32):

     //Simple example, hashing a string using crc32Of helper 
function
     ubyte[4] hash = crc32Of("abc");
     //Let's get a hash string
     assert(crcHexString(hash) == "352441C2");

Here, we know what the result is supposed to be, because we can 
calculate it manually(or with external tools that we know they 
are correct) and write the literal result in the unit test. But 
How will you check that `crc32Of(string.random())` returns a 
correct result?
Aug 14 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/14/14, 6:20 PM, Idan Arye wrote:

 At any rate, random is bad for unit testing. Unit tests need to be
 deterministic, because when a unit test fails you want to debug it to
 see what's wrong.


Just publish the seed upon failure. It's a classic technique. -- Andrei
Aug 14 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/14/14, 3:37 PM, Jonathan M Davis wrote:

 On Thursday, 14 August 2014 at 21:34:00 UTC, Andrei Alexandrescu wrote:

 It follows that once a D template gets instantiated, it's supposed to
 work as expected for the entire range of types it was meant to.


 Yes, but in my experience, it's rarely the case that you can actually
 test a template generically, because construction is not generic, so the
 values to use are hard to generate generically, and then creating the
 values to test the results against are frequently just as hard.


We need a way to generate correct but random values generically.


 Where
 you can test something generically, it's great, but it's rarely
 practical from what I've seen. We should support it regardless (and we
 do currently by separating the template declaration from the function
 declaration), and adding the shorter syntax for it that you're
 suggesting may be a good idea, but I really don't see these types of
 tests as being effective in most cases even though it's fantastic when
 you can write such tests.


My thesis here is, if you get to instantiate it you must be able to 
trigger some testing for it.


 Also, you still have to specifically instantiate functions like this
 outside of the unittest block that you're proposing - particularly if
 this is in a library, because in that case, the function may not
 actually be used outside of its tests, and the unit tests aren't going
 to run when someone links in the library. And if that's the case, you
 arguably might as well just write tests for each of the individual types.


Agreed.


 So, ultimately, while these sort of tests can be good, I really don't
 think that they're effective very often.


I feel this is fertile ground. We should think more about it.


Andrei
Aug 14 2014
parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, 15 August 2014 at 00:51:56 UTC, Andrei Alexandrescu 
wrote:

 So, ultimately, while these sort of tests can be good, I 
 really don't
 think that they're effective very often.


 I feel this is fertile ground. We should think more about it.


Oh, I'm not at all opposed to trying to leverage this. If we can, 
that would be fantastic. I just haven't seen much practical uses 
of it thus far. And I'm not opposed to adding the proposed 
feature, but at the moment, it seems like it's just providing a 
simpler syntax for something that we can already do that hasn't 
yet shown to be very useful.

- Jonathan M Davis
Aug 14 2014
prev sibling next sibling parent reply "Meta" <jared771 gmail.com> writes:
On Thursday, 14 August 2014 at 01:10:54 UTC, Andrei Alexandrescu 
wrote:

 Destroy https://issues.dlang.org/show_bug.cgi?id=13291?

 Andrei


It's worth noting that you can currently do this:

template fun(T)
{
	int fun(T val)
	{
		static if(is(T == string) || is(T == float))
		{
			return 0;
		}
		else
		{
			return 1;
		}
	}
	
	unittest
	{
		assert(fun(T.init) == 0);
	}
}

void main()
{
	fun("test"); //Ok
	fun(3.0f);   //Ok
	fun(1);      //AssertError thrown
}
Aug 13 2014
parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, 14 August 2014 at 05:53:43 UTC, Meta wrote:

 On Thursday, 14 August 2014 at 01:10:54 UTC, Andrei 
 Alexandrescu wrote:

 Destroy https://issues.dlang.org/show_bug.cgi?id=13291?

 Andrei


 It's worth noting that you can currently do this:

 template fun(T)
 {
 	int fun(T val)
 	{
 		static if(is(T == string) || is(T == float))
 		{
 			return 0;
 		}
 		else
 		{
 			return 1;
 		}
 	}
 	
 	unittest
 	{
 		assert(fun(T.init) == 0);
 	}
 }

 void main()
 {
 	fun("test"); //Ok
 	fun(3.0f);   //Ok
 	fun(1);      //AssertError thrown
 }


Yeah. This proposal adds a shortcut to do this sort of thing, but 
it doesn't actually add new functionality. And in my experience, 
it's not even useful functionality. It's just too hard to be able 
to generic types, because construction isn't generic. _Some_ 
things can be tested generically (particularly if all you need is 
the init value for a type), and it's occasionally useful, but in 
most cases, it isn't. So, I don't think that it really hurts to 
add this feature, but I don't think that it really adds much 
either.

- Jonathan M Davis
Aug 14 2014
prev sibling next sibling parent Timothee Cour via Digitalmars-d <digitalmars-d puremagic.com> writes:
Does the proposal handle multiple unittest blocks?



On Wed, Aug 13, 2014 at 9:02 PM, H. S. Teoh via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Wed, Aug 13, 2014 at 06:10:54PM -0700, Andrei Alexandrescu via
 Digitalmars-d wrote:

 Destroy https://issues.dlang.org/show_bug.cgi?id=13291?


 [...]

 1) How to add attributes to the unittest without them clashing with the
 function's attributes?

 2) How to parse ddoc comments for the unittest?

 3) Where should unittest block stand in relation to in/out contracts?

 4) How much can you realistically test on a generic type argument T,
 that you can't already cover with concrete types? I'm finding that the
 per-instantiation behaviour of unittest blocks inside templated
 structs/classes is rarely desired, esp. when you write ddoc unittests
 (because you want code examples in the docs to involve concrete types,
 not abstract types, otherwise they are of limited use to the reader).
 Because of this, I often move unittests outside the aggregate or enclose
 them in static if's. This suggests that perhaps per-instantiation
 unittests are only of limited utility.


 T

 --
 "A one-question geek test. If you get the joke, you're a geek:
 Seen on a California license plate on a VW Beetle: 'FEATURE'..." --
 Joshua D. Wachs - Natural Intelligence, Inc.
Aug 13 2014
prev sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 Destroy https://issues.dlang.org/show_bug.cgi?id=13291?


I'd like a way to test nested functions:


void foo() {
     int bar() { return 0; }

     unittest {
         assert(bar() == 1);
     }
}

void main() {}


Bye,
bearophile
Aug 14 2014
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 08/14/2014 08:26 PM, bearophile wrote:

 Andrei Alexandrescu:

 Destroy https://issues.dlang.org/show_bug.cgi?id=13291?


 I'd like a way to test nested functions:


 void foo() {
      int bar() { return 0; }

      unittest {
          assert(bar() == 1);
      }
 }

 void main() {}


 Bye,
 bearophile


It is already possible, but I doubt that this is intended (where would 
the unit test get the enclosing stack frame of 'foo' from?):

void foo(){
     int bar(){ return 0; }
     mixin barTest;
}
template barTest(){
     unittest {
         assert(bar()==1);
     }
}
void main() {}

You should be safe for static nested functions though, which the 
compiler will enforce are the only ones called if you make the unit test 
'static'. :o)
Aug 14 2014
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 08/14/2014 08:39 PM, Timon Gehr wrote:

 On 08/14/2014 08:26 PM, bearophile wrote:

 Andrei Alexandrescu:

 Destroy https://issues.dlang.org/show_bug.cgi?id=13291?


 I'd like a way to test nested functions:
 ...
 void foo() {
      int bar() { return 0; }

      unittest {
          assert(bar() == 1);
      }
 }

 void main() {}
  ...


 It is already possible, ...


Another way:

void foo(){
     static int bar() { return 0; }
     static struct T{ unittest{ assert(bar()==1); } }
}
void main() {}
Aug 14 2014
prev sibling next sibling parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Thursday, 14 August 2014 at 18:26:06 UTC, bearophile wrote:

 Andrei Alexandrescu:

 Destroy https://issues.dlang.org/show_bug.cgi?id=13291?


 I'd like a way to test nested functions:


 void foo() {
     int bar() { return 0; }

     unittest {
         assert(bar() == 1);
     }
 }

 void main() {}


 Bye,
 bearophile


Nested functions are not really "units" - they are implementation 
details.
Aug 14 2014
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Idan Arye:


 Nested functions are not really "units" - they are 
 implementation details.


It's not important how you call them. I'd still like some nice 
way to test those "details" :-)

----------------

H. S. Teoh:


 When should such unittests be run, and how should the following 
 code behave?


Is just after the post-condition of the enclosing function a good 
moment to run them?

Bye,
bearophile
Aug 14 2014
parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Aug 14, 2014 at 08:38:31PM +0000, bearophile via Digitalmars-d wrote:
[...]

 H. S. Teoh:
 

When should such unittests be run, and how should the following code
behave?


 
 Is just after the post-condition of the enclosing function a good
 moment to run them?


[...]

So you're suggesting that these unittests should run at *every single
function call*? That sounds like total overkill to me.

Furthermore, if the unittest fails, what should the runtime do, now that
the program is actually already running? Aborting seems to be the most
reasonable answer, in which case the question is, why not just write
asserts in the outer function instead?


T

-- 
Skill without imagination is craftsmanship and gives us many useful objects
such as wickerwork picnic baskets.  Imagination without skill gives us modern
art. -- Tom Stoppard
Aug 14 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Aug 14, 2014 at 06:26:05PM +0000, bearophile via Digitalmars-d wrote:

 Andrei Alexandrescu:

Destroy https://issues.dlang.org/show_bug.cgi?id=13291?


 
 I'd like a way to test nested functions:
 
 
 void foo() {
     int bar() { return 0; }
 
     unittest {
         assert(bar() == 1);
     }
 }
 
 void main() {}


[...]

When should such unittests be run, and how should the following code
behave?

	void foo(int x) {
		int bar() {
			return x+1;
		}
		unittest {
			// When is this unittest run? What's the value
			// of x when the unittest runs?
			assert(bar() == 1);
		}
	}


T

-- 
"Uhh, I'm still not here." -- KD, while "away" on ICQ.
Aug 14 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/14/14, 11:26 AM, bearophile wrote:

 Andrei Alexandrescu:

 Destroy https://issues.dlang.org/show_bug.cgi?id=13291?


 I'd like a way to test nested functions:


That's in keep with the turtles principle, but wouldn't practically be 
too much? -- Andrei
Aug 14 2014