www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - documented unit tests as examples

reply Steven Schveighoffer <schveiguy yahoo.com> writes:
Just looking at this PR: https://github.com/dlang/phobos/pull/4319

Now the example, instead of running and producing output (i.e. visual 
feedback) that the program is doing something, just runs and creates no 
feedback.

I'm wondering if we can have a mechanism for documented unit tests to 
have a slightly different showing inside the docs vs. the actual unit test.

For example, let's say we have a function writelnAssert. Used like this:

writelnAssert(someText, "Text You Expect To Output");

When running this function, it's basically just an assert that someText 
== the expected text. However, when DDOC creates the document for this, 
it says:

writeln(someText); // "Text You Expect To Output"

This way, we are actually testing the output, but at the same time, 
giving someone playing with the example the tools to see some feedback.

Thoughts?

-Steve
May 13 2016
next sibling parent reply Meta <jared771 gmail.com> writes:
On Friday, 13 May 2016 at 20:39:56 UTC, Steven Schveighoffer 
wrote:

 Just looking at this PR: 
 https://github.com/dlang/phobos/pull/4319

 Now the example, instead of running and producing output (i.e. 
 visual feedback) that the program is doing something, just runs 
 and creates no feedback.

 I'm wondering if we can have a mechanism for documented unit 
 tests to have a slightly different showing inside the docs vs. 
 the actual unit test.

 For example, let's say we have a function writelnAssert. Used 
 like this:

 writelnAssert(someText, "Text You Expect To Output");

 When running this function, it's basically just an assert that 
 someText == the expected text. However, when DDOC creates the 
 document for this, it says:

 writeln(someText); // "Text You Expect To Output"

 This way, we are actually testing the output, but at the same 
 time, giving someone playing with the example the tools to see 
 some feedback.

 Thoughts?

 -Steve


When I was new to D and I first saw the `assert(...)` idiom in an 
example in the documentation, it confused me for a minute or two, 
but if you know what `assert` does you can quickly wrap your head 
around the fact that it's both a test and an example. This would 
benefit users that are completely new to programming in general, 
however.
May 13 2016
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 5/13/16 4:55 PM, Meta wrote:


 When I was new to D and I first saw the `assert(...)` idiom in an
 example in the documentation, it confused me for a minute or two, but if
 you know what `assert` does you can quickly wrap your head around the
 fact that it's both a test and an example. This would benefit users that
 are completely new to programming in general, however.


Given the fact that asserts aren't always run, it's never comforting to 
me to run a program that tests something and have it give NO feedback. 
In fact, I frequently find myself triggering the assert to make sure 
it's actually being run (and I've caught the build not actually running 
it many times).

This has a negative affect on anyone actually looking to see how a D 
function works. I can write a program that does nothing easily enough, 
why such a complicated example?

-Steve
May 13 2016
parent Lutger <lutger.blijdestijn gmail.com> writes:
On Friday, 13 May 2016 at 21:00:04 UTC, Steven Schveighoffer 
wrote:

 On 5/13/16 4:55 PM, Meta wrote:


 When I was new to D and I first saw the `assert(...)` idiom in 
 an
 example in the documentation, it confused me for a minute or 
 two, but if
 you know what `assert` does you can quickly wrap your head 
 around the
 fact that it's both a test and an example. This would benefit 
 users that
 are completely new to programming in general, however.


 Given the fact that asserts aren't always run, it's never 
 comforting to me to run a program that tests something and have 
 it give NO feedback. In fact, I frequently find myself 
 triggering the assert to make sure it's actually being run (and 
 I've caught the build not actually running it many times).

 This has a negative affect on anyone actually looking to see 
 how a D function works. I can write a program that does nothing 
 easily enough, why such a complicated example?

 -Steve


This is a flaw of the simplistic test runner, not of the idea of 
unittests itself. Every other unittest system I worked with, 
including for example unit-threaded in D, reports a summary of 
the amount of tests that are ran. Very simple and just enough 
information that the test you just added has indeed been 
executed. One line is enough.

Honestly I think keeping asserts in examples is better than the 
assert/writeln hybrid approach, because 1) asserts give the 
reader exact information about the expected behavior and 
contracts of a function (this information is lost to the reader 
when the asserts are reduced to print statements) and 2) having 
unittests compile to something very differently depending on 
context sounds like adding too much accidental complexity.

Ideally there would be a way to print the values of all arguments 
given to an assert, that would be the most informative. And 
zooming out just a one liner with the number of tests or asserts 
ran and the number succeeded. I think this should be the domain 
of an external tool or library though, not the compiler itself. 
It is certainly possible to create such a tool and have phobos 
use it, no reason to add more complexity to the language itself.
May 15 2016
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 13 May 2016 at 20:39:56 UTC, Steven Schveighoffer 
wrote:

 Thoughts?


Yeah, I'm not a big fan of this either. A lot of in-comment 
examples have been moved to unittests lately, and I think it is a 
net negative:

* Running it gives silent output
* Data representation in source isn't always instructive
* Assert just kinda looks weird

The one pro would be that it is automatically tested... but is 
it? Consider the following:

---
import std.stdio;

///
unittest {
    writeln("Hello, world!");
}
---

That passes the test, but if the user copy/pasted the example, it 
wouldn't actually compile because of the missing import.

Certainly, some surrounding boilerplate is expected much of the 
time, but the unittest doesn't even prove it actually runs with 
the same user-expected surrounding code. It just proves it runs 
from the implementation module: it can use private imports, 
private functions, and more.

So it is a dubious win for automatic testing too.



I prefer examples that you can copy/paste and get something 
useful out of more than documented unittests. I actually really 
like either one-liners showing the syntax and/or complete 
programs that do something - MSDN is good about having those 
kinds of examples.

D's documented unittests are somewhere in the middle... and I 
think fails to capture the advantages of either extreme.
May 13 2016
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 5/13/16 5:16 PM, Adam D. Ruppe wrote:

 On Friday, 13 May 2016 at 20:39:56 UTC, Steven Schveighoffer wrote:

 Thoughts?


 Yeah, I'm not a big fan of this either. A lot of in-comment examples
 have been moved to unittests lately, and I think it is a net negative:

 * Running it gives silent output
 * Data representation in source isn't always instructive
 * Assert just kinda looks weird

 The one pro would be that it is automatically tested... but is it?
 Consider the following:

 ---
 import std.stdio;

 ///
 unittest {
     writeln("Hello, world!");
 }
 ---

 That passes the test, but if the user copy/pasted the example, it
 wouldn't actually compile because of the missing import.


This isn't tested, even for DDOC examples. So I'm not super concerned 
about it.

A unit test can use a private module symbol and that won't work in user 
code either.


 Certainly, some surrounding boilerplate is expected much of the time,
 but the unittest doesn't even prove it actually runs with the same
 user-expected surrounding code. It just proves it runs from the
 implementation module: it can use private imports, private functions,
 and more.

 So it is a dubious win for automatic testing too.


The idea is to make sure examples actually compile and run in SOME context.

To get down to the lowest level, you can tell someone to copy and paste 
an example in notepad, and run dmd on it, but if they haven't yet 
downloaded dmd, then it won't work ;) It's impossible to test what the 
user is going to do or know before hand.


 D's documented unittests are somewhere in the middle... and I think
 fails to capture the advantages of either extreme.


I'm not knocking documented unit tests. Certainly, without documented 
unit tests, the situation before was that examples were riddled with 
bugs. What I'm pointing out here is that DDOC examples have some 
advantages that unittests cannot currently use.

A potential way to fix this may be marking a unit test as being a 
complete example program that assumes the user has installed proper 
access to the library. Then it won't compile unless you add the correct 
imports, and it's treated as if it were in a separate module (no private 
symbol access). This is probably the closest we can get to simulating a 
user copying an example unit test into his own file and trying to run it.

-Steve
May 13 2016
parent Seb <seb wilzba.ch> writes:
On Friday, 13 May 2016 at 21:27:25 UTC, Steven Schveighoffer 
wrote:

 A potential way to fix this may be marking a unit test as being 
 a complete example program that assumes the user has installed 
 proper access to the library. Then it won't compile unless you 
 add the correct imports, and it's treated as if it were in a 
 separate module (no private symbol access). This is probably 
 the closest we can get to simulating a user copying an example 
 unit test into his own file and trying to run it.

 -Steve


Maybe I should have stated that I discovered that the in-text 
example is broken, because I was working on this new feature: 
https://github.com/dlang/dlang.org/pull/1297 (allow all unittests 
in Phobos to be run online via DPaste).
So yep for this feature to work smoothly we either need to 
auto-detect what other imports (except for the current module) 
are needed, or force Phobos devs to explicitly import other 
modules in ddoced unittests. I would also prefer the latter.
Anyways this example didn't run, because the import std.conv 
wasn't imported and the idea to put tests whenever possible from 
plaintext to D code is too ensure that are actually working ;-)

 Steve: I do understand you partially - I don't like assert that 
much either, because when it breaks you don't know why. That's 
why I would prefer if we would have something like

assertEqual("a", "b")

in Phobos and most of the tests.
The rationale is that it breaks with a proper error message 
instead of just saying - hey you have some error there. We still 
could automatically replace this to writeln(a); // b, but I think 
having a proper equal should already be a lot better and people 
hopefully can then just put in a different value for 'b' to see 
the test fail.

Here are some numbers, from in total 28761 assert's in Phobos

- 15266 use "==" (53%)
- 1451 use equal (5%)
- 165 use approxEqual (0.6%)

- 452 assert(0
- 5165 static assert
May 14 2016
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Friday, May 13, 2016 16:39:56 Steven Schveighoffer via Digitalmars-d wrote:

 Just looking at this PR: https://github.com/dlang/phobos/pull/4319

 Now the example, instead of running and producing output (i.e. visual
 feedback) that the program is doing something, just runs and creates no
 feedback.

 I'm wondering if we can have a mechanism for documented unit tests to
 have a slightly different showing inside the docs vs. the actual unit test.

 For example, let's say we have a function writelnAssert. Used like this:

 writelnAssert(someText, "Text You Expect To Output");

 When running this function, it's basically just an assert that someText
 == the expected text. However, when DDOC creates the document for this,
 it says:

 writeln(someText); // "Text You Expect To Output"

 This way, we are actually testing the output, but at the same time,
 giving someone playing with the example the tools to see some feedback.

 Thoughts?


I confess that I really don't have a problem with assertions being used in
examples like they normally are. That's how unit tests work, and anyone who
uses D for any length of time is going to find that out quickly. And it's
not like you can see the result of writeln anyway unless you copy-pase the
code into an editor to run it yourself, and if you do that, you can replace
the assertion with a call to writeln easily enough. And the assertion can
actually show you the output in the documentation, whereas you won't get
that with writeln. So, if anything, it seems to me that having assertions
that show you the values is better than having to copy-paste the code into
an editor to run it and see the output.

What's a bigger problem is when you want an example that does something like
talk to a website (like std.net.curl does), since it's frequently not
reasonable to have that in unit tests. Fortunately, it's still perfectly
possible to have an untested example if you need it, and it's not usually
needed.

- Jonathan M Davis
May 13 2016
prev sibling next sibling parent Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 5/13/16, Steven Schveighoffer via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 I'm wondering if we can have a mechanism for documented unit tests to
 have a slightly different showing inside the docs vs. the actual unit test.

 For example, let's say we have a function writelnAssert. Used like this:

 writelnAssert(someText, "Text You Expect To Output");


Hmm.. I like the idea, but I have an alternate proposal. Add a new
compiler switch (or better yet use a built-in version switch) which
makes all assertions chatty.

So for a unittest like this:

----
auto sum (a, b) => a + b;

///
unittest { assert(sum(2, 2) == 4); }
----

Running it with something like this:

----
$ dmd -unittest -version=printasserts -run test.d
----

Would produce:

----

 test.d(4): assert(4 == 4); was successful


----

We could nitpick about the actual output, but the general idea is to
avoid having to modify all the unittests by hand and instead use a
compiler switch for chatty assertions.

And actually if we used a -version switch then I think we could
implement this entirely in Druntime. That is unless I remember
incorrectly how assertions are emitted, I haven't looked at the code
in a while.
May 14 2016
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, May 14, 2016 at 03:32:45PM +0200, Andrej Mitrovic via Digitalmars-d
wrote:
[...]

 Add a new compiler switch (or better yet use a built-in version
 switch) which makes all assertions chatty.
 
 So for a unittest like this:
 
 ----
 auto sum (a, b) => a + b;
 
 ///
 unittest { assert(sum(2, 2) == 4); }
 ----
 
 Running it with something like this:
 
 ----
 $ dmd -unittest -version=printasserts -run test.d
 ----
 
 Would produce:
 
 ----

 test.d(4): assert(4 == 4); was successful


 ----
 
 We could nitpick about the actual output, but the general idea is to
 avoid having to modify all the unittests by hand and instead use a
 compiler switch for chatty assertions.


I like this idea. But wouldn't it be too verbose? I.e., there may be a
lot of non-unittest-related assertions inside other functions and such.
We could say only top-level asserts in a unittest block become chatty,
but that excludes unittests calling unittest-only helper functions to do
the checks.



 And actually if we used a -version switch then I think we could
 implement this entirely in Druntime. That is unless I remember
 incorrectly how assertions are emitted, I haven't looked at the code
 in a while.


How would the assert implementation in druntime access the expression
being tested? I.e., in a way that it can be printed out in
human-readable form.


T

-- 
Gone Chopin. Bach in a minuet.
May 14 2016
prev sibling parent reply Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 5/14/16, H. S. Teoh via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 I like this idea. But wouldn't it be too verbose?


Hmm good point. Alternatively we could perhaps make ddoc inject
writeln calls, something like:

----
auto sum (a, b) => a + b;

///
unittest { assert(sum(2, 2) == 4); }
----

Turns into ddoc output:

----
writeln(sum(2, 2));
assert(sum(2, 2) == 4);
----


 How would the assert implementation in druntime access the expression
 being tested? I.e., in a way that it can be printed out in
 human-readable form.


Ah I was remembering wrong here. The compiler turns the assert
expression into something like "cast(bool)expression || __assert".

So anyway, I think perhaps the simplest solution is to make ddoc
inject the writeln calls (or possibly replace assertions altogether in
the output).
May 14 2016
parent Ryan Frame <dlang ryanjframe.com> writes:
On Saturday, 14 May 2016 at 17:48:48 UTC, Andrej Mitrovic wrote:

 So anyway, I think perhaps the simplest solution is to make 
 ddoc inject the writeln calls (or possibly replace assertions 
 altogether in the output).


The problem with replacing the assert is the loss of information.

assert(sum(2, 2) == 4);

tells me what sum(2, 2) should return merely by reading the 
example, but

writeln(sum(2, 2));

requires me to run it (or re-read the documentation for sum(), 
but the example is supposed to save me from that). Merely 
injecting the writeln() provides full return information when 
reading and executing the example (though more context should be 
provided than just the return value).
May 14 2016