www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - documented unit tests as examples

reply Steven Schveighoffer <schveiguy yahoo.com> writes:
Just looking at this PR: https://github.com/dlang/phobos/pull/4319

Now the example, instead of running and producing output (i.e. visual 
feedback) that the program is doing something, just runs and creates no 
feedback.

I'm wondering if we can have a mechanism for documented unit tests to 
have a slightly different showing inside the docs vs. the actual unit test.

For example, let's say we have a function writelnAssert. Used like this:

writelnAssert(someText, "Text You Expect To Output");

When running this function, it's basically just an assert that someText 
== the expected text. However, when DDOC creates the document for this, 
it says:

writeln(someText); // "Text You Expect To Output"

This way, we are actually testing the output, but at the same time, 
giving someone playing with the example the tools to see some feedback.

Thoughts?

-Steve
May 13 2016
next sibling parent reply Meta <jared771 gmail.com> writes:
On Friday, 13 May 2016 at 20:39:56 UTC, Steven Schveighoffer 
wrote:
 Just looking at this PR: 
 https://github.com/dlang/phobos/pull/4319

 Now the example, instead of running and producing output (i.e. 
 visual feedback) that the program is doing something, just runs 
 and creates no feedback.

 I'm wondering if we can have a mechanism for documented unit 
 tests to have a slightly different showing inside the docs vs. 
 the actual unit test.

 For example, let's say we have a function writelnAssert. Used 
 like this:

 writelnAssert(someText, "Text You Expect To Output");

 When running this function, it's basically just an assert that 
 someText == the expected text. However, when DDOC creates the 
 document for this, it says:

 writeln(someText); // "Text You Expect To Output"

 This way, we are actually testing the output, but at the same 
 time, giving someone playing with the example the tools to see 
 some feedback.

 Thoughts?

 -Steve
When I was new to D and I first saw the `assert(...)` idiom in an example in the documentation, it confused me for a minute or two, but if you know what `assert` does you can quickly wrap your head around the fact that it's both a test and an example. This would benefit users that are completely new to programming in general, however.
May 13 2016
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 5/13/16 4:55 PM, Meta wrote:

 When I was new to D and I first saw the `assert(...)` idiom in an
 example in the documentation, it confused me for a minute or two, but if
 you know what `assert` does you can quickly wrap your head around the
 fact that it's both a test and an example. This would benefit users that
 are completely new to programming in general, however.
Given the fact that asserts aren't always run, it's never comforting to me to run a program that tests something and have it give NO feedback. In fact, I frequently find myself triggering the assert to make sure it's actually being run (and I've caught the build not actually running it many times). This has a negative affect on anyone actually looking to see how a D function works. I can write a program that does nothing easily enough, why such a complicated example? -Steve
May 13 2016
parent Lutger <lutger.blijdestijn gmail.com> writes:
On Friday, 13 May 2016 at 21:00:04 UTC, Steven Schveighoffer 
wrote:
 On 5/13/16 4:55 PM, Meta wrote:

 When I was new to D and I first saw the `assert(...)` idiom in 
 an
 example in the documentation, it confused me for a minute or 
 two, but if
 you know what `assert` does you can quickly wrap your head 
 around the
 fact that it's both a test and an example. This would benefit 
 users that
 are completely new to programming in general, however.
Given the fact that asserts aren't always run, it's never comforting to me to run a program that tests something and have it give NO feedback. In fact, I frequently find myself triggering the assert to make sure it's actually being run (and I've caught the build not actually running it many times). This has a negative affect on anyone actually looking to see how a D function works. I can write a program that does nothing easily enough, why such a complicated example? -Steve
This is a flaw of the simplistic test runner, not of the idea of unittests itself. Every other unittest system I worked with, including for example unit-threaded in D, reports a summary of the amount of tests that are ran. Very simple and just enough information that the test you just added has indeed been executed. One line is enough. Honestly I think keeping asserts in examples is better than the assert/writeln hybrid approach, because 1) asserts give the reader exact information about the expected behavior and contracts of a function (this information is lost to the reader when the asserts are reduced to print statements) and 2) having unittests compile to something very differently depending on context sounds like adding too much accidental complexity. Ideally there would be a way to print the values of all arguments given to an assert, that would be the most informative. And zooming out just a one liner with the number of tests or asserts ran and the number succeeded. I think this should be the domain of an external tool or library though, not the compiler itself. It is certainly possible to create such a tool and have phobos use it, no reason to add more complexity to the language itself.
May 15 2016
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 13 May 2016 at 20:39:56 UTC, Steven Schveighoffer 
wrote:
 Thoughts?
Yeah, I'm not a big fan of this either. A lot of in-comment examples have been moved to unittests lately, and I think it is a net negative: * Running it gives silent output * Data representation in source isn't always instructive * Assert just kinda looks weird The one pro would be that it is automatically tested... but is it? Consider the following: --- import std.stdio; /// unittest { writeln("Hello, world!"); } --- That passes the test, but if the user copy/pasted the example, it wouldn't actually compile because of the missing import. Certainly, some surrounding boilerplate is expected much of the time, but the unittest doesn't even prove it actually runs with the same user-expected surrounding code. It just proves it runs from the implementation module: it can use private imports, private functions, and more. So it is a dubious win for automatic testing too. I prefer examples that you can copy/paste and get something useful out of more than documented unittests. I actually really like either one-liners showing the syntax and/or complete programs that do something - MSDN is good about having those kinds of examples. D's documented unittests are somewhere in the middle... and I think fails to capture the advantages of either extreme.
May 13 2016
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 5/13/16 5:16 PM, Adam D. Ruppe wrote:
 On Friday, 13 May 2016 at 20:39:56 UTC, Steven Schveighoffer wrote:
 Thoughts?
Yeah, I'm not a big fan of this either. A lot of in-comment examples have been moved to unittests lately, and I think it is a net negative: * Running it gives silent output * Data representation in source isn't always instructive * Assert just kinda looks weird The one pro would be that it is automatically tested... but is it? Consider the following: --- import std.stdio; /// unittest { writeln("Hello, world!"); } --- That passes the test, but if the user copy/pasted the example, it wouldn't actually compile because of the missing import.
This isn't tested, even for DDOC examples. So I'm not super concerned about it. A unit test can use a private module symbol and that won't work in user code either.
 Certainly, some surrounding boilerplate is expected much of the time,
 but the unittest doesn't even prove it actually runs with the same
 user-expected surrounding code. It just proves it runs from the
 implementation module: it can use private imports, private functions,
 and more.

 So it is a dubious win for automatic testing too.
The idea is to make sure examples actually compile and run in SOME context. To get down to the lowest level, you can tell someone to copy and paste an example in notepad, and run dmd on it, but if they haven't yet downloaded dmd, then it won't work ;) It's impossible to test what the user is going to do or know before hand.
 D's documented unittests are somewhere in the middle... and I think
 fails to capture the advantages of either extreme.
I'm not knocking documented unit tests. Certainly, without documented unit tests, the situation before was that examples were riddled with bugs. What I'm pointing out here is that DDOC examples have some advantages that unittests cannot currently use. A potential way to fix this may be marking a unit test as being a complete example program that assumes the user has installed proper access to the library. Then it won't compile unless you add the correct imports, and it's treated as if it were in a separate module (no private symbol access). This is probably the closest we can get to simulating a user copying an example unit test into his own file and trying to run it. -Steve
May 13 2016
parent Seb <seb wilzba.ch> writes:
On Friday, 13 May 2016 at 21:27:25 UTC, Steven Schveighoffer 
wrote:
 A potential way to fix this may be marking a unit test as being 
 a complete example program that assumes the user has installed 
 proper access to the library. Then it won't compile unless you 
 add the correct imports, and it's treated as if it were in a 
 separate module (no private symbol access). This is probably 
 the closest we can get to simulating a user copying an example 
 unit test into his own file and trying to run it.

 -Steve
Maybe I should have stated that I discovered that the in-text example is broken, because I was working on this new feature: https://github.com/dlang/dlang.org/pull/1297 (allow all unittests in Phobos to be run online via DPaste). So yep for this feature to work smoothly we either need to auto-detect what other imports (except for the current module) are needed, or force Phobos devs to explicitly import other modules in ddoced unittests. I would also prefer the latter. Anyways this example didn't run, because the import std.conv wasn't imported and the idea to put tests whenever possible from plaintext to D code is too ensure that are actually working ;-) Steve: I do understand you partially - I don't like assert that much either, because when it breaks you don't know why. That's why I would prefer if we would have something like assertEqual("a", "b") in Phobos and most of the tests. The rationale is that it breaks with a proper error message instead of just saying - hey you have some error there. We still could automatically replace this to writeln(a); // b, but I think having a proper equal should already be a lot better and people hopefully can then just put in a different value for 'b' to see the test fail. Here are some numbers, from in total 28761 assert's in Phobos - 15266 use "==" (53%) - 1451 use equal (5%) - 165 use approxEqual (0.6%) - 452 assert(0 - 5165 static assert
May 14 2016
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Friday, May 13, 2016 16:39:56 Steven Schveighoffer via Digitalmars-d wrote:
 Just looking at this PR: https://github.com/dlang/phobos/pull/4319

 Now the example, instead of running and producing output (i.e. visual
 feedback) that the program is doing something, just runs and creates no
 feedback.

 I'm wondering if we can have a mechanism for documented unit tests to
 have a slightly different showing inside the docs vs. the actual unit test.

 For example, let's say we have a function writelnAssert. Used like this:

 writelnAssert(someText, "Text You Expect To Output");

 When running this function, it's basically just an assert that someText
 == the expected text. However, when DDOC creates the document for this,
 it says:

 writeln(someText); // "Text You Expect To Output"

 This way, we are actually testing the output, but at the same time,
 giving someone playing with the example the tools to see some feedback.

 Thoughts?
I confess that I really don't have a problem with assertions being used in examples like they normally are. That's how unit tests work, and anyone who uses D for any length of time is going to find that out quickly. And it's not like you can see the result of writeln anyway unless you copy-pase the code into an editor to run it yourself, and if you do that, you can replace the assertion with a call to writeln easily enough. And the assertion can actually show you the output in the documentation, whereas you won't get that with writeln. So, if anything, it seems to me that having assertions that show you the values is better than having to copy-paste the code into an editor to run it and see the output. What's a bigger problem is when you want an example that does something like talk to a website (like std.net.curl does), since it's frequently not reasonable to have that in unit tests. Fortunately, it's still perfectly possible to have an untested example if you need it, and it's not usually needed. - Jonathan M Davis
May 13 2016
prev sibling next sibling parent Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 5/13/16, Steven Schveighoffer via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 I'm wondering if we can have a mechanism for documented unit tests to
 have a slightly different showing inside the docs vs. the actual unit test.

 For example, let's say we have a function writelnAssert. Used like this:

 writelnAssert(someText, "Text You Expect To Output");
Hmm.. I like the idea, but I have an alternate proposal. Add a new compiler switch (or better yet use a built-in version switch) which makes all assertions chatty. So for a unittest like this: ---- auto sum (a, b) => a + b; /// unittest { assert(sum(2, 2) == 4); } ---- Running it with something like this: ---- $ dmd -unittest -version=printasserts -run test.d ---- Would produce: ----
 test.d(4): assert(4 == 4); was successful
---- We could nitpick about the actual output, but the general idea is to avoid having to modify all the unittests by hand and instead use a compiler switch for chatty assertions. And actually if we used a -version switch then I think we could implement this entirely in Druntime. That is unless I remember incorrectly how assertions are emitted, I haven't looked at the code in a while.
May 14 2016
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, May 14, 2016 at 03:32:45PM +0200, Andrej Mitrovic via Digitalmars-d
wrote:
[...]
 Add a new compiler switch (or better yet use a built-in version
 switch) which makes all assertions chatty.
 
 So for a unittest like this:
 
 ----
 auto sum (a, b) => a + b;
 
 ///
 unittest { assert(sum(2, 2) == 4); }
 ----
 
 Running it with something like this:
 
 ----
 $ dmd -unittest -version=printasserts -run test.d
 ----
 
 Would produce:
 
 ----
 test.d(4): assert(4 == 4); was successful
---- We could nitpick about the actual output, but the general idea is to avoid having to modify all the unittests by hand and instead use a compiler switch for chatty assertions.
I like this idea. But wouldn't it be too verbose? I.e., there may be a lot of non-unittest-related assertions inside other functions and such. We could say only top-level asserts in a unittest block become chatty, but that excludes unittests calling unittest-only helper functions to do the checks.
 And actually if we used a -version switch then I think we could
 implement this entirely in Druntime. That is unless I remember
 incorrectly how assertions are emitted, I haven't looked at the code
 in a while.
How would the assert implementation in druntime access the expression being tested? I.e., in a way that it can be printed out in human-readable form. T -- Gone Chopin. Bach in a minuet.
May 14 2016
prev sibling parent reply Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 5/14/16, H. S. Teoh via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 I like this idea. But wouldn't it be too verbose?
Hmm good point. Alternatively we could perhaps make ddoc inject writeln calls, something like: ---- auto sum (a, b) => a + b; /// unittest { assert(sum(2, 2) == 4); } ---- Turns into ddoc output: ---- writeln(sum(2, 2)); assert(sum(2, 2) == 4); ----
 How would the assert implementation in druntime access the expression
 being tested? I.e., in a way that it can be printed out in
 human-readable form.
Ah I was remembering wrong here. The compiler turns the assert expression into something like "cast(bool)expression || __assert". So anyway, I think perhaps the simplest solution is to make ddoc inject the writeln calls (or possibly replace assertions altogether in the output).
May 14 2016
parent Ryan Frame <dlang ryanjframe.com> writes:
On Saturday, 14 May 2016 at 17:48:48 UTC, Andrej Mitrovic wrote:
 So anyway, I think perhaps the simplest solution is to make 
 ddoc inject the writeln calls (or possibly replace assertions 
 altogether in the output).
The problem with replacing the assert is the loss of information. assert(sum(2, 2) == 4); tells me what sum(2, 2) should return merely by reading the example, but writeln(sum(2, 2)); requires me to run it (or re-read the documentation for sum(), but the example is supposed to save me from that). Merely injecting the writeln() provides full return information when reading and executing the example (though more context should be provided than just the return value).
May 14 2016