www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.experimental.testing formal review

reply "Robert burner Schadek" <rburners gmail.com> writes:
This post marks the start of the two week review process of 
std.experimental.testing.

PR: https://github.com/D-Programming-Language/phobos/pull/3207
Dub: http://code.dlang.org/packages/unit-threaded
Doc: See CyberShadow/DAutoTest for up-to-date documentation build

Previous Thread: 
http://forum.dlang.org/post/uzocokshugchescbawlj forum.dlang.org
Sep 09 2015
next sibling parent reply "Brian Schott" <briancschott gmail.com> writes:
On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner 
Schadek wrote:

 This post marks the start of the two week review process of 
 std.experimental.testing.

 PR: https://github.com/D-Programming-Language/phobos/pull/3207
 Dub: http://code.dlang.org/packages/unit-threaded
 Doc: See CyberShadow/DAutoTest for up-to-date documentation 
 build

 Previous Thread: 
 http://forum.dlang.org/post/uzocokshugchescbawlj forum.dlang.org


Package-level documentation seems to be missing from the 
auto-generated documentation.

The gen_ut_main link on the side bar is also a 404.

std.experimental.testing.options.Options and 
std.experimental.testing.reflection.TestData fields have no DDoc, 
so they don't show up in the generated documentation.

Is there going to be a shouldEqual that's specialized for 
floating point, or should shouldBeTrue(approxEqual(...)) be used 
instead? (If so, this should be documented)

std.experimental.testing.testcase.TestCase.numTestsRun should be 
 property?
Sep 09 2015
next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 9 September 2015 at 18:54:30 UTC, Brian Schott 
wrote:

 On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner 
 Schadek wrote:

 This post marks the start of the two week review process of 
 std.experimental.testing.

 PR: https://github.com/D-Programming-Language/phobos/pull/3207
 Dub: http://code.dlang.org/packages/unit-threaded
 Doc: See CyberShadow/DAutoTest for up-to-date documentation 
 build

 Previous Thread: 
 http://forum.dlang.org/post/uzocokshugchescbawlj forum.dlang.org


 Package-level documentation seems to be missing from the 
 auto-generated documentation.


I think I generated the docs before there were any, let me go see.


 The gen_ut_main link on the side bar is also a 404.


Good catch, I'll take a look. I'm not looking forward to trying 
to recreate the site... it really should be easier.


 std.experimental.testing.options.Options and 
 std.experimental.testing.reflection.TestData fields have no 
 DDoc, so they don't show up in the generated documentation.


I'll add DDoc.



 Is there going to be a shouldEqual that's specialized for 
 floating point, or should shouldBeTrue(approxEqual(...)) be 
 used instead? (If so, this should be documented)


Good question.


 std.experimental.testing.testcase.TestCase.numTestsRun should 
 be  property?


Sure.


Atila
Sep 10 2015
prev sibling parent "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 9 September 2015 at 18:54:30 UTC, Brian Schott 
wrote:

 On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner 
 Schadek wrote:

 This post marks the start of the two week review process of 
 std.experimental.testing.

 PR: https://github.com/D-Programming-Language/phobos/pull/3207
 Dub: http://code.dlang.org/packages/unit-threaded
 Doc: See CyberShadow/DAutoTest for up-to-date documentation 
 build

 Previous Thread: 
 http://forum.dlang.org/post/uzocokshugchescbawlj forum.dlang.org


 Package-level documentation seems to be missing from the 
 auto-generated documentation.

 The gen_ut_main link on the side bar is also a 404.

 std.experimental.testing.options.Options and 
 std.experimental.testing.reflection.TestData fields have no 
 DDoc, so they don't show up in the generated documentation.



The autotester docs are... different from `make html`. I fixed 
it, but just in case here's my version of dlang.org with the docs:

http://atilaneves.github.io/phobos/


 Is there going to be a shouldEqual that's specialized for 
 floating point, or should shouldBeTrue(approxEqual(...)) be 
 used instead? (If so, this should be documented)


Implemented in the latest version and unit tested.


Atila
Sep 11 2015
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 09-Sep-2015 18:20, Robert burner Schadek wrote:

 This post marks the start of the two week review process of
 std.experimental.testing.

 PR: https://github.com/D-Programming-Language/phobos/pull/3207
 Dub: http://code.dlang.org/packages/unit-threaded
 Doc: See CyberShadow/DAutoTest for up-to-date documentation build

 Previous Thread:
 http://forum.dlang.org/post/uzocokshugchescbawlj forum.dlang.org


Where is the announce thread?

-- 
Dmitry Olshansky
Sep 10 2015
parent "wobbles" <grogan.colin gmail.com> writes:
On Thursday, 10 September 2015 at 14:03:31 UTC, Dmitry Olshansky 
wrote:

 On 09-Sep-2015 18:20, Robert burner Schadek wrote:

 This post marks the start of the two week review process of
 std.experimental.testing.

 PR: https://github.com/D-Programming-Language/phobos/pull/3207
 Dub: http://code.dlang.org/packages/unit-threaded
 Doc: See CyberShadow/DAutoTest for up-to-date documentation 
 build

 Previous Thread:
 http://forum.dlang.org/post/uzocokshugchescbawlj forum.dlang.org


 Where is the announce thread?


http://forum.dlang.org/post/tekjnfyvvmrozmqixtro forum.dlang.org
Sep 10 2015
prev sibling next sibling parent reply "NVolcz" <niklas.volcz gmail.com> writes:
On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner 
Schadek wrote:

 This post marks the start of the two week review process of 
 std.experimental.testing.
 </ snip>


Some questions from a Java programmer:
How would I go about making test fixtures. Ex. in JUnit you have 
 Before and  BeforeClass
Is it possible to categorize tests?
How about Fuzz-tests, randomize input for test on each run?
Time limited test? If this tests runs more than 5min then fail
Sep 11 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Friday, 11 September 2015 at 10:02:22 UTC, NVolcz wrote:

 On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner 
 Schadek wrote:

 This post marks the start of the two week review process of 
 std.experimental.testing.
 </ snip>


 Some questions from a Java programmer:
 How would I go about making test fixtures. Ex. in JUnit you 
 have  Before and  BeforeClass


The py.test way: write a function that creates the fixture, call 
it from the unit test.


 Is it possible to categorize tests?


D's module system does that already.


 How about Fuzz-tests, randomize input for test on each run?


Like QuickCheck? Robert has something for that.


 Time limited test? If this tests runs more than 5min then fail


Unit tests should run in a fraction of a second... no, there's no 
such functionality.

Atila
Sep 11 2015
next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 11 September 2015 at 11:27:59 UTC, Atila Neves wrote:

 On Friday, 11 September 2015 at 10:02:22 UTC, NVolcz wrote:

 Is it possible to categorize tests?


 D's module system does that already.


Along with things like version(unittestFeatureA) or 
version(unittestPrecision)
Sep 11 2015
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-09-11 13:27, Atila Neves wrote:


 Unit tests should run in a fraction of a second... no, there's no such
 functionality.


Why would I not use this for other kinds of tests?

-- 
/Jacob Carlborg
Sep 11 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Friday, 11 September 2015 at 15:54:57 UTC, Jacob Carlborg 
wrote:

 On 2015-09-11 13:27, Atila Neves wrote:


 Unit tests should run in a fraction of a second... no, there's 
 no such
 functionality.


 Why would I not use this for other kinds of tests?


I guess you're right. In any case, getting what's in there to get 
accepted is work enough as it is.

Atila
Sep 12 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-09-12 11:52, Atila Neves wrote:


 I guess you're right. In any case, getting what's in there to get
 accepted is work enough as it is.


Yes, absolutely. I wasn't arguing for the feature, just the reason not 
to add it :)

-- 
/Jacob Carlborg
Sep 12 2015
prev sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 09/11/2015 01:27 PM, Atila Neves wrote:

 How about Fuzz-tests, randomize input for test on each run?


 
 Like QuickCheck? Robert has something for that.


There is also https://github.com/MartinNowak/qcheck for that.
Sep 13 2015
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner 
Schadek wrote:

 This post marks the start of the two week review process of 
 std.experimental.testing.

 PR: https://github.com/D-Programming-Language/phobos/pull/3207
 Dub: http://code.dlang.org/packages/unit-threaded
 Doc: See CyberShadow/DAutoTest for up-to-date documentation 
 build

 Previous Thread: 
 http://forum.dlang.org/post/uzocokshugchescbawlj forum.dlang.org


My opinion hasn't changed since last review - it is a very solid 
and useful library but I don't want provided API to become 
standard (== part of Phobos).

I also don't like mixing unittest and higher level functional 
tests (with setup and cleanup phases) into the same buckets - 
this doesn't fit nice with D module system. Latter should be 
placed in a separate modules/package to avoid being picked up by 
rdmd & Co when compiled as dependency.

At the same time actual test running facilities (parallel, 
randomized, etc) look generally applicable and uncontroversial.
Sep 12 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-09-12 15:34, Dicebot wrote:


 I also don't like mixing unittest and higher level functional tests
 (with setup and cleanup phases) into the same buckets - this doesn't fit
 nice with D module system. Latter should be placed in a separate
 modules/package to avoid being picked up by rdmd & Co when compiled as
 dependency.


Not sure I understand the problem. Does this prevent one from writing 
functional tests in a completely separate directory?

-- 
/Jacob Carlborg
Sep 12 2015
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 12 September 2015 at 14:50:32 UTC, Jacob Carlborg 
wrote:

 Not sure I understand the problem. Does this prevent one from 
 writing functional tests in a completely separate directory?


Nope but the fact that they are treated the same (no separate 
category, no output separation, no documentation warning) 
encourages to do so - while in practice it is almost always a bad 
idea.
Sep 12 2015
prev sibling parent reply Dicebot <public dicebot.lv> writes:
On Saturday, 12 September 2015 at 14:50:32 UTC, Jacob Carlborg 
wrote:

 On 2015-09-12 15:34, Dicebot wrote:


 I also don't like mixing unittest and higher level functional 
 tests
 (with setup and cleanup phases) into the same buckets - this 
 doesn't fit
 nice with D module system. Latter should be placed in a 
 separate
 modules/package to avoid being picked up by rdmd & Co when 
 compiled as
 dependency.


 Not sure I understand the problem. Does this prevent one from 
 writing functional tests in a completely separate directory?


On related topic - there are 2 things that are currently missing 
in `TestCase` when applied for functional test purpose:

1) being able to mark test case as fatal (i.e. if internal 
handshake or sanity check fails there is no point in trying to 
run other tests)
2) being able to do weak ordering of tests (by defining strict 
sequence of groups so that parallelization/randomization only 
happens within such group) - I have used something as simple as 
numerical priority value so far for my needs
Sep 13 2015
next sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Sunday, 13 September 2015 at 09:59:18 UTC, Dicebot wrote:

 On Saturday, 12 September 2015 at 14:50:32 UTC, Jacob Carlborg 
 wrote:

 On 2015-09-12 15:34, Dicebot wrote:


 I also don't like mixing unittest and higher level functional 
 tests
 (with setup and cleanup phases) into the same buckets - this 
 doesn't fit
 nice with D module system. Latter should be placed in a 
 separate
 modules/package to avoid being picked up by rdmd & Co when 
 compiled as
 dependency.


 Not sure I understand the problem. Does this prevent one from 
 writing functional tests in a completely separate directory?


 On related topic - there are 2 things that are currently 
 missing in `TestCase` when applied for functional test purpose:

 1) being able to mark test case as fatal (i.e. if internal 
 handshake or sanity check fails there is no point in trying to 
 run other tests)


I've never heard of functionality like that, but should be easy 
to implement.


 2) being able to do weak ordering of tests (by defining strict 
 sequence of groups so that parallelization/randomization only 
 happens within such group) - I have used something as simple as 
 numerical priority value so far for my needs


There's ` singleThreaded` for that: all tests in a module with 
that UDA run in series (other modules are still run in parallel). 
I didn't think one was needed for random ordering.

Atila
Atila
Sep 13 2015
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-09-13 12:44, Atila Neves wrote:


 I've never heard of functionality like that, but should be easy to
 implement.


We're using that at work, but on a different level. We have two separate 
jobs in Jenkins, one depends on the other one. If the first one fails, 
the second one is not run.

-- 
/Jacob Carlborg
Sep 13 2015
prev sibling parent reply Dicebot <public dicebot.lv> writes:
On Sunday, 13 September 2015 at 10:44:30 UTC, Atila Neves wrote:

 2) being able to do weak ordering of tests (by defining strict 
 sequence of groups so that parallelization/randomization only 
 happens within such group) - I have used something as simple 
 as numerical priority value so far for my needs


 There's ` singleThreaded` for that: all tests in a module with 
 that UDA run in series (other modules are still run in 
 parallel). I didn't think one was needed for random ordering.

 Atila


I had inverse thing in mind - all tests within a module / block 
run in parallel, but blocks run sequentially. It is not a good 
feature for unit tests but quite important one to higher level 
ones which deal with nasty environment issues.
Sep 15 2015
parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 15 September 2015 at 08:27:29 UTC, Dicebot wrote:

 On Sunday, 13 September 2015 at 10:44:30 UTC, Atila Neves wrote:

 2) being able to do weak ordering of tests (by defining 
 strict sequence of groups so that 
 parallelization/randomization only happens within such group) 
 - I have used something as simple as numerical priority value 
 so far for my needs


 There's ` singleThreaded` for that: all tests in a module with 
 that UDA run in series (other modules are still run in 
 parallel). I didn't think one was needed for random ordering.

 Atila


 I had inverse thing in mind - all tests within a module / block 
 run in parallel, but blocks run sequentially. It is not a good 
 feature for unit tests but quite important one to higher level 
 ones which deal with nasty environment issues.


I'm not sure we're understanding each other. With the current
implementation and a module like this:

 singleThreaded
 name("serial1") unittest { ... }

 singleThreaded
 name("serial2") unittest { ... }

 name("parallel1") unittest {... }
 name("parallel2") unittest { ...}


3 tasks would get scheduled in parallel: parallel1, parallel2, 
and a composite task that does serial1 then serial2. If there are 
any other modules, all of the other tests run in parallel with 
these 3 tasks.

I'm proposing to extend the same behaviour to randomised running 
of tests, but if that's the case the name would change.

Atila
Sep 15 2015
prev sibling next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Sunday, 13 September 2015 at 09:59:18 UTC, Dicebot wrote:

 On Saturday, 12 September 2015 at 14:50:32 UTC, Jacob Carlborg 
 wrote:

 On 2015-09-12 15:34, Dicebot wrote:


 I also don't like mixing unittest and higher level functional 
 tests
 (with setup and cleanup phases) into the same buckets - this 
 doesn't fit
 nice with D module system. Latter should be placed in a 
 separate
 modules/package to avoid being picked up by rdmd & Co when 
 compiled as
 dependency.


 Not sure I understand the problem. Does this prevent one from 
 writing functional tests in a completely separate directory?


 On related topic - there are 2 things that are currently 
 missing in `TestCase` when applied for functional test purpose:

 1) being able to mark test case as fatal (i.e. if internal 
 handshake or sanity check fails there is no point in trying to 
 run other tests)


I'll see if I can add this before the end of the review phase.


 2) being able to do weak ordering of tests (by defining strict 
 sequence of groups so that parallelization/randomization only 
 happens within such group) - I have used something as simple as 
 numerical priority value so far for my needs


How about I change the name from  singleThreaded to  serial and 
make sure it works with randomised runs as well (it probably 
already does, I just have to check)?

Atila
Sep 15 2015
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Sunday, 13 September 2015 at 09:59:18 UTC, Dicebot wrote:

 1) being able to mark test case as fatal (i.e. if internal 
 handshake or sanity check fails there is no point in trying to 
 run other tests)


I'm leaning towards not including this now and concentrating on 
getting it
approved - a PR to change std.experimental.testing later will be 
much
easier to deal with than doing it now.


 2) being able to do weak ordering of tests (by defining strict 
 sequence of groups so that parallelization/randomization only 
 happens within such group) - I have used something as simple as 
 numerical priority value so far for my needs


I did the check for random runs and renamed  singleThreaded to 
 serial in the PR.

Atila
Sep 17 2015
parent Dicebot <public dicebot.lv> writes:
Sure, it isn't really important and does not impact my opinion 
anyway. Was simply sharing experience of writing similar purpose 
library.
Sep 17 2015
prev sibling next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner 
Schadek wrote:

 This post marks the start of the two week review process of 
 std.experimental.testing.

 PR: https://github.com/D-Programming-Language/phobos/pull/3207
 Dub: http://code.dlang.org/packages/unit-threaded
 Doc: See CyberShadow/DAutoTest for up-to-date documentation 
 build

 Previous Thread: 
 http://forum.dlang.org/post/uzocokshugchescbawlj forum.dlang.org


I've updated the PR with a proposal on the functionality to 
auto-generate the file containing the main function to run the 
unit tests.

Atila
Sep 15 2015
prev sibling next sibling parent reply Robert burner Schadek <rburners gmail.com> writes:
Review of std.experimental.testing formal review

the two weeks of the formal review phase are over.
The review thread was very shallow. Dicebot again expressed this 
disaffection with the assert function names 
"should(BeTrue|BeFalse|...)" No agreement could be found. 
Personally, I'm not sure if an agreement can be found, as this is 
more a personal style question rather then technical question.

Some minor comments still need to be addressed:
std.experimental.testing.reflection.TestData fields have no DDoc

I will start the voting thread a week from now.
Sep 28 2015
parent Atila Neves <atila.neves gmail.com> writes:
On Monday, 28 September 2015 at 10:03:14 UTC, Robert burner 
Schadek wrote:

 Review of std.experimental.testing formal review

 the two weeks of the formal review phase are over.
 The review thread was very shallow. Dicebot again expressed 
 this disaffection with the assert function names 
 "should(BeTrue|BeFalse|...)" No agreement could be found. 
 Personally, I'm not sure if an agreement can be found, as this 
 is more a personal style question rather then technical 
 question.

 Some minor comments still need to be addressed:
 std.experimental.testing.reflection.TestData fields have no DDoc

 I will start the voting thread a week from now.


Added the DDoc.

Atila
Sep 28 2015
prev sibling parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner 
Schadek wrote:

 This post marks the start of the two week review process of 
 std.experimental.testing.


Will `runTests` automatically assert that all pure unittests by 
default are parallellized and all non-pure are serialized? If so 
why is the  serial UDA needed?

Will it make use of D's builtin threadpool or will every unittest 
run in its own thread? IMHO, we should strive for threadpool 
usage here.
Sep 29 2015
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 29 September 2015 at 10:45:23 UTC, Per Nordlöw wrote:

 On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner 
 Schadek wrote:

 This post marks the start of the two week review process of 
 std.experimental.testing.


 Will `runTests` automatically assert that all pure unittests by 
 default are parallellized and all non-pure are serialized? If 
 so why is the  serial UDA needed?


runTests with no optional arguments will run the tests in threads.
There's nothing about purity enforcement there. In fact, I tried
using pure unit tests yesterday with std.experimental.testing
and couldn't. The compiler inferred the functions to not be pure.
I tried adding pure to them and descended into a madness of adding
pure all over phobos until I got fed up. It seems to be something
to do with `format` not being pure, I have no idea why. It really
should be.

 serial is needed to _not_ run tests in threads, which you'd need
if those tests call functions that mutate global state. Even
"thread-global" is bad here since those tests could run in the 
same
or different threads depending on the run.


 Will it make use of D's builtin threadpool or will every 
 unittest run in its own thread? IMHO, we should strive for 
 threadpool usage here.


It uses the thread pool.

Atila
Sep 29 2015
parent reply Meta <jared771 gmail.com> writes:
On Tuesday, 29 September 2015 at 14:34:42 UTC, Atila Neves wrote:

 runTests with no optional arguments will run the tests in 
 threads.
 There's nothing about purity enforcement there. In fact, I tried
 using pure unit tests yesterday with std.experimental.testing
 and couldn't. The compiler inferred the functions to not be 
 pure.
 I tried adding pure to them and descended into a madness of 
 adding pure all over phobos until I got fed up. It seems to be 
 something to do with `format` not being pure, I have no idea 
 why. It really should be.


There are various unit tests in Phobos that are annotated with 
 pure,  nogc, nothrow, etc. to test that those attributes will be 
inferred by the compiler. In light of that, this could pose a 
problem.
Sep 29 2015
parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 29 September 2015 at 19:15:13 UTC, Meta wrote:

 On Tuesday, 29 September 2015 at 14:34:42 UTC, Atila Neves 
 wrote:

 runTests with no optional arguments will run the tests in 
 threads.
 There's nothing about purity enforcement there. In fact, I 
 tried
 using pure unit tests yesterday with std.experimental.testing
 and couldn't. The compiler inferred the functions to not be 
 pure.
 I tried adding pure to them and descended into a madness of 
 adding pure all over phobos until I got fed up. It seems to be 
 something to do with `format` not being pure, I have no idea 
 why. It really should be.


 There are various unit tests in Phobos that are annotated with 
  pure,  nogc, nothrow, etc. to test that those attributes will 
 be inferred by the compiler. In light of that, this could pose 
 a problem.


Which is why I tested with a pure unittest. The problem is that 
`text` and `to` from `std.conv` aren't pure. That can and should 
be fixed, but unless I write my own pure versions, it's got 
nothing to do with this proposal.

Atila
Sep 30 2015