www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Quit running foreign unittests >_<

reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
This is getting to be (or rather, *continuing* to be) a royal PITA:

https://github.com/rejectedsoftware/vibe.d/issues/673

I don't mean to pick on Vibe.d in particular, but can we have a solution 
(that doesn't involve obscure corners of druntime, or demanding everyone 
use the same build system) for running our unittests *without* 
triggering a cascading avalanche of unittests from ALL third party libs 
that don't cooperate with the [frankly] quite clumsy 
version(mylib_unittests) hack?!
Sep 09 2014
next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 09 Sep 2014 22:13:57 -0400
Nick Sabalausky via Digitalmars-d <digitalmars-d puremagic.com> wrote:

write build tool that is capable of doing that...
Sep 09 2014
parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 9/9/2014 10:21 PM, ketmar via Digitalmars-d wrote:
 On Tue, 09 Sep 2014 22:13:57 -0400
 Nick Sabalausky via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 write build tool that is capable of doing that...
??? If you think that's reasonable when it's the COMPILER that inserts all those calls (or druntime, whatever), then go ahead with it yourself. Seriously, DMD needs a "--omit-unittest=some.package.*". I'd try a PR myself (I'd have *already* tried), but last time I brought it up the whole idea was completely shot down in favor of both the version(mylib_unittests) hack (which people don't fucking use half the time, sometimes deliberately) and some goofy druntime trickery (ie, writing your own test runner and getting it to druntime before druntime gets to the "run unittests" phase. Seriously? That's *preferable* to a "--omit-unittest=some.package.*"??).
Sep 09 2014
next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 9/9/2014 10:37 PM, Nick Sabalausky wrote:
 On 9/9/2014 10:21 PM, ketmar via Digitalmars-d wrote:
 On Tue, 09 Sep 2014 22:13:57 -0400
 Nick Sabalausky via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 write build tool that is capable of doing that...
??? If you think that's reasonable when it's the COMPILER that inserts all those calls (or druntime, whatever), then go ahead with it yourself.
Well, ok, I guess it could be done with separate invocations of DMD, but that has non-trivial implications for build performance, and templates probably could still fuck it all up anyway.
Sep 09 2014
next sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 10 September 2014 at 03:00:48 UTC, Nick Sabalausky 
wrote:
 separate invocations […] and templates probably could still 
 fuck it all up anyway.
If so, then this is either a critical bug in DMD, or an issue with your code (e.g. due to version(unittest) use). I'm aware of the issues that affect some ways of doing incremental compilation, but that's mostly a different story. David
Sep 09 2014
parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 9/9/2014 11:07 PM, David Nadlinger wrote:
 On Wednesday, 10 September 2014 at 03:00:48 UTC, Nick Sabalausky wrote:
 separate invocations […] and templates probably could still fuck it
 all up anyway.
If so, then this is either a critical bug in DMD, or an issue with your code (e.g. due to version(unittest) use). I'm aware of the issues that affect some ways of doing incremental compilation, but that's mostly a different story. David
I was thinking about something like this: //a.d import b.d; alias x = Foo!whatever; //b.d struct Foo(Arg) { void foo() {...} unittest { /+...unittest foo()...+/ } } // Attempt to exclude module b's unittests, except...aren't // they compiled when module a's is compiled? The first // dmd invocation here has no idea module b will be compiled // without the -unittest flag.
dmd -c -unittest a.d
dmd -c b.d   // Duplicate a whole buttload of processing here
dmd a.o b.o
Sep 09 2014
prev sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 09 Sep 2014 23:00:06 -0400
Nick Sabalausky via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 Well, ok, I guess it could be done with separate invocations of DMD,
 but that has non-trivial implications for build performance
we can cache analyzed ASTs somewhere. this can speedup separate compilations a little. but i don't see any problems with separate compilation: it's fast enough even with alot of modules.
Sep 10 2014
prev sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 09 Sep 2014 22:37:23 -0400
Nick Sabalausky via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 If you think that's reasonable when it's the COMPILER that inserts
 all those calls (or druntime, whatever), then go ahead with it
 yourself.
sure it's a compiler. that's why we have "-unittest" arg. and separate compilation rocks (why, it's cheap! and windows... ah, i don't care). yes, templates still sux (considering built-in unittests). this is where compiler should be fixed: it should compile unittest invocations only for templates that comes from the module it compiles, i think.
Sep 10 2014
parent "Kagamin" <spam here.lot> writes:
On Wednesday, 10 September 2014 at 08:05:28 UTC, ketmar via 
Digitalmars-d wrote:
 yes, templates still sux (considering built-in unittests). this 
 is
 where compiler should be fixed: it should compile unittest 
 invocations
 only for templates that comes from the module it compiles, i 
 think.
That way it will be clear. By the way, will this link? --- template A(T) { int a; unittest{ f(); } } version(unittest) void f(){} --- If this module is compiled without unittests, `f` function will not be compiled, then if another module is compiled with unittests and instantiates the template, the template's unittest is compiled and it will link with `f` function, which wasn't compiled, so linking will fail?
Sep 10 2014
prev sibling next sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 10 September 2014 at 02:14:39 UTC, Nick Sabalausky 
wrote:
 This is getting to be (or rather, *continuing* to be) a royal 
 PITA:

 https://github.com/rejectedsoftware/vibe.d/issues/673

 I don't mean to pick on Vibe.d in particular, but can we have a 
 solution (that doesn't involve obscure corners of druntime, or 
 demanding everyone use the same build system) for running our 
 unittests *without* triggering a cascading avalanche of 
 unittests from ALL third party libs that don't cooperate with 
 the [frankly] quite clumsy version(mylib_unittests) hack?!
A while ago, I though about adding an optional module/package whitelist as a compiler flag, something like -unittest=myapp,mylib.important. Never really got to implementing it, or even thinking about the implications in any details. David
Sep 09 2014
next sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 10 September 2014 at 02:29:41 UTC, David Nadlinger 
wrote:
 A while ago, I though about adding an optional module/package 
 whitelist as a compiler flag, something like 
 -unittest=myapp,mylib.important. Never really got to 
 implementing it, or even thinking about the implications in any 
 details.
s/compiler/rdmd/
Sep 09 2014
prev sibling parent reply "Kagamin" <spam here.lot> writes:
I think, libraries should support compilation without unittests, 
i.e. write unittests outside of templates. But then one may want 
to test every template instantiation, if it makes sense and 
intended.
Sep 09 2014
parent reply "Kagamin" <spam here.lot> writes:
It's questionable that testing every template instantiation is 
intended: you can't do the same for templated functions.
Sep 09 2014
next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 9/10/2014 1:01 AM, Kagamin wrote:
 It's questionable that testing every template instantiation is intended:
 you can't do the same for templated functions.
Regardless of the (de)merits of deliberately putting unittests in a template, the important thing is this: Whether a person intentionally *or inadvertently* winds up with a unittest inside a template, the whole "causes some of my unittests run when my lib's *users* run *their* project's unittests" is a highly unintuitive non-localized consequence.
Sep 10 2014
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 09/10/2014 07:01 AM, Kagamin wrote:
 It's questionable that testing every template instantiation is intended:
 you can't do the same for templated functions.
Yes you can.
Sep 10 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 10 September 2014 at 02:14:39 UTC, Nick Sabalausky 
wrote:
 This is getting to be (or rather, *continuing* to be) a royal 
 PITA:

 https://github.com/rejectedsoftware/vibe.d/issues/673

 I don't mean to pick on Vibe.d in particular, but can we have a 
 solution (that doesn't involve obscure corners of druntime, or 
 demanding everyone use the same build system) for running our 
 unittests *without* triggering a cascading avalanche of 
 unittests from ALL third party libs that don't cooperate with 
 the [frankly] quite clumsy version(mylib_unittests) hack?!
My opinion stays the same - it is good reliable default and should stay that way. And yes, I absolutely hate when libraries do version(mylib_unittest) hacks forcing me to bother with millions of versions to enable them back. Simple solutions is to provide alternative unit test runner module as part of Phobos which does the module/package filtering.
Sep 10 2014
parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 9/10/2014 3:57 AM, Dicebot wrote:
 My opinion stays the same - it is good reliable default and should stay
 that way. And yes, I absolutely hate when libraries do
 version(mylib_unittest) hacks forcing me to bother with millions of
 versions to enable them back.
Being "default" is one thing, I really don't care what the default is - but for now it's not just "the default", it's the only realistic option without getting into PITA workarounds. Some people like including 3rd party library unittests with their own (apparently), some people like *not* including them. So we have some projects using version(foo_unittest) and other projects not using it. The current result is: *Nobody* can easily get what they want: Not people who like it my way, and not people who like it your way. Not easily enough anyway. So it's a global ecosystem problem. It needs fixed. (And to start with, general agreeance that it should be fixed.)
 Simple solutions is to provide alternative unit test runner module as
 part of Phobos which does the module/package filtering.
It needs to be *crystal* clear and trivially simple: A. Knowing that it exists. B. Sussing out how to use it. C. Actually using it. Otherwise, version(foo_unittest) will continue to linger out of an individual's convenience, and so once again, not everyone gets to have things their way. A related concern I have about a Phobos-only solution: When/where do you configure it? Obviously it can't happen anywhere in the scope of main. Are static ctors sufficient? I would *think* so, but I'm not certain.
Sep 10 2014
parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 10 September 2014 at 08:21:34 UTC, Nick Sabalausky 
wrote:
 Simple solutions is to provide alternative unit test runner 
 module as
 part of Phobos which does the module/package filtering.
It needs to be *crystal* clear and trivially simple: A. Knowing that it exists. B. Sussing out how to use it. C. Actually using it. Otherwise, version(foo_unittest) will continue to linger out of an individual's convenience, and so once again, not everyone gets to have things their way. A related concern I have about a Phobos-only solution: When/where do you configure it? Obviously it can't happen anywhere in the scope of main. Are static ctors sufficient? I would *think* so, but I'm not certain.
Leandro Lucarella has written similar solution for our internal testing purposes. It provides a test runner that is configured by CLI arguments and allows filtering of modules and non-fatal test failures. It is for D1/tango but I am pretty sure it can be ported to D2 runtime with close to 0 changes. I have not managed to convince him it is worth making public though :(
Sep 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 09 Sep 2014 22:13:57 -0400, Nick Sabalausky  
<SeeWebsiteToContactMe semitwist.com> wrote:

 This is getting to be (or rather, *continuing* to be) a royal PITA:

 https://github.com/rejectedsoftware/vibe.d/issues/673

 I don't mean to pick on Vibe.d in particular, but can we have a solution  
 (that doesn't involve obscure corners of druntime, or demanding everyone  
 use the same build system) for running our unittests *without*  
 triggering a cascading avalanche of unittests from ALL third party libs  
 that don't cooperate with the [frankly] quite clumsy  
 version(mylib_unittests) hack?!
Hm... would it be acceptable if unit tests were compiled, but not run? Because the unit tester runtime just foreach's over every module and runs the tests contained in that module. Now, I don't know how it works if you compile a template unit test -- does it get deposited into the module that instantiated or into the module that defined it? Worth looking into I think. -Steve
Sep 16 2014
prev sibling next sibling parent reply "Martin Nowak" <code dawg.eu> writes:
A fairly simple solution to this would be to compile unittests 
only for root modules (the ones that are part of the 
compilation), not for imported modules.
Then everyone can decide for which modules to use -unittest.
Apr 27 2015
parent reply "Martin Nowak" <code dawg.eu> writes:
https://issues.dlang.org/show_bug.cgi?id=13454
Apr 27 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 27 April 2015 at 07:28:28 UTC, Martin Nowak wrote:
 https://issues.dlang.org/show_bug.cgi?id=13454
Great, lets compromise both unittest correctness and convienience at the same time. Can we please just close the issue as `RESOLVED WONTFIX`? Compiling tests of dependencies pretty much never causes any notable slowdown. Trying to be smart about it does harm the users and contributors of the library though.
Apr 27 2015
parent reply "Kagamin" <spam here.lot> writes:
On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:
 Compiling tests of dependencies pretty much never causes any 
 notable slowdown.
This thread doesn't support that view, see the first post.
Apr 27 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote:
 On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:
 Compiling tests of dependencies pretty much never causes any 
 notable slowdown.
This thread doesn't support that view, see the first post.
Which part exactly? I only see comparisons for compiling AND running tests for dependencies. And it is usually running which causes the slowdown.
Apr 27 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/27/15 6:20 AM, Dicebot wrote:
 On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote:
 On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:
 Compiling tests of dependencies pretty much never causes any notable
 slowdown.
This thread doesn't support that view, see the first post.
Which part exactly? I only see comparisons for compiling AND running tests for dependencies. And it is usually running which causes the slowdown.
The problem is as follows: 1. Unit tests for some library are written for that library. They are written to run tests during unit tests of that library only (possibly with certain requirements of environment, including build lines, or expectations of system resource availability). 2. People who import that library's modules are not trying to test the library, they are trying to test their code. 3. The library runs its unit tests for templates in this case, not the other unit tests in the module (those would be run in that module's unit tests, which means you'd have to recompile the library to run unit tests). Most often, unit tests in templates are only expected to be defined by the library. I think ketmar had the most reasonable request here -- only compile/run unit tests in templates for the module that defines the template. If you want to create a unit test that runs for your specialized template version, there should be a way to do that, but it should be opt-in. I envision something like this: unittest { import somelib.somemod : SomeModTemplate, unittest SomeModTemplate!MyLocalType t; // runs templated unit tests } If you look in RedBlackTree, I use templated unit tests to great effect (found about 4-5 compiler bugs that way). But I have a lot of machinery around the unit tests to try and only test unit tests on integral types. It would be nice to not have to write all that shit. And BTW, I didn't even get it right, I forgot about other parameters to the RBTree, necessitating a PR later. -Steve
Apr 27 2015
next sibling parent reply "Ivan Kazmenko" <gassa mail.ru> writes:
On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer 
wrote:
 The problem is as follows:

 1. Unit tests for some library are written for that library. 
 They are written to run tests during unit tests of that library 
 only (possibly with certain requirements of environment, 
 including build lines, or expectations of system resource 
 availability).
By the way, a unittest-related issue still stands in DMD 2.067.1 for RedBlackTree: https://issues.dlang.org/show_bug.cgi?id=12246 A similar matter got resolved quickly for BinaryHeap: https://issues.dlang.org/show_bug.cgi?id=12245 BinaryHeap's case was handled by putting the container's unittests into "debug(BinaryHeap)". RedBlackTree's case is controlled via "debug(RBDoChecks)" (formerly "version(RBDoChecks)"). The difference is the presence of a "version(unittest) debug = RBDoChecks;" line. This looks inconsistent. For RedBlackTree, compile the following with or without -unittest option and run for a visible difference in speed (runs momentarily or for a few seconds): ----- import std.container; void main() { auto t = redBlackTree!int; foreach (i; 0..3000) t.insert(i); } ----- Ivan Kazmenko.
Apr 27 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/27/15 10:30 AM, Ivan Kazmenko wrote:
 On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer wrote:
 The problem is as follows:

 1. Unit tests for some library are written for that library. They are
 written to run tests during unit tests of that library only (possibly
 with certain requirements of environment, including build lines, or
 expectations of system resource availability).
By the way, a unittest-related issue still stands in DMD 2.067.1 for RedBlackTree: https://issues.dlang.org/show_bug.cgi?id=12246 A similar matter got resolved quickly for BinaryHeap: https://issues.dlang.org/show_bug.cgi?id=12245 BinaryHeap's case was handled by putting the container's unittests into "debug(BinaryHeap)". RedBlackTree's case is controlled via "debug(RBDoChecks)" (formerly "version(RBDoChecks)"). The difference is the presence of a "version(unittest) debug = RBDoChecks;" line. This looks inconsistent. For RedBlackTree, compile the following with or without -unittest option and run for a visible difference in speed (runs momentarily or for a few seconds): ----- import std.container; void main() { auto t = redBlackTree!int; foreach (i; 0..3000) t.insert(i); } ----- Ivan Kazmenko.
It's an anecdotal fix. I remember arguing over the change to debug, that was done for purity (pure functions need debug mode to print out something, which rbdochecks will do if there is an issue), but I can't find the conversation. But someone else will complain that when they try to debug their code, adding -debug to the command line debugs RBTree's algorithm (similarly to how they complained BinaryHeap was doing this). This really should only EVER run during phobos unit tests. I don't know how to fix this properly without something like I outlined above, or without doing some global version(PhobosUnitTests) hack. In fact, I don't agree with the BinaryHeap change. At this point, phobos unit tests are NOT testing the binary heap structure. Sure that makes user code run faster, but at the cost of never testing it even when it should be tested. -Steve
Apr 27 2015
parent reply "Ivan Kazmenko" <gassa mail.ru> writes:
On Monday, 27 April 2015 at 15:29:05 UTC, Steven Schveighoffer 
wrote:
 On 4/27/15 10:30 AM, Ivan Kazmenko wrote:
 On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer 
 wrote:
 The problem is as follows:

 1. Unit tests for some library are written for that library. 
 They are
 written to run tests during unit tests of that library only 
 (possibly
 with certain requirements of environment, including build 
 lines, or
 expectations of system resource availability).
By the way, a unittest-related issue still stands in DMD 2.067.1 for RedBlackTree: https://issues.dlang.org/show_bug.cgi?id=12246 A similar matter got resolved quickly for BinaryHeap: https://issues.dlang.org/show_bug.cgi?id=12245 BinaryHeap's case was handled by putting the container's unittests into "debug(BinaryHeap)". RedBlackTree's case is controlled via "debug(RBDoChecks)" (formerly "version(RBDoChecks)"). The difference is the presence of a "version(unittest) debug = RBDoChecks;" line. This looks inconsistent. For RedBlackTree, compile the following with or without -unittest option and run for a visible difference in speed (runs momentarily or for a few seconds): ----- import std.container; void main() { auto t = redBlackTree!int; foreach (i; 0..3000) t.insert(i); } ----- Ivan Kazmenko.
It's an anecdotal fix. I remember arguing over the change to debug, that was done for purity (pure functions need debug mode to print out something, which rbdochecks will do if there is an issue), but I can't find the conversation. But someone else will complain that when they try to debug their code, adding -debug to the command line debugs RBTree's algorithm (similarly to how they complained BinaryHeap was doing this).
I am doing just that, right now =) . But it's -unittest, not -debug, which triggers RBTree's expensive checks.
 This really should only EVER run during phobos unit tests.
A choice would be even better. When a user's code is wrong, for example, a comparison function does not define a comparison in mathematical sense, the user might benefit from container's sanity checks. From personal experience, when I implement my own sorted container and some complex logic using it, I do benefit from the container's checks sometimes.
 I don't know how to fix this properly without something like I 
 outlined above, or without doing some global 
 version(PhobosUnitTests) hack.

 In fact, I don't agree with the BinaryHeap change. At this 
 point, phobos unit tests are NOT testing the binary heap 
 structure. Sure that makes user code run faster, but at the 
 cost of never testing it even when it should be tested.

 -Steve
Right now, it is possible to test binary heap integrity with an explicit "-debug=BinaryHeap", and it is not tested by default. And it is mandatory to test red-black tree integrity when running any unittests involving a red-black tree, as it is tested by default. So, from a user's point of view, I like the binary heap situation better, since it gives me a choice with a reasonable default. How Phobos' unittests should be handled is another matter which, in my opinion, should not take away choices, and reasonable defaults too, from the user. To me, the proposed version(PhobosUnitTests) hack looks good for the particular case. But I don't see whether such trick scales well for the whole ecosystem of libraries. Ivan Kazmenko.
Apr 27 2015
next sibling parent "Ivan Kazmenko" <gassa mail.ru> writes:
 To me, the proposed version(PhobosUnitTests) hack looks good 
 for the particular case.  But I don't see whether such trick 
 scales well for the whole ecosystem of libraries.
Another approach that might work specifically for containers is to pass as an additional template parameter whether the particular RedBlackTree must continuously check its integrity or not. That way, the parameter can be explicitly set to "doCheck" from the unittest instantiations, default to "doNotCheck" for other code, and be enabled on a case-by-case basis when debugging. Again, I don't see whether it fits the larger picture, but still think it's worth to consider. Ivan Kazmenko.
Apr 27 2015
prev sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/27/15 12:00 PM, Ivan Kazmenko wrote:

 Right now, it is possible to test binary heap integrity with an explicit
 "-debug=BinaryHeap", and it is not tested by default.
Right, but phobos unit tests are not doing that. If we added a version(unittest) debug = BinaryHeap; to fix that problem there, then you would still be complaining ;) -Steve
Apr 27 2015
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer 
wrote:
 On 4/27/15 6:20 AM, Dicebot wrote:
 On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote:
 On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:
 Compiling tests of dependencies pretty much never causes any 
 notable
 slowdown.
This thread doesn't support that view, see the first post.
Which part exactly? I only see comparisons for compiling AND running tests for dependencies. And it is usually running which causes the slowdown.
The problem is as follows: 1. Unit tests for some library are written for that library. They are written to run tests during unit tests of that library only (possibly with certain requirements of environment, including build lines, or expectations of system resource availability). 2. People who import that library's modules are not trying to test the library, they are trying to test their code.
Those are two points I fundamentally disagree with. It doesn't matter where the code comes from - in the end only thing that matters is correctness of your application as a whole. And considering tests are not necessarily pure the results may very well differ between running those tests spearately and as part of application test suite a whole. Unless compiling some specific tests causes some proven _compilation_ slowdown (I have yet to see that) those all must be compiled and filtered by runtime test runner optionally. And if tests are written in a weird way that they can only be ran within that library test step, those are not really unittests. Usage of version(MyLibTests) in Nick SDL library annoyed me so much that I forked it to never deal with those pesky versions again. Don't want to do that with Phobos too.
Apr 28 2015
next sibling parent reply "Ivan Kazmenko" <gassa mail.ru> writes:
On Tuesday, 28 April 2015 at 16:40:05 UTC, Dicebot wrote:
 On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer 
 wrote:
 On 4/27/15 6:20 AM, Dicebot wrote:
 On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote:
 On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:
 Compiling tests of dependencies pretty much never causes 
 any notable
 slowdown.
This thread doesn't support that view, see the first post.
Which part exactly? I only see comparisons for compiling AND running tests for dependencies. And it is usually running which causes the slowdown.
The problem is as follows: 1. Unit tests for some library are written for that library. They are written to run tests during unit tests of that library only (possibly with certain requirements of environment, including build lines, or expectations of system resource availability). 2. People who import that library's modules are not trying to test the library, they are trying to test their code.
Those are two points I fundamentally disagree with. It doesn't matter where the code comes from - in the end only thing that matters is correctness of your application as a whole. And considering tests are not necessarily pure the results may very well differ between running those tests spearately and as part of application test suite a whole. Unless compiling some specific tests causes some proven _compilation_ slowdown (I have yet to see that) those all must be compiled and filtered by runtime test runner optionally. And if tests are written in a weird way that they can only be ran within that library test step, those are not really unittests. Usage of version(MyLibTests) in Nick SDL library annoyed me so much that I forked it to never deal with those pesky versions again. Don't want to do that with Phobos too.
Then how do you propose to approach the containers problem? On one hand, a unittest on containers themselves involves testing the container for integrity after each operation. On the other hand, a unittest on another module may involve heavy use of containers (say, N operations with a container), and if integrity checks are enabled at this time, it totals to N^2 trivial operations which may not be feasible. Ivan Kazmenko.
Apr 28 2015
parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 28 April 2015 at 20:57:01 UTC, Ivan Kazmenko wrote:
 Then how do you propose to approach the containers problem?

 On one hand, a unittest on containers themselves involves 
 testing the container for integrity after each operation.

 On the other hand, a unittest on another module may involve 
 heavy use of containers (say, N operations with a container), 
 and if integrity checks are enabled at this time, it totals to 
 N^2 trivial operations which may not be feasible.
If it is that slow I tend to put such tests outside of the tested symbol so that it won't be repeated over and over again for templates. There is still full access to private symbols so it is always possible. Though that is very exceptional case reserved only for most critical cases (everything below 30-60 seconds total is ok to me)
Apr 28 2015
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/28/15 12:40 PM, Dicebot wrote:
 On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer wrote:
 On 4/27/15 6:20 AM, Dicebot wrote:
 On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote:
 On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:
 Compiling tests of dependencies pretty much never causes any notable
 slowdown.
This thread doesn't support that view, see the first post.
Which part exactly? I only see comparisons for compiling AND running tests for dependencies. And it is usually running which causes the slowdown.
The problem is as follows: 1. Unit tests for some library are written for that library. They are written to run tests during unit tests of that library only (possibly with certain requirements of environment, including build lines, or expectations of system resource availability). 2. People who import that library's modules are not trying to test the library, they are trying to test their code.
Those are two points I fundamentally disagree with.
I think by default, nobody wants to test already-tested code in their application. That's not the point of unit tests. For example, if there's a module that has: struct S(T) { unittest {...} } unittest { S!int; // run unit tests for int } Then I can assume that unit test was run and passed. If I want to test it to be sure, I'll run that library's unit tests! But I don't want my unit test that uses S!int to also test S's unit tests for S!int. That doesn't make sense. I'm wasting cycles on something that is already proven. Now, if I want to run unit tests for S!MyCustomType, that makes sense. The library didn't test that. Which is why there should be a way to do it (if it's valid!). Right now, there isn't a way to control what runs and what doesn't. I don't care where it's decided or how it's decided, but somehow I should be able to NOT run S!int tests again.
 Unless compiling some specific tests causes some proven _compilation_
 slowdown (I have yet to see that) those all must be compiled and
 filtered by runtime test runner optionally.
For example, RedBlackTree when running unit tests does a sanity check of the tree for every operation. If you then use a RedBlackTree in your unit tests, you are doing that again. Maybe it's worth it to you, but it can increase the runtime of your test by orders of magnitude. Last time I checked, performance was high on people's priority lists. Or, potentially you could be running INVALID TESTS, because the tests weren't written for your specific usages. I ran into this when others complained that they couldn't test their code that uses RedBlackTree!string, because all the unit tests instantiated with int literals. This is simply a "does not work" thing. You can't turn them off, and you can't test your code. Is it worth it to you for a library to try to compile tests that prevent your test build from happening? Do you enjoy waiting on others to fix their problems so you can test your code? This is a very less than ideal situation. And it's not something the compiler tells you will happen.
 And if tests are written in a weird way that they can only be ran within
 that library test step, those are not really unittests.
This means you should never ever write unit tests into a template. Because it's impossible to create tests that will work for every single instantiation. See RedBlackTree for all the crud you have to put in for this to be viable today. I'd love to get rid of all that. In which case, let's disallow that capability. I'm fine with that too. This simply means your "all encompassing tests" will not be all encompassing any more, they will only test a select few instantiations, and you can't control that either from your application code. I want a way to control it as a template designer and as a template instantiator. Flexibility is king. -Steve
Apr 28 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 28 April 2015 at 21:28:09 UTC, Steven Schveighoffer 
wrote:
 I think by default, nobody wants to test already-tested code in 
 their application. That's not the point of unit tests.
How many more times should I repeat that I am exactly that nobody? I do want do test everything as part of my app tests, including all possible dependencies, transitively. This is awesome default. With a simple `rdmd -main -unittest` call I can ensure that certain app/module works correctly without having to trust maintainers of dependencies to run tests regularly and without even knowing what those dependencies are. It is beautiful in its simplicity which makes it good default.
 struct S(T)
 {
   unittest {...}
 }

 unittest
 {
    S!int; // run unit tests for int
 }
If this a very slow test (and there are expected many S) simply put the test blocks out of the aggregate.
 Now, if I want to run unit tests for S!MyCustomType, that makes 
 sense. The library didn't test that. Which is why there should 
 be a way to do it (if it's valid!).
There is something fundamentally broken with a template that needs to be tested for each new user type argument. Built-in tests must cover all possible type classes or they are incomplete.
 Unless compiling some specific tests causes some proven 
 _compilation_
 slowdown (I have yet to see that) those all must be compiled 
 and
 filtered by runtime test runner optionally.
For example, RedBlackTree when running unit tests does a sanity check of the tree for every operation. If you then use a RedBlackTree in your unit tests, you are doing that again. Maybe it's worth it to you, but it can increase the runtime of your test by orders of magnitude. Last time I checked, performance was high on people's priority lists.
It must be very slow sanity checks :X Sounds weird but I will accept it as given. Move the tests out of the RBL definition then. Also performance has nothing in common with test performance. I can't comment about rest because from the very beginning you seem to make statements about testing in general which do not match my vision at all.
Apr 28 2015
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/28/15 7:04 PM, Dicebot wrote:
 On Tuesday, 28 April 2015 at 21:28:09 UTC, Steven Schveighoffer wrote:
 I think by default, nobody wants to test already-tested code in their
 application. That's not the point of unit tests.
How many more times should I repeat that I am exactly that nobody?
OK, sorry. nodbody-1 :)
 I do want do test everything as part of my app tests, including all
 possible dependencies, transitively. This is awesome default. With a
 simple `rdmd -main -unittest` call I can ensure that certain app/module
 works correctly without having to trust maintainers of dependencies to
 run tests regularly and without even knowing what those dependencies
 are. It is beautiful in its simplicity which makes it good default.
or rdmd -main -unittest -> fail to build because the templated unit test doesn't work on your code. Good luck with that. Again, there are so many reasons I should not have to worry about unit tests in my library being run with your code. That's on you. I didn't write it for your code to run, if you want to run it, run my unit test script. -Steve
Apr 28 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 29 April 2015 at 04:53:47 UTC, Steven Schveighoffer 
wrote:
 or rdmd -main -unittest -> fail to build because the templated 
 unit test doesn't work on your code. Good luck with that.
I will create an upstream PR to fix it, problem solved. Have never had a need to do so though, not even a single time. Also : can you please point me again what part of RBT causes compilation slowdowns with version(unittest)? I looked through and found only runtime checks. And for that "move out of the aggregate" + "runtime test filtering" does what you want.
 Again, there are so many reasons I should not have to worry 
 about unit tests in my library being run with your code. That's 
 on you. I didn't write it for your code to run, if you want to 
 run it, run my unit test script.
If you don't wan't to run it, filter it out in the test runner. I assure you, there are at least as much reasons why I shouldn't worry if you actually run tests for your library and how those need to be run. Both defaults can be circumvented.
Apr 29 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/29/15 7:43 AM, Dicebot wrote:
 On Wednesday, 29 April 2015 at 04:53:47 UTC, Steven Schveighoffer wrote:
 or rdmd -main -unittest -> fail to build because the templated unit
 test doesn't work on your code. Good luck with that.
I will create an upstream PR to fix it, problem solved. Have never had a need to do so though, not even a single time. Also : can you please point me again what part of RBT causes compilation slowdowns with version(unittest)? I looked through and found only runtime checks.
It's runtime checks that slow down, the unit test compilation is not slow (although it was at one point, but that was for the proper full unit tests).
 And for that "move out of the aggregate" + "runtime test
 filtering" does what you want.
Move out of the aggregate makes it oh so ugly. I want my unit tests right where they belong. Runtime test filtering seems like it adds more complexity to the system unnecessarily. All I need is a way to say "yes compiler, I want to run those templated unit tests I imported" or "no compiler, I don't want to do that". What the default is really isn't important.
 Again, there are so many reasons I should not have to worry about unit
 tests in my library being run with your code. That's on you. I didn't
 write it for your code to run, if you want to run it, run my unit test
 script.
If you don't wan't to run it, filter it out in the test runner. I assure you, there are at least as much reasons why I shouldn't worry if you actually run tests for your library and how those need to be run. Both defaults can be circumvented.
I don't *care* if you run it. But the compiler shouldn't force you to. The funny thing is, you DON'T run imported non-template unit tests, because the compiler doesn't make you (and actually doesn't allow you to). So why should this be any different? A unit test is supposed to test the unit, not the world. -Steve
Apr 29 2015
parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 30 April 2015 at 02:26:19 UTC, Steven Schveighoffer 
wrote:
 On 4/29/15 7:43 AM, Dicebot wrote:
 On Wednesday, 29 April 2015 at 04:53:47 UTC, Steven 
 Schveighoffer wrote:
 or rdmd -main -unittest -> fail to build because the 
 templated unit
 test doesn't work on your code. Good luck with that.
I will create an upstream PR to fix it, problem solved. Have never had a need to do so though, not even a single time. Also : can you please point me again what part of RBT causes compilation slowdowns with version(unittest)? I looked through and found only runtime checks.
It's runtime checks that slow down, the unit test compilation is not slow (although it was at one point, but that was for the proper full unit tests).
Then test separation + custom runtime filter fixes the issue without language changes.
 And for that "move out of the aggregate" + "runtime test
 filtering" does what you want.
Move out of the aggregate makes it oh so ugly. I want my unit tests right where they belong. Runtime test filtering seems like it adds more complexity to the system unnecessarily. All I need is a way to say "yes compiler, I want to run those templated unit tests I imported" or "no compiler, I don't want to do that". What the default is really isn't important.
It is exactly the other way around. Implementing such compile-time filtering would mean adding totally new feature to the language, considerably complicating version system which is very simple right now. And remembering all the issues we had with -allinst it will take a while to figure out. You need something better than "ugly" to justify such language change. Especially when there is a working solution within existing rules - even if it seems ugly, it is achievable without language changes and addresses the issue.
May 05 2015
prev sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 04/28/2015 07:04 PM, Dicebot wrote:
 I do want do test everything as part of my app tests, including all
 possible dependencies, transitively.
Even Phobos?
 This is awesome default. With a
 simple `rdmd -main -unittest` call I can ensure that certain app/module
 works correctly without having to trust maintainers of dependencies to
 run tests regularly and without even knowing what those dependencies
 are. It is beautiful in its simplicity which makes it good default.
Apr 30 2015
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Tuesday, 28 April 2015 at 16:40:05 UTC, Dicebot wrote:
 Those are two points I fundamentally disagree with. It doesn't 
 matter where the code comes from - in the end only thing that 
 matters is correctness of your application as a whole.
3rd party libraries are supposed to be tested already, if you want to test it, you should go and properly run its test suite. Whatever template unittests you accidentally instantiated in your code mean nothing with respect to overall correctness.
 And if tests are written in a weird way that they can only be 
 ran within that library test step, those are not really 
 unittests.
The library can be tested only when it's compiled in unittest mode as a whole. When you link with its release version its unittests are not even compiled at all.
Apr 29 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 29 April 2015 at 07:44:17 UTC, Kagamin wrote:
 On Tuesday, 28 April 2015 at 16:40:05 UTC, Dicebot wrote:
 Those are two points I fundamentally disagree with. It doesn't 
 matter where the code comes from - in the end only thing that 
 matters is correctness of your application as a whole.
3rd party libraries are supposed to be tested already, if you want to test it, you should go and properly run its test suite. Whatever template unittests you accidentally instantiated in your code mean nothing with respect to overall correctness.
And software is supposed to not have bugs, right. I can put some trust in regular testing of Phobos but it ends there. If "accidental" template tests from user code break any 3d party library and its author refuses to accept PR with the fix I will simply call it broken and fork.
 And if tests are written in a weird way that they can only be 
 ran within that library test step, those are not really 
 unittests.
The library can be tested only when it's compiled in unittest mode as a whole. When you link with its release version its unittests are not even compiled at all.
Which is exactly why "all source builds with same flags" is the only reasonable compilation model for D. Providing release static libraries causes only problems and never benefits. Even for Phobos it feels like a mistake in the long run.
Apr 29 2015
parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 29 April 2015 at 11:46:57 UTC, Dicebot wrote:
 Which is exactly why "all source builds with same flags" is the 
 only reasonable compilation model for D.
If you compile all libraries in unittest mode, you get full set of unit tests from those libraries and the proposed change doesn't affect you in the least, it only affects linking with a library compiled in release mode.
Apr 29 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 29 April 2015 at 14:10:32 UTC, Kagamin wrote:
 On Wednesday, 29 April 2015 at 11:46:57 UTC, Dicebot wrote:
 Which is exactly why "all source builds with same flags" is 
 the only reasonable compilation model for D.
If you compile all libraries in unittest mode, you get full set of unit tests from those libraries and the proposed change doesn't affect you in the least, it only affects linking with a library compiled in release mode.
The core of Nick proposal (and what he does in his own libraries) is to not compile tests of dependnecies even when those are all compiled at once in -unittest mode. He uses `version(MyLibUnittest)` to disable all those completely and to run all tests you need to provide full set of such version flags for each dependency (transitively). This is what affects me and what I don't see implemented as a default. (P.S. release and -unittest are not exclusive in D, I presume you have meant "release non-test mode" :P)
Apr 29 2015
parent "Kagamin" <spam here.lot> writes:
On Wednesday, 29 April 2015 at 15:00:53 UTC, Dicebot wrote:
 The core of Nick proposal (and what he does in his own 
 libraries) is to not compile tests of dependnecies even when 
 those are all compiled at once in -unittest mode.
If they are compiled in unittest mode, all their unittests are compiled and run. What's problem?
 He uses `version(MyLibUnittest)` to disable all those 
 completely and to run all tests you need to provide full set of 
 such version flags for each dependency (transitively). This is 
 what affects me and what I don't see implemented as a default.
If the enhancement is implemented, that hack with versions won't be needed, all tests will be plain unittests, which will suit both you and Nick.
Apr 30 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-09-10 04:13, Nick Sabalausky wrote:
 This is getting to be (or rather, *continuing* to be) a royal PITA:

 https://github.com/rejectedsoftware/vibe.d/issues/673

 I don't mean to pick on Vibe.d in particular, but can we have a solution
 (that doesn't involve obscure corners of druntime, or demanding everyone
 use the same build system) for running our unittests *without*
 triggering a cascading avalanche of unittests from ALL third party libs
 that don't cooperate with the [frankly] quite clumsy
 version(mylib_unittests) hack?!
The most simple solution that ever body will hate (except me), put the unit tests in a separate directory. -- /Jacob Carlborg
Apr 29 2015