www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Quit running foreign unittests >_<

reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
This is getting to be (or rather, *continuing* to be) a royal PITA:

https://github.com/rejectedsoftware/vibe.d/issues/673

I don't mean to pick on Vibe.d in particular, but can we have a solution 
(that doesn't involve obscure corners of druntime, or demanding everyone 
use the same build system) for running our unittests *without* 
triggering a cascading avalanche of unittests from ALL third party libs 
that don't cooperate with the [frankly] quite clumsy 
version(mylib_unittests) hack?!
Sep 09 2014
next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 09 Sep 2014 22:13:57 -0400
Nick Sabalausky via Digitalmars-d <digitalmars-d puremagic.com> wrote:

write build tool that is capable of doing that...
Sep 09 2014
parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 9/9/2014 10:21 PM, ketmar via Digitalmars-d wrote:

 On Tue, 09 Sep 2014 22:13:57 -0400
 Nick Sabalausky via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 write build tool that is capable of doing that...


???
If you think that's reasonable when it's the COMPILER that inserts all 
those calls (or druntime, whatever), then go ahead with it yourself.

Seriously, DMD needs a "--omit-unittest=some.package.*". I'd try a PR 
myself (I'd have *already* tried), but last time I brought it up the 
whole idea was completely shot down in favor of both the 
version(mylib_unittests) hack (which people don't fucking use half the 
time, sometimes deliberately) and some goofy druntime trickery (ie, 
writing your own test runner and getting it to druntime before druntime 
gets to the "run unittests" phase. Seriously? That's *preferable* to a 
"--omit-unittest=some.package.*"??).
Sep 09 2014
next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 9/9/2014 10:37 PM, Nick Sabalausky wrote:

 On 9/9/2014 10:21 PM, ketmar via Digitalmars-d wrote:

 On Tue, 09 Sep 2014 22:13:57 -0400
 Nick Sabalausky via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 write build tool that is capable of doing that...


 ???
 If you think that's reasonable when it's the COMPILER that inserts all
 those calls (or druntime, whatever), then go ahead with it yourself.


Well, ok, I guess it could be done with separate invocations of DMD, but 
that has non-trivial implications for build performance, and templates 
probably could still fuck it all up anyway.
Sep 09 2014
next sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 10 September 2014 at 03:00:48 UTC, Nick Sabalausky 
wrote:

 separate invocations […] and templates probably could still 
 fuck it all up anyway.


If so, then this is either a critical bug in DMD, or an issue 
with your code (e.g. due to version(unittest) use).

I'm aware of the issues that affect some ways of doing 
incremental compilation, but that's mostly a different story.

David
Sep 09 2014
parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 9/9/2014 11:07 PM, David Nadlinger wrote:

 On Wednesday, 10 September 2014 at 03:00:48 UTC, Nick Sabalausky wrote:

 separate invocations […] and templates probably could still fuck it
 all up anyway.


 If so, then this is either a critical bug in DMD, or an issue with your
 code (e.g. due to version(unittest) use).

 I'm aware of the issues that affect some ways of doing incremental
 compilation, but that's mostly a different story.

 David


I was thinking about something like this:

//a.d
import b.d;
alias x = Foo!whatever;

//b.d
struct Foo(Arg) {
     void foo() {...}
     unittest { /+...unittest foo()...+/ }
}

// Attempt to exclude module b's unittests, except...aren't
// they compiled when module a's is compiled? The first
// dmd invocation here has no idea module b will be compiled
// without the -unittest flag.

dmd -c -unittest a.d
dmd -c b.d   // Duplicate a whole buttload of processing here
dmd a.o b.o
Sep 09 2014
prev sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 09 Sep 2014 23:00:06 -0400
Nick Sabalausky via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 Well, ok, I guess it could be done with separate invocations of DMD,
 but that has non-trivial implications for build performance


we can cache analyzed ASTs somewhere. this can speedup
separate compilations a little. but i don't see any problems with
separate compilation: it's fast enough even with alot of modules.
Sep 10 2014
prev sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 09 Sep 2014 22:37:23 -0400
Nick Sabalausky via Digitalmars-d <digitalmars-d puremagic.com> wrote:


 If you think that's reasonable when it's the COMPILER that inserts
 all those calls (or druntime, whatever), then go ahead with it
 yourself.


sure it's a compiler. that's why we have "-unittest" arg. and separate
compilation rocks (why, it's cheap! and windows... ah, i don't care).

yes, templates still sux (considering built-in unittests). this is
where compiler should be fixed: it should compile unittest invocations
only for templates that comes from the module it compiles, i think.
Sep 10 2014
parent "Kagamin" <spam here.lot> writes:
On Wednesday, 10 September 2014 at 08:05:28 UTC, ketmar via 
Digitalmars-d wrote:

 yes, templates still sux (considering built-in unittests). this 
 is
 where compiler should be fixed: it should compile unittest 
 invocations
 only for templates that comes from the module it compiles, i 
 think.


That way it will be clear.

By the way, will this link?
---
template A(T)
{
   int a;
   unittest{ f(); }
}

version(unittest) void f(){}
---
If this module is compiled without unittests, `f` function will 
not be compiled, then if another module is compiled with 
unittests and instantiates the template, the template's unittest 
is compiled and it will link with `f` function, which wasn't 
compiled, so linking will fail?
Sep 10 2014
prev sibling next sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 10 September 2014 at 02:14:39 UTC, Nick Sabalausky 
wrote:

 This is getting to be (or rather, *continuing* to be) a royal 
 PITA:

 https://github.com/rejectedsoftware/vibe.d/issues/673

 I don't mean to pick on Vibe.d in particular, but can we have a 
 solution (that doesn't involve obscure corners of druntime, or 
 demanding everyone use the same build system) for running our 
 unittests *without* triggering a cascading avalanche of 
 unittests from ALL third party libs that don't cooperate with 
 the [frankly] quite clumsy version(mylib_unittests) hack?!


A while ago, I though about adding an optional module/package 
whitelist as a compiler flag, something like 
-unittest=myapp,mylib.important. Never really got to implementing 
it, or even thinking about the implications in any details.

David
Sep 09 2014
next sibling parent "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 10 September 2014 at 02:29:41 UTC, David Nadlinger 
wrote:

 A while ago, I though about adding an optional module/package 
 whitelist as a compiler flag, something like 
 -unittest=myapp,mylib.important. Never really got to 
 implementing it, or even thinking about the implications in any 
 details.


s/compiler/rdmd/
Sep 09 2014
prev sibling parent reply "Kagamin" <spam here.lot> writes:
I think, libraries should support compilation without unittests, 
i.e. write unittests outside of templates. But then one may want 
to test every template instantiation, if it makes sense and 
intended.
Sep 09 2014
parent reply "Kagamin" <spam here.lot> writes:
It's questionable that testing every template instantiation is 
intended: you can't do the same for templated functions.
Sep 09 2014
next sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 9/10/2014 1:01 AM, Kagamin wrote:

 It's questionable that testing every template instantiation is intended:
 you can't do the same for templated functions.


Regardless of the (de)merits of deliberately putting unittests in a 
template, the important thing is this:

Whether a person intentionally *or inadvertently* winds up with a 
unittest inside a template, the whole "causes some of my unittests run 
when my lib's *users* run *their* project's unittests" is a highly 
unintuitive non-localized consequence.
Sep 10 2014
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 09/10/2014 07:01 AM, Kagamin wrote:

 It's questionable that testing every template instantiation is intended:
 you can't do the same for templated functions.


Yes you can.
Sep 10 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 10 September 2014 at 02:14:39 UTC, Nick Sabalausky 
wrote:

 This is getting to be (or rather, *continuing* to be) a royal 
 PITA:

 https://github.com/rejectedsoftware/vibe.d/issues/673

 I don't mean to pick on Vibe.d in particular, but can we have a 
 solution (that doesn't involve obscure corners of druntime, or 
 demanding everyone use the same build system) for running our 
 unittests *without* triggering a cascading avalanche of 
 unittests from ALL third party libs that don't cooperate with 
 the [frankly] quite clumsy version(mylib_unittests) hack?!


My opinion stays the same - it is good reliable default and 
should stay that way. And yes, I absolutely hate when libraries 
do version(mylib_unittest) hacks forcing me to bother with 
millions of versions to enable them back.

Simple solutions is to provide alternative unit test runner 
module as part of Phobos which does the module/package filtering.
Sep 10 2014
parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 9/10/2014 3:57 AM, Dicebot wrote:

 My opinion stays the same - it is good reliable default and should stay
 that way. And yes, I absolutely hate when libraries do
 version(mylib_unittest) hacks forcing me to bother with millions of
 versions to enable them back.


Being "default" is one thing, I really don't care what the default is - 
but for now it's not just "the default", it's the only realistic option 
without getting into PITA workarounds.

Some people like including 3rd party library unittests with their own 
(apparently), some people like *not* including them. So we have some 
projects using version(foo_unittest) and other projects not using it. 
The current result is: *Nobody* can easily get what they want: Not 
people who like it my way, and not people who like it your way. Not 
easily enough anyway.

So it's a global ecosystem problem. It needs fixed. (And to start with, 
general agreeance that it should be fixed.)


 Simple solutions is to provide alternative unit test runner module as
 part of Phobos which does the module/package filtering.


It needs to be *crystal* clear and trivially simple:

A. Knowing that it exists.
B. Sussing out how to use it.
C. Actually using it.

Otherwise, version(foo_unittest) will continue to linger out of an 
individual's convenience, and so once again, not everyone gets to have 
things their way.

A related concern I have about a Phobos-only solution: When/where do you 
configure it? Obviously it can't happen anywhere in the scope of main. 
Are static ctors sufficient? I would *think* so, but I'm not certain.
Sep 10 2014
parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 10 September 2014 at 08:21:34 UTC, Nick Sabalausky 
wrote:

 Simple solutions is to provide alternative unit test runner 
 module as
 part of Phobos which does the module/package filtering.


 It needs to be *crystal* clear and trivially simple:

 A. Knowing that it exists.
 B. Sussing out how to use it.
 C. Actually using it.

 Otherwise, version(foo_unittest) will continue to linger out of 
 an individual's convenience, and so once again, not everyone 
 gets to have things their way.

 A related concern I have about a Phobos-only solution: 
 When/where do you configure it? Obviously it can't happen 
 anywhere in the scope of main. Are static ctors sufficient? I 
 would *think* so, but I'm not certain.


Leandro Lucarella has written similar solution for our internal 
testing purposes. It provides a test runner that is configured by 
CLI arguments and allows filtering of modules and non-fatal test 
failures. It is for D1/tango but I am pretty sure it can be 
ported to D2 runtime with close to 0 changes. I have not managed 
to convince him it is worth making public though :(
Sep 10 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 09 Sep 2014 22:13:57 -0400, Nick Sabalausky  
<SeeWebsiteToContactMe semitwist.com> wrote:


 This is getting to be (or rather, *continuing* to be) a royal PITA:

 https://github.com/rejectedsoftware/vibe.d/issues/673

 I don't mean to pick on Vibe.d in particular, but can we have a solution  
 (that doesn't involve obscure corners of druntime, or demanding everyone  
 use the same build system) for running our unittests *without*  
 triggering a cascading avalanche of unittests from ALL third party libs  
 that don't cooperate with the [frankly] quite clumsy  
 version(mylib_unittests) hack?!


Hm... would it be acceptable if unit tests were compiled, but not run?  
Because the unit tester runtime just foreach's over every module and runs  
the tests contained in that module.

Now, I don't know how it works if you compile a template unit test -- does  
it get deposited into the module that instantiated or into the module that  
defined it? Worth looking into I think.

-Steve
Sep 16 2014
prev sibling next sibling parent reply "Martin Nowak" <code dawg.eu> writes:
A fairly simple solution to this would be to compile unittests 
only for root modules (the ones that are part of the 
compilation), not for imported modules.
Then everyone can decide for which modules to use -unittest.
Apr 27 2015
parent reply "Martin Nowak" <code dawg.eu> writes:
https://issues.dlang.org/show_bug.cgi?id=13454
Apr 27 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 27 April 2015 at 07:28:28 UTC, Martin Nowak wrote:

 https://issues.dlang.org/show_bug.cgi?id=13454


Great, lets compromise both unittest correctness and convienience 
at the same time. Can we please just close the issue as `RESOLVED 
WONTFIX`? Compiling tests of dependencies pretty much never 
causes any notable slowdown. Trying to be smart about it does 
harm the users and contributors of the library though.
Apr 27 2015
parent reply "Kagamin" <spam here.lot> writes:
On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:

 Compiling tests of dependencies pretty much never causes any 
 notable slowdown.


This thread doesn't support that view, see the first post.
Apr 27 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote:

 On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:

 Compiling tests of dependencies pretty much never causes any 
 notable slowdown.


 This thread doesn't support that view, see the first post.


Which part exactly? I only see comparisons for compiling AND 
running tests for dependencies. And it is usually running which 
causes the slowdown.
Apr 27 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/27/15 6:20 AM, Dicebot wrote:

 On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote:

 On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:

 Compiling tests of dependencies pretty much never causes any notable
 slowdown.


 This thread doesn't support that view, see the first post.


 Which part exactly? I only see comparisons for compiling AND running
 tests for dependencies. And it is usually running which causes the
 slowdown.


The problem is as follows:

1. Unit tests for some library are written for that library. They are 
written to run tests during unit tests of that library only (possibly 
with certain requirements of environment, including build lines, or 
expectations of system resource availability).
2. People who import that library's modules are not trying to test the 
library, they are trying to test their code.
3. The library runs its unit tests for templates in this case, not the 
other unit tests in the module (those would be run in that module's unit 
tests, which means you'd have to recompile the library to run unit 
tests). Most often, unit tests in templates are only expected to be 
defined by the library.

I think ketmar had the most reasonable request here -- only compile/run 
unit tests in templates for the module that defines the template.

If you want to create a unit test that runs for your specialized 
template version, there should be a way to do that, but it should be 
opt-in. I envision something like this:

unittest
{
	import somelib.somemod : SomeModTemplate, unittest
	SomeModTemplate!MyLocalType t; // runs templated unit tests
}

If you look in RedBlackTree, I use templated unit tests to great effect 
(found about 4-5 compiler bugs that way). But I have a lot of machinery 
around the unit tests to try and only test unit tests on integral types. 
It would be nice to not have to write all that shit.

And BTW, I didn't even get it right, I forgot about other parameters to 
the RBTree, necessitating a PR later.

-Steve
Apr 27 2015
next sibling parent reply "Ivan Kazmenko" <gassa mail.ru> writes:
On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer 
wrote:

 The problem is as follows:

 1. Unit tests for some library are written for that library. 
 They are written to run tests during unit tests of that library 
 only (possibly with certain requirements of environment, 
 including build lines, or expectations of system resource 
 availability).


By the way, a unittest-related issue still stands in DMD 2.067.1 
for RedBlackTree:
https://issues.dlang.org/show_bug.cgi?id=12246

A similar matter got resolved quickly for BinaryHeap:
https://issues.dlang.org/show_bug.cgi?id=12245

BinaryHeap's case was handled by putting the container's 
unittests into "debug(BinaryHeap)".

RedBlackTree's case is controlled via "debug(RBDoChecks)" 
(formerly "version(RBDoChecks)").  The difference is the presence 
of a "version(unittest) debug = RBDoChecks;" line.  This looks 
inconsistent.

For RedBlackTree, compile the following with or without -unittest 
option and run for a visible difference in speed (runs 
momentarily or for a few seconds):
-----
import std.container;
void main() {
	auto t = redBlackTree!int;
	foreach (i; 0..3000) t.insert(i);
}
-----

Ivan Kazmenko.
Apr 27 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/27/15 10:30 AM, Ivan Kazmenko wrote:

 On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer wrote:

 The problem is as follows:

 1. Unit tests for some library are written for that library. They are
 written to run tests during unit tests of that library only (possibly
 with certain requirements of environment, including build lines, or
 expectations of system resource availability).


 By the way, a unittest-related issue still stands in DMD 2.067.1 for
 RedBlackTree:
 https://issues.dlang.org/show_bug.cgi?id=12246

 A similar matter got resolved quickly for BinaryHeap:
 https://issues.dlang.org/show_bug.cgi?id=12245

 BinaryHeap's case was handled by putting the container's unittests into
 "debug(BinaryHeap)".

 RedBlackTree's case is controlled via "debug(RBDoChecks)" (formerly
 "version(RBDoChecks)").  The difference is the presence of a
 "version(unittest) debug = RBDoChecks;" line.  This looks inconsistent.

 For RedBlackTree, compile the following with or without -unittest option
 and run for a visible difference in speed (runs momentarily or for a few
 seconds):
 -----
 import std.container;
 void main() {
      auto t = redBlackTree!int;
      foreach (i; 0..3000) t.insert(i);
 }
 -----

 Ivan Kazmenko.


It's an anecdotal fix. I remember arguing over the change to debug, that 
was done for purity (pure functions need debug mode to print out 
something, which rbdochecks will do if there is an issue), but I can't 
find the conversation.

But someone else will complain that when they try to debug their code, 
adding -debug to the command line debugs RBTree's algorithm (similarly 
to how they complained BinaryHeap was doing this). This really should 
only EVER run during phobos unit tests.

I don't know how to fix this properly without something like I outlined 
above, or without doing some global version(PhobosUnitTests) hack.

In fact, I don't agree with the BinaryHeap change. At this point, phobos 
unit tests are NOT testing the binary heap structure. Sure that makes 
user code run faster, but at the cost of never testing it even when it 
should be tested.

-Steve
Apr 27 2015
parent reply "Ivan Kazmenko" <gassa mail.ru> writes:
On Monday, 27 April 2015 at 15:29:05 UTC, Steven Schveighoffer 
wrote:

 On 4/27/15 10:30 AM, Ivan Kazmenko wrote:

 On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer 
 wrote:

 The problem is as follows:

 1. Unit tests for some library are written for that library. 
 They are
 written to run tests during unit tests of that library only 
 (possibly
 with certain requirements of environment, including build 
 lines, or
 expectations of system resource availability).


 By the way, a unittest-related issue still stands in DMD 
 2.067.1 for
 RedBlackTree:
 https://issues.dlang.org/show_bug.cgi?id=12246

 A similar matter got resolved quickly for BinaryHeap:
 https://issues.dlang.org/show_bug.cgi?id=12245

 BinaryHeap's case was handled by putting the container's 
 unittests into
 "debug(BinaryHeap)".

 RedBlackTree's case is controlled via "debug(RBDoChecks)" 
 (formerly
 "version(RBDoChecks)").  The difference is the presence of a
 "version(unittest) debug = RBDoChecks;" line.  This looks 
 inconsistent.

 For RedBlackTree, compile the following with or without 
 -unittest option
 and run for a visible difference in speed (runs momentarily or 
 for a few
 seconds):
 -----
 import std.container;
 void main() {
     auto t = redBlackTree!int;
     foreach (i; 0..3000) t.insert(i);
 }
 -----

 Ivan Kazmenko.


 It's an anecdotal fix. I remember arguing over the change to 
 debug, that was done for purity (pure functions need debug mode 
 to print out something, which rbdochecks will do if there is an 
 issue), but I can't find the conversation.

 But someone else will complain that when they try to debug 
 their code, adding -debug to the command line debugs RBTree's 
 algorithm (similarly to how they complained BinaryHeap was 
 doing this).


I am doing just that, right now =) .  But it's -unittest, not 
-debug, which triggers RBTree's expensive checks.


 This really should only EVER run during phobos unit tests.


A choice would be even better.  When a user's code is wrong, for 
example, a comparison function does not define a comparison in 
mathematical sense, the user might benefit from container's 
sanity checks.  From personal experience, when I implement my own 
sorted container and some complex logic using it, I do benefit 
from the container's checks sometimes.


 I don't know how to fix this properly without something like I 
 outlined above, or without doing some global 
 version(PhobosUnitTests) hack.

 In fact, I don't agree with the BinaryHeap change. At this 
 point, phobos unit tests are NOT testing the binary heap 
 structure. Sure that makes user code run faster, but at the 
 cost of never testing it even when it should be tested.

 -Steve


Right now, it is possible to test binary heap integrity with an 
explicit "-debug=BinaryHeap", and it is not tested by default.

And it is mandatory to test red-black tree integrity when running 
any unittests involving a red-black tree, as it is tested by 
default.

So, from a user's point of view, I like the binary heap situation 
better, since it gives me a choice with a reasonable default.

How Phobos' unittests should be handled is another matter which, 
in my opinion, should not take away choices, and reasonable 
defaults too, from the user.

To me, the proposed version(PhobosUnitTests) hack looks good for 
the particular case.  But I don't see whether such trick scales 
well for the whole ecosystem of libraries.

Ivan Kazmenko.
Apr 27 2015
next sibling parent "Ivan Kazmenko" <gassa mail.ru> writes:
 To me, the proposed version(PhobosUnitTests) hack looks good 
 for the particular case.  But I don't see whether such trick 
 scales well for the whole ecosystem of libraries.


Another approach that might work specifically for containers is 
to pass as an additional template parameter whether the 
particular RedBlackTree must continuously check its integrity or 
not.  That way, the parameter can be explicitly set to "doCheck" 
from the unittest instantiations, default to "doNotCheck" for 
other code, and be enabled on a case-by-case basis when debugging.

Again, I don't see whether it fits the larger picture, but still 
think it's worth to consider.

Ivan Kazmenko.
Apr 27 2015
prev sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/27/15 12:00 PM, Ivan Kazmenko wrote:


 Right now, it is possible to test binary heap integrity with an explicit
 "-debug=BinaryHeap", and it is not tested by default.


Right, but phobos unit tests are not doing that. If we added a 
version(unittest) debug = BinaryHeap; to fix that problem there, then 
you would still be complaining ;)

-Steve
Apr 27 2015
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer 
wrote:

 On 4/27/15 6:20 AM, Dicebot wrote:

 On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote:

 On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:

 Compiling tests of dependencies pretty much never causes any 
 notable
 slowdown.


 This thread doesn't support that view, see the first post.


 Which part exactly? I only see comparisons for compiling AND 
 running
 tests for dependencies. And it is usually running which causes 
 the
 slowdown.


 The problem is as follows:

 1. Unit tests for some library are written for that library. 
 They are written to run tests during unit tests of that library 
 only (possibly with certain requirements of environment, 
 including build lines, or expectations of system resource 
 availability).
 2. People who import that library's modules are not trying to 
 test the library, they are trying to test their code.


Those are two points I fundamentally disagree with. It doesn't 
matter where the code comes from - in the end only thing that 
matters is correctness of your application as a whole. And 
considering tests are not necessarily pure the results may very 
well differ between running those tests spearately and as part of 
application test suite a whole. Unless compiling some specific 
tests causes some proven _compilation_ slowdown (I have yet to 
see that) those all must be compiled and filtered by runtime test 
runner optionally.

And if tests are written in a weird way that they can only be ran 
within that library test step, those are not really unittests.

Usage of version(MyLibTests) in Nick SDL library annoyed me so 
much that I forked it to never deal with those pesky versions 
again. Don't want to do that with Phobos too.
Apr 28 2015
next sibling parent reply "Ivan Kazmenko" <gassa mail.ru> writes:
On Tuesday, 28 April 2015 at 16:40:05 UTC, Dicebot wrote:

 On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer 
 wrote:

 On 4/27/15 6:20 AM, Dicebot wrote:

 On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote:

 On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:

 Compiling tests of dependencies pretty much never causes 
 any notable
 slowdown.


 This thread doesn't support that view, see the first post.


 Which part exactly? I only see comparisons for compiling AND 
 running
 tests for dependencies. And it is usually running which 
 causes the
 slowdown.


 The problem is as follows:

 1. Unit tests for some library are written for that library. 
 They are written to run tests during unit tests of that 
 library only (possibly with certain requirements of 
 environment, including build lines, or expectations of system 
 resource availability).
 2. People who import that library's modules are not trying to 
 test the library, they are trying to test their code.


 Those are two points I fundamentally disagree with. It doesn't 
 matter where the code comes from - in the end only thing that 
 matters is correctness of your application as a whole. And 
 considering tests are not necessarily pure the results may very 
 well differ between running those tests spearately and as part 
 of application test suite a whole. Unless compiling some 
 specific tests causes some proven _compilation_ slowdown (I 
 have yet to see that) those all must be compiled and filtered 
 by runtime test runner optionally.

 And if tests are written in a weird way that they can only be 
 ran within that library test step, those are not really 
 unittests.

 Usage of version(MyLibTests) in Nick SDL library annoyed me so 
 much that I forked it to never deal with those pesky versions 
 again. Don't want to do that with Phobos too.


Then how do you propose to approach the containers problem?

On one hand, a unittest on containers themselves involves testing 
the container for integrity after each operation.

On the other hand, a unittest on another module may involve heavy 
use of containers (say, N operations with a container), and if 
integrity checks are enabled at this time, it totals to N^2 
trivial operations which may not be feasible.

Ivan Kazmenko.
Apr 28 2015
parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 28 April 2015 at 20:57:01 UTC, Ivan Kazmenko wrote:

 Then how do you propose to approach the containers problem?

 On one hand, a unittest on containers themselves involves 
 testing the container for integrity after each operation.

 On the other hand, a unittest on another module may involve 
 heavy use of containers (say, N operations with a container), 
 and if integrity checks are enabled at this time, it totals to 
 N^2 trivial operations which may not be feasible.


If it is that slow I tend to put such tests outside of the tested 
symbol so that it won't be repeated over and over again for 
templates. There is still full access to private symbols so it is 
always possible. Though that is very exceptional case reserved 
only for most critical cases (everything below 30-60 seconds 
total is ok to me)
Apr 28 2015
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/28/15 12:40 PM, Dicebot wrote:

 On Monday, 27 April 2015 at 11:30:04 UTC, Steven Schveighoffer wrote:

 On 4/27/15 6:20 AM, Dicebot wrote:

 On Monday, 27 April 2015 at 10:15:20 UTC, Kagamin wrote:

 On Monday, 27 April 2015 at 09:22:48 UTC, Dicebot wrote:

 Compiling tests of dependencies pretty much never causes any notable
 slowdown.


 This thread doesn't support that view, see the first post.


 Which part exactly? I only see comparisons for compiling AND running
 tests for dependencies. And it is usually running which causes the
 slowdown.


 The problem is as follows:

 1. Unit tests for some library are written for that library. They are
 written to run tests during unit tests of that library only (possibly
 with certain requirements of environment, including build lines, or
 expectations of system resource availability).
 2. People who import that library's modules are not trying to test the
 library, they are trying to test their code.


 Those are two points I fundamentally disagree with.


I think by default, nobody wants to test already-tested code in their 
application. That's not the point of unit tests.

For example, if there's a module that has:

struct S(T)
{
   unittest {...}
}

unittest
{
    S!int; // run unit tests for int
}

Then I can assume that unit test was run and passed. If I want to test 
it to be sure, I'll run that library's unit tests!

But I don't want my unit test that uses S!int to also test S's unit 
tests for S!int. That doesn't make sense. I'm wasting cycles on 
something that is already proven.

Now, if I want to run unit tests for S!MyCustomType, that makes sense. 
The library didn't test that. Which is why there should be a way to do 
it (if it's valid!).

Right now, there isn't a way to control what runs and what doesn't. I 
don't care where it's decided or how it's decided, but somehow I should 
be able to NOT run S!int tests again.


 Unless compiling some specific tests causes some proven _compilation_
 slowdown (I have yet to see that) those all must be compiled and
 filtered by runtime test runner optionally.


For example, RedBlackTree when running unit tests does a sanity check of 
the tree for every operation. If you then use a RedBlackTree in your 
unit tests, you are doing that again. Maybe it's worth it to you, but it 
can increase the runtime of your test by orders of magnitude. Last time 
I checked, performance was high on people's priority lists.

Or, potentially you could be running INVALID TESTS, because the tests 
weren't written for your specific usages. I ran into this when others 
complained that they couldn't test their code that uses 
RedBlackTree!string, because all the unit tests instantiated with int 
literals. This is simply a "does not work" thing. You can't turn them 
off, and you can't test your code. Is it worth it to you for a library 
to try to compile tests that prevent your test build from happening? Do 
you enjoy waiting on others to fix their problems so you can test your 
code? This is a very less than ideal situation. And it's not something 
the compiler tells you will happen.


 And if tests are written in a weird way that they can only be ran within
 that library test step, those are not really unittests.


This means you should never ever write unit tests into a template. 
Because it's impossible to create tests that will work for every single 
instantiation. See RedBlackTree for all the crud you have to put in for 
this to be viable today. I'd love to get rid of all that.

In which case, let's disallow that capability. I'm fine with that too. 
This simply means your "all encompassing tests" will not be all 
encompassing any more, they will only test a select few instantiations, 
and you can't control that either from your application code.

I want a way to control it as a template designer and as a template 
instantiator. Flexibility is king.

-Steve
Apr 28 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 28 April 2015 at 21:28:09 UTC, Steven Schveighoffer 
wrote:

 I think by default, nobody wants to test already-tested code in 
 their application. That's not the point of unit tests.


How many more times should I repeat that I am exactly that nobody?

I do want do test everything as part of my app tests, including 
all possible dependencies, transitively. This is awesome default. 
With a simple `rdmd -main -unittest` call I can ensure that 
certain app/module works correctly without having to trust 
maintainers of dependencies to run tests regularly and without 
even knowing what those dependencies are. It is beautiful in its 
simplicity which makes it good default.


 struct S(T)
 {
   unittest {...}
 }

 unittest
 {
    S!int; // run unit tests for int
 }


If this a very slow test (and there are expected many S) simply 
put the test blocks out of the aggregate.


 Now, if I want to run unit tests for S!MyCustomType, that makes 
 sense. The library didn't test that. Which is why there should 
 be a way to do it (if it's valid!).


There is something fundamentally broken with a template that 
needs to be tested for each new user type argument. Built-in 
tests must cover all possible type classes or they are incomplete.


 Unless compiling some specific tests causes some proven 
 _compilation_
 slowdown (I have yet to see that) those all must be compiled 
 and
 filtered by runtime test runner optionally.


 For example, RedBlackTree when running unit tests does a sanity 
 check of the tree for every operation. If you then use a 
 RedBlackTree in your unit tests, you are doing that again. 
 Maybe it's worth it to you, but it can increase the runtime of 
 your test by orders of magnitude. Last time I checked, 
 performance was high on people's priority lists.


It must be very slow sanity checks :X Sounds weird but I will 
accept it as given. Move the tests out of the RBL definition then.

Also performance has nothing in common with test performance. I 
can't comment about rest because from the very beginning you seem 
to make statements about testing in general which do not match my 
vision at all.
Apr 28 2015
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/28/15 7:04 PM, Dicebot wrote:

 On Tuesday, 28 April 2015 at 21:28:09 UTC, Steven Schveighoffer wrote:

 I think by default, nobody wants to test already-tested code in their
 application. That's not the point of unit tests.


 How many more times should I repeat that I am exactly that nobody?


OK, sorry. nodbody-1 :)


 I do want do test everything as part of my app tests, including all
 possible dependencies, transitively. This is awesome default. With a
 simple `rdmd -main -unittest` call I can ensure that certain app/module
 works correctly without having to trust maintainers of dependencies to
 run tests regularly and without even knowing what those dependencies
 are. It is beautiful in its simplicity which makes it good default.


or rdmd -main -unittest -> fail to build because the templated unit test 
doesn't work on your code. Good luck with that.

Again, there are so many reasons I should not have to worry about unit 
tests in my library being run with your code. That's on you. I didn't 
write it for your code to run, if you want to run it, run my unit test 
script.

-Steve
Apr 28 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 29 April 2015 at 04:53:47 UTC, Steven Schveighoffer 
wrote:

 or rdmd -main -unittest -> fail to build because the templated 
 unit test doesn't work on your code. Good luck with that.


I will create an upstream PR to fix it, problem solved. Have 
never had a need to do so though, not even a single time.

Also : can you please point me again what part of RBT causes 
compilation slowdowns with version(unittest)? I looked through 
and found only runtime checks. And for that "move out of the 
aggregate" + "runtime test filtering" does what you want.


 Again, there are so many reasons I should not have to worry 
 about unit tests in my library being run with your code. That's 
 on you. I didn't write it for your code to run, if you want to 
 run it, run my unit test script.


If you don't wan't to run it, filter it out in the test runner. I 
assure you, there are at least as much reasons why I shouldn't 
worry if you actually run tests for your library and how those 
need to be run. Both defaults can be circumvented.
Apr 29 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 4/29/15 7:43 AM, Dicebot wrote:

 On Wednesday, 29 April 2015 at 04:53:47 UTC, Steven Schveighoffer wrote:

 or rdmd -main -unittest -> fail to build because the templated unit
 test doesn't work on your code. Good luck with that.


 I will create an upstream PR to fix it, problem solved. Have never had a
 need to do so though, not even a single time.

 Also : can you please point me again what part of RBT causes compilation
 slowdowns with version(unittest)? I looked through and found only
 runtime checks.


It's runtime checks that slow down, the unit test compilation is not 
slow (although it was at one point, but that was for the proper full 
unit tests).


 And for that "move out of the aggregate" + "runtime test
 filtering" does what you want.


Move out of the aggregate makes it oh so ugly. I want my unit tests 
right where they belong.

Runtime test filtering seems like it adds more complexity to the system 
unnecessarily. All I need is a way to say "yes compiler, I want to run 
those templated unit tests I imported" or "no compiler, I don't want to 
do that". What the default is really isn't important.


 Again, there are so many reasons I should not have to worry about unit
 tests in my library being run with your code. That's on you. I didn't
 write it for your code to run, if you want to run it, run my unit test
 script.


 If you don't wan't to run it, filter it out in the test runner. I assure
 you, there are at least as much reasons why I shouldn't worry if you
 actually run tests for your library and how those need to be run. Both
 defaults can be circumvented.


I don't *care* if you run it. But the compiler shouldn't force you to. 
The funny thing is, you DON'T run imported non-template unit tests, 
because the compiler doesn't make you (and actually doesn't allow you 
to). So why should this be any different? A unit test is supposed to 
test the unit, not the world.

-Steve
Apr 29 2015
parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 30 April 2015 at 02:26:19 UTC, Steven Schveighoffer 
wrote:

 On 4/29/15 7:43 AM, Dicebot wrote:

 On Wednesday, 29 April 2015 at 04:53:47 UTC, Steven 
 Schveighoffer wrote:

 or rdmd -main -unittest -> fail to build because the 
 templated unit
 test doesn't work on your code. Good luck with that.


 I will create an upstream PR to fix it, problem solved. Have 
 never had a
 need to do so though, not even a single time.

 Also : can you please point me again what part of RBT causes 
 compilation
 slowdowns with version(unittest)? I looked through and found 
 only
 runtime checks.


 It's runtime checks that slow down, the unit test compilation 
 is not slow (although it was at one point, but that was for the 
 proper full unit tests).


Then test separation + custom runtime filter fixes the issue 
without language changes.


 And for that "move out of the aggregate" + "runtime test
 filtering" does what you want.


 Move out of the aggregate makes it oh so ugly. I want my unit 
 tests right where they belong.

 Runtime test filtering seems like it adds more complexity to 
 the system unnecessarily. All I need is a way to say "yes 
 compiler, I want to run those templated unit tests I imported" 
 or "no compiler, I don't want to do that". What the default is 
 really isn't important.


It is exactly the other way around. Implementing such 
compile-time filtering would mean adding totally new feature to 
the language, considerably complicating version system which is 
very simple right now. And remembering all the issues we had with 
-allinst it will take a while to figure out. You need something 
better than "ugly" to justify such language change. Especially 
when there is a working solution within existing rules - even if 
it seems ugly, it is achievable without language changes and 
addresses the issue.
May 05 2015
prev sibling parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 04/28/2015 07:04 PM, Dicebot wrote:

 I do want do test everything as part of my app tests, including all
 possible dependencies, transitively.


Even Phobos?


 This is awesome default. With a
 simple `rdmd -main -unittest` call I can ensure that certain app/module
 works correctly without having to trust maintainers of dependencies to
 run tests regularly and without even knowing what those dependencies
 are. It is beautiful in its simplicity which makes it good default.
Apr 30 2015
prev sibling parent reply "Kagamin" <spam here.lot> writes:
On Tuesday, 28 April 2015 at 16:40:05 UTC, Dicebot wrote:

 Those are two points I fundamentally disagree with. It doesn't 
 matter where the code comes from - in the end only thing that 
 matters is correctness of your application as a whole.


3rd party libraries are supposed to be tested already, if you 
want to test it, you should go and properly run its test suite. 
Whatever template unittests you accidentally instantiated in your 
code mean nothing with respect to overall correctness.


 And if tests are written in a weird way that they can only be 
 ran within that library test step, those are not really 
 unittests.


The library can be tested only when it's compiled in unittest 
mode as a whole. When you link with its release version its 
unittests are not even compiled at all.
Apr 29 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 29 April 2015 at 07:44:17 UTC, Kagamin wrote:

 On Tuesday, 28 April 2015 at 16:40:05 UTC, Dicebot wrote:

 Those are two points I fundamentally disagree with. It doesn't 
 matter where the code comes from - in the end only thing that 
 matters is correctness of your application as a whole.


 3rd party libraries are supposed to be tested already, if you 
 want to test it, you should go and properly run its test suite. 
 Whatever template unittests you accidentally instantiated in 
 your code mean nothing with respect to overall correctness.


And software is supposed to not have bugs, right. I can put some 
trust in regular testing of Phobos but it ends there. If 
"accidental" template tests from user code break any 3d party 
library and its author refuses to accept PR with the fix I will 
simply call it broken and fork.


 And if tests are written in a weird way that they can only be 
 ran within that library test step, those are not really 
 unittests.


 The library can be tested only when it's compiled in unittest 
 mode as a whole. When you link with its release version its 
 unittests are not even compiled at all.


Which is exactly why "all source builds with same flags" is the 
only reasonable compilation model for D. Providing release static 
libraries causes only problems and never benefits. Even for 
Phobos it feels like a mistake in the long run.
Apr 29 2015
parent reply "Kagamin" <spam here.lot> writes:
On Wednesday, 29 April 2015 at 11:46:57 UTC, Dicebot wrote:

 Which is exactly why "all source builds with same flags" is the 
 only reasonable compilation model for D.


If you compile all libraries in unittest mode, you get full set 
of unit tests from those libraries and the proposed change 
doesn't affect you in the least, it only affects linking with a 
library compiled in release mode.
Apr 29 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 29 April 2015 at 14:10:32 UTC, Kagamin wrote:

 On Wednesday, 29 April 2015 at 11:46:57 UTC, Dicebot wrote:

 Which is exactly why "all source builds with same flags" is 
 the only reasonable compilation model for D.


 If you compile all libraries in unittest mode, you get full set 
 of unit tests from those libraries and the proposed change 
 doesn't affect you in the least, it only affects linking with a 
 library compiled in release mode.


The core of Nick proposal (and what he does in his own libraries) 
is to not compile tests of dependnecies even when those are all 
compiled at once in -unittest mode. He uses 
`version(MyLibUnittest)` to disable all those completely and to 
run all tests you need to provide full set of such version flags 
for each dependency (transitively). This is what affects me and 
what I don't see implemented as a default.

(P.S. release and -unittest are not exclusive in D, I presume you 
have meant "release non-test mode" :P)
Apr 29 2015
parent "Kagamin" <spam here.lot> writes:
On Wednesday, 29 April 2015 at 15:00:53 UTC, Dicebot wrote:

 The core of Nick proposal (and what he does in his own 
 libraries) is to not compile tests of dependnecies even when 
 those are all compiled at once in -unittest mode.


If they are compiled in unittest mode, all their unittests are 
compiled and run. What's problem?


 He uses `version(MyLibUnittest)` to disable all those 
 completely and to run all tests you need to provide full set of 
 such version flags for each dependency (transitively). This is 
 what affects me and what I don't see implemented as a default.


If the enhancement is implemented, that hack with versions won't 
be needed, all tests will be plain unittests, which will suit 
both you and Nick.
Apr 30 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-09-10 04:13, Nick Sabalausky wrote:

 This is getting to be (or rather, *continuing* to be) a royal PITA:

 https://github.com/rejectedsoftware/vibe.d/issues/673

 I don't mean to pick on Vibe.d in particular, but can we have a solution
 (that doesn't involve obscure corners of druntime, or demanding everyone
 use the same build system) for running our unittests *without*
 triggering a cascading avalanche of unittests from ALL third party libs
 that don't cooperate with the [frankly] quite clumsy
 version(mylib_unittests) hack?!


The most simple solution that ever body will hate (except me), put the 
unit tests in a separate directory.

-- 
/Jacob Carlborg
Apr 29 2015