www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Named unittests

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
We're having a strong need for named unittests at Facebook for multiple 
reasons.

1. We have sophisticated tooling that verifies whether unittests are 
flaky. The automated monitor (for e.g. C++) figures whether a given 
unittest fails several times across several commits. Unittests are 
identified by name; relying on file/line is impossible because the line 
of a failure is not stable across changes.

2. Again for efficient automated testing and flakiness detection, one 
should be able to run only a subset of unittests by mentioning them by 
line in the command line. Note that this implies there's no 
interdependency between distinct unittests, which is fine because the 
new ability is opt-on; I'd say is pure style anyway.

3. Mentioning unittest names in failure messages helps human 
communication (e.g. "AddPeer is failing after your change"). This is 
impossible with file and line numbers.

I'd like to make a DIP for named unittests. Who can help me with that?


Andrei
Mar 30 2015
next sibling parent reply "Panke" <tobias pankrath.net> writes:
On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu 
wrote:

 We're having a strong need for named unittests at Facebook for 
 multiple reasons.

 1. We have sophisticated tooling that verifies whether 
 unittests are flaky. The automated monitor (for e.g. C++) 
 figures whether a given unittest fails several times across 
 several commits. Unittests are identified by name; relying on 
 file/line is impossible because the line of a failure is not 
 stable across changes.

 2. Again for efficient automated testing and flakiness 
 detection, one should be able to run only a subset of unittests 
 by mentioning them by line in the command line. Note that this 
 implies there's no interdependency between distinct unittests, 
 which is fine because the new ability is opt-on; I'd say is 
 pure style anyway.

 3. Mentioning unittest names in failure messages helps human 
 communication (e.g. "AddPeer is failing after your change"). 
 This is impossible with file and line numbers.

 I'd like to make a DIP for named unittests. Who can help me 
 with that?


 Andrei


I've implemented this in a library and I'm sure others have as 
well. Are you sure, you want a language solution?
Mar 30 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/30/15 2:55 PM, Panke wrote:

 I've implemented this in a library and I'm sure others have as well. Are
 you sure, you want a language solution?


With attributes? That might be palatable but only as a standard 
solution. I'd want to add that to Phobos. -- Andrei
Mar 30 2015
next sibling parent "Panke" <tobias pankrath.net> writes:
On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu 
wrote:

 On 3/30/15 2:55 PM, Panke wrote:

 I've implemented this in a library and I'm sure others have as 
 well. Are
 you sure, you want a language solution?


 With attributes? That might be palatable but only as a standard 
 solution. I'd want to add that to Phobos. -- Andrei


In my case: no. I wrote it before UDA and wanted to be able to 
generate testcases at runtime. Used this to register a test case 
for a SAT solver for every instance in a subdirectory etc.
Mar 30 2015
prev sibling parent reply "tcak" <tcak gmail.com> writes:
On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu 
wrote:

 On 3/30/15 2:55 PM, Panke wrote:

 I've implemented this in a library and I'm sure others have as 
 well. Are
 you sure, you want a language solution?


 With attributes? That might be palatable but only as a standard 
 solution. I'd want to add that to Phobos. -- Andrei


The "unittest" is a language thing. Why would name support be put 
into library I don't get it.

unittest{}  => Unnamed

unittest!"Testing new classes" {}  => Named (Explained)
Mar 31 2015
parent reply "w0rp" <devw0rp gmail.com> writes:
On Tuesday, 31 March 2015 at 11:39:02 UTC, tcak wrote:

 On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu 
 wrote:

 On 3/30/15 2:55 PM, Panke wrote:

 I've implemented this in a library and I'm sure others have 
 as well. Are
 you sure, you want a language solution?


 With attributes? That might be palatable but only as a 
 standard solution. I'd want to add that to Phobos. -- Andrei


 The "unittest" is a language thing. Why would name support be 
 put into library I don't get it.

 unittest{}  => Unnamed

 unittest!"Testing new classes" {}  => Named (Explained)


Becaused adding language features takes longer than using a 
library, and every single feature, no matter how seemingly 
simple, will increase the number of langauge bugs and lead to 
more odd things happening.
Mar 31 2015
parent "tcak" <tcak gmail.com> writes:
On Tuesday, 31 March 2015 at 12:10:15 UTC, w0rp wrote:

 On Tuesday, 31 March 2015 at 11:39:02 UTC, tcak wrote:

 On Monday, 30 March 2015 at 21:57:33 UTC, Andrei Alexandrescu 
 wrote:

 On 3/30/15 2:55 PM, Panke wrote:

 I've implemented this in a library and I'm sure others have 
 as well. Are
 you sure, you want a language solution?


 With attributes? That might be palatable but only as a 
 standard solution. I'd want to add that to Phobos. -- Andrei


 The "unittest" is a language thing. Why would name support be 
 put into library I don't get it.

 unittest{}  => Unnamed

 unittest!"Testing new classes" {}  => Named (Explained)


 Becaused adding language features takes longer than using a 
 library, and every single feature, no matter how seemingly 
 simple, will increase the number of langauge bugs and lead to 
 more odd things happening.


I can understand this, and I agree with you. But this is no 
different then ordinary businesses those do not care about 
quality of their products or customer service as long as they 
make profit. And as you know, we mostly swear at them due to this.
Mar 31 2015
prev sibling parent reply "Panke" <tobias pankrath.net> writes:
 I've implemented this in a library and I'm sure others have as 
 well. Are you sure, you want a language solution?


Basic principle:

---
unittest {

     testCase("a testcase",
     {
         // actual test code here
     });
}
---

Testcase registers the structure and the runtime just runs the 
unittest as is does today and than calls all closures registered 
during the unittest. Makes this fully backward compatible.
Mar 30 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/30/15 2:59 PM, Panke wrote:

 I've implemented this in a library and I'm sure others have as well.
 Are you sure, you want a language solution?


 Basic principle:

 ---
 unittest {

      testCase("a testcase",
      {
          // actual test code here
      });
 }
 ---


Looks a bit too much work.


 Testcase registers the structure and the runtime just runs the unittest
 as is does today and than calls all closures registered during the
 unittest. Makes this fully backward compatible.


Backward compatibility is not an issue seeing this is a pure addition.


Andrei
Mar 30 2015
parent "Panke" <tobias pankrath.net> writes:
 Backward compatibility is not an issue seeing this is a pure 
 addition.


 Andrei


I mean that it lives peacefully alongside normal unittest blocks.
Mar 30 2015
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
I'd prefer putting alternative test runner into Phobos instead 
which will support ` name("Something") unittest { }`
Mar 30 2015
next sibling parent "Brian Schott" <briancschott gmail.com> writes:
On Monday, 30 March 2015 at 21:58:13 UTC, Dicebot wrote:

 I'd prefer putting alternative test runner into Phobos instead 
 which will support ` name("Something") unittest { }`


I agree that an annotation (Probably defined in object.d) is the 
best way to handle this.

Here's something I don't think many people know: You can't use 
`~` in deprecated() attributes. You can only put a string literal 
in the parenthesis. If you want to break messages over multiple 
lines you need to rely on implicit string concatenation.
Mar 30 2015
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/30/15 5:58 PM, Dicebot wrote:

 I'd prefer putting alternative test runner into Phobos instead which
 will support ` name("Something") unittest { }`


Yes, this is one of the benefits I touted 2 years ago when I asked for 
module RTInfo -- we can use this information in the runtime to 
instrument how we run unit tests.

We still don't have module RTInfo.

And yes, then it can be a library solution. unittests are a language 
feature, but only in how they are compiled and linked. The runtime is 
fully responsible for how they are run. All we need is a way to tell the 
compiler how to describe them to the runtime.

-Steve
Mar 31 2015
parent reply "w0rp" <devw0rp gmail.com> writes:
On Tuesday, 31 March 2015 at 12:33:31 UTC, Steven Schveighoffer 
wrote:

 On 3/30/15 5:58 PM, Dicebot wrote:

 I'd prefer putting alternative test runner into Phobos instead 
 which
 will support ` name("Something") unittest { }`


 Yes, this is one of the benefits I touted 2 years ago when I 
 asked for module RTInfo -- we can use this information in the 
 runtime to instrument how we run unit tests.

 We still don't have module RTInfo.

 And yes, then it can be a library solution. unittests are a 
 language feature, but only in how they are compiled and linked. 
 The runtime is fully responsible for how they are run. All we 
 need is a way to tell the compiler how to describe them to the 
 runtime.

 -Steve


ModuleInfo does actually exist, but it's not documented. I'm not 
sure if it's usable for this purpose though. Maybe?
Mar 31 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/31/15 9:05 AM, w0rp wrote:

 On Tuesday, 31 March 2015 at 12:33:31 UTC, Steven Schveighoffer wrote:

 On 3/30/15 5:58 PM, Dicebot wrote:

 I'd prefer putting alternative test runner into Phobos instead which
 will support ` name("Something") unittest { }`


 Yes, this is one of the benefits I touted 2 years ago when I asked for
 module RTInfo -- we can use this information in the runtime to
 instrument how we run unit tests.

 We still don't have module RTInfo.

 And yes, then it can be a library solution. unittests are a language
 feature, but only in how they are compiled and linked. The runtime is
 fully responsible for how they are run. All we need is a way to tell
 the compiler how to describe them to the runtime.


 ModuleInfo does actually exist, but it's not documented. I'm not sure if
 it's usable for this purpose though. Maybe?


No, I mean this: https://github.com/D-Programming-Language/dmd/pull/2271

Essentially, you have user-defined generation of runtime info stored 
inside the ModuleInfo. When this is working, we can do whatever we want 
for unit tests via attributes.

-Steve
Mar 31 2015
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/31/15 9:28 AM, Steven Schveighoffer wrote:

 On 3/31/15 9:05 AM, w0rp wrote:

 On Tuesday, 31 March 2015 at 12:33:31 UTC, Steven Schveighoffer wrote:

 On 3/30/15 5:58 PM, Dicebot wrote:

 I'd prefer putting alternative test runner into Phobos instead which
 will support ` name("Something") unittest { }`


 Yes, this is one of the benefits I touted 2 years ago when I asked for
 module RTInfo -- we can use this information in the runtime to
 instrument how we run unit tests.

 We still don't have module RTInfo.

 And yes, then it can be a library solution. unittests are a language
 feature, but only in how they are compiled and linked. The runtime is
 fully responsible for how they are run. All we need is a way to tell
 the compiler how to describe them to the runtime.


 ModuleInfo does actually exist, but it's not documented. I'm not sure if
 it's usable for this purpose though. Maybe?


 No, I mean this: https://github.com/D-Programming-Language/dmd/pull/2271

 Essentially, you have user-defined generation of runtime info stored
 inside the ModuleInfo. When this is working, we can do whatever we want
 for unit tests via attributes.


See here too:

https://issues.dlang.org/show_bug.cgi?id=10023

-Steve
Mar 31 2015
prev sibling parent reply "Meta" <jared771 gmail.com> writes:
On Monday, 30 March 2015 at 21:58:13 UTC, Dicebot wrote:

 I'd prefer putting alternative test runner into Phobos instead 
 which will support ` name("Something") unittest { }`


Aren't unittest blocks just special functions? If that's the 
case, there should be no problem being able to give them names. 
It seems to me that it would entail the lifting of a restriction 
rather than a real language change.

Before:
unittest
{
     assert(1 == 1);
}

After:
unittest checkBasicLaw
{
     assert(1 == 1);
}
Mar 31 2015
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 7:55 AM, Meta wrote:

 On Monday, 30 March 2015 at 21:58:13 UTC, Dicebot wrote:

 I'd prefer putting alternative test runner into Phobos instead which
 will support ` name("Something") unittest { }`


 Aren't unittest blocks just special functions? If that's the case, there
 should be no problem being able to give them names. It seems to me that
 it would entail the lifting of a restriction rather than a real language
 change.

 Before:
 unittest
 {
      assert(1 == 1);
 }

 After:
 unittest checkBasicLaw
 {
      assert(1 == 1);
 }


That's what I had in mind, too. -- Andrei
Mar 31 2015
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2015-03-31 16:55, Meta wrote:


 Aren't unittest blocks just special functions? If that's the case, there
 should be no problem being able to give them names. It seems to me that
 it would entail the lifting of a restriction rather than a real language
 change.

 Before:
 unittest
 {
      assert(1 == 1);
 }

 After:
 unittest checkBasicLaw
 {
      assert(1 == 1);
 }


I prefer a UDA accepting a string, this can contain spaces and it's not 
limited to identifier names.

-- 
/Jacob Carlborg
Mar 31 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 1:04 PM, Jacob Carlborg wrote:

 On 2015-03-31 16:55, Meta wrote:


 Aren't unittest blocks just special functions? If that's the case, there
 should be no problem being able to give them names. It seems to me that
 it would entail the lifting of a restriction rather than a real language
 change.

 Before:
 unittest
 {
      assert(1 == 1);
 }

 After:
 unittest checkBasicLaw
 {
      assert(1 == 1);
 }


 I prefer a UDA accepting a string, this can contain spaces and it's not
 limited to identifier names.


I used to think the same, but then I figured a bit of structure might be 
preferable. -- Andrei
Mar 31 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 31 March 2015 at 20:18:41 UTC, Andrei Alexandrescu 
wrote:

 On 3/31/15 1:04 PM, Jacob Carlborg wrote:

 On 2015-03-31 16:55, Meta wrote:


 Aren't unittest blocks just special functions? If that's the 
 case, there
 should be no problem being able to give them names. It seems 
 to me that
 it would entail the lifting of a restriction rather than a 
 real language
 change.

 Before:
 unittest
 {
     assert(1 == 1);
 }

 After:
 unittest checkBasicLaw
 {
     assert(1 == 1);
 }


 I prefer a UDA accepting a string, this can contain spaces and 
 it's not
 limited to identifier names.


 I used to think the same, but then I figured a bit of structure 
 might be preferable. -- Andrei


I see no value in test names limited to valid identifiers. It is 
only tiny bit more informative than `unittestXXX` we have 
already. If we add names, please, let them be proper names that 
are easy to read.
Mar 31 2015
parent "krzaq" <dlangmailinglist krzaq.cc> writes:
On Tuesday, 31 March 2015 at 21:24:20 UTC, Dicebot wrote:

 I see no value in test names limited to valid identifiers. It 
 is only tiny bit more informative than `unittestXXX` we have 
 already. If we add names, please, let them be proper names that 
 are easy to read.


I'd rather have the name obey the same restrictions as normal 
function names and keep additional information in optional 
 description("...")
Apr 02 2015
prev sibling parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Tuesday, 31 March 2015 at 20:04:01 UTC, Jacob Carlborg wrote:

 On 2015-03-31 16:55, Meta wrote:


 Aren't unittest blocks just special functions? If that's the 
 case, there
 should be no problem being able to give them names. It seems 
 to me that
 it would entail the lifting of a restriction rather than a 
 real language
 change.

 Before:
 unittest
 {
     assert(1 == 1);
 }

 After:
 unittest checkBasicLaw
 {
     assert(1 == 1);
 }


 I prefer a UDA accepting a string, this can contain spaces and 
 it's not limited to identifier names.


Limiting unittest names to legal identifiers will save a lot of 
headache when we set our tools to actually use these names. 
Matching a legal identifier in a text stream is much easier than 
matching an arbitrary unicode string, even if that string is 
escaped.
Mar 31 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-03-31 23:08, Idan Arye wrote:


 Limiting unittest names to legal identifiers will save a lot of headache
 when we set our tools to actually use these names. Matching a legal
 identifier in a text stream is much easier than matching an arbitrary
 unicode string, even if that string is escaped.


I don't see any problem with arbitrary strings. It works perfectly fine 
in RSpec, which has much better tooling than D.

-- 
/Jacob Carlborg
Apr 01 2015
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
Sounds like something that should be added as a library rather 
than the core language.
Mar 30 2015
prev sibling next sibling parent reply "Kapps" <opantm2+spam gmail.com> writes:
On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu 
wrote:

 I'd like to make a DIP for named unittests. Who can help me 
 with that?


 Andrei


I agree that using library-defined annotations would be a better 
approach than language changes. Currently things like tested use 
the form
 name("AddPeer") unittest { /* ... */ }
Which is nice, because then you can extend it as desired, such as
 parallel  name("AddPeer") unittest

The main issue is that at this point, practically every single 
person has defined:
struct Name {
     string val;
}
string name(string val) { return Name(val); }

Adding common attributes such as this to Phobos or druntime and 
potentially have the default unittest runner include them would 
be good.
Mar 30 2015
parent reply Mathias Lang via Digitalmars-d <digitalmars-d puremagic.com> writes:
We do have an ` name` as UDA in Vibe.d, so that'll be a breaking change
(But ` NamedUnittest("name")` will do).
I also think it should be a library solution.

2015-03-31 0:21 GMT+02:00 Kapps via Digitalmars-d <
digitalmars-d puremagic.com>:


 On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu wrote:


 I'd like to make a DIP for named unittests. Who can help me with that?


 Andrei


 I agree that using library-defined annotations would be a better approach
 than language changes. Currently things like tested use the form
  name("AddPeer") unittest { /* ... */ }
 Which is nice, because then you can extend it as desired, such as
  parallel  name("AddPeer") unittest

 The main issue is that at this point, practically every single person has
 defined:
 struct Name {
     string val;
 }
 string name(string val) { return Name(val); }

 Adding common attributes such as this to Phobos or druntime and
 potentially have the default unittest runner include them would be good.
Mar 30 2015
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 30 March 2015 at 23:02:51 UTC, Mathias Lang wrote:

 We do have an ` name` as UDA in Vibe.d, so that'll be a 
 breaking change
 (But ` NamedUnittest("name")` will do).
 I also think it should be a library solution.


I can't afford to care about used names for attributes, same as 
we still add new symbols to Phobos (though those do break user 
code for same reason). They can be disambugated using module 
system.

(and yes, this is HUGE drawback of D module system)
Mar 30 2015
prev sibling next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 31/03/2015 10:52 a.m., Andrei Alexandrescu wrote:

 We're having a strong need for named unittests at Facebook for multiple
 reasons.

 1. We have sophisticated tooling that verifies whether unittests are
 flaky. The automated monitor (for e.g. C++) figures whether a given
 unittest fails several times across several commits. Unittests are
 identified by name; relying on file/line is impossible because the line
 of a failure is not stable across changes.

 2. Again for efficient automated testing and flakiness detection, one
 should be able to run only a subset of unittests by mentioning them by
 line in the command line. Note that this implies there's no
 interdependency between distinct unittests, which is fine because the
 new ability is opt-on; I'd say is pure style anyway.

 3. Mentioning unittest names in failure messages helps human
 communication (e.g. "AddPeer is failing after your change"). This is
 impossible with file and line numbers.

 I'd like to make a DIP for named unittests. Who can help me with that?


 Andrei


/**
  * Does something funny
  *
  * Uses writeln, to tell the user off.
  * And perhaps something else too.
  */
unittest {
	import std.stdio;
	import std.ddoc;
	writeln("BAD USER, now say \"NO\"");

	assert(readln() == "NO");

	assert(ddocParse(__DDOC__).summary == "Does something funny");
}


Only one addition required. __DDOC__ to the compiler. The rest is all 
library solution. Oh and assert should probably be aware of it.

void assert(bool value, string text="", string mod = __MODULE__, uint 
line = __LINE__, string ddoc = __DDOC__);

Well along those lines for a prototype.
Mar 30 2015
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-03-30 23:52, Andrei Alexandrescu wrote:

 We're having a strong need for named unittests at Facebook for multiple
 reasons.

 1. We have sophisticated tooling that verifies whether unittests are
 flaky. The automated monitor (for e.g. C++) figures whether a given
 unittest fails several times across several commits. Unittests are
 identified by name; relying on file/line is impossible because the line
 of a failure is not stable across changes.

 2. Again for efficient automated testing and flakiness detection, one
 should be able to run only a subset of unittests by mentioning them by
 line in the command line. Note that this implies there's no
 interdependency between distinct unittests, which is fine because the
 new ability is opt-on; I'd say is pure style anyway.

 3. Mentioning unittest names in failure messages helps human
 communication (e.g. "AddPeer is failing after your change"). This is
 impossible with file and line numbers.

 I'd like to make a DIP for named unittests. Who can help me with that?


I completely agree. I always thought that the built-in unit test support 
wasn't sufficient. All of the above should be possible to implement in 
library code without needing to change the language.

Now, it depends on how far you want to go. We could do something simple 
as adding a UDA, which has already been suggested by others:

 name("this is a test") unittest
{
     assert(false);
}

Personally I would like a complete testing framework like RSpec. I have 
a very simple implementation [1] of this:

unittest
{
   describe("std.uni", {
     describe("toUpper", {
       it("converts a string to uppercase", {
         "foo".toUpper.should.eq("FOO")
       });
     });

     describe("toLower", {
       it("converts a string to lowercase", {
         "Foo".toLower.should.eq("foo")
       });
     });
   });
}

With RSpec, which supports multiple formatters, it can look like this:

---
Randomized with seed 28149

std.uni
   toUpper
     converts a string to uppercase
   toLower
     converts a string to lowercase

Finished in 0.00078 seconds (files took 0.0931 seconds to load)
2 examples, 0 failures

Randomized with seed 28149
---

This shows a failing test:

---
Randomized with seed 57730

std.uni
   toLower
     converts a string to lowercase
   toUpper
     converts a string to uppercase (FAILED - 1)

Failures:

   1) std.uni toUpper converts a string to uppercase
      Failure/Error: 'foo'.should == 'FOO'
        expected: "FOO"
             got: "foo" (using ==)


Finished in 0.00298 seconds (files took 0.08553 seconds to load)
2 examples, 1 failure

Failed examples:



Randomized with seed 57730
---

Or with the TextMate formatter [2]. It shows both passing and failing 
test. When a test fails to get a link pointing back to the editor and a 
syntax highlighted snippet of the failing source code.

With RSpec it's also possible to specify the --line-number flag which 
will only run the tests matching a given line number. Or the --example 
flag which will run all examples (tests) matching the given string.

Is any of this interesting to have in Phobos? Otherwise I'll continue 
working on my own framework.

[1] https://github.com/jacob-carlborg/dspec
[2] http://thejqr.com/2009/02/06/textmate-rspec-and-dot-spec-party.html

-- 
/Jacob Carlborg
Mar 31 2015
prev sibling next sibling parent "ezneh" <petitv.isat gmail.com> writes:
On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu 
wrote:

 We're having a strong need for named unittests at Facebook for 
 multiple reasons.

 1. We have sophisticated tooling that verifies whether 
 unittests are flaky. The automated monitor (for e.g. C++) 
 figures whether a given unittest fails several times across 
 several commits. Unittests are identified by name; relying on 
 file/line is impossible because the line of a failure is not 
 stable across changes.

 2. Again for efficient automated testing and flakiness 
 detection, one should be able to run only a subset of unittests 
 by mentioning them by line in the command line. Note that this 
 implies there's no interdependency between distinct unittests, 
 which is fine because the new ability is opt-on; I'd say is 
 pure style anyway.

 3. Mentioning unittest names in failure messages helps human 
 communication (e.g. "AddPeer is failing after your change"). 
 This is impossible with file and line numbers.

 I'd like to make a DIP for named unittests. Who can help me 
 with that?


 Andrei



Would it be doable to make something like

unittest("Say my name") {
  // tests
}

and still be backward compatible ?

In the other hand, other proposed solutions seem to be good imho.
Mar 31 2015
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Mon, 30 Mar 2015 14:52:36 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 We're having a strong need for named unittests at Facebook for
 multiple reasons.
 


As there have already been suggestions to use UDAs I think we should
discuss the fundamental difficulty with unittests:

Right now the default implementation works by putting pointers to a
test function into ModuleInfo. We could instead add arrays of some
'unittest information' struct to ModuleInfo to support names etc. But
we can't make this as extensible and powerful as it should be: In order
to support arbitrary UDAs we'd always need some kind of UDA=>runtime
serialization.

The other option is getting a list of unittests at compile time.
(__traits allMEmbers, etc). AFAIK all unittest frameworks
supporting UDA use this approach. This is much more powerful and
extensible. It might make sense to switch the default implementation.

But here's the problem:

1) The compile time approach requires some kind
   of explicit registration of the unittests. At least one mixin per
   module.
2) This mixin will usually provide a module constructor. But
   using module constructors will cause issues with cycle detection.

There are some similar usecases with exactly the same issues (benchmark
functions, ...). So we'd need a general solution for this.


One possible solution for problem 1 would be to have mixins that
automatically get mixed in once at module scope in imported modules (a
little bit like RTInfo):

-----------------------
module benchmark;

 automodule mixin template Foo()
{
    shared static this()  nocycle
    {
        //traits allmembers, ...
    }
}
-----------------------
module test;
import benchmark; //=> automatically inserts mixin Foo;
-----------------------

To solve problem problem 2 a nocycle UDA could be used to ignore
cycle checking (or  depends("benchmark") to only list explicit
dependencies).
Mar 31 2015
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 31 March 2015 at 09:08:47 UTC, Johannes Pfau wrote:

 Am Mon, 30 Mar 2015 14:52:36 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 We're having a strong need for named unittests at Facebook for
 multiple reasons.





 Right now the default implementation works by putting pointers 
 to a
 test function into ModuleInfo. We could instead add arrays of 
 some
 'unittest information' struct to ModuleInfo to support names 
 etc. But
 we can't make this as extensible and powerful as it should be: 
 In order
 to support arbitrary UDAs we'd always need some kind of 
 UDA=>runtime
 serialization.


Most powerful solution would be to simply put attributes for 
unittest blocks in runtime information for tests (using RTTI it 
should be possible to define such variadic structure in similar 
manner as D-style variadic function arguments).


 The other option is getting a list of unittests at compile time.
 (__traits allMEmbers, etc). AFAIK all unittest frameworks
 supporting UDA use this approach. This is much more powerful and
 extensible. It might make sense to switch the default 
 implementation.

 But here's the problem:

 1) The compile time approach requires some kind
    of explicit registration of the unittests. At least one 
 mixin per
    module.
 2) This mixin will usually provide a module constructor. But
    using module constructors will cause issues with cycle 
 detection.


This is not really true. You only need one mixin in the root 
module(s), rest can be iterated recursively by test runner itself 
using __traits(allMembers) reflection. Only issue with that 
approach is that last time I checked there was a DMD bug which 
prevented from getting complete list of imported modules via 
allMembers. Should be fixable.

And module constructors are not needed at all for this, there is 
http://dlang.org/traits.html#getUnitTests
Mar 31 2015
next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 31 Mar 2015 13:31:58 +0000
schrieb "Dicebot" <public dicebot.lv>:


 But here's the problem:

 1) The compile time approach requires some kind
    of explicit registration of the unittests. At least one 
 mixin per
    module.
 2) This mixin will usually provide a module constructor. But
    using module constructors will cause issues with cycle 
 detection.


 
 This is not really true. You only need one mixin in the root 
 module(s), rest can be iterated recursively by test runner itself 
 using __traits(allMembers) reflection. Only issue with that 
 approach is that last time I checked there was a DMD bug which 
 prevented from getting complete list of imported modules via 
 allMembers. Should be fixable.


But then you still have to explicitly import (or at least name) all
modules that should be tested. This is a drawback compared to the
current builtin-unittests where you do not have to explicitly import
to-be-tested modules.

I was thinking of a fully-automated way where you only have to import a
module (could even be object-d => no explicit import required) and
have all tests run. This can be done by mixing in a module constructor
in every module. From that constructor you'd call
std.unittest.registerTest(...) and in the main function you could then
call std.unittest.runTests(). This way you never need to name or know
the tested modules.

IIRC std.benchmark used that approach, with the drawback of manual
mixins and module constructor cycle issues.


 
 And module constructors are not needed at all for this, there is 
 http://dlang.org/traits.html#getUnitTests


Sure, but I was thinking about a runtime registration scheme as
explained above.
Mar 31 2015
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 31 March 2015 at 14:13:29 UTC, Johannes Pfau wrote:

 Am Tue, 31 Mar 2015 13:31:58 +0000
 schrieb "Dicebot" <public dicebot.lv>:


 But here's the problem:

 1) The compile time approach requires some kind
    of explicit registration of the unittests. At least one 
 mixin per
    module.
 2) This mixin will usually provide a module constructor. But
    using module constructors will cause issues with cycle 
 detection.


 
 This is not really true. You only need one mixin in the root 
 module(s), rest can be iterated recursively by test runner 
 itself using __traits(allMembers) reflection. Only issue with 
 that approach is that last time I checked there was a DMD bug 
 which prevented from getting complete list of imported modules 
 via allMembers. Should be fixable.


 But then you still have to explicitly import (or at least name) 
 all
 modules that should be tested. This is a drawback compared to 
 the
 current builtin-unittests where you do not have to explicitly 
 import
 to-be-tested modules.


This is true, but importing modules by name can be code that is 
generated. Which is exactly what I do with dtest [1] and 
unit-threaded [2]. It's not that big of a deal when it's part of 
the build system.

Atila

[1] http://code.dlang.org/my_packages/dtest
[2] http://code.dlang.org/my_packages/unit-threaded


 I was thinking of a fully-automated way where you only have to 
 import a
 module (could even be object-d => no explicit import required) 
 and
 have all tests run. This can be done by mixing in a module 
 constructor
 in every module. From that constructor you'd call
 std.unittest.registerTest(...) and in the main function you 
 could then
 call std.unittest.runTests(). This way you never need to name 
 or know
 the tested modules.

 IIRC std.benchmark used that approach, with the drawback of 
 manual
 mixins and module constructor cycle issues.


 
 And module constructors are not needed at all for this, there 
 is http://dlang.org/traits.html#getUnitTests


 Sure, but I was thinking about a runtime registration scheme as
 explained above.
Mar 31 2015
parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 03/31/2015 11:01 PM, Atila Neves wrote:

 This is true, but importing modules by name can be code that is
 generated. Which is exactly what I do with dtest [1] and unit-threaded
 [2]. It's not that big of a deal when it's part of the build system.


Dub does it as well (dub test), because if you don't import a module, it
might not get linked too.
Mar 31 2015
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Tue, 31 Mar 2015 13:31:58 +0000
schrieb "Dicebot" <public dicebot.lv>:


 On Tuesday, 31 March 2015 at 09:08:47 UTC, Johannes Pfau wrote:

 Am Mon, 30 Mar 2015 14:52:36 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 We're having a strong need for named unittests at Facebook for
 multiple reasons.




 

 Right now the default implementation works by putting pointers 
 to a
 test function into ModuleInfo. We could instead add arrays of 
 some
 'unittest information' struct to ModuleInfo to support names 
 etc. But
 we can't make this as extensible and powerful as it should be: 
 In order
 to support arbitrary UDAs we'd always need some kind of 
 UDA=>runtime
 serialization.


 
 Most powerful solution would be to simply put attributes for 
 unittest blocks in runtime information for tests (using RTTI it 
 should be possible to define such variadic structure in similar 
 manner as D-style variadic function arguments).


Yes, one array of TestAttribute[]
struct TestAttribute
{
    TypeInfo ti;
    void* value;
}
per unittest and an array of unittests in ModuleInfo would likely work.

But this works only for unittests and it's still restricted*. I'd prefer
a more general solution which also works for benchmarks and similar
usecases.

*For example you can extract file/line
information (useful for IDEs) but you'll have to add explicit  fileline
UDAs to the tests. With compile-time reflection you can gather this
information without additional UDAs.
Mar 31 2015
parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 31 March 2015 at 14:27:45 UTC, Johannes Pfau wrote:

 Am Tue, 31 Mar 2015 13:31:58 +0000
 schrieb "Dicebot" <public dicebot.lv>:


 On Tuesday, 31 March 2015 at 09:08:47 UTC, Johannes Pfau wrote:

 Am Mon, 30 Mar 2015 14:52:36 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 We're having a strong need for named unittests at Facebook 
 for
 multiple reasons.




 

 Right now the default implementation works by putting 
 pointers to a
 test function into ModuleInfo. We could instead add arrays 
 of some
 'unittest information' struct to ModuleInfo to support names 
 etc. But
 we can't make this as extensible and powerful as it should 
 be: In order
 to support arbitrary UDAs we'd always need some kind of 
 UDA=>runtime
 serialization.


 
 Most powerful solution would be to simply put attributes for 
 unittest blocks in runtime information for tests (using RTTI 
 it should be possible to define such variadic structure in 
 similar manner as D-style variadic function arguments).


 Yes, one array of TestAttribute[]
 struct TestAttribute
 {
     TypeInfo ti;
     void* value;
 }
 per unittest and an array of unittests in ModuleInfo would 
 likely work.

 But this works only for unittests and it's still restricted*. 
 I'd prefer
 a more general solution which also works for benchmarks and 
 similar
 usecases.

 *For example you can extract file/line
 information (useful for IDEs) but you'll have to add explicit 
  fileline
 UDAs to the tests. With compile-time reflection you can gather 
 this
 information without additional UDAs.


Problem with more generic solution is that it is also note 
demanding and potentially more intrusive - right now UDA don't 
affect actual types at all. I'd prefer something simple and 
practical that can work for texts right now - those are special 
enough case to justify dedicated approach (even if generic rtinfo 
gets added later)
Mar 31 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-03-31 15:31, Dicebot wrote:


 Most powerful solution would be to simply put attributes for unittest
 blocks in runtime information for tests (using RTTI it should be
 possible to define such variadic structure in similar manner as D-style
 variadic function arguments).


I think the most powerful and most generic solution would be to:

1. Remove the unittest keyword
2. Make code executable at module scope
3. Add support for trailing delegate syntax

module foo;

unittest("foobar")
{
}

Would be lowered to

unittest("foobar", {

});

Works with benchmark and other things as well.

-- 
/Jacob Carlborg
Apr 01 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-03-31 11:08, Johannes Pfau wrote:


 But here's the problem:

 1) The compile time approach requires some kind
     of explicit registration of the unittests. At least one mixin per
     module.
 2) This mixin will usually provide a module constructor. But
     using module constructors will cause issues with cycle detection.


When we get RTInfo for modules [1] there shouldn't be any problems. A 
template will be instantiated once for each module the compiler sees. 
With that module __traits(getUnitTests) and __traits(allMembers) can be 
used to access the unit tests, benchmarks, or whatever. Not need for any 
registration or module constructors.

[1] https://github.com/D-Programming-Language/dmd/pull/2271

-- 
/Jacob Carlborg
Apr 01 2015
prev sibling next sibling parent reply "Idan Arye" <GenericNPC gmail.com> writes:
I understand the preference to librarize as much as possible, but 
I don't think the desire to sacrifice every possible bit of 
convenience to avoid the tiniest changes to the language is 
always beneficial. I don't say that implementing everything 
inside the compiler is good either though, but in many cases some 
slight changes to the language can make the library solution so 
much more simple and elegant.

In this case, allowing to name a unittest should be a very simple 
language change that'll make any library implementation of the 
rest of the feature more elegant to use, simpler to implement, 
and more consistent with alternative library implementations.


Another argument in favor of language solution - sometimes the 
unittests take the better part of the compilation process, 
especially when the project is a heavily-templated library(Phobos 
is a prime example). When working on a bug discovered by a 
unittest, that means you either have to either build all 
unittests every time you want to check your changes, or copy the 
unittest code a `main` function and work on it there. If unittest 
names are part of the language, it could be possible to instruct 
the compiler to only build a single unittest, which will make the 
iterations much faster.
Mar 31 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote:

 I understand the preference to librarize as much as possible, 
 but I don't think the desire to sacrifice every possible bit of 
 convenience to avoid the tiniest changes to the language is 
 always beneficial. I don't say that implementing everything 
 inside the compiler is good either though, but in many cases 
 some slight changes to the language can make the library 
 solution so much more simple and elegant.

 In this case, allowing to name a unittest should be a very 
 simple language change that'll make any library implementation 
 of the rest of the feature more elegant to use, simpler to 
 implement, and more consistent with alternative library 
 implementations.


It isn't simple at all. Name is just one of many meta-values you 
commonly want to attach to unittest block. Some others: 
description, dependency, parallelization, benchmark tag, I/O 
indicator. It is simply impossible to foresee it all in a 
language feature - but it is exactly kind of data UDA are 
designed for. All we need is to enhance/fix the language to 
actually make using of that information convenient.
Mar 31 2015
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Tuesday, 31 March 2015 at 13:34:24 UTC, Dicebot wrote:

 On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote:

 I understand the preference to librarize as much as possible, 
 but I don't think the desire to sacrifice every possible bit 
 of convenience to avoid the tiniest changes to the language is 
 always beneficial. I don't say that implementing everything 
 inside the compiler is good either though, but in many cases 
 some slight changes to the language can make the library 
 solution so much more simple and elegant.

 In this case, allowing to name a unittest should be a very 
 simple language change that'll make any library implementation 
 of the rest of the feature more elegant to use, simpler to 
 implement, and more consistent with alternative library 
 implementations.


 It isn't simple at all. Name is just one of many meta-values 
 you commonly want to attach to unittest block. Some others: 
 description, dependency, parallelization, benchmark tag, I/O 
 indicator. It is simply impossible to foresee it all in a 
 language feature - but it is exactly kind of data UDA are 
 designed for. All we need is to enhance/fix the language to 
 actually make using of that information convenient.


But unittests already have 
names(http://dpaste.dzfl.pl/b15e94000f15), so the only required 
change is to allow the user to specify that name. This should be 
much simpler than adding entirely new fields.
Mar 31 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 7:45 AM, Idan Arye wrote:

 On Tuesday, 31 March 2015 at 13:34:24 UTC, Dicebot wrote:

 On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote:

 I understand the preference to librarize as much as possible, but I
 don't think the desire to sacrifice every possible bit of convenience
 to avoid the tiniest changes to the language is always beneficial. I
 don't say that implementing everything inside the compiler is good
 either though, but in many cases some slight changes to the language
 can make the library solution so much more simple and elegant.

 In this case, allowing to name a unittest should be a very simple
 language change that'll make any library implementation of the rest
 of the feature more elegant to use, simpler to implement, and more
 consistent with alternative library implementations.


 It isn't simple at all. Name is just one of many meta-values you
 commonly want to attach to unittest block. Some others: description,
 dependency, parallelization, benchmark tag, I/O indicator. It is
 simply impossible to foresee it all in a language feature - but it is
 exactly kind of data UDA are designed for. All we need is to
 enhance/fix the language to actually make using of that information
 convenient.


 But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so
 the only required change is to allow the user to specify that name. This
 should be much simpler than adding entirely new fields.


Interesting point! -- Andrei
Mar 31 2015
parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 31/03/2015 20:21, Andrei Alexandrescu a écrit :

 On 3/31/15 7:45 AM, Idan Arye wrote:

 On Tuesday, 31 March 2015 at 13:34:24 UTC, Dicebot wrote:

 On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote:

 I understand the preference to librarize as much as possible, but I
 don't think the desire to sacrifice every possible bit of convenience
 to avoid the tiniest changes to the language is always beneficial. I
 don't say that implementing everything inside the compiler is good
 either though, but in many cases some slight changes to the language
 can make the library solution so much more simple and elegant.

 In this case, allowing to name a unittest should be a very simple
 language change that'll make any library implementation of the rest
 of the feature more elegant to use, simpler to implement, and more
 consistent with alternative library implementations.


 It isn't simple at all. Name is just one of many meta-values you
 commonly want to attach to unittest block. Some others: description,
 dependency, parallelization, benchmark tag, I/O indicator. It is
 simply impossible to foresee it all in a language feature - but it is
 exactly kind of data UDA are designed for. All we need is to
 enhance/fix the language to actually make using of that information
 convenient.


 But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so
 the only required change is to allow the user to specify that name. This
 should be much simpler than adding entirely new fields.


 Interesting point! -- Andrei


It will be nice to have named unittest.

And better if IDEs will be able to retrieve those names, to allow replay 
by name, displaying results by names,...
Mar 31 2015
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Tuesday, 31 March 2015 at 21:05:40 UTC, Xavier Bigand wrote:

 Le 31/03/2015 20:21, Andrei Alexandrescu a écrit :

 On 3/31/15 7:45 AM, Idan Arye wrote:

 On Tuesday, 31 March 2015 at 13:34:24 UTC, Dicebot wrote:

 On Tuesday, 31 March 2015 at 10:25:57 UTC, Idan Arye wrote:

 I understand the preference to librarize as much as 
 possible, but I
 don't think the desire to sacrifice every possible bit of 
 convenience
 to avoid the tiniest changes to the language is always 
 beneficial. I
 don't say that implementing everything inside the compiler 
 is good
 either though, but in many cases some slight changes to the 
 language
 can make the library solution so much more simple and 
 elegant.

 In this case, allowing to name a unittest should be a very 
 simple
 language change that'll make any library implementation of 
 the rest
 of the feature more elegant to use, simpler to implement, 
 and more
 consistent with alternative library implementations.


 It isn't simple at all. Name is just one of many meta-values 
 you
 commonly want to attach to unittest block. Some others: 
 description,
 dependency, parallelization, benchmark tag, I/O indicator. 
 It is
 simply impossible to foresee it all in a language feature - 
 but it is
 exactly kind of data UDA are designed for. All we need is to
 enhance/fix the language to actually make using of that 
 information
 convenient.


 But unittests already have 
 names(http://dpaste.dzfl.pl/b15e94000f15), so
 the only required change is to allow the user to specify that 
 name. This
 should be much simpler than adding entirely new fields.


 Interesting point! -- Andrei


 It will be nice to have named unittest.

 And better if IDEs will be able to retrieve those names, to 
 allow replay by name, displaying results by names,...


Building by unittest name! Imagine - instead of placing temporary 
code in `main` to develop a new feature or fix a bug, you put in 
a named unittest and tell your IDE/build-system to only build 
that unittest(and whatever code needed for it to run). You 
`writeln` stuff to the console, and when you get some changes to 
output what you want you change the `writeln`s to `assert`s and 
proceed to work on the next step. When you are done, all you have 
to do is tidy it up a bit and BAM - without any special effort 
you get a unittest that tests that feature/bug you just worked on.
Mar 31 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-03-31 23:14, Idan Arye wrote:


 Building by unittest name! Imagine - instead of placing temporary code
 in `main` to develop a new feature or fix a bug, you put in a named
 unittest and tell your IDE/build-system to only build that unittest(and
 whatever code needed for it to run). You `writeln` stuff to the console,
 and when you get some changes to output what you want you change the
 `writeln`s to `assert`s and proceed to work on the next step. When you
 are done, all you have to do is tidy it up a bit and BAM - without any
 special effort you get a unittest that tests that feature/bug you just
 worked on.


You just reinvented test driven development ;). It's perfectly possible 
to do this with a UDA an a text string as well. UDA's also allows you to 
tag the tests. Basically a short name you put on multiple tests, then 
tell the test runner to run only those tests, or ignore those.

 tag("foo")  name("this is my test name") unittest {}

$ run-tests -t foo

-- 
/Jacob Carlborg
Apr 01 2015
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Wednesday, 1 April 2015 at 14:05:46 UTC, Jacob Carlborg wrote:

 On 2015-03-31 23:14, Idan Arye wrote:


 Building by unittest name! Imagine - instead of placing 
 temporary code
 in `main` to develop a new feature or fix a bug, you put in a 
 named
 unittest and tell your IDE/build-system to only build that 
 unittest(and
 whatever code needed for it to run). You `writeln` stuff to 
 the console,
 and when you get some changes to output what you want you 
 change the
 `writeln`s to `assert`s and proceed to work on the next step. 
 When you
 are done, all you have to do is tidy it up a bit and BAM - 
 without any
 special effort you get a unittest that tests that feature/bug 
 you just
 worked on.


 You just reinvented test driven development ;). It's perfectly 
 possible to do this with a UDA an a text string as well. UDA's 
 also allows you to tag the tests. Basically a short name you 
 put on multiple tests, then tell the test runner to run only 
 those tests, or ignore those.

  tag("foo")  name("this is my test name") unittest {}

 $ run-tests -t foo


The problem is not with running the tests, it's with building 
them. In heavily templated libraries(like, for example Phobos), 
building without unittests takes seconds and building with 
unitetests takes minutes - mainly because the tests need to to 
many template instantiations. If we could tell the compiler to 
only build a single, specific test the development cycle can 
become orders of magnitude faster.
Apr 01 2015
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-04-01 20:35, Idan Arye wrote:


 If we could tell the compiler to only build a single, specific test the
 development cycle can become orders of magnitude faster.


There should be a lot of option to run tests:

* Base on a file
* Line number
* Name
* Tag

And so on.

-- 
/Jacob Carlborg
Apr 01 2015
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 1 April 2015 at 18:35:14 UTC, Idan Arye wrote:

 The problem is not with running the tests, it's with building 
 them. In heavily templated libraries(like, for example Phobos), 
 building without unittests takes seconds and building with 
 unitetests takes minutes - mainly because the tests need to to 
 many template instantiations. If we could tell the compiler to 
 only build a single, specific test the development cycle can 
 become orders of magnitude faster.


This is fixed by having smaller modules. If test for a single 
module takes more than few blinks of an eye, something is wrong 
already.
Apr 01 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-01 21:16, Dicebot wrote:


 This is fixed by having smaller modules. If test for a single module
 takes more than few blinks of an eye, something is wrong already.


For unit tests, yes. But there are other kinds of tests as well. 
Integration, functional, user acceptance test and so on.

-- 
/Jacob Carlborg
Apr 01 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 2 April 2015 at 06:21:53 UTC, Jacob Carlborg wrote:

 On 2015-04-01 21:16, Dicebot wrote:


 This is fixed by having smaller modules. If test for a single 
 module
 takes more than few blinks of an eye, something is wrong 
 already.


 For unit tests, yes. But there are other kinds of tests as 
 well. Integration, functional, user acceptance test and so on.


I don't think anyone is going to put those in a same inline style 
as unittest blocks, so this is not truly relevant. At least I 
hope so.
Apr 02 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-04-02 22:31, Dicebot wrote:


 I don't think anyone is going to put those in a same inline style as
 unittest blocks, so this is not truly relevant. At least I hope so.


You mean inline with the code it tests? No, I hope so too. I put my unit 
tests in separate files as well, but that's just the way I prefer it.

-- 
/Jacob Carlborg
Apr 03 2015
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 31 March 2015 at 14:45:50 UTC, Idan Arye wrote:

 But unittests already have 
 names(http://dpaste.dzfl.pl/b15e94000f15), so the only required 
 change is to allow the user to specify that name. This should 
 be much simpler than adding entirely new fields.


And does not really help. Of all metadata that one may want to 
attach to a test block, names are actually _least_ important in 
practice. It will be matter of time until Andrei creates new 
topic about adding descriptions to tests or way to mark flakey 
ones. Better to invest into something that scales at least a bit.
Mar 31 2015
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Tuesday, 31 March 2015 at 21:22:09 UTC, Dicebot wrote:

 On Tuesday, 31 March 2015 at 14:45:50 UTC, Idan Arye wrote:

 But unittests already have 
 names(http://dpaste.dzfl.pl/b15e94000f15), so the only 
 required change is to allow the user to specify that name. 
 This should be much simpler than adding entirely new fields.


 And does not really help. Of all metadata that one may want to 
 attach to a test block, names are actually _least_ important in 
 practice. It will be matter of time until Andrei creates new 
 topic about adding descriptions to tests or way to mark flakey 
 ones. Better to invest into something that scales at least a 
 bit.


I think you and I work under different assumptions of the goals 
for this feature. If we only want unittest names to be something 
that can be printed when the unittest runner runs the unittests, 
than a UDA with a string is indeed preferable. If we want 
something that tools can actually use to refer to a specific 
unittest, we need a proper identifier(yes, even though it can be 
implemented in library code because D is Turing-complete...)
Mar 31 2015
parent reply Jacob Carlborg <doob me.com> writes:
On 2015-04-01 00:02, Idan Arye wrote:


 I think you and I work under different assumptions of the goals for this
 feature. If we only want unittest names to be something that can be
 printed when the unittest runner runs the unittests, than a UDA with a
 string is indeed preferable. If we want something that tools can
 actually use to refer to a specific unittest, we need a proper
 identifier(yes, even though it can be implemented in library code
 because D is Turing-complete...)


The tools should, mostly, use the file and line information.

-- 
/Jacob Carlborg
Apr 01 2015
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Wednesday, 1 April 2015 at 14:07:14 UTC, Jacob Carlborg wrote:

 On 2015-04-01 00:02, Idan Arye wrote:


 I think you and I work under different assumptions of the 
 goals for this
 feature. If we only want unittest names to be something that 
 can be
 printed when the unittest runner runs the unittests, than a 
 UDA with a
 string is indeed preferable. If we want something that tools 
 can
 actually use to refer to a specific unittest, we need a proper
 identifier(yes, even though it can be implemented in library 
 code
 because D is Turing-complete...)


 The tools should, mostly, use the file and line information.


So now we just need a way to keep a line number unmodified when 
you add lines above it...
Apr 01 2015
parent Jacob Carlborg <doob me.com> writes:
On 2015-04-01 20:49, Idan Arye wrote:


 So now we just need a way to keep a line number unmodified when you add
 lines above it...


I'm not sure what kind of tools you have in mind.

* Print a link which points back to a failing test, requires file and 
line information

* Running a specific test, requires file and line information, name or a tag

What else do you have in mind?

-- 
/Jacob Carlborg
Apr 01 2015
prev sibling next sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 03/31/2015 04:45 PM, Idan Arye wrote:

 
 But unittests already have names(http://dpaste.dzfl.pl/b15e94000f15), so
 the only required change is to allow the user to specify that name. This
 should be much simpler than adding entirely new fields.


That's the line number, which can't be used as the OP pointed out.
Mar 31 2015
prev sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tuesday, March 31, 2015 14:45:49 Idan Arye via Digitalmars-d wrote:

 But unittests already have
 names(http://dpaste.dzfl.pl/b15e94000f15), so the only required
 change is to allow the user to specify that name. This should be
 much simpler than adding entirely new fields.


Exactly what I was thinking. If we just made

unittest myTest
{
}

or

unittest(myTest)
{
}

legal, then the unit test function would then be named myTest instead of
whatever it's already named right now. It's a lot less ugly than using
attirbutes, and it fits really well into what we already have. Certainly,
it seems like a very straightforward and simple solution to the problem.

- Jonathan M Davis
Mar 31 2015
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
I already have a library that finds all unittest blocks (as well 
as its own) and names them by appending a monotonically 
increasing integer to mymodulename.unittest. I've been thinking 
of using a UDA to decorate unittest blocks so that they show up 
with that instead but never got around to it because I don't 
really use unittest blocks and nobody is asking for the feature. 
It'd take 30min to add though.

Obviously 1st class support would be preferable, I like the

unittest nameGoesHere {
    //...
}

that was mentioned later in the thread.

Atila

On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu 
wrote:

 We're having a strong need for named unittests at Facebook for 
 multiple reasons.

 1. We have sophisticated tooling that verifies whether 
 unittests are flaky. The automated monitor (for e.g. C++) 
 figures whether a given unittest fails several times across 
 several commits. Unittests are identified by name; relying on 
 file/line is impossible because the line of a failure is not 
 stable across changes.

 2. Again for efficient automated testing and flakiness 
 detection, one should be able to run only a subset of unittests 
 by mentioning them by line in the command line. Note that this 
 implies there's no interdependency between distinct unittests, 
 which is fine because the new ability is opt-on; I'd say is 
 pure style anyway.

 3. Mentioning unittest names in failure messages helps human 
 communication (e.g. "AddPeer is failing after your change"). 
 This is impossible with file and line numbers.

 I'd like to make a DIP for named unittests. Who can help me 
 with that?


 Andrei
Mar 31 2015
prev sibling next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:

 I'd like to make a DIP for named unittests. Who can help me with that?


Why a DIP, isn't the only question what syntax to use.

unittest (myname) vs. unittest ("mynameexpression")


 Andrei


There is an ER and an half of an implementation.

https://issues.dlang.org/show_bug.cgi?id=4747
https://github.com/D-Programming-Language/dmd/pull/1131
https://github.com/D-Programming-Language/druntime/pull/308
Mar 31 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 2:50 PM, Martin Nowak wrote:

 On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:

 I'd like to make a DIP for named unittests. Who can help me with that?


 Why a DIP, isn't the only question what syntax to use.

 unittest (myname) vs. unittest ("mynameexpression")


Also unittest myname and unittest "mynameexpression". There will be no 
shortage of folks willing to debate this to smithereens.


 Andrei


 There is an ER and an half of an implementation.

 https://issues.dlang.org/show_bug.cgi?id=4747
 https://github.com/D-Programming-Language/dmd/pull/1131
 https://github.com/D-Programming-Language/druntime/pull/308


Nice, I'll check that out.


Andrei
Mar 31 2015
parent David Gileadi <gileadis NSPMgmail.com> writes:
On 3/31/15 3:28 PM, Andrei Alexandrescu wrote:

 On 3/31/15 2:50 PM, Martin Nowak wrote:

 On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:

 I'd like to make a DIP for named unittests. Who can help me with that?


 Why a DIP, isn't the only question what syntax to use.

 unittest (myname) vs. unittest ("mynameexpression")


 Also unittest myname and unittest "mynameexpression". There will be no
 shortage of folks willing to debate this to smithereens.


One argument for myname (using standard identifier naming rules) is that 
it's command-line friendly: it won't require quoting when invoking a 
single unit test, and tends toward shorter names. It may also be more 
obvious that standard identifier names would use the existing lookup 
rules for guaranteeing uniqueness among unittest names.

One argument for "mynameexpression" is that it allows for nice 
descriptions (e.g. unittest "count should accept a custom comparator 
function").

One compromise might be something like:
      description("count should accept a custom comparator function")
     unittest countCustomComparator
Under this example the unittest could be invoked from the command line 
by the countCustomComparator name or possibly by the description string 
as an alternative.

For the very little it's worth my bikeshed is painted without 
parentheses, assuming that's easily implemented.
Mar 31 2015
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 31 March 2015 at 21:50:56 UTC, Martin Nowak wrote:

 On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:

 I'd like to make a DIP for named unittests. Who can help me 
 with that?


 Why a DIP, isn't the only question what syntax to use.

 unittest (myname) vs. unittest ("mynameexpression")


 Andrei


 There is an ER and an half of an implementation.

 https://issues.dlang.org/show_bug.cgi?id=4747
 https://github.com/D-Programming-Language/dmd/pull/1131
 https://github.com/D-Programming-Language/druntime/pull/308


So now we are going to change the language for this ?

There is a natural name for unitests, the name of the module. We 
have way to break module into pieces in a backward compatible 
manner now, so it's all good.

We may want to add various annotation to a test, and we have UDA 
for that.

So the only things that is really needed is a way to customize 
the test runner from client code to output whatever everybody 
needs.
Mar 31 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 3:58 PM, deadalnix wrote:

 On Tuesday, 31 March 2015 at 21:50:56 UTC, Martin Nowak wrote:

 On 03/30/2015 11:52 PM, Andrei Alexandrescu wrote:

 I'd like to make a DIP for named unittests. Who can help me with that?


 Why a DIP, isn't the only question what syntax to use.

 unittest (myname) vs. unittest ("mynameexpression")


 Andrei


 There is an ER and an half of an implementation.

 https://issues.dlang.org/show_bug.cgi?id=4747
 https://github.com/D-Programming-Language/dmd/pull/1131
 https://github.com/D-Programming-Language/druntime/pull/308


 So now we are going to change the language for this ?

 There is a natural name for unitests, the name of the module. We have
 way to break module into pieces in a backward compatible manner now, so
 it's all good.

 We may want to add various annotation to a test, and we have UDA for that.

 So the only things that is really needed is a way to customize the test
 runner from client code to output whatever everybody needs.


The reasonable course is to see how far we can get with a library-only 
solution. Amaury, want to work on that? -- Andrei
Mar 31 2015
next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote:

 The reasonable course is to see how far we can get with a library-only
 solution. Amaury, want to work on that? -- Andrei


In any case you should talk to Atila Neves who wrote a really good
unittest library.

http://code.dlang.org/packages/unit-threaded
http://code.dlang.org/packages/dtest

It's also used by quite a lot of people already.
http://code.dlang.org/api/packages/unit-threaded/stats
Mar 31 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 4:38 PM, Martin Nowak wrote:

 On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote:

 The reasonable course is to see how far we can get with a library-only
 solution. Amaury, want to work on that? -- Andrei


 In any case you should talk to Atila Neves who wrote a really good
 unittest library.

 http://code.dlang.org/packages/unit-threaded
 http://code.dlang.org/packages/dtest

 It's also used by quite a lot of people already.
 http://code.dlang.org/api/packages/unit-threaded/stats


Atila, is there interest in moving your library to std.experimental? -- 
Andrei
Mar 31 2015
next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Tuesday, 31 March 2015 at 23:53:29 UTC, Andrei Alexandrescu 
wrote:

 On 3/31/15 4:38 PM, Martin Nowak wrote:

 On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote:

 The reasonable course is to see how far we can get with a 
 library-only
 solution. Amaury, want to work on that? -- Andrei


 In any case you should talk to Atila Neves who wrote a really 
 good
 unittest library.

 http://code.dlang.org/packages/unit-threaded
 http://code.dlang.org/packages/dtest

 It's also used by quite a lot of people already.
 http://code.dlang.org/api/packages/unit-threaded/stats


 Atila, is there interest in moving your library to 
 std.experimental? -- Andrei


Sure, I can have the PR ready by next week. As mentioned earlier, 
a UDA to attach a name to a unittest block and then run with that 
name in the output is 30min work. Making the whole thing 
std.experimental ready will take a few days.

TBH, if I'd known it had a chance of making it in there, I'd've 
done it ages ago!

Atila
Apr 01 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/1/15 12:46 AM, Atila Neves wrote:

 On Tuesday, 31 March 2015 at 23:53:29 UTC, Andrei Alexandrescu wrote:

 On 3/31/15 4:38 PM, Martin Nowak wrote:

 On 04/01/2015 01:11 AM, Andrei Alexandrescu wrote:

 The reasonable course is to see how far we can get with a library-only
 solution. Amaury, want to work on that? -- Andrei


 In any case you should talk to Atila Neves who wrote a really good
 unittest library.

 http://code.dlang.org/packages/unit-threaded
 http://code.dlang.org/packages/dtest

 It's also used by quite a lot of people already.
 http://code.dlang.org/api/packages/unit-threaded/stats


 Atila, is there interest in moving your library to std.experimental?
 -- Andrei


 Sure, I can have the PR ready by next week. As mentioned earlier, a UDA
 to attach a name to a unittest block and then run with that name in the
 output is 30min work. Making the whole thing std.experimental ready will
 take a few days.

 TBH, if I'd known it had a chance of making it in there, I'd've done it
 ages ago!

 Atila


Sounds like a plan. Thanks in advance! -- Andrei
Apr 01 2015
prev sibling parent "Robert burner Schadek" <rburners gmail.com> writes:
I have PR 
https://github.com/D-Programming-Language/phobos/pull/2995 open 
since October 2014.

it includes:
* extensible haskell like quickcheck
* benchmarking with "names" and record keeping to see progress
* offline tool to plot benchmark results with gnuplot
* most std.string functions already have benchmarks
* makefiles got a new BUILD=benchmark

IMO continuous benchmarking of phobos is a must-have feature and 
a nice marketing tool ( benchmark graphs on dlang.org). Something 
similar to quickcheck is equally important, to get the test input 
you didn't expect.

I got one and a half comments so far.
Apr 01 2015
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 31 March 2015 at 23:11:46 UTC, Andrei Alexandrescu 
wrote:

 The reasonable course is to see how far we can get with a 
 library-only solution. Amaury, want to work on that? -- Andrei


If nobody picks it up, I can, but I have very little bandwidth 
right now with the work involved on SDC.
Mar 31 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 3/31/15 5:58 PM, deadalnix wrote:

 On Tuesday, 31 March 2015 at 23:11:46 UTC, Andrei Alexandrescu wrote:

 The reasonable course is to see how far we can get with a library-only
 solution. Amaury, want to work on that? -- Andrei


 If nobody picks it up, I can, but I have very little bandwidth right now
 with the work involved on SDC.


I'll quote that when you post a long rant here :o). -- Andrei
Mar 31 2015
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/01/2015 12:58 AM, deadalnix wrote:

 So now we are going to change the language for this ?
 
 There is a natural name for unitests, the name of the module. We have
 way to break module into pieces in a backward compatible manner now, so
 it's all good.


Are you saying one should split off unittests into submodules?

Or are you saying
https://github.com/D-Programming-Language/druntime/blob/3656ba9469a60b4b23bb4a3cd95812b4f410b8a1/src/test_runner.d
is enough for your needs?

I think having multiple tests per module makes sense, e.g. because it's
better to parallelize the tests.


 We may want to add various annotation to a test, and we have UDA for that.


Sure use a UDA, I don't care. It seems important though, that the
compiler recognizes it and emits it as metadata to the ModuleInfo.
And if you need some compiler magic anyhow, a UDA just looks redundant.

 unittest("my test") unittest

unittest("my test")


 So the only things that is really needed is a way to customize the test
 runner from client code to output whatever everybody needs.


You mean this?
Mar 31 2015
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 31 March 2015 at 23:29:28 UTC, Martin Nowak wrote:

 On 04/01/2015 12:58 AM, deadalnix wrote:

 So now we are going to change the language for this ?
 
 There is a natural name for unitests, the name of the module. 
 We have
 way to break module into pieces in a backward compatible 
 manner now, so
 it's all good.


 Are you saying one should split off unittests into submodules?


No necessarily.

Typically, in most test framework, you have a class, and various 
method, each of them testing various aspect of what you want to 
test.

I see the class here as the module, and each of the method as a 
unitest block.


 Or are you saying
 https://github.com/D-Programming-Language/druntime/blob/3656ba9469a60b4b23bb4a3cd95812b4f410b8a1/src/test_runner.d
 is enough for your needs?

 I think having multiple tests per module makes sense, e.g. 
 because it's
 better to parallelize the tests.


See my previous comment, that is not needed. In fact i also think 
having several unitest blocks is useful.


 We may want to add various annotation to a test, and we have 
 UDA for that.


 Sure use a UDA, I don't care. It seems important though, that 
 the
 compiler recognizes it and emits it as metadata to the 
 ModuleInfo.
 And if you need some compiler magic anyhow, a UDA just looks 
 redundant.


How I see it, it should be possible for a user to specify a test 
runner template. This test runner template can do compile time 
reflection and format its output the way to wants.

We must not bake all required attribute we need to associate a 
test in the language. What if we want to associate the test with 
a team name, or give it a name, or give an id that my tools uses 
to check for tests, or whatever.

Every user will want to add various metadata, and these will 
vary. Every users will want various output, and these will vary.

It should be possible to have a test runner that use template and 
compile time reflection to make sense of the unitest block. Yes, 
there is a bit of compiler magic involved, but nothing terribly 
fancy.


 You mean this?



We need template to use compile time reflection.
Mar 31 2015
parent "Panke" <tobias pankrath.net> writes:
 We need template to use compile time reflection.


Please don't make it all compile time. It is useful to register 
tests at runtime as well. Like one tests per SAT instance in a 
given SAT benchmark.

Practically only thing we need is a standard way to register a 
test function as a unittest and associate a key-value store with 
it.
Mar 31 2015
prev sibling next sibling parent reply Shammah Chancellor <anonymous coward.com> writes:
On 2015-03-30 21:52:36 +0000, Andrei Alexandrescu said:


 We're having a strong need for named unittests at Facebook for multiple 
 reasons.
 
 1. We have sophisticated tooling that verifies whether unittests are 
 flaky. The automated monitor (for e.g. C++) figures whether a given 
 unittest fails several times across several commits. Unittests are 
 identified by name; relying on file/line is impossible because the line 
 of a failure is not stable across changes.
 
 2. Again for efficient automated testing and flakiness detection, one 
 should be able to run only a subset of unittests by mentioning them by 
 line in the command line. Note that this implies there's no 
 interdependency between distinct unittests, which is fine because the 
 new ability is opt-on; I'd say is pure style anyway.
 
 3. Mentioning unittest names in failure messages helps human 
 communication (e.g. "AddPeer is failing after your change"). This is 
 impossible with file and line numbers.
 
 I'd like to make a DIP for named unittests. Who can help me with that?
 
 
 Andrei


This is a fantastic idea -- and something I was concerned about lately.  +1

I can possibly help with a DIP, but it seems like a very 
straightforward request?

-Shammah
Mar 31 2015
parent "Idan Arye" <GenericNPC gmail.com> writes:
On Wednesday, 1 April 2015 at 04:42:33 UTC, Shammah Chancellor 
wrote:

 I can possibly help with a DIP, but it seems like a very 
 straightforward request?

 -Shammah


Considering the heated debate going on here, I'd say it's not as 
straightforward as it seems...
Apr 01 2015
prev sibling parent reply KnightMare <black80 bk.ru> writes:
On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu 
wrote:

 we need named unittests


4 years have passed. what the status of subj?
May 17 2019
next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Friday, 17 May 2019 at 17:42:16 UTC, KnightMare wrote:

 On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu 
 wrote:

 we need named unittests


 4 years have passed. what the status of subj?


I am not sure wheter adding named unittests to the language give 
that much benefit.

Projects developers want to integrate unittest results into build 
pipelines and having nice reports which shows directly the source 
code of failing tests. The luxury version is e.g. that Jenkins 
sent in case of unittest failures an email to the relevant 
developers.

I use the dub package dunit. It is small, easy, without any 
dependencies and it creates an QUnit compatible XML report which 
opens all possibilities mentioned above.

Kind regards
Andre
May 17 2019
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, May 17, 2019 at 05:42:16PM +0000, KnightMare via Digitalmars-d wrote:

 On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu wrote:

 we need named unittests


 
 4 years have passed. what the status of subj?


Use a UDA to name your unittest, then write a custom unittest runner to
invoke it by name.

There are already several alternative unittest runners on
code.dlang.org, such as unit-threaded.


T

-- 
Heads I win, tails you lose.
May 17 2019
next sibling parent KnightMare <black80 bk.ru> writes:
next all is IMO.


well in one place and only once (I am not sure only in percent 
number).

Dlang says "We have built-in unit tests!"
well, they looks like https://pasteboard.co/If9olU6.png

(1) sometimes I want to see every named test is OK - its some 
meditation thing.
compiler options (its just my suggestion):
- no unittest extension option - use as now.
- -unittest:M - prints OK for each tested module/file.
- -unittest:N - prints OK for each named test (failed tests 
prints always - that is a point of unit tests). u can use 
-unittest:MN too.
maybe use coloring too.
and I dont thought yet what to do with /SUBSYSTEM:Windows (non 
CONSOLE), maybe create console for tests. maybe such thing 
implemented already.
failed tests should be detailed after all tests.

(2) default assert message "unittest failure" for DMD or 
"Assertion failure" for LDC is shitty.
assert has checked expression already. use it as assert message. 
why not? too many string literals in EXE? ok. add this string 
literals only for version(unittest) only and use current assert 
behavior in other case.

(3) stack traces dont need in unit test. well, let say in 90% of 
running tests (I threw the dice for that number).
and we have here shitty stack trace - no useful info at all. lets 
increase number to 99.99999%.

(4) we can sum totally unit tests in module/file, isn't it?
well, lets print valid numbers for tests
as we can see I have a few blocks of tests: "first tests" is ok, 
"second tests" is failed and I have "3rd tests" that wasn't 
testes after failed second. and in any failed case we see "1/1 
unittests FAILED". are the one is string literal? "1/1" here is 
completely lying. well, I have assert( false ) in 2 unit tests.


Dlang says "We have built-in unit tests!"
lets imagine we come to buy yacht and we see "We have toilet in 
each room!" (weird, but just imagine it)
- and where is toilet? - we asked when we looked at the interior.
- do u see bucket in the corner?..
well, the bucket does its job. well, current unittest does its 
jobs.

when u dont use any unit tests u dont care how it looks like.
80% does. it a big number. I think unittest infrastructure 
should/must be written right once. it is necessary, it deserves 
it and it is worth it.

u can to say "use any unittest library from DUB".
well, lets remove buckets from rooms at all. and in each room 
(3rd-side library) will use biotoilet, japan toilet pan or 
another stuff. and now our projects depends from 3-4 unittest 
libraries even indirectly.
again: needed for 70% people things must be done once for all and 
for any case.

OFFTOPIC 1:
lets talk about C++ std::string. they are 30 years old. and still 
shitty. they needed in probably for 99% of all programs. and they 
are shitty. well, if u need just store chars - its ok, u can live 
with it. but if u need some string manipulation (trim, lower,..) 
- its shitty! (maybe something was changed since C++14+?). u(or 
somebody else) can write awesome string class/type with almost 
needed methods for all of us with using ICU BUT when another man 
brings some another 3rd library with some another string class 
(or even std::string) u will be cursed by your coworkers - adding 
interop between 2-3 types of strings... well, u just cannot use 
another type of string in reality, 3rd libraries used default 
std::string. u have only one(?) exception - u writing ur 
application with Qt with QString and use 3rd libraries for Qt too.
just add extension methods to C++ and add ICU to std::string 
class - C++ strings will be great for all of us.

OFFTOPIC 2:
things that needed for 70% people should/must be written very 
well once in one place.
if u dont use UDA u dont care it. if u use UDA most probably u 
will add some UDAs into project.
lets add  name,  description,  tag,  default(Variant) to Phobos.
why?
somebody wrote awesome JSON serialization library that use or 
field names or some UDAs. and we need another BINARY 
serialization library from another company with another UDAs. 
both UDAs are almost same - name, description... and this is pain 
in ass - override, rename, wrap, interop one library types to 
another. ok, we will not use UDAs for such things, lets library 
use field names.. its same like "we will not use string, we will 
use const char* and we solve the problem"

again:
what is needed at least 70% of people must be done very well.
- Dlang' unittest should be brilliant. they needed for 80+% 
developers. shouldn't be used 3rd libraries for compiler with 
native supports of unit tests. 3rd libraries brings dunit, 
unit-threaded or something else. well, this means Dlang haven't 
built-in unit tests, just the bucket in the corner. change it.
- C++ strings should be brilliant. they needed for 95+% 
C++-developers. compare QString and std::string. I want to cry 
with std::string.
- add some useful UDAs holding in a head XML/JSON. u will make 
friends a bunch of libraries with same UDAs. next UDAs -  name, 
 description,  tag,  defaultValue - will be helpful. add some 
another to this list. and add it to Phobos. dont need bring same 
UDAs from 3rd libraries.
May 17 2019
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 5/17/19 2:17 PM, H. S. Teoh wrote:

 On Fri, May 17, 2019 at 05:42:16PM +0000, KnightMare via Digitalmars-d wrote:

 On Monday, 30 March 2015 at 21:52:35 UTC, Andrei Alexandrescu wrote:

 we need named unittests


 4 years have passed. what the status of subj?


 
 Use a UDA to name your unittest, then write a custom unittest runner to
 invoke it by name.
 
 There are already several alternative unittest runners on
 code.dlang.org, such as unit-threaded.


It would be great if the default test runner would, in case of failure, 
print the file and line of the failing unittest. If there are string 
UDAs associated with the unittest, those should be printed too. 
Something like:

#line 42
 ("Is this a pigeon?") unittest
{
     assert(0);
}

would print out something like:

program.d(42): failed unittest "Is this a pigeon?"
program.d(44): Error: core.exception.AssertError: Assertion failure

Currently, we print:

core.exception.AssertError onlineapp.d(103): Assertion failure
----------------
??:? _d_assertp [0x5648d4103839]
onlineapp.d:103 _Dmain [0x5648d410252a]

which is terrible, and gratuitously so. This is because the format of 
the file/line information is not compatible with the format of 
compile-time errors, so people cannot use a variety of tools (IDEs, 
emacs/vim modes, editor plug-ins) to quickly jump to the offending line. 
And that should be the case, because what are unittests if not a natural 
extrapolation of compile-time diagnostics? We should - must! - market 
unittests consistently like that: really an extension of what the 
compiler can check. (All that nonsense with running unittests and then 
the application should go too, though there's scripts relying on that.)

Not to mention the whole thing with stopping all unittesting once one 
unittest failed. Does the compiler stop at the first error? No? Then 
running tests should not, either.

Again: the entire unittest workflow should be designed, handled, and 
marketed as an extension of the semantic checking process. The fact that 
it's done after code generation is a minor detail.

This matter bubbles up with some frequency to the top of our community's 
consciousness. Yet there's always something that prevents it getting 
fixed. Like that curse in Eastern European mythology with the builders 
who work on a church all day but then the walls fall at night.

People went off and created their own test runners, which is very nice, 
but there's a word to be said about choosing good defaults.
May 17 2019
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Also: Failing assertions at unittest level must NOT print a stacktrace. 
It's useless - the stack frame above the unittest is only internal 
function calls and stuff.
May 17 2019
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
https://github.com/dlang/druntime/pull/2611
May 17 2019
parent Johannes Loher <johannes.loher fg4f.de> writes:
Am 18.05.19 um 01:31 schrieb Andrei Alexandrescu:

 https://github.com/dlang/druntime/pull/2611


This is a great improvement. Thanks Andrei!
May 18 2019
prev sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Friday, 17 May 2019 at 21:56:45 UTC, Andrei Alexandrescu wrote:

 Again: the entire unittest workflow should be designed, 
 handled, and marketed as an extension of the semantic checking 
 process. The fact that it's done after code generation is a 
 minor detail.


I like this a lot. And I think I have a plan to make it work 
nicely with only one change to the compiler (when -unittest is 
used, the compiler automatically runs the test program before 
exiting successfully) and a few small changes to druntime (the 
tests are run only when given a particular command line flag).

D's built in unittests are easy enough you might as well write 
them, it is little effort.

This would make running them so seamless you don't even realize 
it is happening too. (heck I kinda want -unittest to become a 
default, so dmd runs it unless you specifically tell it not to. 
but that's another conversation).

i might play with this over the weekend.
May 17 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, May 18, 2019 at 02:50:52AM +0000, Adam D. Ruppe via Digitalmars-d wrote:

 On Friday, 17 May 2019 at 21:56:45 UTC, Andrei Alexandrescu wrote:

 Again: the entire unittest workflow should be designed, handled, and
 marketed as an extension of the semantic checking process. The fact
 that it's done after code generation is a minor detail.


 
 I like this a lot. And I think I have a plan to make it work nicely
 with only one change to the compiler (when -unittest is used, the
 compiler automatically runs the test program before exiting
 successfully) and a few small changes to druntime (the tests are run
 only when given a particular command line flag).


Dmd already has -run, so it's not too much of a stretch to change the
behaviour of -unittest into the equivalent of today's:

	dmd -unittest -run blah.d

Furthermore, main() shouldn't run when unittests are run, otherwise this
wouldn't make sense (compiling the program shouldn't execute main() as a
side-effect).  Perhaps some kind of hidden switch to suppress the
emission of main() so that the compiler can run the unittests after
compilation without also triggering main().  Perhaps tentatively we can
call it --DRT-run-unittests, which will start the unittest runner and
skip over calling _Dmain.



 D's built in unittests are easy enough you might as well write them,
 it is little effort.


D's built-in unittests are currently *the* reason I write any unittests
for my programs at all.  I wouldn't want to go without them.

True, one of my recent projects needed external testing, so I've written
a simplistic external test driver for this purpose (it's driven by a
bunch of files in a subdir called test/, and figures out what to do
based on what input files exist, so creating a test is a matter of
writing an input file and one or more output files containing expected
output). But for targeted functionality I still prefer the built-in
unittests, even in addition to the external test suite.



 This would make running them so seamless you don't even realize it is
 happening too. (heck I kinda want -unittest to become a default, so
 dmd runs it unless you specifically tell it not to. but that's another
 conversation).


[...]

I like the idea of -unittest by default.

One useful pattern that we could consider, that I've developed over
time, is to have the compiler compile *two* executables for the same
code, one with unittests (with no main()) and the other without (and
with main()). My build script runs both in parallel, and automatically
executes the unittest executable as part of the build. If a unittest
fails, the build aborts with an error. Otherwise, it deletes the
unittest executable, leaving the "real" one ready to run.

Of course, currently this involves running two instances of the
compiler, one with -unittest and one without; but if there was some way
to unify them, and perhaps make running the unittests automatic, that
would be *really* nice.

Then we could make this the default behaviour, and have a -no-unittest
to suppress it and revert to today's behaviour.


T

-- 
If it tastes good, it's probably bad for you.
May 17 2019
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 18 May 2019 at 05:49:00 UTC, H. S. Teoh wrote:

 Dmd already has -run, so it's not too much of a stretch to 
 change the behaviour of -unittest into the equivalent of 
 today's:

 	dmd -unittest -run blah.d


Actually, right after I went to bed, I realized the solution to 
everyone's problem.

We all want `-unittest=package,list` to control which tests are 
run. That doesn't exist yet which means it is our opportunity to 
change other things along with the new syntax.



triggers new druntime behavior


The help text might be
   -unittest         compile in unit tests
   -unittest=<pattern> run automatic unittests for <pattern> 
modules as part of the build



I think this will keep unit-threaded from experiencing any 
breakage (though I wouldn't be surprised if Atila would be 
onboard for some changes too. I actually could go a little 
further and make the built in tests more customizable, but right 
now I wanna do the simplest thing that can possibly work.)




 Perhaps tentatively we can
 call it --DRT-run-unittests, which will start the unittest 
 runner and skip over calling _Dmain.


Yes, that's what I'm thinking.

So the dmd auto run uses the --DRT-run-unittests flag which runs 
them but NOT main.

But if you don't specify that, you run main like normal, without 
tests.

And this can be done only on the new switch syntax so no legacy / 
third party runner breakage.



 Of course, currently this involves running two instances of the 
 compiler, one with -unittest and one without; but if there was 
 some way to unify them, and perhaps make running the unittests 
 automatic, that would be *really* nice.


Yes, I thought about that too, I just figured it would be easier 
today to just keep the existing plumbing for the most part.

version(unittest) existing makes it kinda hard to imagine doing 
to builds with a lot of reuse of compiler time. Well, I'm sure we 
can do some but each semantic could be different with the 
presence of that version thus making me think the compiler is 
basically run twice regardless.

but maybe that can be solved, just again I wanna go for a MVP we 
can try out in the ~2ish hours I have available this weekend to 
hack on D.
May 18 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/18/19 2:20 PM, Adam D. Ruppe wrote:

 On Saturday, 18 May 2019 at 05:49:00 UTC, H. S. Teoh wrote:

 Dmd already has -run, so it's not too much of a stretch to change the 
 behaviour of -unittest into the equivalent of today's:

     dmd -unittest -run blah.d


 
 Actually, right after I went to bed, I realized the solution to 
 everyone's problem.
 
 We all want `-unittest=package,list` to control which tests are run.


Doesn't that seem a bit much? It seems to me you either want to run 
unittests or not, why run just a few? Going with the typechecking 
metaphor - do we want to check some modules but not others?

If including/excluding certain unittests is needed e.g. for time 
reasons, "version" seems the right tool for the job. Or compiling some 
modules with -unittest and others without. But that should be the 
special case, not something supported at the command line level.


 That doesn't exist yet which means it is our opportunity to change other 
 things along with the new syntax.
 


 new druntime behavior


Neither seems to help the simple case, which is "compile, link, and run 
all unittests without running main". I'm afraid this is heading toward a 
gallop of overengineering.
May 18 2019
next sibling parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Saturday, 18 May 2019 at 14:56:16 UTC, Andrei Alexandrescu 
wrote:

 Doesn't that seem a bit much? It seems to me you either want to 
 run unittests or not, why run just a few? Going with the 
 typechecking metaphor - do we want to check some modules but 
 not others?


You almost always want to run _your_ tests not your dependencies 
tests, presumably they have already been tested and therefore do 
not need to be run.
May 18 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/18/19 4:02 PM, Nicholas Wilson wrote:

 On Saturday, 18 May 2019 at 14:56:16 UTC, Andrei Alexandrescu wrote:

 Doesn't that seem a bit much? It seems to me you either want to run 
 unittests or not, why run just a few? Going with the typechecking 
 metaphor - do we want to check some modules but not others?


 
 You almost always want to run _your_ tests not your dependencies tests, 
 presumably they have already been tested and therefore do not need to be 
 run.


Wouldn't the dependencies be built separately (i.e. with a different 
compiler invocation)?
May 18 2019
next sibling parent reply Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Saturday, 18 May 2019 at 17:52:40 UTC, Andrei Alexandrescu 
wrote:

 On 5/18/19 4:02 PM, Nicholas Wilson wrote:

 On Saturday, 18 May 2019 at 14:56:16 UTC, Andrei Alexandrescu 
 wrote:

 Doesn't that seem a bit much? It seems to me you either want 
 to run unittests or not, why run just a few? Going with the 
 typechecking metaphor - do we want to check some modules but 
 not others?


 
 You almost always want to run _your_ tests not your 
 dependencies tests, presumably they have already been tested 
 and therefore do not need to be run.


 Wouldn't the dependencies be built separately (i.e. with a 
 different compiler invocation)?


If the dependencies are templated, like large portions of Phobos, 
then no, they and their unit tests will be compiled together with 
your code.

When you import a module, the compiler adds its unit tests to the 
list of all unit tests that need to be run. At the moment the 
compiler can't know if you're importing your own module (that you 
want to test) or a third-party one (that you don't care about its 
unit tests).
May 18 2019
parent Johannes Loher <johannes.loher fg4f.de> writes:
Am 19.05.19 um 07:59 schrieb Petar Kirov [ZombineDev]:
[...]

 
 If the dependencies are templated, like large portions of Phobos, then
 no, they and their unit tests will be compiled together with your code.
 
 When you import a module, the compiler adds its unit tests to the list
 of all unit tests that need to be run. At the moment the compiler can't
 know if you're importing your own module (that you want to test) or a
 third-party one (that you don't care about its unit tests).


This is not entirely true. Only unittest blocks that are included in the
modules that are actually compiled are run. There is one nasty corner
case however. Consider the following excample:

--- main.d

import foo;
import std.stdio;

void main()
{
    writeln(add(1, 1));
}

unittest
{
    writeln("unittest in main");
    assert(add(1, 2) == 3);
}

--- foo.d

template add(T)
{
    T add(T a, T b)
    {
        return a + b;
    }

    unittest
    {
        import std.stdio : writeln;

        writeln("unittest in foo.add");
        assert(add(1, 2) == 3);
    }

}

unittest
{
    import std.stdio : writeln;

    writeln("unittest in foo");
    assert(add(1, 2) == 3);
}

When compiling and running with `dmd -unittest -run main.d`, this prints

unittest in main
unittest in foo.add
2

As you can see, the unittest block located in the template is executed,
while the one at module level in foo is not. This is because templates
are expanded at call site, so from the compilers perspective it looks
like the unittest was inside main.d, which is being compiled, so the
unittest is also compiled (for this particular instantiation of add).

There have been ideas to fix this by moving the unittests outside the
template before expanding it by modifying the AST or something like that
(I believe I hear Jonathan talk about that during DConf) but nobody has
gotten around tom implementing that yet and it also sound a bit hackish
to me...
May 18 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/18/2019 10:52 AM, Andrei Alexandrescu wrote:

 On 5/18/19 4:02 PM, Nicholas Wilson wrote:

 On Saturday, 18 May 2019 at 14:56:16 UTC, Andrei Alexandrescu wrote:

 Doesn't that seem a bit much? It seems to me you either want to run unittests 
 or not, why run just a few? Going with the typechecking metaphor - do we want 
 to check some modules but not others?


 You almost always want to run _your_ tests not your dependencies tests, 
 presumably they have already been tested and therefore do not need to be run.


 
 Wouldn't the dependencies be built separately (i.e. with a different compiler 
 invocation)?


That's right. Also, running the unittests shouldn't be a time consuming 
operation, so there shouldn't be an issue even if the dependency unittests were
run.
May 19 2019
parent Adam D. Ruppe <destructionator gmail.com> writes:
On Sunday, 19 May 2019 at 16:17:05 UTC, Walter Bright wrote:

 On 5/18/2019 10:52 AM, Andrei Alexandrescu wrote:

 Wouldn't the dependencies be built separately (i.e. with a 
 different compiler invocation)?


 That's right. Also, running the unittests shouldn't be a time 
 consuming operation, so there shouldn't be an issue even if the 
 dependency unittests were run.


These aren't necessarily true in practice. Many D builds include 
third party code in the main invocation (dub calls this a 
"sourceLibrary" target type) and sometimes unittests are awfully 
slow.

But that said, I think we can run with it as a pilot program and 
see how it works, then review if the real world data confirms or 
debunks our ideas. Might have to reshuffle some unit tests with 
templates but that might be a good idea anyway to get more 
guarantees out of our code.
May 19 2019
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, May 18, 2019 at 10:56:16AM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 5/18/19 2:20 PM, Adam D. Ruppe wrote:


[...]

 Actually, right after I went to bed, I realized the solution to
 everyone's problem.
 
 We all want `-unittest=package,list` to control which tests are run.


 
 Doesn't that seem a bit much? It seems to me you either want to run
 unittests or not, why run just a few? Going with the typechecking
 metaphor - do we want to check some modules but not others?


This is actually one of the big warts in D's otherwise very nice
built-in unittests: when you compile with -unittest, it compiles ALL
unittests, including those of 3rd party libraries that you really don't
care about.  You *really* want to be able to say "hey, compile *my*
unittests, which are in modules X, Y, Z, and don't bother with
everything else, 'cos they are external code that have already been
unittested to death".  I'm really *not* interested in running Phobos
unittests for the 100th time which the CI's have already run 1000 times
over, every single darned time I compile my own code!



 If including/excluding certain unittests is needed e.g. for time
 reasons, "version" seems the right tool for the job. Or compiling some
 modules with -unittest and others without. But that should be the
 special case, not something supported at the command line level.


[...]

Wrong. Compiling with -unittest currently compiles unittests for ALL
imports.  Meaning, (practically) all Phobos unittests get compiled and
run as soon as you have the audacity to `import std.xyz` (Phobos being
the tangled hairball of dependencies that it is right now).  Using
"version" is really not practical since I'd have to modify Phobos code
everytime I compile my own code and/or have knowledge of Phobos internal
details such as version identifiers to turn unittests on/off.

This isn't just Phobos, of course, but any 3rd party library the code
may depend on. I shouldn't have to know secret magic version identifiers
for every single 3rd party library I use just so I can turn off their
unittests!

Far from being the special case, this should be the *default* behaviour:
nobody wants to recompile and rerun 3rd party unittests for the 100th
time which the upstream authors should already have done; they want to
unittest their *own* code.


T

-- 
Too many people have open minds but closed eyes.
May 18 2019
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/18/19 4:32 PM, H. S. Teoh wrote:

 This is actually one of the big warts in D's otherwise very nice
 built-in unittests: when you compile with -unittest, it compiles ALL
 unittests, including those of 3rd party libraries that you really don't
 care about.  You*really*  want to be able to say "hey, compile*my*
 unittests, which are in modules X, Y, Z, and don't bother with
 everything else, 'cos they are external code that have already been
 unittested to death".  I'm really*not*  interested in running Phobos
 unittests for the 100th time which the CI's have already run 1000 times
 over, every single darned time I compile my own code!


Wait, if you build a program with -unittest will it run some/all of 
phobos' unittests? That would be indeed undesirable!

I wonder how often people compile external libraries together with the 
application within the same command line.
May 18 2019
parent reply Johannes Pfau <nospam example.com> writes:
Am Sat, 18 May 2019 13:55:39 -0400 schrieb Andrei Alexandrescu:


 
 Wait, if you build a program with -unittest will it run some/all of
 phobos' unittests? That would be indeed undesirable!
 
 I wonder how often people compile external libraries together with the
 application within the same command line.


It runs some of the phobos unittests, those which are in templates 
instantiated in user code:
-------------------------------------------------
module phobos;
struct Foo(T)
{
    unittest {import std.stdio; writeln("Test");}
}

module user;
import phobos;
void main(){Foo!int a;}
-------------------------------------------------
dmd -unittest user.d

If you remove the Foo template parameter, the test will no longer run. 
The reason for this behavior is simple: We can only run the test for 
template instances, not for declarations.

The compiler could easily check whether the template declaration of a 
unittest in a template instance is part of the root modules (modules 
present on DMD command line) and skip other tests. However, this will 
break the test patterns where people intentionally instantiate templates 
in other modules. Tests in the same module do not cover visibility issues 
and especially if we ever get proper DLL support for windows, where we 
would have to mark functions explicitly as export, you really do want to 
test in another module, likely even in another library.




I think the biggest problem with D's unittest implementation is that it 
isn't built on small, orthogonal and composable features. Instead we 
collect all unittests in a module, generate a function which calls them 
one by one and the add a pointer to that function to ModuleInfo. This 
implementation is highly specific and very inflexible: With the current 
framework, we simply cannot continue running tests after one failed, as 
the compiler essentially concatenated all of them into one function.

I posted a DMD PR which allows running tests individually seven/five 
years ago:
https://github.com/dlang/dmd/pull/1131 https://github.com/dlang/dmd/pull/
3518
Back then this was shot down on details whether we should allow to 
disable unittests and whether we should add file/line information. With 
the current implementation (storing information in ModuleInfo), this is 
something we as compiler developers have to decide on. But all these 
issues are detail decisions, which should really be left to unittest 
library/runner authors. For the core language/compiler we should rather 
provide reusable building blocks which are flexible enough to allow 
different library implementations instead of a highly specific, 
inflexible implementation.



So why do we have this specific compiler implementation? What does it do 
that library code can't do? Basically it does two things:

1) Test discovery: Decide which tests should be run and collect all of 
them. The selection criteria is essentially this: Unittests in files 
passed to dmd like this: 'dmd -unittest file.d' are selected. Also some 
tests in other modules because of the template problem. 

2) Test registration: For each discovered tests, make it somehow possible 
for the runtime to run it.


Now we do have library test runners*. They use compile time reflection to 
find tests, build on UDAs and other well-known language features and I 
really think this is the way to go. In order to switch druntime to such 
an implementation, we would need to reach feature parity with the current 
implementation though. The main problem here is in 1), as reflection 
based runners require you to somehow list all modules which should be 
tested.
I do not think we want to mess with ModuleInfo here, as this always 
involves serializing compile-time type information to runtime 
information. We want a completely compile time solution here. So what we 
need is some way for a library to reflect on every other application 
module automatically. I'd propose to leverage template mixins for this 
and extend them in one crucial point. Template mixins almost do what we 
want:


-------------------------------------------------
module test;

void registerTest(string name)
{
    import std.stdio;
    writeln(name);
}

mixin template registerMixin(string name)
{
    pragma(crt_constructor)
    extern(C) void registerUnittest()
    //FIXME: Need a unique mangle for this function
    {
        mixin("alias members = __traits(allMembers, " ~ name ~ ");");
        foreach(member; members)
            registerTest(member.stringof);
    }
}
-------------------------------------------------
module user;
import test;

mixin registerMixin!"user";

void main() {}
-------------------------------------------------
dmd main.d test.d


The only problem here is that we have to manually invoke `mixin 
registerMixin!"user";` in every module we want to test. Now what if we add 
`import mixins`? Simply change the `registerMixin` signature to `import 
mixin template registerMixin(string name)` and whenever the module is 
imported, the compiler automatically inserts the mixin line into the 
importing module. Essentially an import mixin template is a template 
automatically mixed in into importing modules and able to reflect on 
these modules. This small addition now allows us to use all the compile 
time, template based reflection stuff and have automatically registered 
tests. As this feature is generic we could also use it to auto-register 
 benchmark(UDA) functions. Or automatically generate serialization stuff. 
Or generate runtime typeinfo. Or .... And every library author can do the 
same.

I realize this feature is a bit controversial, as it essentially allows 
library authors to silently hook code into user modules. OTOH I think 
this is a very orthogonal feature enabling many new idioms to be explored.



2) is then easily solved, we already have crt_constructor. You could also 
place pointers into a named section for low level targets and there may 
be other options. Registration is basically a solved problem.



For the default druntime tester, we could just add such an import mixin 
template to object.d, so it's included in every file. Then wrap the code 
in the mixin template in version(unittest) and we're mostly done. Not 
including a main function / running the main application logic would 
likely be the users responsibility then, as a D library cannot really 
prevent the runtime from calling main. The remaining problem are tests in 
templates (see *), as we can't automatically register these without 
additional compiler help. For now, the compiler could explicity emit 
registerTest calls for these and we could add a deprecation for this, 
advising to explicitly test using RegisterTests!(T).


* Note that CTFE reflection based test runners do not run tests in 
templated aggregates, so they are not affected by the template issue. 
Actually, this seems to be a very good thing: You could easily create a 
template which runs nested tests on demand using reflection on the type:
RegisterTests!(Foo!int);
You can easily place this into different libraries, modules, files, ... 
and it will always work as expected.

-- 
Johannes
May 19 2019
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/19/19 10:25 AM, Johannes Pfau wrote:

 Am Sat, 18 May 2019 13:55:39 -0400 schrieb Andrei Alexandrescu:
 

 Wait, if you build a program with -unittest will it run some/all of
 phobos' unittests? That would be indeed undesirable!

 I wonder how often people compile external libraries together with the
 application within the same command line.


 
 It runs some of the phobos unittests, those which are in templates
 instantiated in user code:


That is correct, and seems desirable. Instantiations of library 
templated types with user types should be tested during unittesting.
May 19 2019
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/19/2019 6:13 AM, Andrei Alexandrescu wrote:

 On 5/19/19 10:25 AM, Johannes Pfau wrote:

 It runs some of the phobos unittests, those which are in templates
 instantiated in user code:


 
 That is correct, and seems desirable. Instantiations of library templated
types 
 with user types should be tested during unittesting.


Which suggests a "best practices" of:

1. Unittesting the logic of the template in a unittest outside the scope of the 
template.

2. Unittests in the scope of the template should be testing the parts of the 
user types that the template needs.
May 19 2019
prev sibling parent Johannes Pfau <nospam example.com> writes:
Am Sun, 19 May 2019 09:13:21 -0400 schrieb Andrei Alexandrescu:


 On 5/19/19 10:25 AM, Johannes Pfau wrote:

 Am Sat, 18 May 2019 13:55:39 -0400 schrieb Andrei Alexandrescu:
 
 

 Wait, if you build a program with -unittest will it run some/all of
 phobos' unittests? That would be indeed undesirable!

 I wonder how often people compile external libraries together with the
 application within the same command line.


 
 It runs some of the phobos unittests, those which are in templates
 instantiated in user code:


 
 That is correct, and seems desirable. Instantiations of library
 templated types with user types should be tested during unittesting.


Most of the time that's true. However, this can get very annoying if the 
phobos unit tests fail and you can't run your own tests as the runner 
aborts after the first failed test. You also can't disable the phobos 
test, as we don't have a way to disable tests.

So if you have a flaky phobos test (depends on network, datetime 
(timezone files) or something similar) pulled into your application and 
it fails, this might prevent you from testing application code. I think 
this happened to me only once, but it was really, really annoying.

-- 
Johannes
May 21 2019
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2019-05-19 11:25, Johannes Pfau wrote:


 Now we do have library test runners*. They use compile time reflection to
 find tests, build on UDAs and other well-known language features and I
 really think this is the way to go. In order to switch druntime to such
 an implementation, we would need to reach feature parity with the current
 implementation though. The main problem here is in 1), as reflection
 based runners require you to somehow list all modules which should be
 tested.
 I do not think we want to mess with ModuleInfo here, as this always
 involves serializing compile-time type information to runtime
 information. We want a completely compile time solution here. So what we
 need is some way for a library to reflect on every other application
 module automatically. I'd propose to leverage template mixins for this
 and extend them in one crucial point. Template mixins almost do what we
 want:


Have a look at my reply in a new thread [1].

[1] https://forum.dlang.org/thread/qbs8t1$jo0$1 digitalmars.com

-- 
/Jacob Carlborg
May 19 2019
prev sibling next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 18 May 2019 at 14:56:16 UTC, Andrei Alexandrescu 
wrote:

 Doesn't that seem a bit much? It seems to me you either want to 
 run unittests or not, why run just a few?


The others already mentioned that this is a common pain point 
among D users with third party code being included, but even with 
your own code, there might be times when you'll want to take 
shortcuts for a quicker test iteration cycle. Unittests are 
supposed to be independent, after all, so no point running 
hundreds of them when you are in the middle of working on fixing 
just one.

This specific -unittest=pattern syntax has been requested a 
couple times before, and there was at least one PR for it a 
couple years ago. Here it is: 
https://github.com/dlang/dmd/pull/6375

One of the suggestions back then was to make -unittest only work 
on modules explicitly compiled in (open question: how does this 
interact with the new -i switch added in 2018, after that 2017 
PR?), but Atila said that breaks the utility 
__traits(getUnitTests). You (Andrei) at the time said that is a 
good trade off, and a few agreed.

I lean toward that being a good solution too, just Atila's 
unit-threaded is an important project that we should give careful 
consideration to when changing anything about D's unittest 
facilities.

Another possibility is to compile in all the code, and use a 
runtime switch to the program (instead of the compiler) to 
determine which ones actually run. But I'd prefer doing it on the 
compiler side because fast build times are one of D's key 
advantages. Again, very frustrating to have to wait several 
seconds each edit/build/debug iteration when you are working on 
fixing one unit test at a time.
May 18 2019
parent Johannes Loher <johannes.loher fg4f.de> writes:
Am 18.05.19 um 19:49 schrieb Adam D. Ruppe:

 [...]
 One of the suggestions back then was to make -unittest only work on
 modules explicitly compiled in (open question: how does this interact
 with the new -i switch added in 2018, after that 2017 PR?), but Atila
 said that breaks the utility __traits(getUnitTests). You (Andrei) at the
 time said that is a good trade off, and a few agreed.
 
 [...]


This is already implemented and the default behavior. See my other
recent post in this thread. The interaction with the -i switch is that
all unittests from the dependencies are compiled. This is consistent
with the description of the -i switch: "include imported modules in the
compilation".
May 18 2019
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-05-18 16:56, Andrei Alexandrescu wrote:


 Doesn't that seem a bit much? It seems to me you either want to run 
 unittests or not, why run just a few? 


It's useful when you're just focusing on a single test. No point in 
running the other tests then. The test suite can take quite a long time 
to finish. The DMD test suite takes, I don't know, 10-20 minutes or 
something (a contributor mentioned it took 30 minutes). I would also use 
the "unittest" blocks for non-unit testing as well, like functional, 
integration and end to end testing. Those kind of tests can take hours.


 Going with the typechecking 
 metaphor - do we want to check some modules but not others?


Yes, definitely. Having the compiler type check the file I'm currently 
editing in my editor as I type would be great. If it can do that without 
checking the other files, sure why not? It would be faster.


 If including/excluding certain unittests is needed e.g. for time 
 reasons, "version" seems the right tool for the job. Or compiling some 
 modules with -unittest and others without. But that should be the 
 special case, not something supported at the command line level.


No, not good enough. It should not required to edit the code when 
selecting which tests to run.

-- 
/Jacob Carlborg
May 20 2019
parent Dejan Lekic <dejan.lekic gmail.com> writes:
On Monday, 20 May 2019 at 09:32:38 UTC, Jacob Carlborg wrote:

 No, not good enough. It should not required to edit the code 
 when selecting which tests to run.


+1
This is one of the things I needed many times...
What I did in the past was to extract the function and unittest 
or unittests I currently work on in a separate module, and once I 
am done I move them to the right place.
May 20 2019
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Saturday, 18 May 2019 at 14:56:16 UTC, Andrei Alexandrescu 
wrote:

 On 5/18/19 2:20 PM, Adam D. Ruppe wrote:

 On Saturday, 18 May 2019 at 05:49:00 UTC, H. S. Teoh wrote:

 Dmd already has -run, so it's not too much of a stretch to 
 change the behaviour of -unittest into the equivalent of 
 today's:

     dmd -unittest -run blah.d


 
 Actually, right after I went to bed, I realized the solution 
 to everyone's problem.
 
 We all want `-unittest=package,list` to control which tests 
 are run.


 Doesn't that seem a bit much? It seems to me you either want to 
 run unittests or not, why run just a few? Going with the 
 typechecking metaphor - do we want to check some modules but 
 not others?


I frequently only run a few unit tests - usually the ones that 
are currently failing. To me at least it's a question of screen 
real estate and attention.
May 21 2019
next sibling parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 5/21/19 5:20 AM, Atila Neves wrote:

 On Saturday, 18 May 2019 at 14:56:16 UTC, Andrei Alexandrescu wrote:

 On 5/18/19 2:20 PM, Adam D. Ruppe wrote:

 We all want `-unittest=package,list` to control which tests are run.


 Doesn't that seem a bit much? It seems to me you either want to run 
 unittests or not, why run just a few? Going with the typechecking 
 metaphor - do we want to check some modules but not others?


 
 I frequently only run a few unit tests - usually the ones that are 
 currently failing. To me at least it's a question of screen real estate 
 and attention.
 


It can also be an issue of quick debug-edit cycles. Mysql-native's tests 
can easily take a couple minutes or so to run. If I'm debugging a 
specific failure, I absolutely don't want to waste all that time waiting 
on all those irrelevant tests, every single time, before the testrunner 
finally gets around to the test I care about. (Which is why 
mysql-native's tests now use unit-threaded).
May 21 2019
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/21/19 10:20 AM, Atila Neves wrote:

 On Saturday, 18 May 2019 at 14:56:16 UTC, Andrei Alexandrescu wrote:

 On 5/18/19 2:20 PM, Adam D. Ruppe wrote:

 On Saturday, 18 May 2019 at 05:49:00 UTC, H. S. Teoh wrote:

 Dmd already has -run, so it's not too much of a stretch to change 
 the behaviour of -unittest into the equivalent of today's:

     dmd -unittest -run blah.d


 Actually, right after I went to bed, I realized the solution to 
 everyone's problem.

 We all want `-unittest=package,list` to control which tests are run.


 Doesn't that seem a bit much? It seems to me you either want to run 
 unittests or not, why run just a few? Going with the typechecking 
 metaphor - do we want to check some modules but not others?


 
 I frequently only run a few unit tests - usually the ones that are 
 currently failing. To me at least it's a question of screen real estate 
 and attention.


That's a good argument, thanks.
May 21 2019
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/18/19 6:49 AM, H. S. Teoh wrote:

 One useful pattern that we could consider, that I've developed over
 time, is to have the compiler compile*two*  executables for the same
 code, one with unittests (with no main()) and the other without (and
 with main()). My build script runs both in parallel, and automatically
 executes the unittest executable as part of the build. If a unittest
 fails, the build aborts with an error. Otherwise, it deletes the
 unittest executable, leaving the "real" one ready to run.


This is very nice, and very close do being doable today with scripting 
without modifying the codebase being built. What's needed is that main 
is not run after unittesting.
May 18 2019
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, May 18, 2019 at 10:58:36AM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:

 On 5/18/19 6:49 AM, H. S. Teoh wrote:

 One useful pattern that we could consider, that I've developed over
 time, is to have the compiler compile*two*  executables for the same
 code, one with unittests (with no main()) and the other without (and
 with main()). My build script runs both in parallel, and
 automatically executes the unittest executable as part of the build.
 If a unittest fails, the build aborts with an error. Otherwise, it
 deletes the unittest executable, leaving the "real" one ready to
 run.


 
 This is very nice, and very close do being doable today with scripting
 without modifying the codebase being built. What's needed is that main
 is not run after unittesting.


This should be pretty easy once we implement --DRT-run-unittests to only
run unittests and skip over main().


T

-- 
If Java had true garbage collection, most programs would delete themselves upon
execution. -- Robert Sewell
May 18 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/18/2019 8:21 AM, H. S. Teoh wrote:

 This should be pretty easy once we implement --DRT-run-unittests to only
 run unittests and skip over main().


Actually, I'd just change the behavior so if any unittests are in the 
executable, they are run and main() is not. main() is run only if there are no 
unittests.

In my experience, I've never wanted to run both in the same executable.
May 19 2019
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/19/19 5:27 PM, Walter Bright wrote:

 On 5/18/2019 8:21 AM, H. S. Teoh wrote:

 This should be pretty easy once we implement --DRT-run-unittests to only
 run unittests and skip over main().


 
 Actually, I'd just change the behavior so if any unittests are in the 
 executable, they are run and main() is not. main() is run only if there 
 are no unittests.


I'd find that confusing, and difficult to even describe. "-unittest does 
not run main if there are any unittests, but if there aren't, main will 
run". Randomly running main or not in a -unittest build seems just odd. 
A -unittest build with no unittests does nothing. Just like the plain 
English sentence says. KISS and all that. -unittest means build the unit 
test executable, period.


 In my experience, I've never wanted to run both in the same executable.


Let's do it!!!
May 19 2019