www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dmd -unittest=<pattern> (same syntax as -i)

reply Timothee Cour <thelastmammoth gmail.com> writes:
would a PR for `dmd -unittest=<pattern> (same syntax as -i)` be welcome?
wouldn't that avoid all the complicatiosn with version(StdUnittest) ?
eg use case:


dmd -unittest=foo -unittest=-foo.bar -i main.d
Mar 14 2018
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Wednesday, 14 March 2018 at 21:22:01 UTC, Timothee Cour wrote:

 would a PR for `dmd -unittest=<pattern> (same syntax as -i)` be 
 welcome?


so when this came up on irc earlier (was that you?) this was the 
first thought that came to my mind. I'd support it, tho I'm no 
decision maker.
Mar 14 2018
next sibling parent Timothee Cour <thelastmammoth gmail.com> writes:
I originally proposed it here:
https://forum.dlang.org/post/mailman.3166.1517969180.9493.digitalmars-d puremagic.com
but it was buried under another thread

On Wed, Mar 14, 2018 at 3:04 PM, Adam D. Ruppe via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Wednesday, 14 March 2018 at 21:22:01 UTC, Timothee Cour wrote:

 would a PR for `dmd -unittest=<pattern> (same syntax as -i)` be welcome?



 so when this came up on irc earlier (was that you?) this was the first
 thought that came to my mind. I'd support it, tho I'm no decision maker.
Mar 14 2018
prev sibling parent reply Dejan Lekic <dejan.lekic gmail.com> writes:
On Wednesday, 14 March 2018 at 22:04:50 UTC, Adam D. Ruppe wrote:

 On Wednesday, 14 March 2018 at 21:22:01 UTC, Timothee Cour 
 wrote:

 would a PR for `dmd -unittest=<pattern> (same syntax as -i)` 
 be welcome?


 so when this came up on irc earlier (was that you?) this was 
 the first thought that came to my mind. I'd support it, tho I'm 
 no decision maker.


I guess it was me talking about it two days ago on IRC...

Almost exclusively I need to run unittests only in the module I 
currently work on and it really makes no sense to run other 
unittests at that point of time (unless I explicitly want to). I 
would even go further to say that we basically need to be able to 
run particular unittest.

What I do at the moment is that I developed this kind of practice 
to have a test_runner.d top-level module in every D project of 
mine that contains tests I do during the development of 
particular feature TDD style, and then once I am done, I move 
this code to appropriate unittest blocks in my final module...

There are bunch of issues with existing support for unittests 
that could be solved to make unit-testing a really pleasant 
activity.
Mar 16 2018
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/16/18 10:32 AM, Dejan Lekic wrote:

 On Wednesday, 14 March 2018 at 22:04:50 UTC, Adam D. Ruppe wrote:

 On Wednesday, 14 March 2018 at 21:22:01 UTC, Timothee Cour wrote:

 would a PR for `dmd -unittest=<pattern> (same syntax as -i)` be welcome?


 so when this came up on irc earlier (was that you?) this was the first 
 thought that came to my mind. I'd support it, tho I'm no decision maker.


 
 I guess it was me talking about it two days ago on IRC...
 
 Almost exclusively I need to run unittests only in the module I 
 currently work on and it really makes no sense to run other unittests at 
 that point of time (unless I explicitly want to). I would even go 
 further to say that we basically need to be able to run particular 
 unittest.
 
 What I do at the moment is that I developed this kind of practice to 
 have a test_runner.d top-level module in every D project of mine that 
 contains tests I do during the development of particular feature TDD 
 style, and then once I am done, I move this code to appropriate unittest 
 blocks in my final module...
 
 There are bunch of issues with existing support for unittests that could 
 be solved to make unit-testing a really pleasant activity.


You can do most of this via custom unit test handlers[1]. We could 
potentially add more features to unit testing in druntime, but the more 
we add, the more complex the code gets.

-Steve

[1] 
https://dlang.org/phobos/core_runtime.html#.Runtime.extendedModuleUnitTester
Mar 16 2018
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Friday, 16 March 2018 at 14:32:47 UTC, Dejan Lekic wrote:

 On Wednesday, 14 March 2018 at 22:04:50 UTC, Adam D. Ruppe 
 wrote:

 [...]


 I guess it was me talking about it two days ago on IRC...

 [...]


There are a bunch of alternative test runners on code.dlang.org. 
Obviously I prefer mine:

http://code.dlang.org/packages/unit-threaded

Atila
Mar 16 2018
prev sibling next sibling parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Wednesday, 14 March 2018 at 21:22:01 UTC, Timothee Cour wrote:

 would a PR for `dmd -unittest=<pattern> (same syntax as -i)` be 
 welcome?
 wouldn't that avoid all the complicatiosn with 
 version(StdUnittest) ?
 eg use case:


 subpackage foo.bar) dmd -unittest=foo -unittest=-foo.bar -i 
 main.d


I'm in favor. If you decide to create a PR, all the match logic 
is in mars.d at the end of the file. Might want to move it to 
another module.
Mar 14 2018
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 15, 2018 at 01:08:11AM +0000, Jonathan Marler via Digitalmars-d
wrote:

 On Wednesday, 14 March 2018 at 21:22:01 UTC, Timothee Cour wrote:

 would a PR for `dmd -unittest=<pattern> (same syntax as -i)` be
 welcome?  wouldn't that avoid all the complicatiosn with
 version(StdUnittest) ?  eg use case:
 

 foo.bar) dmd -unittest=foo -unittest=-foo.bar -i main.d


 
 I'm in favor. If you decide to create a PR, all the match logic is in
 mars.d at the end of the file. Might want to move it to another
 module.


I'm also in favor.  Having recently migrated an old C++ project (well,
actually, more like rewriting it D, and still not complete yet), I've
increasingly felt the need to selectively compile unittests for a subset
of modules and test them separately from the others. Especially when one
of the modules involves really expensive tests (basically, it constructs
objects with data from rather large files, and almost all non-trivial
tests require rather large datasets), having -unittest compile unittests
for everything (including Phobos!) is a bit too blunt a hammer. Once the
module has been thoroughly tested, it seems excessive to have to rerun
its unittests over and over again just because I need to test another
modules that depends on that module, since -unittest doesn't let me run
only the tests for the module I'm currently working on.


T

-- 
The irony is that Bill Gates claims to be making a stable operating system and
Linus Torvalds claims to be trying to take over the world. -- Anonymous
Mar 14 2018
prev sibling next sibling parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 14 March 2018 at 21:22:01 UTC, Timothee Cour wrote:

 would a PR for `dmd -unittest=<pattern> (same syntax as -i)` be 
 welcome?
 wouldn't that avoid all the complicatiosn with 
 version(StdUnittest) ?
 eg use case:


 subpackage foo.bar) dmd -unittest=foo -unittest=-foo.bar -i 
 main.d


Hmm how would this solve the StdUnittest use case? I.e. that 
templated phobos unittests and private unittest symbols are 
compiled into the users unittests?

See also:

https://github.com/dlang/phobos/pull/6202
https://github.com/dlang/phobos/pull/6159
Mar 14 2018
parent reply Jacob Carlborg <doob me.com> writes:
On Thursday, 15 March 2018 at 05:22:45 UTC, Seb wrote:


 Hmm how would this solve the StdUnittest use case? I.e. that 
 templated phobos unittests and private unittest symbols are 
 compiled into the users unittests?

 See also:

 https://github.com/dlang/phobos/pull/6202
 https://github.com/dlang/phobos/pull/6159


I would hope this would be solvable without the having the user 
do something, like `-unittest=<pattern>`.

--
/Jacob Carlborg
Mar 15 2018
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Mar 15, 2018 at 12:14:12PM +0000, Jacob Carlborg via Digitalmars-d
wrote:

 On Thursday, 15 March 2018 at 05:22:45 UTC, Seb wrote:
 

 Hmm how would this solve the StdUnittest use case? I.e. that
 templated phobos unittests and private unittest symbols are compiled
 into the users unittests?
 
 See also:
 
 https://github.com/dlang/phobos/pull/6202
 https://github.com/dlang/phobos/pull/6159


 
 I would hope this would be solvable without the having the user do
 something, like `-unittest=<pattern>`.


[...]

Even that would be preferable to the current situation.


T

-- 
Truth, Sir, is a cow which will give [skeptics] no more milk, and so they are
gone to milk the bull. -- Sam. Johnson
Mar 15 2018
prev sibling next sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Thursday, March 15, 2018 07:17:47 H. S. Teoh via Digitalmars-d wrote:

 On Thu, Mar 15, 2018 at 12:14:12PM +0000, Jacob Carlborg via Digitalmars-d 


wrote:

 On Thursday, 15 March 2018 at 05:22:45 UTC, Seb wrote:

 Hmm how would this solve the StdUnittest use case? I.e. that
 templated phobos unittests and private unittest symbols are compiled
 into the users unittests?

 See also:

 https://github.com/dlang/phobos/pull/6202
 https://github.com/dlang/phobos/pull/6159


 I would hope this would be solvable without the having the user do
 something, like `-unittest=<pattern>`.


 [...]

 Even that would be preferable to the current situation.


Being able to run easily run the unit tests for a specific module without
much effort would be nice, but as far as the overall unittest problem goes,
it doesn't really fix it, just work around it.

And if all we're looking for is a workaround, we can always use a
project-specific version identifier and version off the unit test code - and
that works right now without any compiler changes. It gives full control of
the situation and gives essentially what we really want in terms of the
resulting binary. It also ensures that projects importing you're library
don't end up with your unit test code in their project no matter how they
compile theirs (short of figuring out your project-specific version
identifier and explicitly compiling with it for some, weird reason anyway).
It's just a really sucky thing to have to do. I was forced to do it with
dxml though due to a weird linker error that projects importing dxml got
when building their unit tests.

What we really want is essentially what such a project-specific version
identifier does but without having to use a project-specific identfier.

- Jonathan M Davis
Mar 15 2018
prev sibling next sibling parent Seb <seb wilzba.ch> writes:
On Thursday, 15 March 2018 at 12:14:12 UTC, Jacob Carlborg wrote:

 On Thursday, 15 March 2018 at 05:22:45 UTC, Seb wrote:


 Hmm how would this solve the StdUnittest use case? I.e. that 
 templated phobos unittests and private unittest symbols are 
 compiled into the users unittests?

 See also:

 https://github.com/dlang/phobos/pull/6202
 https://github.com/dlang/phobos/pull/6159


 I would hope this would be solvable without the having the user 
 do something, like `-unittest=<pattern>`.

 --
 /Jacob Carlborg


... exactly my point, but I think I misunderstood the goal of 
-unittest=<pattern>. It's solely about being able to filter 
selectively for which unittests to run, which is a noble goal 
itself.

A bit of background regarding StdUnittest
-----------------------------------------

To explain the need for StdUnittest to people who haven't been 
following up on the discussion have a look at 
https://github.com/dlang/phobos/pull/5927 which introduced 
StdUnittest initially.

There are a few examples in this PR:

1) short-path version(unittest) in a symbol was done in 
production code too (see 
https://github.com/dlang/phobos/pull/5932)

2) The unittest in templated aggregates are compiled and executed 
whenever the users uses the templates (see also: 
https://wiki.dlang.org/DIP82)

3) version(unitest) symbols which are added for unittest are 
still visible and compiled into user code. Of course this should 
be fixed with `private`, but even then the compiler still looks 
at the symbol, which it wouldn't do with StdUnittest (see 
Vladimir's comment about import time 
https://github.com/dlang/phobos/pull/6159#issuecomment-365290004)

Now StdUnitest was reverted because -deps compiles ALL code and 
thus lead to breakage in downstream packages, see 
https://github.com/dlang/phobos/pull/6159 for the discussion
Mar 15 2018
prev sibling parent Jonathan Marler <johnnymarler gmail.com> writes:
On Thursday, 15 March 2018 at 12:14:12 UTC, Jacob Carlborg wrote:

 On Thursday, 15 March 2018 at 05:22:45 UTC, Seb wrote:


 Hmm how would this solve the StdUnittest use case? I.e. that 
 templated phobos unittests and private unittest symbols are 
 compiled into the users unittests?

 See also:

 https://github.com/dlang/phobos/pull/6202
 https://github.com/dlang/phobos/pull/6159


 I would hope this would be solvable without the having the user 
 do something, like `-unittest=<pattern>`.

 --
 /Jacob Carlborg


We could do the same thing for -unittest that we did with -i, 
which is to implicitly add:

-unittest=-std -unittest=-core -unittest=-etc -unittest=-obj
Mar 15 2018
prev sibling next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 3/14/18 5:22 PM, Timothee Cour wrote:

 would a PR for `dmd -unittest=<pattern> (same syntax as -i)` be welcome?
 wouldn't that avoid all the complicatiosn with version(StdUnittest) ?
 eg use case:
 

 dmd -unittest=foo -unittest=-foo.bar -i main.d
 


Note, we can do a lot with just runtime options as well. The code to run 
all the unit tests could easily be changed to deal with such patterns. 
But this only affects unit test functions inside those modules, it 
doesn't affect unit tests inside template types (or version(unittest) 
statements).

We really should have a way to control both static unittest compilation 
on a modular level, and also which modules' unit tests are run at runtime.

For example, you may actually wish to enable version(unittest) for 
module foo, so you can test using a template inside foo with a type from 
bar, but you may only then want to run bar's unittest functions (where 
the template is actually instantiated and the unittest would be run).

-Steve
Mar 15 2018
prev sibling next sibling parent =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Wednesday, 14 March 2018 at 21:22:01 UTC, Timothee Cour wrote:

 would a PR for `dmd -unittest=<pattern> (same syntax as -i)` be 
 welcome?
 wouldn't that avoid all the complicatiosn with 
 version(StdUnittest) ?
 eg use case:


 subpackage foo.bar) dmd -unittest=foo -unittest=-foo.bar -i 
 main.d


I've already tried to adding support for `-unittest=foo` this at

https://github.com/dlang/dmd/pull/6375

enabling even faster edit-save-check turnarounds when using

https://github.com/flycheck/flycheck-d-unittest/blob/master/flycheck-d-unittest.el

but it hasn't (yet) been accepted. There are quite a few people 
who have brought up this a good idea in the forums. Keep up the 
discussions and claim the importance of adding this perhaps by 
adding a note in the pull request linked above.
Mar 15 2018
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 14 Mar 2018 14:22:01 -0700 schrieb Timothee Cour:


 would a PR for `dmd -unittest=<pattern> (same syntax as -i)` be welcome?
 wouldn't that avoid all the complicatiosn with version(StdUnittest) ?
 eg use case:
 

 foo.bar)
 dmd -unittest=foo -unittest=-foo.bar -i main.d


And then we'll have to add yet another "-import" switch for DLL support. 
Now we have 3 switches doing essentially the same: Telling the compiler 
which modules are currently compiled and which modules are part of an 
external library. Instead of just using the next best simple solution, I 
think we should take a step back, think about this and design a proper, 
generic solution.

Then instead of having to use up to 6 flags(1) to link a library we can 
use one and even end up with better user experience than what we have now 
or what C++ provides:

(1)
dmd -I /libfoo -L-Bstatic -L-llibfoo -i main.d -unittest main.d -import 
libfoo*

(2)
dmd -library=foo:static

The key here is to realize that all the necessary information for 
unittests, DLL imports, library linking and automatically finding source 
dependencies is knowing the library layout. The compiler needs to know 
which modules are externally in libraries and which are currently build*.

In order to achieve this, we should define a standardized library 
metadata file which lists:
* source files belonging to that library (replaces -i and -import and -  
  unittest=)
* (relative) source path (to replace -I)
* library name (to replace -L)
* optional linker flags

Some of these are actually covered by the pkg-config format which is 
extensible and already used by meson when building D projects. Then we 
simply need the compiler to parse these files (optionally also generate 
them) and we can have a much cleaner user experience.

As these files are the extensible, you can also add documentation URIs 
for example. In the end, you could simply open your IDE, browse a list of 
libraries (even online, if we map names 1:1 to dub names), select the 
library and the IDE could provide documentation. The compiler then 
transparently handles includes, library linking, ....

I think I'll start writing a DIP for this on Sunday but given the state 
of the DIP queue it'll take some time till this will get reviewed. 


* (There actually may be some corner cases where you may want to exclude 
unittests for some modules which are actually compiled. I think version 
statements are fine for that usecase. We should try to find other special 
use cases and see whether this simple, generic approach works for all of 
them.)

-- 
Johannes
Mar 15 2018
parent reply Jonathan Marler <johnnymarler gmail.com> writes:
On Thursday, 15 March 2018 at 23:11:41 UTC, Johannes Pfau wrote:

 Am Wed, 14 Mar 2018 14:22:01 -0700 schrieb Timothee Cour:


 [...]


 And then we'll have to add yet another "-import" switch for DLL 
 support. Now we have 3 switches doing essentially the same: 
 Telling the compiler which modules are currently compiled and 
 which modules are part of an external library. Instead of just 
 using the next best simple solution, I think we should take a 
 step back, think about this and design a proper, generic 
 solution.

 [...]


I had the same idea but mine was to add this metadata in the 
library file itself instead of having it as a separate file.  
However, this design is "orthogonal" to -i= and -unittest=,  in 
both cases you may want to include/exclude certain modules 
regardless of whether or not they are in a library.
Mar 15 2018
parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 15 Mar 2018 23:21:42 +0000 schrieb Jonathan Marler:


 On Thursday, 15 March 2018 at 23:11:41 UTC, Johannes Pfau wrote:

 Am Wed, 14 Mar 2018 14:22:01 -0700 schrieb Timothee Cour:


 [...]


 And then we'll have to add yet another "-import" switch for DLL
 support. Now we have 3 switches doing essentially the same: Telling the
 compiler which modules are currently compiled and which modules are
 part of an external library. Instead of just using the next best simple
 solution, I think we should take a step back, think about this and
 design a proper, generic solution.

 [...]


 
 I had the same idea but mine was to add this metadata in the library
 file itself instead of having it as a separate file.


This is to some degree nicer, as it allows for self contained 
distribution. But then you have to support different library formats, 
it's more difficult to include support in IDEs and it's more difficult to 
extend the format.


 However, this
 design is "orthogonal" to -i= and -unittest=,  in both cases you may
 want to include/exclude certain modules regardless of whether or not
 they are in a library.


When would this be the case for -i? You never want to include modules in 
compilation which are in an external library you also link to, as you'll 
get duplicate symbol errors. I also don't see why you would want to 
exclude a module from compilation which is imported somewhere and not in 
any external library. Maybe to avoid generating the ModuleInfo and 
TypeInfo for 'header only' modules. But then you're breaking the 
assumption that typeid(X) works for any type, so excluding modules from 
compilation can't be a recommended practice.

For -unittest, I can see that you may sometimes want to test across 
library boundaries, but then you'd have to keep the tests out of the 
library anyway. I guess there could be cases where you want to instantiate 
a templated unittest for some type specializitaion, But I've never seen a 
real world use of that. Excluding local module from unittesting may be 
more useful, but I think version() and runtime selection of modules to be 
tested should cover basically all use cases.

Can you explain in some more detail what use cases you think of?

-- 
Johannes
Mar 16 2018
parent Jonathan Marler <johnnymarler gmail.com> writes:
On Friday, 16 March 2018 at 07:47:31 UTC, Johannes Pfau wrote:

 Am Thu, 15 Mar 2018 23:21:42 +0000 schrieb Jonathan Marler:


 On Thursday, 15 March 2018 at 23:11:41 UTC, Johannes Pfau 
 wrote:

 Am Wed, 14 Mar 2018 14:22:01 -0700 schrieb Timothee Cour:


 [...]


 And then we'll have to add yet another "-import" switch for 
 DLL support. Now we have 3 switches doing essentially the 
 same: Telling the compiler which modules are currently 
 compiled and which modules are part of an external library. 
 Instead of just using the next best simple solution, I think 
 we should take a step back, think about this and design a 
 proper, generic solution.

 [...]


 
 I had the same idea but mine was to add this metadata in the 
 library file itself instead of having it as a separate file.


 This is to some degree nicer, as it allows for self contained 
 distribution. But then you have to support different library 
 formats, it's more difficult to include support in IDEs and 
 it's more difficult to extend the format.


It makes the data more difficult to pull from the file but 
doesn't make it more difficult to extend.  Every library format 
supports generic "comment" type data blobs used for things like 
this.  You could provide a small library to "pull" this data blob 
from each supported library format.  You would already want to 
provide a library for the format itself so that same library 
could include the code to pull it from each library format (i.e. 
ELF/OMF/COFF).  I actually started writing a library in 
anticipation for this.  Currently it can and print the modules in 
an OMF/ELF file by finding the TypeInfo symbols.  This could be a 
fallback mechanism to use when a library didn't have any 
metadata, or even be used to "patch" a library to include 
metadata:

https://github.com/marler8997/dlangmodulereader

Of course TypeInfo goes away for code compiled with -betterC so 
it doesn't always work, hence why you'd want to add the metadata 
beforehand.


 However, this
 design is "orthogonal" to -i= and -unittest=,  in both cases 
 you may
 want to include/exclude certain modules regardless of whether 
 or not
 they are in a library.


 When would this be the case for -i?


First, if we were to add the functionality you've talked about 
(which I hope we do at some point) this would work alongside -i 
not be an alternative to it.  The new mechanism would allow us to 
ALWAYS EXCLUDE modules that exist in a pre-compiled library.  So 
we could remove the standard exclusions from -i (-i=-std -i=-core 
-i=-etc -i=-object) because those modules would already be 
excluded since they would be in phobos.  And if a program wasn't 
using phobos, then they wouldn't erroneously be excluded.  It 
just works as it should.

I agree there is no use case where you would want to compile 
modules that are in a library passed to the compiler.  However, 
it's easy to come up with use cases where you want to exclude 
imported modules from the compilation even if they aren't in any 
library passed to the compiler. In fact, if you're doing any type 
of incremental compilation, this will most certainly be the more 
common use case since libraries are only passed to the 
compiler/linker during the final link stage. For example, maybe 
you are compiling a "plugin" that will be linked to another 
program but uses a common library that you want to exclude from 
your initial compilation, call it "library_for_plugins" i.e.

--- myplugin.d
static import library_for_plugins; // DO NOT compile this module 
into the
                                    // plugin library
static import some_other_library;  // DO compile this module into 
the
                                    // plugin library

void foo()
{
     // uses symbols from both modules but that doesn't mean you 
want
     // to include all of them in compilation
     library_for_plugins.foo();
     some_other_library.bar();
}

dmd -c -od=obj -i=-library_for_plugins myplugin.d
lib -o myplugin.lib obj\*.o

Of course this is just one example I came up with on the fly. You 
could come up with any number of use cases where you would want 
this.  The point is, this is a compiler, it's job is to compile 
modules and there's alot more use cases than just "compile 
everything except what's in the libraries I've provided".
Mar 16 2018