www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Named unittests and __traits(getModules)

reply Jacob Carlborg <doob me.com> writes:
I'm staring a new thread here on the topic of Name unittests because the 
existing one is getting too long [1].

The best way to add support for named unit tests, without adding any new 
syntax to the language, is to leverage UDAs. This has already been 
discussed in the previous post [1]. The problem with UDAs and the 
current unit test runner is by the time the unit test runner runs the 
unit tests, the UDAs are long gone. They need to be fetched at 
compile-time. It's also not possible to continue running unit tests in 
the same module after a failed unit test with the current runner. This 
is because the compiler generates one function that calls all the unit 
tests in the whole module.

To solve these problems `__traits(getUnitTests)` can be used to access 
the unit tests at compile-time. This leads to the next problem. To 
access the unit tests one needs to import the modules where the unit 
tests are defined. There are currently no way to get a list of all 
modules. Existing third party unit test runners usually use a pre-build 
script that collects all files with unit tests and generates a new file 
with all the imports. This is also noted in the previous thread [2].

The solution of running a pre-build script is not workable to include in 
druntime. I purpose we add two new traits: `__traits(getRootModules)` 
and `__traits(getImportedModules)`. `getRootModules` will return all 
root modules, i.e. the files which were passed to the compiler on the 
command line. `getImportedModules` will return all imported modules, 
i.e. the non-root modules. Adding these two together will give access to 
all modules the compiler has processed. Note that these traits are 
useful for other things than building a unit test runner.

With these two traits (only getRootModules might be necessary) and using 
`static foreach` to generate the imports, `getUnitTests` can later be 
used to retrieve the unit tests at compile-time which also gives access 
to the UDAs. Using this technique to run the unit tests allows to add 
additional features, like continue running after a failed test, before 
and after callbacks and other features.

There's already a unit test runner in the DMD test suite that uses this 
technique (but with the pre-build script). With these new traits this 
unit test runner could be added pretty easily to druntime.

Thoughts?

[1] https://forum.dlang.org/thread/mfcgj3$12a0$1 digitalmars.com
[2] https://forum.dlang.org/post/qbr7dd$16fd$1 digitalmars.com

-- 
/Jacob Carlborg
May 19 2019
next sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:

 I'm staring a new thread here on the topic of Name unittests 
 because the existing one is getting too long [1].

 [...]


As far as i remember there was another suggestion of Andrei (in 
another context). By importing a module B in module A, the module 
B can specify coding which is executed and gets the module A as 
info.

This might could also solve this issue.

Kind regards
Andre
May 19 2019
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/19/19 8:13 PM, Andre Pany wrote:

 On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:

 I'm staring a new thread here on the topic of Name unittests because 
 the existing one is getting too long [1].

 [...]


 
 As far as i remember there was another suggestion of Andrei (in another 
 context). By importing a module B in module A, the module B can specify 
 coding which is executed and gets the module A as info.
 
 This might could also solve this issue.


Yah, it's quite a universal pattern. It can be used as "import 
core.rtti; to add RTTI support for this module" etc.
May 19 2019
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/19/19 11:41 PM, Andrei Alexandrescu wrote:

 On 5/19/19 8:13 PM, Andre Pany wrote:

 On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:

 I'm staring a new thread here on the topic of Name unittests because 
 the existing one is getting too long [1].

 [...]


 As far as i remember there was another suggestion of Andrei (in 
 another context). By importing a module B in module A, the module B 
 can specify coding which is executed and gets the module A as info.

 This might could also solve this issue.


 
 Yah, it's quite a universal pattern. It can be used as "import 
 core.rtti; to add RTTI support for this module" etc.


Related: https://github.com/dlang/dmd/pull/9814. There, the idea is to 
predicate an import for each language feature or groups of features. 
Many, however, can be done as templates and realized naturally by means 
of template instantiation (e.g. a[] = b[] triggers the generation of the 
array assign support function). RTTI is different because it instructs 
the compiler to generate code a specific way without a detectable action 
in the source code. That kind of feature could be done with import + 
hook "execute this on the importing module".
May 19 2019
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Monday, 20 May 2019 at 00:14:01 UTC, Andrei Alexandrescu wrote:


 RTTI is different because it instructs the compiler to generate 
 code a specific way without a detectable action in the source 
 code. That kind of feature could be done with import + hook 
 "execute this on the importing module".


I'm not sure I fully understand this, so forgive me if I'm just 
making noise, but does the pattern in this PR give you what you 
need? https://github.com/dlang/dmd/pull/7799

The idea is to put a detectable action in the source code.  In 
the case of the PR referenced above, the detectable action is the 
existence of the `class TypeInfo` declaration.  The compiler has 
been programmed to look for that declaration, and generate code 
based on whether or not it exists.  Because the declaration 
exists in druntime's source code, users can also check for its 
existence in their source code.

Basically, the compiler is doing design-by-introspection just 
like the user.  The runtime's source code informs the compiler 
what to do rather than the other way around.

So the compiler does have a detectable action in the source code 
(i.e. the existence of `class TypeInfo`) and that same detectable 
action is available to the user as well.

Mike
May 19 2019
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 5/19/19 8:55 PM, Mike Franklin wrote:

 On Monday, 20 May 2019 at 00:14:01 UTC, Andrei Alexandrescu wrote:
 

 RTTI is different because it instructs the compiler to generate code a 
 specific way without a detectable action in the source code. That kind 
 of feature could be done with import + hook "execute this on the 
 importing module".


 
 I'm not sure I fully understand this, so forgive me if I'm just making 
 noise, but does the pattern in this PR give you what you need? 
 https://github.com/dlang/dmd/pull/7799
 
 The idea is to put a detectable action in the source code.  In the case 
 of the PR referenced above, the detectable action is the existence of 
 the `class TypeInfo` declaration.  The compiler has been programmed to 
 look for that declaration, and generate code based on whether or not it 
 exists.  Because the declaration exists in druntime's source code, users 
 can also check for its existence in their source code.
 
 Basically, the compiler is doing design-by-introspection just like the 
 user.  The runtime's source code informs the compiler what to do rather 
 than the other way around.
 
 So the compiler does have a detectable action in the source code (i.e. 
 the existence of `class TypeInfo`) and that same detectable action is 
 available to the user as well.


Having the compiler detect constructs in code would work, as would 
built-in attributes such as:

 noRTTI module mymodule;

and/or

 noRTTI class BareBones { ... }

The disadvantage is that the compiler would need to be modified for each 
of these, whereas if we define and avail ourselves of powerful 
introspection, we can achieve this kind of stuff in library code.
May 20 2019
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 19 May 2019 at 22:41:52 UTC, Andrei Alexandrescu wrote:

 On 5/19/19 8:13 PM, Andre Pany wrote:

 On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:

 I'm staring a new thread here on the topic of Name unittests 
 because the existing one is getting too long [1].

 [...]


 
 As far as i remember there was another suggestion of Andrei 
 (in another context). By importing a module B in module A, the 
 module B can specify coding which is executed and gets the 
 module A as info.
 
 This might could also solve this issue.


 Yah, it's quite a universal pattern. It can be used as "import 
 core.rtti; to add RTTI support for this module" etc.


Allowing import statements to have side effects seems like a 
pretty big can of worms to open just so that we can avoid typing 
"mixin RTTISupport!()". One of the great things about D's module 
system is that a D module can't meddle around in another module's 
global scope the way a header file can in C or C++. I hope that 
guarantee won't be broken simply for the sake of syntactic 
convenience.
May 22 2019
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/22/19 5:08 PM, Paul Backus wrote:

 On Sunday, 19 May 2019 at 22:41:52 UTC, Andrei Alexandrescu wrote:

 On 5/19/19 8:13 PM, Andre Pany wrote:

 On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:

 I'm staring a new thread here on the topic of Name unittests because 
 the existing one is getting too long [1].

 [...]


 As far as i remember there was another suggestion of Andrei (in 
 another context). By importing a module B in module A, the module B 
 can specify coding which is executed and gets the module A as info.

 This might could also solve this issue.


 Yah, it's quite a universal pattern. It can be used as "import 
 core.rtti; to add RTTI support for this module" etc.


 
 Allowing import statements to have side effects seems like a pretty big 
 can of worms to open just so that we can avoid typing "mixin 
 RTTISupport!()". One of the great things about D's module system is that 
 a D module can't meddle around in another module's global scope the way 
 a header file can in C or C++. I hope that guarantee won't be broken 
 simply for the sake of syntactic convenience.


Good point, thanks.
May 23 2019
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2019-05-19 21:13, Andre Pany wrote:


 As far as i remember there was another suggestion of Andrei (in another 
 context). By importing a module B in module A, the module B can specify 
 coding which is executed and gets the module A as info.


Ok, that sounds useful as well. But in this case I think my solution is 
easier to implement, works less like magic and is better suited to 
implement a unit test runner.

With Andrei's proposal, as far as I understand, would require to add an 
import. The implementation would also iterate the unit tests locally for 
each module. I'm not sure how you would be able to combine all of them 
into a single list. Also how to know when all unit tests have been 
collected and the runner can actually start running the tests.

-- 
/Jacob Carlborg
May 20 2019
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 19 May 2019 19:13:31 +0000 schrieb Andre Pany:


 On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:

 I'm staring a new thread here on the topic of Name unittests because
 the existing one is getting too long [1].

 [...]


 
 As far as i remember there was another suggestion of Andrei (in another
 context). By importing a module B in module A, the module B can specify
 coding which is executed and gets the module A as info.
 
 This might could also solve this issue.
 
 Kind regards Andre


Do you happen to have a link for that original post?

 Andrei how exactly should something like this be implemented, do you 
agree that mixin templates are the right tool for this (e.g. as described 
here: https://forum.dlang.org/post/qbr7dd$16fd$1 digitalmars.com)? If so, 
I could write a DIP and the compiler implementation for that. Of course 
the DIP would have to adress the details (selective imports, public 
imports, ...), but I'd like to have some feedback on the general idea* 
first.

* annotate template mixin with import, then automatically add 
mixin(template(thisModule))) in every importing module

-- 
Johannes
May 21 2019
parent Andre Pany <andre s-e-a-p.de> writes:
On Tuesday, 21 May 2019 at 17:54:36 UTC, Johannes Pfau wrote:

 Am Sun, 19 May 2019 19:13:31 +0000 schrieb Andre Pany:


 [...]


 Do you happen to have a link for that original post?

  Andrei how exactly should something like this be implemented, 
 do you agree that mixin templates are the right tool for this 
 (e.g. as described here: 
 https://forum.dlang.org/post/qbr7dd$16fd$1 digitalmars.com)? If 
 so, I could write a DIP and the compiler implementation for 
 that. Of course the DIP would have to adress the details 
 (selective imports, public imports, ...), but I'd like to have 
 some feedback on the general idea* first.

 * annotate template mixin with import, then automatically add 
 mixin(template(thisModule))) in every importing module


Here the link
https://forum.dlang.org/post/q7l56r$bo6$1 digitalmars.com

Kind regards
Andre
May 21 2019
prev sibling next sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 5/19/19 2:56 PM, Jacob Carlborg wrote:

 I'm staring a new thread here on the topic of Name unittests because the 
 existing one is getting too long [1].
 
 [1] https://forum.dlang.org/thread/mfcgj3$12a0$1 digitalmars.com


Keep in mind, most of that thread is from 4 years ago.

But, umm, seriously dudes...we have unit-threaded now. Takes care of all 
that and then some. Just make it official, bake to into Phobos or 
whatever, and be done with it. Don't we have more important things to do 
than re-implement stuff we already have that's *already* working quite well?

I mean seriously, we've got people highly averse to *small, but actual, 
improvements*, but completely re-implementing something that's already 
working very well is worthwhile??? WAT???

(Or am I misunderstanding the gist of the small, non-outdated branch of 
that thread?)
May 20 2019
parent reply Johannes Loher <johannes.loher fg4f.de> writes:
On Monday, 20 May 2019 at 16:38:39 UTC, Nick Sabalausky 
(Abscissa) wrote:

 On 5/19/19 2:56 PM, Jacob Carlborg wrote:

 I'm staring a new thread here on the topic of Name unittests 
 because the existing one is getting too long [1].
 
 [1] 
 https://forum.dlang.org/thread/mfcgj3$12a0$1 digitalmars.com


 Keep in mind, most of that thread is from 4 years ago.

 But, umm, seriously dudes...we have unit-threaded now. Takes 
 care of all that and then some. Just make it official, bake to 
 into Phobos or whatever, and be done with it. Don't we have 
 more important things to do than re-implement stuff we already 
 have that's *already* working quite well?

 I mean seriously, we've got people highly averse to *small, but 
 actual, improvements*, but completely re-implementing something 
 that's already working very well is worthwhile??? WAT???

 (Or am I misunderstanding the gist of the small, non-outdated 
 branch of that thread?)


unit-threaded is great but it requires a prebuild command to 
gather all the modules. Can’t really make that official...

This is also not currently fixable on a library level because 
there is no way to reflect on what modules are being compiled. 
Adding a way to do this is basically what Jacob was suggesting.
May 20 2019
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 5/20/19 4:07 PM, Johannes Loher wrote:

 
 unit-threaded is great but it requires a prebuild command to gather all 
 the modules. Can’t really make that official...
 
 This is also not currently fixable on a library level because there is 
 no way to reflect on what modules are being compiled. Adding a way to do 
 this is basically what Jacob was suggesting.


Ahh, right, right, I gotcha. And TBH, that's been LOOOOONG standing 
giant gaping hole in D's reflection. Would be fantastic to finally get 
that fixed even regardless of the unittest stuff. IIRC, that was also 
the one main technical issue in the way of being able to build a 
Java-style runtime reflection system on top of D's compiletime 
reflection (well, that and the odd choice of reflection not being able 
to see through private, which has luckily been fixed now AIUI).
May 20 2019
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:

 I'm staring a new thread here on the topic of Name unittests 
 because the existing one is getting too long [1].

 [...]


How would this work with separate compilation?
May 21 2019
next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Tuesday, 21 May 2019 at 09:43:32 UTC, Atila Neves wrote:

 On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:

 I'm staring a new thread here on the topic of Name unittests 
 because the existing one is getting too long [1].

 [...]


 How would this work with separate compilation?


I assume with the proposal of Andrei, even the separate 
compilation can be solved.
Of course the points of Jacob 
(https://forum.dlang.org/post/qbtrre$ilr$1 digitalmars.com) have 
to be investigated and addressed in detail.

Kind regards
André
May 21 2019
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-05-21 11:43, Atila Neves wrote:


 How would this work with separate compilation?


Hmm, I'm not sure. That would be tested when it's implemented :)

-- 
/Jacob Carlborg
May 21 2019
parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 21 May 2019 at 20:04:38 UTC, Jacob Carlborg wrote:

 On 2019-05-21 11:43, Atila Neves wrote:


 How would this work with separate compilation?


 Hmm, I'm not sure. That would be tested when it's implemented :)


Testing in production is the best way to test things. It is 
known. :P
May 22 2019
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-05-21 11:43, Atila Neves wrote:


 How would this work with separate compilation?


I have given this some more thought. For my idea to work, regardless of 
separate compilation, I think the compiler needs to invoke a template 
that uses `__traits(getRootModules)` which collects all tests. If it's 
not a template I'm guessing this would contain the files passed to the 
compiler when druntime was built and not when the user application is built.

Then it would collect the unit tests to a global variable, that should 
work with separate compilation as well, I think.

-- 
/Jacob Carlborg
May 23 2019
parent reply Andre Pany <andre s-e-a-p.de> writes:
On Thursday, 23 May 2019 at 18:01:32 UTC, Jacob Carlborg wrote:

 On 2019-05-21 11:43, Atila Neves wrote:


 How would this work with separate compilation?


 I have given this some more thought. For my idea to work, 
 regardless of separate compilation, I think the compiler needs 
 to invoke a template that uses `__traits(getRootModules)` which 
 collects all tests. If it's not a template I'm guessing this 
 would contain the files passed to the compiler when druntime 
 was built and not when the user application is built.

 Then it would collect the unit tests to a global variable, that 
 should work with separate compilation as well, I think.


One thing I do not understand. Once module A, which contains the 
new traits, is compiled using separate compilation, the list of 
found modules is fixed. If now new module B is added, nothing 
will cause a rebuild of A and therefore the list of found modules 
is incomplete.

This can of course be solved by telling the user to not use 
separate compilation, which might be a fair tradeoff, I am not 
sure.

Kind regards
Andre
May 23 2019
next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 23 May 2019 at 20:42:29 UTC, Andre Pany wrote:

 On Thursday, 23 May 2019 at 18:01:32 UTC, Jacob Carlborg wrote:

 [...]


 One thing I do not understand. Once module A, which contains 
 the new traits, is compiled using separate compilation, the 
 list of found modules is fixed. If now new module B is added, 
 nothing will cause a rebuild of A and therefore the list of 
 found modules is incomplete.

 This can of course be solved by telling the user to not use 
 separate compilation, which might be a fair tradeoff, I am not 
 sure.


Not being able to use separate compilation is a non-starter.
May 24 2019
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-05-23 22:42, Andre Pany wrote:


 One thing I do not understand. Once module A, which contains the new 
 traits, is compiled using separate compilation, the list of found 
 modules is fixed. If now new module B is added, nothing will cause a 
 rebuild of A and therefore the list of found modules is incomplete.
 
 This can of course be solved by telling the user to not use separate 
 compilation, which might be a fair tradeoff, I am not sure.


It works with separate compilation if the unit tests are collected into 
a global variable. Here's an example using the existing 
__traits(allMembers):

$ cat main.d
module bar;

extern (C) __gshared string[] members;

void main()
{
     import foo;

     members ~= [__traits(allMembers, bar)];
     foobar();
     assert(members == ["object", "members", "main", "object", 
"members", "foobar"]);
}

$ cat foo.d
module foo;

extern (C) extern __gshared string[] members;

void foobar()
{
     members ~= [__traits(allMembers, foo)];
}

$ dmd -c foo.d && dmd foo.o -run main.d

The assertion passes as expected.

-- 
/Jacob Carlborg
May 24 2019
parent reply Andre Pany <andre s-e-a-p.de> writes:
On Friday, 24 May 2019 at 15:32:15 UTC, Jacob Carlborg wrote:

 On 2019-05-23 22:42, Andre Pany wrote:


 One thing I do not understand. Once module A, which contains 
 the new traits, is compiled using separate compilation, the 
 list of found modules is fixed. If now new module B is added, 
 nothing will cause a rebuild of A and therefore the list of 
 found modules is incomplete.
 
 This can of course be solved by telling the user to not use 
 separate compilation, which might be a fair tradeoff, I am not 
 sure.


 It works with separate compilation if the unit tests are 
 collected into a global variable. Here's an example using the 
 existing __traits(allMembers):

 $ cat main.d
 module bar;

 extern (C) __gshared string[] members;

 void main()
 {
     import foo;

     members ~= [__traits(allMembers, bar)];
     foobar();
     assert(members == ["object", "members", "main", "object", 
 "members", "foobar"]);
 }

 $ cat foo.d
 module foo;

 extern (C) extern __gshared string[] members;

 void foobar()
 {
     members ~= [__traits(allMembers, foo)];
 }

 $ dmd -c foo.d && dmd foo.o -run main.d

 The assertion passes as expected.


Maybe my understanding is wrong. As far as I understand, your 
example only works as long as foo.d is compiled first and and 
main.d last. The order is guaranteed by the import foo statement.

But if now use __traits(getModules) in main.d and compile with 
this command:
dmd -c main.d && dmd main.o -run foo.d
I assume it won't work anymore.

The order of compilation will then matter and I do not how you 
can ensure the file containing the __traits(getModules) is always 
compiled as last.

Kind regards
Andre
May 24 2019
parent Jacob Carlborg <doob me.com> writes:
On 2019-05-24 20:53, Andre Pany wrote:


 Maybe my understanding is wrong. As far as I understand, your example 
 only works as long as foo.d is compiled first and and main.d last. 


No, please give it a try. It works.


 The order is guaranteed by the import foo statement.
 
 But if now use __traits(getModules) in main.d and compile with this 
 command:
 dmd -c main.d && dmd main.o -run foo.d
 I assume it won't work anymore.
 
 The order of compilation will then matter and I do not how you can 
 ensure the file containing the __traits(getModules) is always compiled 
 as last.


The compiler would need to invoke a druntime function every time the 
compiler is invoked. This druntime function will call run 
`__traits(getModules)`. It's basically the same thing as I've done in my 
example.

-- 
/Jacob Carlborg
May 24 2019