www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Named unittests and __traits(getModules)

reply Jacob Carlborg <doob me.com> writes:
I'm staring a new thread here on the topic of Name unittests because the 
existing one is getting too long [1].

The best way to add support for named unit tests, without adding any new 
syntax to the language, is to leverage UDAs. This has already been 
discussed in the previous post [1]. The problem with UDAs and the 
current unit test runner is by the time the unit test runner runs the 
unit tests, the UDAs are long gone. They need to be fetched at 
compile-time. It's also not possible to continue running unit tests in 
the same module after a failed unit test with the current runner. This 
is because the compiler generates one function that calls all the unit 
tests in the whole module.

To solve these problems `__traits(getUnitTests)` can be used to access 
the unit tests at compile-time. This leads to the next problem. To 
access the unit tests one needs to import the modules where the unit 
tests are defined. There are currently no way to get a list of all 
modules. Existing third party unit test runners usually use a pre-build 
script that collects all files with unit tests and generates a new file 
with all the imports. This is also noted in the previous thread [2].

The solution of running a pre-build script is not workable to include in 
druntime. I purpose we add two new traits: `__traits(getRootModules)` 
and `__traits(getImportedModules)`. `getRootModules` will return all 
root modules, i.e. the files which were passed to the compiler on the 
command line. `getImportedModules` will return all imported modules, 
i.e. the non-root modules. Adding these two together will give access to 
all modules the compiler has processed. Note that these traits are 
useful for other things than building a unit test runner.

With these two traits (only getRootModules might be necessary) and using 
`static foreach` to generate the imports, `getUnitTests` can later be 
used to retrieve the unit tests at compile-time which also gives access 
to the UDAs. Using this technique to run the unit tests allows to add 
additional features, like continue running after a failed test, before 
and after callbacks and other features.

There's already a unit test runner in the DMD test suite that uses this 
technique (but with the pre-build script). With these new traits this 
unit test runner could be added pretty easily to druntime.

Thoughts?

[1] https://forum.dlang.org/thread/mfcgj3$12a0$1 digitalmars.com
[2] https://forum.dlang.org/post/qbr7dd$16fd$1 digitalmars.com

-- 
/Jacob Carlborg
May 19
next sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:
 I'm staring a new thread here on the topic of Name unittests 
 because the existing one is getting too long [1].

 [...]
As far as i remember there was another suggestion of Andrei (in another context). By importing a module B in module A, the module B can specify coding which is executed and gets the module A as info. This might could also solve this issue. Kind regards Andre
May 19
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/19/19 8:13 PM, Andre Pany wrote:
 On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:
 I'm staring a new thread here on the topic of Name unittests because 
 the existing one is getting too long [1].

 [...]
As far as i remember there was another suggestion of Andrei (in another context). By importing a module B in module A, the module B can specify coding which is executed and gets the module A as info. This might could also solve this issue.
Yah, it's quite a universal pattern. It can be used as "import core.rtti; to add RTTI support for this module" etc.
May 19
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/19/19 11:41 PM, Andrei Alexandrescu wrote:
 On 5/19/19 8:13 PM, Andre Pany wrote:
 On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:
 I'm staring a new thread here on the topic of Name unittests because 
 the existing one is getting too long [1].

 [...]
As far as i remember there was another suggestion of Andrei (in another context). By importing a module B in module A, the module B can specify coding which is executed and gets the module A as info. This might could also solve this issue.
Yah, it's quite a universal pattern. It can be used as "import core.rtti; to add RTTI support for this module" etc.
Related: https://github.com/dlang/dmd/pull/9814. There, the idea is to predicate an import for each language feature or groups of features. Many, however, can be done as templates and realized naturally by means of template instantiation (e.g. a[] = b[] triggers the generation of the array assign support function). RTTI is different because it instructs the compiler to generate code a specific way without a detectable action in the source code. That kind of feature could be done with import + hook "execute this on the importing module".
May 19
parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Monday, 20 May 2019 at 00:14:01 UTC, Andrei Alexandrescu wrote:

 RTTI is different because it instructs the compiler to generate 
 code a specific way without a detectable action in the source 
 code. That kind of feature could be done with import + hook 
 "execute this on the importing module".
I'm not sure I fully understand this, so forgive me if I'm just making noise, but does the pattern in this PR give you what you need? https://github.com/dlang/dmd/pull/7799 The idea is to put a detectable action in the source code. In the case of the PR referenced above, the detectable action is the existence of the `class TypeInfo` declaration. The compiler has been programmed to look for that declaration, and generate code based on whether or not it exists. Because the declaration exists in druntime's source code, users can also check for its existence in their source code. Basically, the compiler is doing design-by-introspection just like the user. The runtime's source code informs the compiler what to do rather than the other way around. So the compiler does have a detectable action in the source code (i.e. the existence of `class TypeInfo`) and that same detectable action is available to the user as well. Mike
May 19
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.com> writes:
On 5/19/19 8:55 PM, Mike Franklin wrote:
 On Monday, 20 May 2019 at 00:14:01 UTC, Andrei Alexandrescu wrote:
 
 RTTI is different because it instructs the compiler to generate code a 
 specific way without a detectable action in the source code. That kind 
 of feature could be done with import + hook "execute this on the 
 importing module".
I'm not sure I fully understand this, so forgive me if I'm just making noise, but does the pattern in this PR give you what you need? https://github.com/dlang/dmd/pull/7799 The idea is to put a detectable action in the source code.  In the case of the PR referenced above, the detectable action is the existence of the `class TypeInfo` declaration.  The compiler has been programmed to look for that declaration, and generate code based on whether or not it exists.  Because the declaration exists in druntime's source code, users can also check for its existence in their source code. Basically, the compiler is doing design-by-introspection just like the user.  The runtime's source code informs the compiler what to do rather than the other way around. So the compiler does have a detectable action in the source code (i.e. the existence of `class TypeInfo`) and that same detectable action is available to the user as well.
Having the compiler detect constructs in code would work, as would built-in attributes such as: noRTTI module mymodule; and/or noRTTI class BareBones { ... } The disadvantage is that the compiler would need to be modified for each of these, whereas if we define and avail ourselves of powerful introspection, we can achieve this kind of stuff in library code.
May 20
prev sibling parent reply Paul Backus <snarwin gmail.com> writes:
On Sunday, 19 May 2019 at 22:41:52 UTC, Andrei Alexandrescu wrote:
 On 5/19/19 8:13 PM, Andre Pany wrote:
 On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:
 I'm staring a new thread here on the topic of Name unittests 
 because the existing one is getting too long [1].

 [...]
As far as i remember there was another suggestion of Andrei (in another context). By importing a module B in module A, the module B can specify coding which is executed and gets the module A as info. This might could also solve this issue.
Yah, it's quite a universal pattern. It can be used as "import core.rtti; to add RTTI support for this module" etc.
Allowing import statements to have side effects seems like a pretty big can of worms to open just so that we can avoid typing "mixin RTTISupport!()". One of the great things about D's module system is that a D module can't meddle around in another module's global scope the way a header file can in C or C++. I hope that guarantee won't be broken simply for the sake of syntactic convenience.
May 22
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/22/19 5:08 PM, Paul Backus wrote:
 On Sunday, 19 May 2019 at 22:41:52 UTC, Andrei Alexandrescu wrote:
 On 5/19/19 8:13 PM, Andre Pany wrote:
 On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:
 I'm staring a new thread here on the topic of Name unittests because 
 the existing one is getting too long [1].

 [...]
As far as i remember there was another suggestion of Andrei (in another context). By importing a module B in module A, the module B can specify coding which is executed and gets the module A as info. This might could also solve this issue.
Yah, it's quite a universal pattern. It can be used as "import core.rtti; to add RTTI support for this module" etc.
Allowing import statements to have side effects seems like a pretty big can of worms to open just so that we can avoid typing "mixin RTTISupport!()". One of the great things about D's module system is that a D module can't meddle around in another module's global scope the way a header file can in C or C++. I hope that guarantee won't be broken simply for the sake of syntactic convenience.
Good point, thanks.
May 23
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2019-05-19 21:13, Andre Pany wrote:

 As far as i remember there was another suggestion of Andrei (in another 
 context). By importing a module B in module A, the module B can specify 
 coding which is executed and gets the module A as info.
Ok, that sounds useful as well. But in this case I think my solution is easier to implement, works less like magic and is better suited to implement a unit test runner. With Andrei's proposal, as far as I understand, would require to add an import. The implementation would also iterate the unit tests locally for each module. I'm not sure how you would be able to combine all of them into a single list. Also how to know when all unit tests have been collected and the runner can actually start running the tests. -- /Jacob Carlborg
May 20
prev sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Sun, 19 May 2019 19:13:31 +0000 schrieb Andre Pany:

 On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:
 I'm staring a new thread here on the topic of Name unittests because
 the existing one is getting too long [1].

 [...]
As far as i remember there was another suggestion of Andrei (in another context). By importing a module B in module A, the module B can specify coding which is executed and gets the module A as info. This might could also solve this issue. Kind regards Andre
Do you happen to have a link for that original post? Andrei how exactly should something like this be implemented, do you agree that mixin templates are the right tool for this (e.g. as described here: https://forum.dlang.org/post/qbr7dd$16fd$1 digitalmars.com)? If so, I could write a DIP and the compiler implementation for that. Of course the DIP would have to adress the details (selective imports, public imports, ...), but I'd like to have some feedback on the general idea* first. * annotate template mixin with import, then automatically add mixin(template(thisModule))) in every importing module -- Johannes
May 21
parent Andre Pany <andre s-e-a-p.de> writes:
On Tuesday, 21 May 2019 at 17:54:36 UTC, Johannes Pfau wrote:
 Am Sun, 19 May 2019 19:13:31 +0000 schrieb Andre Pany:

 [...]
Do you happen to have a link for that original post? Andrei how exactly should something like this be implemented, do you agree that mixin templates are the right tool for this (e.g. as described here: https://forum.dlang.org/post/qbr7dd$16fd$1 digitalmars.com)? If so, I could write a DIP and the compiler implementation for that. Of course the DIP would have to adress the details (selective imports, public imports, ...), but I'd like to have some feedback on the general idea* first. * annotate template mixin with import, then automatically add mixin(template(thisModule))) in every importing module
Here the link https://forum.dlang.org/post/q7l56r$bo6$1 digitalmars.com Kind regards Andre
May 21
prev sibling next sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 5/19/19 2:56 PM, Jacob Carlborg wrote:
 I'm staring a new thread here on the topic of Name unittests because the 
 existing one is getting too long [1].
 
 [1] https://forum.dlang.org/thread/mfcgj3$12a0$1 digitalmars.com
Keep in mind, most of that thread is from 4 years ago. But, umm, seriously dudes...we have unit-threaded now. Takes care of all that and then some. Just make it official, bake to into Phobos or whatever, and be done with it. Don't we have more important things to do than re-implement stuff we already have that's *already* working quite well? I mean seriously, we've got people highly averse to *small, but actual, improvements*, but completely re-implementing something that's already working very well is worthwhile??? WAT??? (Or am I misunderstanding the gist of the small, non-outdated branch of that thread?)
May 20
parent reply Johannes Loher <johannes.loher fg4f.de> writes:
On Monday, 20 May 2019 at 16:38:39 UTC, Nick Sabalausky 
(Abscissa) wrote:
 On 5/19/19 2:56 PM, Jacob Carlborg wrote:
 I'm staring a new thread here on the topic of Name unittests 
 because the existing one is getting too long [1].
 
 [1] 
 https://forum.dlang.org/thread/mfcgj3$12a0$1 digitalmars.com
Keep in mind, most of that thread is from 4 years ago. But, umm, seriously dudes...we have unit-threaded now. Takes care of all that and then some. Just make it official, bake to into Phobos or whatever, and be done with it. Don't we have more important things to do than re-implement stuff we already have that's *already* working quite well? I mean seriously, we've got people highly averse to *small, but actual, improvements*, but completely re-implementing something that's already working very well is worthwhile??? WAT??? (Or am I misunderstanding the gist of the small, non-outdated branch of that thread?)
unit-threaded is great but it requires a prebuild command to gather all the modules. Can’t really make that official... This is also not currently fixable on a library level because there is no way to reflect on what modules are being compiled. Adding a way to do this is basically what Jacob was suggesting.
May 20
parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 5/20/19 4:07 PM, Johannes Loher wrote:
 
 unit-threaded is great but it requires a prebuild command to gather all 
 the modules. Can’t really make that official...
 
 This is also not currently fixable on a library level because there is 
 no way to reflect on what modules are being compiled. Adding a way to do 
 this is basically what Jacob was suggesting.
Ahh, right, right, I gotcha. And TBH, that's been LOOOOONG standing giant gaping hole in D's reflection. Would be fantastic to finally get that fixed even regardless of the unittest stuff. IIRC, that was also the one main technical issue in the way of being able to build a Java-style runtime reflection system on top of D's compiletime reflection (well, that and the odd choice of reflection not being able to see through private, which has luckily been fixed now AIUI).
May 20
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:
 I'm staring a new thread here on the topic of Name unittests 
 because the existing one is getting too long [1].

 [...]
How would this work with separate compilation?
May 21
next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Tuesday, 21 May 2019 at 09:43:32 UTC, Atila Neves wrote:
 On Sunday, 19 May 2019 at 18:56:33 UTC, Jacob Carlborg wrote:
 I'm staring a new thread here on the topic of Name unittests 
 because the existing one is getting too long [1].

 [...]
How would this work with separate compilation?
I assume with the proposal of Andrei, even the separate compilation can be solved. Of course the points of Jacob (https://forum.dlang.org/post/qbtrre$ilr$1 digitalmars.com) have to be investigated and addressed in detail. Kind regards André
May 21
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-05-21 11:43, Atila Neves wrote:

 How would this work with separate compilation?
Hmm, I'm not sure. That would be tested when it's implemented :) -- /Jacob Carlborg
May 21
parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 21 May 2019 at 20:04:38 UTC, Jacob Carlborg wrote:
 On 2019-05-21 11:43, Atila Neves wrote:

 How would this work with separate compilation?
Hmm, I'm not sure. That would be tested when it's implemented :)
Testing in production is the best way to test things. It is known. :P
May 22
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-05-21 11:43, Atila Neves wrote:

 How would this work with separate compilation?
I have given this some more thought. For my idea to work, regardless of separate compilation, I think the compiler needs to invoke a template that uses `__traits(getRootModules)` which collects all tests. If it's not a template I'm guessing this would contain the files passed to the compiler when druntime was built and not when the user application is built. Then it would collect the unit tests to a global variable, that should work with separate compilation as well, I think. -- /Jacob Carlborg
May 23
parent reply Andre Pany <andre s-e-a-p.de> writes:
On Thursday, 23 May 2019 at 18:01:32 UTC, Jacob Carlborg wrote:
 On 2019-05-21 11:43, Atila Neves wrote:

 How would this work with separate compilation?
I have given this some more thought. For my idea to work, regardless of separate compilation, I think the compiler needs to invoke a template that uses `__traits(getRootModules)` which collects all tests. If it's not a template I'm guessing this would contain the files passed to the compiler when druntime was built and not when the user application is built. Then it would collect the unit tests to a global variable, that should work with separate compilation as well, I think.
One thing I do not understand. Once module A, which contains the new traits, is compiled using separate compilation, the list of found modules is fixed. If now new module B is added, nothing will cause a rebuild of A and therefore the list of found modules is incomplete. This can of course be solved by telling the user to not use separate compilation, which might be a fair tradeoff, I am not sure. Kind regards Andre
May 23
next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 23 May 2019 at 20:42:29 UTC, Andre Pany wrote:
 On Thursday, 23 May 2019 at 18:01:32 UTC, Jacob Carlborg wrote:
 [...]
One thing I do not understand. Once module A, which contains the new traits, is compiled using separate compilation, the list of found modules is fixed. If now new module B is added, nothing will cause a rebuild of A and therefore the list of found modules is incomplete. This can of course be solved by telling the user to not use separate compilation, which might be a fair tradeoff, I am not sure.
Not being able to use separate compilation is a non-starter.
May 24
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-05-23 22:42, Andre Pany wrote:

 One thing I do not understand. Once module A, which contains the new 
 traits, is compiled using separate compilation, the list of found 
 modules is fixed. If now new module B is added, nothing will cause a 
 rebuild of A and therefore the list of found modules is incomplete.
 
 This can of course be solved by telling the user to not use separate 
 compilation, which might be a fair tradeoff, I am not sure.
It works with separate compilation if the unit tests are collected into a global variable. Here's an example using the existing __traits(allMembers): $ cat main.d module bar; extern (C) __gshared string[] members; void main() { import foo; members ~= [__traits(allMembers, bar)]; foobar(); assert(members == ["object", "members", "main", "object", "members", "foobar"]); } $ cat foo.d module foo; extern (C) extern __gshared string[] members; void foobar() { members ~= [__traits(allMembers, foo)]; } $ dmd -c foo.d && dmd foo.o -run main.d The assertion passes as expected. -- /Jacob Carlborg
May 24
parent reply Andre Pany <andre s-e-a-p.de> writes:
On Friday, 24 May 2019 at 15:32:15 UTC, Jacob Carlborg wrote:
 On 2019-05-23 22:42, Andre Pany wrote:

 One thing I do not understand. Once module A, which contains 
 the new traits, is compiled using separate compilation, the 
 list of found modules is fixed. If now new module B is added, 
 nothing will cause a rebuild of A and therefore the list of 
 found modules is incomplete.
 
 This can of course be solved by telling the user to not use 
 separate compilation, which might be a fair tradeoff, I am not 
 sure.
It works with separate compilation if the unit tests are collected into a global variable. Here's an example using the existing __traits(allMembers): $ cat main.d module bar; extern (C) __gshared string[] members; void main() { import foo; members ~= [__traits(allMembers, bar)]; foobar(); assert(members == ["object", "members", "main", "object", "members", "foobar"]); } $ cat foo.d module foo; extern (C) extern __gshared string[] members; void foobar() { members ~= [__traits(allMembers, foo)]; } $ dmd -c foo.d && dmd foo.o -run main.d The assertion passes as expected.
Maybe my understanding is wrong. As far as I understand, your example only works as long as foo.d is compiled first and and main.d last. The order is guaranteed by the import foo statement. But if now use __traits(getModules) in main.d and compile with this command: dmd -c main.d && dmd main.o -run foo.d I assume it won't work anymore. The order of compilation will then matter and I do not how you can ensure the file containing the __traits(getModules) is always compiled as last. Kind regards Andre
May 24
parent Jacob Carlborg <doob me.com> writes:
On 2019-05-24 20:53, Andre Pany wrote:

 Maybe my understanding is wrong. As far as I understand, your example 
 only works as long as foo.d is compiled first and and main.d last. 
No, please give it a try. It works.
 The order is guaranteed by the import foo statement.
 
 But if now use __traits(getModules) in main.d and compile with this 
 command:
 dmd -c main.d && dmd main.o -run foo.d
 I assume it won't work anymore.
 
 The order of compilation will then matter and I do not how you can 
 ensure the file containing the __traits(getModules) is always compiled 
 as last.
The compiler would need to invoke a druntime function every time the compiler is invoked. This druntime function will call run `__traits(getModules)`. It's basically the same thing as I've done in my example. -- /Jacob Carlborg
May 24