www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Phobos Unittest

reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
When is a phobos unittest supposed to be qualified with version 
`(StdUnittest)`? Ideally, always? I don't see that their current 
use is consistenly following a rule. If so, is the purpose of its 
presence to reduce the burden of the compiler when the 
application using phobos is compiled with -unittest? (edited).

If we were to put a `version (StdUnittest)` in front of every 
`unittest` in phobos (via an automated refactoring of course) 
would that speed building large applications or tests-suites with 
the `-unittest` flag provide they import many/all phobos 
libraries via, for instance, import `std`?

When compiling non-phobos modules with `-unittest` flag, how are 
the `unittest`-blocks in imported phobos modules processed by the 
compiler? Are they only lexed (and parsed) but nothing else?
Sep 03 2021
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Sep 03, 2021 at 11:39:44PM +0000, Per Nordlöw via Digitalmars-d-learn
wrote:

 When is a phobos unittest supposed to be qualified with version
 `(StdUnittest)`? Ideally, always? I don't see that their current use
 is consistenly following a rule. If so, is the purpose of its presence
 to reduce the burden of the compiler when the application using phobos
 is compiled with -unittest? (edited).


[...]

This is related to the bogonity of the current behaviour of -unittest,
which compiles *all* unittests of *all* imported modules, even when
you're compiling user code that has no interest in Phobos unittests.
StdUnittest is a hack introduced to suppress Phobos unittests in user
programs.  Theoretically it's supposed to apply to all unittests, but
obviously whoever did it failed to cover every case.

It's still up in the air whether or not this should even be fixed.
Ideally, we should be fixing the behaviour of -unittest instead of
living with this hack.


T

-- 
I'm still trying to find a pun for "punishment"...
Sep 03 2021
parent reply Paul Backus <snarwin gmail.com> writes:
On Saturday, 4 September 2021 at 00:09:37 UTC, H. S. Teoh wrote:

 This is related to the bogonity of the current behaviour of 
 -unittest, which compiles *all* unittests of *all* imported 
 modules, even when you're compiling user code that has no 
 interest in Phobos unittests.


Well, no; it compiles all unittests of all *compiled* modules, 
not all *imported* modules. So it does not actually include 
Phobos unittests.


 StdUnittest is a hack introduced to suppress Phobos unittests 
 in user programs.


The point of `StdUnittest` is to suppress

* `version (unittest)` blocks in Phobos
* Phobos unittests in the bodies of templates that are 
instantiated by user code

As Steven Schveighoffer [pointed out][1], Phobos unittests are 
never included in user code, regardless of whether `StdUnittest` 
is used.

[1]: https://forum.dlang.org/post/sglmk0$33m$1 digitalmars.com
Sep 03 2021
next sibling parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Saturday, 4 September 2021 at 03:18:01 UTC, Paul Backus wrote:

 As Steven Schveighoffer [pointed out][1], Phobos unittests are 
 never included in user code, regardless of whether 
 `StdUnittest` is used.


Yes, but they are lexed and parsed, right?
Sep 04 2021
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 4 September 2021 at 09:42:46 UTC, Per Nordlöw wrote:

 Yes, but they are lexed and parsed, right?


Right, but that's the case regardless of `version(StdUnittest)`.
Sep 04 2021
next sibling parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Saturday, 4 September 2021 at 12:31:33 UTC, Dennis wrote:

 Right, but that's the case regardless of `version(StdUnittest)`.


I don't think so. My guess is that `version(StdUnittest)` does 
maximum parsing, maybe only lexing. How can one easily determine 
if this is the fact? Behaviour of `pragma(msg)`?
Sep 04 2021
prev sibling parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Saturday, 4 September 2021 at 12:31:33 UTC, Dennis wrote:

 On Saturday, 4 September 2021 at 09:42:46 UTC, Per Nordlöw 
 wrote:

 Yes, but they are lexed and parsed, right?


 Right, but that's the case regardless of `version(StdUnittest)`.


Omg. It really seems like it's motivated to do a benchmark with 
phobos unittests prefixed with

     version(StdUnittest)

then. I can make that refactoring in my favorite choice of editor 
and see the result differ. In terms of binary size, and speed and 
memory usage of check, build and link time.
Sep 04 2021
next sibling parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Saturday, 4 September 2021 at 12:40:02 UTC, Per Nordlöw wrote:

 then. I can make that refactoring in my favorite choice of 
 editor and see the result differ. In terms of binary size, and 
 speed and memory usage of check, build and link time.


Automated that is.

If we want this, shall we format as

```d
version (StdUnittest) [QUALIFIER...] unittest
```

instead of current

```d
version (StdUnittest)
[QUALIFIER...] unittest
```

?
Sep 04 2021
prev sibling parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Saturday, 4 September 2021 at 12:40:02 UTC, Per Nordlöw wrote:

 Omg. It really seems like it's motivated to do a benchmark with 
 phobos unittests prefixed with


Should have been a reply to


 The "never" is false, https://d.godbolt.org/z/c4oeYM7rG
Sep 04 2021
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/4/21 5:42 AM, Per Nordlöw wrote:

 On Saturday, 4 September 2021 at 03:18:01 UTC, Paul Backus wrote:

 As Steven Schveighoffer [pointed out][1], Phobos unittests are never 
 included in user code, regardless of whether `StdUnittest` is used.


 
 Yes, but they are lexed and parsed, right?


Yes, and I think this is trivially so for any versioned code in the file.

Note that lexing and parsing is extremely quick, and I wouldn't focus on 
trying to trim this out, you won't get much performance out of that.

-Steve
Sep 04 2021
parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Saturday, 4 September 2021 at 13:12:49 UTC, Steven 
Schveighoffer wrote:

 Note that lexing and parsing is extremely quick, and I wouldn't 
 focus on trying to trim this out, you won't get much 
 performance out of that.

 -Steve


For the record, a D file containing only `import std;` type 
checks via

```sh
time dmd import_std.d -o-
```

in 0.20s whereas

```sh
time dmd import_std.d -o-
```

in 0.45s on my ThreadRipper.
Sep 04 2021
next sibling parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Saturday, 4 September 2021 at 20:05:17 UTC, Per Nordlöw wrote:

 ```sh
 time dmd import_std.d -o-
 ```


should be

```sh
time dmd -unittest import_std.d -o-
```
Sep 04 2021
parent jfondren <julian.fondren gmail.com> writes:
On Saturday, 4 September 2021 at 20:06:27 UTC, Per Nordlöw wrote:

 On Saturday, 4 September 2021 at 20:05:17 UTC, Per Nordlöw 
 wrote:

 ```sh
 time dmd import_std.d -o-
 ```


 should be

 ```sh
 time dmd -unittest import_std.d -o-
 ```


When you generate the object files, I get 13K vs. 75K from a file 
containing just `import std;`. More than parsing is happening 
differently.

```
$ objdump -dwr --no-show-raw-insn importstd.o |ddemangle | grep 
-oP 'std[\w.]+'|sort|uniq -c|sort -n|awk '$1 > 1'
      28 std.uni.InversionList
      28 std.uni.MultiArray
      30 std.encoding.BOM
      31 std.array.Appender
      35 std.concurrency.List
      35 std.concurrency.Message
      38 std.uni.CowArray
      44 std.variant.VariantN
      56 std.typecons.Tuple
      56 std.uni.BitPacked
      66 std.uni.GcPolicy
```
Sep 04 2021
prev sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/4/21 4:05 PM, Per Nordlöw wrote:

 On Saturday, 4 September 2021 at 13:12:49 UTC, Steven Schveighoffer wrote:

 Note that lexing and parsing is extremely quick, and I wouldn't focus 
 on trying to trim this out, you won't get much performance out of that.

 -Steve


 
 For the record, a D file containing only `import std;` type checks via
 
 ```sh
 time dmd import_std.d -o-
 ```
 
 in 0.20s whereas
 
 ```sh
 time dmd import_std.d -o-
 ```
 
 in 0.45s on my ThreadRipper.


I doubt that's because of the parsing. I've [gone 
down](https://github.com/schveiguy/dmd/tree/debugunitteststuff) these 
kinds of rabbit holes. It's one of the reasons I tried to remove ALL 
version(unittest) blocks from phobos that import other modules. But you 
just can't remove them all. And any unittests inside templates are going 
to compile and get included, even if you don't instantiate anything. I 
also discovered that CTFE initializers run, even if you don't use the 
imported symbol, and even if you don't need type inference. Can't 
remember if that got fixed.

I think you would find you could probably attribute the 0.25s there to a 
few modules that do things when unittests are turned on.

According to 
https://github.com/dlang/dmd/blob/1ae5fee06ddd0599fb187595f8b0cebf8c840ebd/src/dm
/parse.d#L642-L683, 
if unittests aren't turned on, indeed the parser simplifies its parsing 
of the unittest. It skips over the unittest block, and doesn't allocate 
any AST nodes for it. But that difference I don't think is going to be 
the cause for a significant slowdown.

One can construct a test to check the parsing:

1. Generate a huge source file with 10,000 (or maybe more?) non-trivial 
unittests. You need enough to move the needle on parsing.
2. Import that file, and build the main file with and without the 
-unittest flag.
3. This should not actually compile the unittests, but just parse them.

Using phobos as a test case is rife with things that may be contributing 
besides the parser.

-Steve
Sep 05 2021
prev sibling parent reply Johan <j j.nl> writes:
On Saturday, 4 September 2021 at 03:18:01 UTC, Paul Backus wrote:

 On Saturday, 4 September 2021 at 00:09:37 UTC, H. S. Teoh wrote:

 This is related to the bogonity of the current behaviour of 
 -unittest, which compiles *all* unittests of *all* imported 
 modules, even when you're compiling user code that has no 
 interest in Phobos unittests.


 Well, no; it compiles all unittests of all *compiled* modules, 
 not all *imported* modules. So it does not actually include 
 Phobos unittests.

 [...]

 As Steven Schveighoffer [pointed out][1], Phobos unittests are 
 never included in user code, regardless of whether 
 `StdUnittest` is used.


The "never" is false, https://d.godbolt.org/z/c4oeYM7rG

Unittests inside template code will be added to user code, unless 
the compiler has determined that the template is already 
instantiated in Phobos code (the "template culling" that the 
frontend does, whose behavior is not easily influenced by the 
programmer).

-Johan
Sep 04 2021
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 9/4/21 7:43 AM, Johan wrote:

 On Saturday, 4 September 2021 at 03:18:01 UTC, Paul Backus wrote:

 On Saturday, 4 September 2021 at 00:09:37 UTC, H. S. Teoh wrote:

 This is related to the bogonity of the current behaviour of 
 -unittest, which compiles *all* unittests of *all* imported modules, 
 even when you're compiling user code that has no interest in Phobos 
 unittests.


 Well, no; it compiles all unittests of all *compiled* modules, not all 
 *imported* modules. So it does not actually include Phobos unittests.

 [...]

 As Steven Schveighoffer [pointed out][1], Phobos unittests are never 
 included in user code, regardless of whether `StdUnittest` is used.


 
 The "never" is false, https://d.godbolt.org/z/c4oeYM7rG
 
 Unittests inside template code will be added to user code, unless the 
 compiler has determined that the template is already instantiated in 
 Phobos code (the "template culling" that the frontend does, whose 
 behavior is not easily influenced by the programmer).


It's always included (or at least it was when I last looked at it), as 
-unittest implies some form of -allinst. There was some effort to fix 
this, but I can't remember if the outcome was that it was merged or not.

-Steve
Sep 04 2021
prev sibling next sibling parent Paul Backus <snarwin gmail.com> writes:
On Friday, 3 September 2021 at 23:39:44 UTC, Per Nordlöw wrote:

 When is a phobos unittest supposed to be qualified with version 
 `(StdUnittest)`?


Almost never.

`version (StdUnittest)` should be used in Phobos wherever you 
would normally use a `version (unittest)` block. It is not for 
the tests themselves, but for test fixtures and support code.
Sep 03 2021
prev sibling parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Friday, 3 September 2021 at 23:39:44 UTC, Per Nordlöw wrote:

 When is a phobos unittest supposed to be qualified with version 
 `(StdUnittest)`? Ideally, always? I don't see that their 
 current use is consistenly following a rule. If so, is the 
 purpose of its presence to reduce the burden of the compiler 
 when the application using phobos is compiled with -unittest? 
 (edited).


For reference, see my previous attempt at making dmd more 
flexible at compiling unittests at

https://github.com/dlang/dmd/pull/6375
Sep 07 2021