• Jacob Carlborg (77/77) Feb 12 2019 This post is mostly directed to DMD contributors.
• kinke (2/2) Feb 12 2019 Thanks, I think it's a nice tool to improve quality via more

Jacob Carlborg <doob me.com> writes:

This post is mostly directed to DMD contributors.

Currently most of the tests for DMD are end-to-end like tests. The test 
invokes the compiler as a new process and the test asserts the exit code 
and/or error messages outputted by the compiler. This means that 
basically for each test file a new process is created which runs the 
compiler. There are two problems with this: a lot of processes need to 
be started and executed and since the compiler runs in a separate 
process from the test, it's not possible to access the internals of the 
compiler.

DMD also contains a few more of traditional unit tests (using the 
`unittest` blocks) which tests individual functions. Mostly utility 
functions.

I would like to announce that a new test runner has recently been added 
for testing DMD. It's like a mix of both of the existing tests types. It 
uses the `unittest` blocks and uses the compiler as a library. It runs 
the test and the compiler in the same process. It allows to compile code 
without starting a new process, then the test is free to assert whatever 
it likes. For example, the structure of internal data that are only 
available in the same process as the compiler.

The test runner compiles all files of this test type into a single 
executable. This should hopefully speed up running the tests because 
only a single executable is compiled and run, instead of hundreds.

This type of tests lives in the `test/unit` directory [1]. They're using 
the `unittest` block to declare a test. It supports attaching a UDA to 
filter tests. It also supports before and after hooks that are executed 
before each `unittest` block within the same module and after each 
`unittest` block (regardless of the test failed or passed). The test 
runner will print a report at the end of which tests that failed.

The tests are executed by default when `make -C test` or `./test/run.d` 
is executed. The runner supports a more finer grained control of running 
the tests:

* To run all the unit test (without running the other tests), run: 
`./test/run.d -u`

* To only run the unit tests in one or more specific files:
`./test/run.d -u test/unit/self_test.d`

* To only run a subset of the unit tests in a single file:
./test/run.d -u test/unit/self_test.d --filter "self test"

In the above example, the `--filter` flag will filter to only run the 
tests with a UDA matching the given value, in this case `self test`.

An example can look like this [2]:

module self_test;

import support : afterEach, beforeEach, defaultImportPaths;

 beforeEach initializeFrontend()
{
     import dmd.frontend : initDMD;
     initDMD();
}

 afterEach deinitializeFrontend()
{
     import dmd.frontend : deinitializeDMD;
     deinitializeDMD();
}

 ("self test")
unittest
{
     import std.algorithm : each;
     import dmd.frontend;

     defaultImportPaths.each!addImport;

     auto t = parseModule("test.d", q{
         int a = 3;
     });

     assert(!t.diagnostics.hasErrors);
     assert(!t.diagnostics.hasWarnings);
}

You're free to test whatever you like inside the `unittest` block, the 
`parseModule` function is not required to use. Here's another example 
that does not use it [3].

I highly recommend splitting up the `unittest` blocks to only test one 
thing per `unittest` block.

I encourage all DMD contributors to give this a try. Using this kind of 
test will hopefully speed up the DMD test suite and make it possible to 
test things that are not tested today.

[1] https://github.com/dlang/dmd/tree/master/test/unit
[2] https://github.com/dlang/dmd/blob/master/test/unit/self_test.d
[3] https://github.com/dlang/dmd/blob/master/test/unit/deinitialization.d

-- 
/Jacob Carlborg

kinke <noone nowhere.com> writes:

Thanks, I think it's a nice tool to improve quality via more 
thorough/faster testing.