• Johan Engelen (17/17) Jan 03 2016 Hi all,
• David Nadlinger via digitalmars-d-ldc (14/16) Jan 03 2016 For the tests that are truly specific to LDC this certainly makes sense.
• Johan Engelen (14/33) Jan 03 2016 For non-zero return values, there is the "not" LLVM utility:

Johan Engelen <j j.nl> writes:

Hi all,
   I've come to like the "lit"-based test suite a lot. It is much 
easier (for me at least) to run those tests than to run the DMD 
testsuite, especially on Windows.

I propose:
- Move the lit python script from ./tests/ir to ./tests.
- Exclude the "d2" folder from lit testing
- Move the LDC-specific tests from the DMD testsuite into a few 
other ./tests/abc directories and run them with lit instead

A big immediate benefit that I see is that it will be easier to 
add testcases and that testcases can be added in the same 
merge/commit as the implementation/fix for the testcase. (the 
submodule thing is a little annoying)
Also, I think the lit system has more flexibility for testing.

What do you think?

cheers,
   Johan

David Nadlinger via digitalmars-d-ldc <digitalmars-d-ldc puremagic.com> writes:

On 3 Jan 2016, at 19:36, Johan Engelen via digitalmars-d-ldc wrote:
 - Move the LDC-specific tests from the DMD testsuite into a few other 
 ./tests/abc directories and run them with lit instead

For the tests that are truly specific to LDC this certainly makes sense.

As fas as the general D tests are concerned that just happened to cause 
an issue in the LDC glue code but not DMD, I think we should actually 
try to upstream them to the DMD testsuite, as other compilers will also 
benefit from the improved coverage. (Not sure whether the powers that be 
agree with that, though.) For new test cases, we could add them to the 
lit-based suite first and upstream them every so often.

How suitable is the lit-based workflow for fail_compilation diagnostic 
checks? I'm not very familiar with it (yet).

Another question is that of organization. Dumping everything (dmd 
testsuite runner, general CMake stuff, lit config, individual lit test 
case files) directly into tests/ might get a bit messy in the long run.

  — David

Johan Engelen <j j.nl> writes:

On Sunday, 3 January 2016 at 18:59:16 UTC, David Nadlinger wrote:
 On 3 Jan 2016, at 19:36, Johan Engelen via digitalmars-d-ldc 
 wrote:
 - Move the LDC-specific tests from the DMD testsuite into a 
 few other ./tests/abc directories and run them with lit instead

 For the tests that are truly specific to LDC this certainly 
 makes sense.

 As fas as the general D tests are concerned that just happened 
 to cause an issue in the LDC glue code but not DMD, I think we 
 should actually try to upstream them to the DMD testsuite, as 
 other compilers will also benefit from the improved coverage. 
 (Not sure whether the powers that be agree with that, though.) 
 For new test cases, we could add them to the lit-based suite 
 first and upstream them every so often.

Agreed.

 How suitable is the lit-based workflow for fail_compilation 
 diagnostic checks? I'm not very familiar with it (yet).

For non-zero return values, there is the "not" LLVM utility:
    not ldc2 compile_error.d
returns zero if ldc2 errored, and returns non-zero if ldc2 
succeeded. There are many tests like that in LLVM/Clang. So then 
I think you'd get something like:
// RUN:  not %ldc2 %s 2&>1 | FileCheck %s
// Test error diagnostic:
// CHECK:  error: bla bla.

 Another question is that of organization. Dumping everything 
 (dmd testsuite runner, general CMake stuff, lit config, 
 individual lit test case files) directly into tests/ might get 
 a bit messy in the long run.

I want to pull lit config, lit runner, cmake stuff out of 
/tests/ir, and put it in /tests. Then all testcases should go 
into a subdirectory of /tests: /tests/codegen, /tests/profile, 
tests/runnable, /tests/whatever.