www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 5091] New: main runs after unittests

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5091

           Summary: main runs after unittests
           Product: D
           Version: D1 & D2
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: enhancement
          Priority: P2
         Component: DMD
        AssignedTo: nobody puremagic.com
        ReportedBy: simen.kjaras gmail.com



PDT ---
When compiling a program with dmd -unittest, after the unittests are run,
main() is called. This is rarely wanted behavior, so should not be the default.

Currently, it is possible to customize main to sidestep the problem:

void main( ) {
    version( unittest ) {
    } else {
        // Your program
    }
}

However, this is cluttering and mixes your application code with versioning
code.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 20 2010
next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5091


Peter Alexander <peter.alexander.au gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |peter.alexander.au gmail.co
                   |                            |m



01:07:14 PDT ---


 When compiling a program with dmd -unittest, after the unittests are run,
 main() is called. This is rarely wanted behavior, so should not be the default.


Why is this rarely wanted?

If your way was the default then your work flow would be:

- Make change
- Run with unit test
- Run without unit test
- Make change
- Run with unit test
- Run without unit test
- etc.

The whole point of unit tests is that they are run regularly, preferably after
ever change. If you only run them after lots of changes then it makes it much
more difficult to find out what broke the build.

If you have a lot of time-consuming unit tests in your code then these should
be put inside a slowtest version block so that you can leave those for your
nightly testing (or whatever you want to do). In general however, unittest
should always be on.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 21 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5091




01:11:22 PDT ---
Even if you don't buy that argument, consider this: with the current unittest,
you only have to add a few lines of code to turn it off (as you described).
This is something that you'll only ever have to do once, and really isn't that
much of a burden.

On the other hand, if running main was off by default then there would be no
way to turn it back on! You would have to resort to scripting your build to get
it working again.

Therefore, the current unittest is unarguably more flexible to people's needs,
and arguably more desirable anyway.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 21 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5091


bearophile_hugs eml.cc changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |bearophile_hugs eml.cc





 Why is this rarely wanted?


D unittesting needs a global redesign, not small changes.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 21 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5091




PDT ---




 When compiling a program with dmd -unittest, after the unittests are run,
 main() is called. This is rarely wanted behavior, so should not be the default.


 
 Why is this rarely wanted?


Because I never want my program to run after I run my unittests? If the
unittests pass, my program should be working, apart from things that are
impossible/hard to unittest.


 If your way was the default then your work flow would be:
 
 - Make change
 - Run with unit test
 - Run without unit test
 - Make change
 - Run with unit test
 - Run without unit test
 - etc.


That's one possibility. When I program, I aim for such coverage with my
unittests that running the program to see if things work is not necessary.
Hence, a more likely scenario:

1 Make change
2 Run unittests (quick)
3 If unittests fail, goto 1
4 Run normally (slow)
5 Goto 1

I have not once wanted to run both the full application and the unittests at
the same time. when I want to run the unittests, running the application would
be a waste of time. When I want to run the program, I don't care about the
unittests.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 21 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5091




00:43:56 PDT ---


 I have not once wanted to run both the full application and the unittests at
 the same time. when I want to run the unittests, running the application would
 be a waste of time. When I want to run the program, I don't care about the
 unittests.


Well I have to say that this is totally alien to me.

Putting our disagreement on exactly when unit tests should be run, we can at
least agree on 3 things:

1. Some people want to run the program and unit tests at the same time, and
others don't.
2. Leaving things as they are, you have the choice between the two (although
one choice requires a few extra lines of code).
3. Changing things makes only one option possible.

Unless I've missed anything, I think this is a clear argument for leaving
things as they are, regardless of which choice we may be biased towards.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Oct 22 2010
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5091


Andrej Mitrovic <andrej.mitrovich gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |andrej.mitrovich gmail.com



17:14:36 PST ---
You can override this behavior by defining your own unittest function:

import core.runtime;
import std.stdio;
import std.string;
import std.path;

bool testRunner()
{
    foreach (m; ModuleInfo)
    {
        if (m is null)
            continue;

        if (auto fp = m.unitTest)
        {
            try
            {
                fp();
            }
            catch (Throwable e)
            {
                writeln(e);
            }
        }
    }

    // stop execution after tests are done
    return false;
}

shared static this()
{
    Runtime.moduleUnitTester(&testRunner);
}

unittest
{
    writeln("in unittest");
    assert(1);
}

void main()
{
    writeln("not in main");
}

--------------

I've written a similar test system where I can also set these at runtime via
getopt switches:

- specify which modules to run the tests on (--testMods=foo.bar)
- ditto but set modules which should *not* have their tests ran on
(--ignoreMods=foo.bar.doo)
- specify whether to run the unittests, and whether to run main() if the tests
were successful

I might write a DWiki entry later on how to do this if there's any interest.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jan 07 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5091


Martin Nowak <code dawg.eu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |code dawg.eu



It's a little hacky though, because returning false is interpreted as failed
tests.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Mar 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5091




12:17:24 PDT ---


 It's a little hacky though, because returning false is interpreted as failed
 tests.


Yeah, I've realized that after posting. Currently I use getOpt and a --runMain
switch in my own code.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Mar 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5091




16:27:36 PDT ---


 It's a little hacky though, because returning false is interpreted as failed
 tests.


Perhaps we could change ModuleUnitTester to return an int instead of a bool.
Then 0 would mean tests failed, 1 would mean continue execution to main, and a
new return (say 2) would mean don't execute main().

runModuleUnitTests would also have to return this status code, and then in
dmain2.d we'd change:

if (runModuleUnitTests())

to:

if (runModuleUnitTests() == 1)

Of course it might be best to use an enumeration for this:

TestStatus
{
    Fail, Ok, SkipMain
}

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Mar 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5091




This always confused me a little because the documentation is inconsistent.
http://dlang.org/phobos/core_runtime.html#moduleUnitTester
http://dlang.org/phobos/core_runtime.html#runModuleUnitTests
One could also require that test failure is indicated by an exception.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Mar 13 2013
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5091




20:49:45 PDT ---


 This always confused me a little because the documentation is inconsistent.
 http://dlang.org/phobos/core_runtime.html#moduleUnitTester
 http://dlang.org/phobos/core_runtime.html#runModuleUnitTests


Yes me too.


 One could also require that test failure is indicated by an exception.


Speaking of which I just noticed that `runAll` doesn't wrap
`runModuleUnitTests()` in a try/catch. That seems like a bug to me.

rt_init does however.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Mar 13 2013
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=5091






 Speaking of which I just noticed that `runAll` doesn't wrap
 `runModuleUnitTests()` in a try/catch. That seems like a bug to me.
 


It does
https://github.com/D-Programming-Language/druntime/blob/master/src/rt/dmain2.d#L613.

tryExec(&runAll);

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Mar 13 2013