www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Should unittests run as logical part of compilation?

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
There's this simple realization that unittests could (should?) be 
considered an intrinsic part of the build process. In order for an 
executable to be worth running, it should pass the regular semantic 
checks and also the unittests, which in a sense are extended semantic 
checks that fall outside the traditional charter of the compiler.

In that view, asserts inside unittests should fail with the same message 
format as regular compilation errors, i.e.

./modulename.d(103): Unittest failed: user-defined message

Stack traces and other artifacts printed by failing asserts should come 
after, indented. An IDE user should run unittest with the usual "build" 
command and be able to click on the unittest failures just the same as 
build errors.

In particular, this view of unittests declares our current stance on 
running unittests ("run unittests just before main()") as meaningless. 
Indeed that has bothered me for quite a while - unittests are part of 
the build/acceptance, not part of every run. To wit, this is a growing 
idiom in D programs:

version(unittest) void main() {}
else void main()
{
     ...
}

What do you think? Logistically it shouldn't be too hard to arrange 
things to cater to this approach.


Andrei
Jan 25 2014
next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:

 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process. In order 
 for an executable to be worth running, it should pass the 
 regular semantic checks and also the unittests, which in a 
 sense are extended semantic checks that fall outside the 
 traditional charter of the compiler.

 In that view, asserts inside unittests should fail with the 
 same message format as regular compilation errors, i.e.

 ./modulename.d(103): Unittest failed: user-defined message

 Stack traces and other artifacts printed by failing asserts 
 should come after, indented. An IDE user should run unittest 
 with the usual "build" command and be able to click on the 
 unittest failures just the same as build errors.

 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless. Indeed that has bothered me for quite 
 a while - unittests are part of the build/acceptance, not part 
 of every run. To wit, this is a growing idiom in D programs:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

 What do you think? Logistically it shouldn't be too hard to 
 arrange things to cater to this approach.


 Andrei


+1, unittest passed as command line argument to dmd should run 
unittests as compilation step and main should remain untouched.
Jan 25 2014
prev sibling next sibling parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:

 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless. Indeed that has bothered me for quite 
 a while - unittests are part of the build/acceptance, not part 
 of every run. To wit, this is a growing idiom in D programs:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }


This idiom is probably mostly obsolete now that we have the -main 
flag. An IDE can build with -unittest -main -run and exclude the 
source file that normally defines `main`.


 What do you think? Logistically it shouldn't be too hard to 
 arrange things to cater to this approach.


I like it, as long as it doesn't mean unit tests are enabled by 
default, as that would probably be very surprising both for 
building release binaries and from a security perspective.
Jan 25 2014
parent =?UTF-8?B?U2ltZW4gS2rDpnLDpXM=?= <simen.kjaras gmail.com> writes:
On 2014-01-25 23:06, Jakob Ovrum wrote:

 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:

 In particular, this view of unittests declares our current stance on
 running unittests ("run unittests just before main()") as meaningless.
 Indeed that has bothered me for quite a while - unittests are part of
 the build/acceptance, not part of every run. To wit, this is a growing
 idiom in D programs:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }


 This idiom is probably mostly obsolete now that we have the -main flag.
 An IDE can build with -unittest -main -run and exclude the source file
 that normally defines `main`.


Except that would exclude any unit tests in the module that defines main.

Of course, if you want to be evil, you could even write unit tests for main:

int main() {
     return 3;
} unittest {
     assert(main() == 3);
}

--
   Simen
Jan 26 2014
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Sat, 25 Jan 2014 14:55:30 -0800
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 There's this simple realization that unittests could (should?) be 
 considered an intrinsic part of the build process. In order for an 
 executable to be worth running, it should pass the regular semantic 
 checks and also the unittests, which in a sense are extended semantic 
 checks that fall outside the traditional charter of the compiler.
 
 [....]
 
 Andrei


But please keep some way to manually run unittests - otherwise cross
compilers will have lots of fun running unittests ;-)

(IMHO this is the responsibility of a build tool, not of the compiler)
Jan 25 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:


 version(unittest) void main() {}
 else void main()
 {
     ...
 }


This is an example of bit a more general need: in D I'd like to 
create modules that can be imported from other modules to use 
their functionality; and to run their own main() when they are 
not imported from other modules. It's a simple feature that is 
still missing in D.

Bye,
bearophile
Jan 25 2014
prev sibling next sibling parent reply "Mike" <none none.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:

 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process. In order 
 for an executable to be worth running, it should pass the 
 regular semantic checks and also the unittests, which in a 
 sense are extended semantic checks that fall outside the 
 traditional charter of the compiler.
 [...]
 What do you think? Logistically it shouldn't be too hard to 
 arrange things to cater to this approach.


How does this work for those who build on one machine and test on 
another?  I'm currently building on a Linux host, and deploying 
to a bare-metal ARM Cortex-M processor.  Also, with limited 
amounts of flash, I may only have enough room for unit tests or 
my program, but not both.  (Just so you know, I don't actually 
use the unit test feature yet, but I intend to)

This seems more like a workflow feature rather than a build 
feature.  But integration into the toolchain could be pretty cool 
(like you said, clicking on a messages in the output window, and 
zooming right to the point of failure).

It seems to me that having an additional tool in the tool chain 
(the unit tester) along with the compiler, linker, debugger, 
etc.. that could be initiated by the build process, is the way to 
go.  If compiling unit tests, the build process could generate a 
unit test executable, run the unit test executable, and, assuming 
all tests pass, then generate the production executable.  I think 
such a tool could scale to my platform, somehow, I suppose.

Mike
Jan 25 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/25/14 4:26 PM, Mike wrote:

 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:

 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.
 [...]
 What do you think? Logistically it shouldn't be too hard to arrange
 things to cater to this approach.


 How does this work for those who build on one machine and test on
 another?


Yah, thanks (and Johannes too) for making this point. That means we 
should not prevent the option of running them separately.


 This seems more like a workflow feature rather than a build feature.
 But integration into the toolchain could be pretty cool (like you said,
 clicking on a messages in the output window, and zooming right to the
 point of failure).


That is correct - an build environment thing.


Andrei
Jan 26 2014
prev sibling next sibling parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:

 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process. In order 
 for an executable to be worth running, it should pass the 
 regular semantic checks and also the unittests, which in a 
 sense are extended semantic checks that fall outside the 
 traditional charter of the compiler.

 In that view, asserts inside unittests should fail with the 
 same message format as regular compilation errors, i.e.

 ./modulename.d(103): Unittest failed: user-defined message


I already attempt to get unittests to fail at compile time by 
using static assert. I think running it during compilation would 
be a great change. Though cross compiling is a valid concern and 
should be addressed.

Many are use to the notion that unittest are separate from 
compilation, I think this is only due to familiarity. Other than 
the cross-compile issue, I believe moving it closer to the time 
of compilation is the correct direction, but it will get distaste 
from those familiar with JUnit/NUnit and other unittesting 
frameworks just as it currently does.
Jan 25 2014
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-01-26 04:52, Jesse Phillips wrote:


 I already attempt to get unittests to fail at compile time by using
 static assert. I think running it during compilation would be a great
 change. Though cross compiling is a valid concern and should be addressed.


It's already possible to do CTFE unit tests:

http://forum.dlang.org/thread/ks1brj$1l6c$1 digitalmars.com

-- 
/Jacob Carlborg
Jan 26 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/25/14 7:52 PM, Jesse Phillips wrote:

 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:

 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.

 In that view, asserts inside unittests should fail with the same
 message format as regular compilation errors, i.e.

 ./modulename.d(103): Unittest failed: user-defined message


 I already attempt to get unittests to fail at compile time by using
 static assert. I think running it during compilation would be a great
 change.


That's great but I wasn't thinking of CTFEing unittests as much as just 
running them right after building.

Andrei
Jan 26 2014
parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Monday, 27 January 2014 at 03:45:00 UTC, Andrei Alexandrescu 
wrote:

 On 1/25/14 7:52 PM, Jesse Phillips wrote:

 I already attempt to get unittests to fail at compile time by 
 using
 static assert. I think running it during compilation would be 
 a great
 change.


 That's great but I wasn't thinking of CTFEing unittests as much 
 as just running them right after building.

 Andrei


I'm aware of that, my point being that I'm already trying to get 
them to run as frequently as possible.
Jan 26 2014
prev sibling next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:

 ./modulename.d(103): Unittest failed: user-defined message


My idea for pragma(error,) and pragma(warning,) would definately 
be invaluable to doing this without adding this specific 
behaviour. (Really should deal with that.)


 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless. Indeed that has bothered me for quite 
 a while - unittests are part of the build/acceptance, not part 
 of every run. To wit, this is a growing idiom in D programs:


Having the ability to run unittests at both compile time and 
runtime would be useful. Because what happens when you need to 
test e.g. an OS feature with it? Or have a dependency that simply 
cannot run at compile time?

I'm all for being able to selectively run unittests and having 
the ability to have some run at compile time.
Jan 25 2014
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Jan 26, 2014 at 04:29:10AM +0000, Rikki Cattermole wrote:

 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu
 wrote:


[...]

In particular, this view of unittests declares our current stance
on running unittests ("run unittests just before main()") as
meaningless. Indeed that has bothered me for quite a while -
unittests are part of the build/acceptance, not part of every run.
To wit, this is a growing idiom in D programs:


 Having the ability to run unittests at both compile time and runtime
 would be useful. Because what happens when you need to test e.g. an
 OS feature with it? Or have a dependency that simply cannot run at
 compile time?
 
 I'm all for being able to selectively run unittests and having the
 ability to have some run at compile time.


+1. Some of my unittests can only be run at runtime: like testing file
I/O.  Obviously, other unittests can also be run at compile-time, so
it's useful to have both.


T

-- 
Кто везде - тот нигде.
Jan 25 2014
next sibling parent reply "anonymous" <anonymous example.com> writes:
On Sunday, 26 January 2014 at 05:17:15 UTC, H. S. Teoh wrote:

 On Sun, Jan 26, 2014 at 04:29:10AM +0000, Rikki Cattermole 
 wrote:

 Having the ability to run unittests at both compile time and 
 runtime
 would be useful. Because what happens when you need to test 
 e.g. an
 OS feature with it? Or have a dependency that simply cannot 
 run at
 compile time?
 
 I'm all for being able to selectively run unittests and having 
 the
 ability to have some run at compile time.


 +1. Some of my unittests can only be run at runtime: like 
 testing file
 I/O.  Obviously, other unittests can also be run at 
 compile-time, so
 it's useful to have both.


I don't think the tests are supposed to be CTFE'd. They'd be
compiled (to a temporary executable?) and then run normally.
Jan 26 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 3:20 AM, anonymous wrote:

 On Sunday, 26 January 2014 at 05:17:15 UTC, H. S. Teoh wrote:

 On Sun, Jan 26, 2014 at 04:29:10AM +0000, Rikki Cattermole wrote:

 Having the ability to run unittests at both compile time and runtime
 would be useful. Because what happens when you need to test e.g. an
 OS feature with it? Or have a dependency that simply cannot run at
 compile time?

 I'm all for being able to selectively run unittests and having the
 ability to have some run at compile time.


 +1. Some of my unittests can only be run at runtime: like testing file
 I/O.  Obviously, other unittests can also be run at compile-time, so
 it's useful to have both.


 I don't think the tests are supposed to be CTFE'd. They'd be
 compiled (to a temporary executable?) and then run normally.


Yah, that was my idea.

Andrei
Jan 26 2014
prev sibling parent "Atila Neves" <atila.neves gmail.com> writes:
 +1. Some of my unittests can only be run at runtime: like 
 testing file
 I/O.  Obviously, other unittests can also be run at 
 compile-time, so
 it's useful to have both.


Tests that do I/O aren't unit tests. They're more likely 
integration tests.

Atila
Jan 27 2014
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-01-26 05:29, Rikki Cattermole wrote:


 I'm all for being able to selectively run unittests and having the
 ability to have some run at compile time.


That's already possible: 
http://forum.dlang.org/thread/ks1brj$1l6c$1 digitalmars.com

-- 
/Jacob Carlborg
Jan 26 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/25/14 8:29 PM, Rikki Cattermole wrote:

 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:

 ./modulename.d(103): Unittest failed: user-defined message


 My idea for pragma(error,) and pragma(warning,) would definately be
 invaluable to doing this without adding this specific behaviour. (Really
 should deal with that.)


What about static assert(false, "message")?

Andrei
Jan 26 2014
prev sibling next sibling parent reply "simendsjo" <simendsjo gmail.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:

 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process.


(...)


 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless.


(...)

I wouldn't mind having unittests be a part of the compilation 
process, but I really don't think "running before main" is 
useless.
This lets me compile a version on my desktop and push the binary 
to a small VPS staging for running runtime tests. Compiling any 
non-trivial code on 512MB with the DMD frontend is impossible. I 
can also send a fully working application with unittests to a 
client as a beta before giving a production version.
Jan 26 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Sunday, 26 January 2014 at 11:25:25 UTC, simendsjo wrote:

 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei 
 Alexandrescu wrote:

 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process.


 (...)


 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless.


 (...)

 I wouldn't mind having unittests be a part of the compilation 
 process, but I really don't think "running before main" is 
 useless.
 This lets me compile a version on my desktop and push the 
 binary to a small VPS staging for running runtime tests. 
 Compiling any non-trivial code on 512MB with the DMD frontend 
 is impossible. I can also send a fully working application with 
 unittests to a client as a beta before giving a production 
 version.


I simply think that "-unittest" should not emit any existing 
"main" function by default (only stub one if --main is set) and 
other than that we are pretty good at current stage. Running 
tests is necessary part of build system indeed but dmd is not a 
build system and should not force any assumptions about it.

Initial idea to make tests always run before the program starts 
does not work though, that is true. But currently we have enough 
tools to do it in any way specific project needs.
Jan 26 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 3:25 AM, simendsjo wrote:

 On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:

 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process.


 (...)


 In particular, this view of unittests declares our current stance on
 running unittests ("run unittests just before main()") as meaningless.


 (...)

 I wouldn't mind having unittests be a part of the compilation process,
 but I really don't think "running before main" is useless.
 This lets me compile a version on my desktop and push the binary to a
 small VPS staging for running runtime tests. Compiling any non-trivial
 code on 512MB with the DMD frontend is impossible. I can also send a
 fully working application with unittests to a client as a beta before
 giving a production version.


Does it help that the client runs them every time? I.e. does the n+1th 
run of the unittests bring more information than the first?

Andrei
Jan 26 2014
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Jan 26, 2014 at 07:53:30PM -0800, Andrei Alexandrescu wrote:

 On 1/26/14 3:25 AM, simendsjo wrote:

On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu wrote:

There's this simple realization that unittests could (should?) be
considered an intrinsic part of the build process.


(...)


In particular, this view of unittests declares our current stance on
running unittests ("run unittests just before main()") as
meaningless.


(...)

I wouldn't mind having unittests be a part of the compilation
process, but I really don't think "running before main" is useless.
This lets me compile a version on my desktop and push the binary to a
small VPS staging for running runtime tests. Compiling any
non-trivial code on 512MB with the DMD frontend is impossible. I can
also send a fully working application with unittests to a client as a
beta before giving a production version.


 
 Does it help that the client runs them every time? I.e. does the n+1th
 run of the unittests bring more information than the first?


[...]

It's very useful when you're in a code-compile-run development cycle,
when you're likely to be making some further changes after every
execution. Any regressions are quickly caught by unittests.

Obviously, it's useless to run the same unittests twice when nothing has
changed, but then, I thought the idea was that you *don't* compile with
-unittest when shipping the executable.


T

-- 
Once the bikeshed is up for painting, the rainbow won't suffice. -- Andrei
Alexandrescu
Jan 26 2014
prev sibling next sibling parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:

 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process. In order 
 for an executable to be worth running, it should pass the 
 regular semantic checks and also the unittests, which in a 
 sense are extended semantic checks that fall outside the 
 traditional charter of the compiler.

 In that view, asserts inside unittests should fail with the 
 same message format as regular compilation errors, i.e.

 ./modulename.d(103): Unittest failed: user-defined message

 Stack traces and other artifacts printed by failing asserts 
 should come after, indented. An IDE user should run unittest 
 with the usual "build" command and be able to click on the 
 unittest failures just the same as build errors.

 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless. Indeed that has bothered me for quite 
 a while - unittests are part of the build/acceptance, not part 
 of every run. To wit, this is a growing idiom in D programs:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

 What do you think? Logistically it shouldn't be too hard to 
 arrange things to cater to this approach.


 Andrei


I don't think unit tests should be run on *each* build - if I'm 
fidgeting with a specific piece of my code, and building&running 
it after each change, I don't want to waste time running the 
unittests of the entire project.

I do agree that unittests shouldn't be part of each run though. I 
think the `-unittest` flag should ignore the `main` function 
altogether, creating a unittests only executable, and the build 
system will be responsible for running it.

How about keeping the old behavior of `-unittest` for backward 
compatibility, and having a new syntax - `-unittest=filename`, 
where "filename" is the name of the unittests executable. That 
way both the unittest and the regular executables will be created 
at the same time - which will speed things up, as most of the 
compilation work can be shared between those two builds - and 
have different file names, so the build system - which will make 
configuring the build system more straightforward.

If we make the unittest a separate executable, there won't be a 
`main` function waiting for command line arguments, so we can use 
the command line arguments for the unittests. We can have flags 
for only running tests of specific packages, or for running the 
tests under certain restrictions etc.
Jan 26 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 4:58 AM, Idan Arye wrote:

 I don't think unit tests should be run on *each* build - if I'm
 fidgeting with a specific piece of my code, and building&running it
 after each change, I don't want to waste time running the unittests of
 the entire project.


Hah, interesting. So unittests of one module should run again iff there 
were changes in that module or other modules that that depends on.


 How about keeping the old behavior of `-unittest` for backward
 compatibility, and having a new syntax - `-unittest=filename`, where
 "filename" is the name of the unittests executable.


Sounds interesting.


 That way both the
 unittest and the regular executables will be created at the same time -
 which will speed things up, as most of the compilation work can be
 shared between those two builds - and have different file names, so the
 build system - which will make configuring the build system more
 straightforward.


Yeppers. One other thought I had was to define a special flag e.g. 
--4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run unittests".


 If we make the unittest a separate executable, there won't be a `main`
 function waiting for command line arguments, so we can use the command
 line arguments for the unittests. We can have flags for only running
 tests of specific packages, or for running the tests under certain
 restrictions etc.


There we go :o).


Andrei
Jan 26 2014
parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Monday, 27 January 2014 at 03:58:54 UTC, Andrei Alexandrescu 
wrote:

 Yeppers. One other thought I had was to define a special flag 
 e.g. --4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run 
 unittests".


I really think it would be better to use 
--4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the 
intent clearer.
Jan 26 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 8:08 PM, Jesse Phillips wrote:

 On Monday, 27 January 2014 at 03:58:54 UTC, Andrei Alexandrescu wrote:

 Yeppers. One other thought I had was to define a special flag e.g.
 --4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run unittests".


 I really think it would be better to use
 --4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the intent
 clearer.


I'm just saying it should not clash with any application argument.

Andrei
Jan 26 2014
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sun, Jan 26, 2014 at 08:15:23PM -0800, Andrei Alexandrescu wrote:

 On 1/26/14 8:08 PM, Jesse Phillips wrote:

On Monday, 27 January 2014 at 03:58:54 UTC, Andrei Alexandrescu wrote:

Yeppers. One other thought I had was to define a special flag e.g.
--4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run unittests".


I really think it would be better to use
--4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the intent
clearer.


 
 I'm just saying it should not clash with any application argument.


[...]

That won't work if the application is a git client looking for an SHA
hash argument...

Let's not add arbitrary things like this to D programs, it doesn't
address the issue and only introduces needless special cases.


T

-- 
"Hi." "'Lo."
Jan 26 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 10:08 PM, H. S. Teoh wrote:

 On Sun, Jan 26, 2014 at 08:15:23PM -0800, Andrei Alexandrescu wrote:

 On 1/26/14 8:08 PM, Jesse Phillips wrote:

 On Monday, 27 January 2014 at 03:58:54 UTC, Andrei Alexandrescu wrote:

 Yeppers. One other thought I had was to define a special flag e.g.
 --4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run unittests".


 I really think it would be better to use
 --4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the intent
 clearer.


 I'm just saying it should not clash with any application argument.


 [...]

 That won't work if the application is a git client looking for an SHA
 hash argument...


Of course it works unless there's a clash with this particular SHA.


 Let's not add arbitrary things like this to D programs, it doesn't
 address the issue and only introduces needless special cases.


I don't see that a special case. The only idea here is to produce one 
single debug executable containing unittests and running them on demand 
(e.g. by the build system).



Andrei
Jan 27 2014
prev sibling parent reply Alix Pexton <alix.DOT.pexton gmail.DOT.com> writes:
On 27/01/2014 4:15 AM, Andrei Alexandrescu wrote:

 On 1/26/14 8:08 PM, Jesse Phillips wrote:

 On Monday, 27 January 2014 at 03:58:54 UTC, Andrei Alexandrescu wrote:

 Yeppers. One other thought I had was to define a special flag e.g.
 --4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run unittests".


 I really think it would be better to use
 --4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the intent
 clearer.


 I'm just saying it should not clash with any application argument.

 Andrei


Hows about making it so that unittests are only run if the executable 
name has a "_ut" suffix, or some other special name/convention?

A...

---
This email is free from viruses and malware because avast! Antivirus protection
is active.
http://www.avast.com
Jan 27 2014
next sibling parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Monday, 27 January 2014 at 11:10:04 UTC, Alix Pexton wrote:

 On 27/01/2014 4:15 AM, Andrei Alexandrescu wrote:

 On 1/26/14 8:08 PM, Jesse Phillips wrote:

 On Monday, 27 January 2014 at 03:58:54 UTC, Andrei 
 Alexandrescu wrote:

 Yeppers. One other thought I had was to define a special 
 flag e.g.
 --4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run 
 unittests".


 I really think it would be better to use
 --4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the 
 intent
 clearer.


 I'm just saying it should not clash with any application 
 argument.

 Andrei


 Hows about making it so that unittests are only run if the 
 executable name has a "_ut" suffix, or some other special 
 name/convention?

 A...


How about exposing the sybmol of the function that runs the 
unittest, and having a "dunittest" tool for running the tests 
stored inside a regular executable? (I think it's possible?)
Jan 27 2014
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 27 January 2014 at 12:18:55 UTC, Idan Arye wrote:

 How about exposing the sybmol of the function that runs the 
 unittest, and having a "dunittest" tool for running the tests 
 stored inside a regular executable? (I think it's possible?)


It is already possible via library solution.
Jan 27 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/27/14 3:10 AM, Alix Pexton wrote:

 On 27/01/2014 4:15 AM, Andrei Alexandrescu wrote:

 On 1/26/14 8:08 PM, Jesse Phillips wrote:

 On Monday, 27 January 2014 at 03:58:54 UTC, Andrei Alexandrescu wrote:

 Yeppers. One other thought I had was to define a special flag e.g.
 --4c5ad7908c2aa1b3de32ea25968cdf49 that says "just run unittests".


 I really think it would be better to use
 --4c5ad7908c2aa1b3de42ea25968cdf49 instead, it just makes the intent
 clearer.


 I'm just saying it should not clash with any application argument.

 Andrei


 Hows about making it so that unittests are only run if the executable
 name has a "_ut" suffix, or some other special name/convention?

 A...


Something like that would work nicely with a build system - the 
executable name has a specific suffix and is promoted to the 
non-unittest suffix once it has passed unittests.

Andrei
Jan 27 2014
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 25 Jan 2014 17:55:30 -0500, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:


 What do you think? Logistically it shouldn't be too hard to arrange  
 things to cater to this approach.


After reading other opinions, this is what I think:

1. unit tests should be built as a separate binary. So when you build foo  
with -unittest, you get foo_unittest in addition to foo.
2. If you want to run unit tests, run foo_unittest. If you want simply the  
program, run foo.

I don't see a huge need for having unit tests run before the normal  
program. But maybe we should add a switch to make that work.

Building and running it as part of compilation seems like an incorrect  
function of the compiler. This is best left to an IDE/build script. Also,  
building unit tests still needs to be opt-in. Some projects can take a  
long time to build unit tests (dcollections takes about 20x longer to  
compile unit tests, last time I checked), and a quick compile-test-debug  
cycle is a key feature of D.

-Steve
Jan 26 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 12:43 PM, Steven Schveighoffer wrote:

 On Sat, 25 Jan 2014 17:55:30 -0500, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:


 What do you think? Logistically it shouldn't be too hard to arrange
 things to cater to this approach.


 After reading other opinions, this is what I think:

 1. unit tests should be built as a separate binary. So when you build
 foo with -unittest, you get foo_unittest in addition to foo.
 2. If you want to run unit tests, run foo_unittest. If you want simply
 the program, run foo.


Makes sense, though we could massage the unittests into the executable.


 I don't see a huge need for having unit tests run before the normal
 program. But maybe we should add a switch to make that work.


Guess we'd need to keep things running as they do for backwards 
compatibility.


 Building and running it as part of compilation seems like an incorrect
 function of the compiler. This is best left to an IDE/build script.
 Also, building unit tests still needs to be opt-in. Some projects can
 take a long time to build unit tests (dcollections takes about 20x
 longer to compile unit tests, last time I checked), and a quick
 compile-test-debug cycle is a key feature of D.


Yah it should be all nicely controllable by the build system.


Andrei
Jan 26 2014
prev sibling next sibling parent reply "Gary Willoughby" <dev nomad.so> writes:
On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:

 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process. In order 
 for an executable to be worth running, it should pass the 
 regular semantic checks and also the unittests, which in a 
 sense are extended semantic checks that fall outside the 
 traditional charter of the compiler.

 In that view, asserts inside unittests should fail with the 
 same message format as regular compilation errors, i.e.

 ./modulename.d(103): Unittest failed: user-defined message

 Stack traces and other artifacts printed by failing asserts 
 should come after, indented. An IDE user should run unittest 
 with the usual "build" command and be able to click on the 
 unittest failures just the same as build errors.

 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless. Indeed that has bothered me for quite 
 a while - unittests are part of the build/acceptance, not part 
 of every run. To wit, this is a growing idiom in D programs:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

 What do you think? Logistically it shouldn't be too hard to 
 arrange things to cater to this approach.


 Andrei


I don't fully understand this. Are you proposing to disconnect 
running unittest and evoking main? So unittests can be compiled 
and run without actually running the program?

If this is the case then yes it would be nice to be able to do 
this but i would also recommend keeping the current behaviour of 
invoking main after the unittests have run, purely because 
developers (myself included) like to run the application often in 
debug/unittest mode during development.
Jan 26 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 1:04 PM, Gary Willoughby wrote:

 I don't fully understand this. Are you proposing to disconnect running
 unittest and evoking main? So unittests can be compiled and run without
 actually running the program?


Right.


 If this is the case then yes it would be nice to be able to do this but
 i would also recommend keeping the current behaviour of invoking main
 after the unittests have run, purely because developers (myself
 included) like to run the application often in debug/unittest mode
 during development.


We'd need to preserve backward compatibility.


Andrei
Jan 26 2014
prev sibling next sibling parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
On 1/25/14, 7:55 PM, Andrei Alexandrescu wrote:

 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.


I can imagine someone who discovered a bug late at night, has a fix and 
needs to upload the new executable as soon as possible: he quickly 
comments all failing unit tests to make them pass. The next morning he 
uncomments them and fixes them with tranquility.
Jan 26 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/26/14 5:36 PM, Ary Borenszweig wrote:

 On 1/25/14, 7:55 PM, Andrei Alexandrescu wrote:

 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.


 I can imagine someone who discovered a bug late at night, has a fix and
 needs to upload the new executable as soon as possible: he quickly
 comments all failing unit tests to make them pass. The next morning he
 uncomments them and fixes them with tranquility.


The point being?

Andrei
Jan 26 2014
parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
On 1/27/14, 1:04 AM, Andrei Alexandrescu wrote:

 On 1/26/14 5:36 PM, Ary Borenszweig wrote:

 On 1/25/14, 7:55 PM, Andrei Alexandrescu wrote:

 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.


 I can imagine someone who discovered a bug late at night, has a fix and
 needs to upload the new executable as soon as possible: he quickly
 comments all failing unit tests to make them pass. The next morning he
 uncomments them and fixes them with tranquility.


 The point being?

 Andrei


That it's annoying if you can't build an executable because some tests 
fail. Sometimes you know tests fail but you didn't have time to fix them 
(but you did fix the code).
Jan 27 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/27/14 3:39 AM, Ary Borenszweig wrote:

 On 1/27/14, 1:04 AM, Andrei Alexandrescu wrote:

 On 1/26/14 5:36 PM, Ary Borenszweig wrote:

 On 1/25/14, 7:55 PM, Andrei Alexandrescu wrote:

 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.


 I can imagine someone who discovered a bug late at night, has a fix and
 needs to upload the new executable as soon as possible: he quickly
 comments all failing unit tests to make them pass. The next morning he
 uncomments them and fixes them with tranquility.


 The point being?

 Andrei


 That it's annoying if you can't build an executable because some tests
 fail. Sometimes you know tests fail but you didn't have time to fix them
 (but you did fix the code).


Got it, thanks.

Andrei
Jan 27 2014
prev sibling next sibling parent Knud Soerensen <4tuu4k002 sneakemail.com> writes:
I am already doing something very similar.

When I make a module I add this to the top.

(I would like to add -cov=100 but I can't get it to work)
Then I just run the module to compile and run unit tests.

My programs with main is always very small,
typical calling a module function with the arguments.

Doing the unit testing in the end of the compile would
mean that we could ensure that dependencies  was unit testing before
use. That would be nice.

Another thing I find really tedious about D unit testing can you read
about here.

http://forum.dlang.org/post/lc514a$sd7$1 digitalmars.com

Knud



On 2014-01-25 23:55, Andrei Alexandrescu wrote:

 There's this simple realization that unittests could (should?) be
 considered an intrinsic part of the build process. In order for an
 executable to be worth running, it should pass the regular semantic
 checks and also the unittests, which in a sense are extended semantic
 checks that fall outside the traditional charter of the compiler.
 
 In that view, asserts inside unittests should fail with the same message
 format as regular compilation errors, i.e.
 
 ./modulename.d(103): Unittest failed: user-defined message
 
 Stack traces and other artifacts printed by failing asserts should come
 after, indented. An IDE user should run unittest with the usual "build"
 command and be able to click on the unittest failures just the same as
 build errors.
 
 In particular, this view of unittests declares our current stance on
 running unittests ("run unittests just before main()") as meaningless.
 Indeed that has bothered me for quite a while - unittests are part of
 the build/acceptance, not part of every run. To wit, this is a growing
 idiom in D programs:
 
 version(unittest) void main() {}
 else void main()
 {
     ...
 }
 
 What do you think? Logistically it shouldn't be too hard to arrange
 things to cater to this approach.
 
 
 Andrei



-- 
Join me on
Skype	   knudhs
Facebook   http://www.facebook.com/profile.php?id=1198821880
Linkedin   http://www.linkedin.com/pub/0/117/a54
Twitter    http://twitter.com/knudsoerensen
bitcoin donations: 13ofyUKqFL43uRJHZtNozyMVP4qxKPsAR2
Jan 26 2014
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
I share the opinion others have expressed here that running unit 
tests should be the responsability of the build system.

Atila

On Saturday, 25 January 2014 at 22:55:33 UTC, Andrei Alexandrescu 
wrote:

 There's this simple realization that unittests could (should?) 
 be considered an intrinsic part of the build process. In order 
 for an executable to be worth running, it should pass the 
 regular semantic checks and also the unittests, which in a 
 sense are extended semantic checks that fall outside the 
 traditional charter of the compiler.

 In that view, asserts inside unittests should fail with the 
 same message format as regular compilation errors, i.e.

 ./modulename.d(103): Unittest failed: user-defined message

 Stack traces and other artifacts printed by failing asserts 
 should come after, indented. An IDE user should run unittest 
 with the usual "build" command and be able to click on the 
 unittest failures just the same as build errors.

 In particular, this view of unittests declares our current 
 stance on running unittests ("run unittests just before 
 main()") as meaningless. Indeed that has bothered me for quite 
 a while - unittests are part of the build/acceptance, not part 
 of every run. To wit, this is a growing idiom in D programs:

 version(unittest) void main() {}
 else void main()
 {
     ...
 }

 What do you think? Logistically it shouldn't be too hard to 
 arrange things to cater to this approach.


 Andrei
Jan 27 2014
prev sibling parent =?windows-1252?Q?Simen_Kj=E6r=E5s?= <simen.kjaras gmail.com> writes:
On 2014-01-25 22:55, Andrei Alexandrescu wrote:
[Good stuff]

 What do you think? Logistically it shouldn't be too hard to arrange
 things to cater to this approach.


As evinced by Bugzilla 5091[0], I always run unit tests separately from 
the rest of the program, and so fully support this motion. As others 
have pointed out, there are possible problems with cross-compilation, 
but all in all, I think treating unit tests as a build step rather than 
something to do with execution is a Good Thing™.


[0]: https://d.puremagic.com/issues/show_bug.cgi?id=5091

--
   Simen
Jan 27 2014