www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dub test

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Why does "dub test" rebuild the library? It should just re-run 
unittests. -- Andrei
Aug 23 2016
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-08-23 19:55, Andrei Alexandrescu wrote:

 Why does "dub test" rebuild the library? It should just re-run
 unittests. -- Andrei


Rebuilds it with unit tests enabled? You can the "--verbose" and/or the 
"--vverbose" to see exactly the commands that Dub are running.

-- 
/Jacob Carlborg
Aug 23 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/23/2016 03:14 PM, Jacob Carlborg wrote:

 On 2016-08-23 19:55, Andrei Alexandrescu wrote:

 Why does "dub test" rebuild the library? It should just re-run
 unittests. -- Andrei


 Rebuilds it with unit tests enabled? You can the "--verbose" and/or the
 "--vverbose" to see exactly the commands that Dub are running.


It does this with every run per --verbose (it should not rebuild the 
test program):

Using dub registry url 'http://code.dlang.org/'
Refreshing local packages (refresh existing: true)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at 
/home/andrei/.dub/packages/local-packages.json
Determined package version using GIT: checkedint_andralex 0.0.2
Refreshing local packages (refresh existing: false)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at 
/home/andrei/.dub/packages/local-packages.json
Checking for missing dependencies.
Refreshing local packages (refresh existing: false)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at 
/home/andrei/.dub/packages/local-packages.json
Checking for upgrades.
Using cached upgrade results...
Generating test runner configuration '__test__library__' for 'library' 
(library).
Get module name from path: 
/home/andrei/d/dub/checkedint_andralex/source/checkedint.d
Refreshing local packages (refresh existing: false)...
Looking for local package map at /var/lib/dub/packages/local-packages.json
Looking for local package map at 
/home/andrei/.dub/packages/local-packages.json
Generate target checkedint_andralex (executable 
/home/andrei/d/dub/checkedint_andralex __test__library__)
Performing "unittest" build using dmd for x86_64.
File 
'../../../../../tmp/dub_test_root-8422d71d-ca6f-4781-9307-b7ff54bb0b8f.d' 
modified, need rebuild.
checkedint_andralex 0.0.2: building configuration "__test__library__"...
dmd -c 
-of.dub/build/__test__library__-unittest-linux.posix-x86_64-dmd_2072-0DDE52948EB4C282B88B8F001B09B726
__test__library__.o 
-debug -g -unittest -w -version=VibeCustomMain 
-version=Have_checkedint_andralex -Isource/ 
../../../../../tmp/dub_test_root-8422d71d-ca6f-4781-9307-b7ff54bb0b8f.d 
source/checkedint.d -vcolumns
Linking...
dmd 
-of.dub/build/__test__library__-unittest-linux.posix-x86_64-dmd_2072-0DDE52948EB4C282B88B8F001B09B7
6/__test__library__ 
.dub/build/__test__library__-unittest-linux.posix-x86_64-dmd_2072-0DDE52948EB4C282B88B8F001B09B726
__test__library__.o 
-L--no-as-needed -g
Copying target from 
/home/andrei/d/dub/checkedint_andralex/.dub/build/__test__library__-unittest-linux.posix-x86_64-dmd_2072-0DDE52948EB4C282B88B8F001B09B7
6/__test__library__ 
to /home/andrei/d/dub/checkedint_andralex
Running ./__test__library__
All unit tests have been run successfully.



Andrei
Aug 23 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-08-23 21:26, Andrei Alexandrescu wrote:

 On 08/23/2016 03:14 PM, Jacob Carlborg wrote:

 On 2016-08-23 19:55, Andrei Alexandrescu wrote:

 Why does "dub test" rebuild the library? It should just re-run
 unittests. -- Andrei


 Rebuilds it with unit tests enabled? You can the "--verbose" and/or the
 "--vverbose" to see exactly the commands that Dub are running.


 It does this with every run per --verbose (it should not rebuild the
 test program):


You mean this line?

dmd -c -of.dub/build/__test__library__ ...

That's builds a generated module which imports all modules in the 
project. It looks like this is a temporary file that is removed after 
each run.

As Walter would say, if it's not in bugzilla it will never be fixed ;)

-- 
/Jacob Carlborg
Aug 23 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/23/2016 04:06 PM, Jacob Carlborg wrote:

 On 2016-08-23 21:26, Andrei Alexandrescu wrote:

 On 08/23/2016 03:14 PM, Jacob Carlborg wrote:

 On 2016-08-23 19:55, Andrei Alexandrescu wrote:

 Why does "dub test" rebuild the library? It should just re-run
 unittests. -- Andrei


 Rebuilds it with unit tests enabled? You can the "--verbose" and/or the
 "--vverbose" to see exactly the commands that Dub are running.


 It does this with every run per --verbose (it should not rebuild the
 test program):


 You mean this line?

 dmd -c -of.dub/build/__test__library__ ...

 That's builds a generated module which imports all modules in the
 project. It looks like this is a temporary file that is removed after
 each run.

 As Walter would say, if it's not in bugzilla it will never be fixed ;)


https://issues.dlang.org/show_bug.cgi?id=16422 -- Andrei
Aug 23 2016
parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 23 August 2016 at 20:26:12 UTC, Andrei Alexandrescu 
wrote:

 As Walter would say, if it's not in bugzilla it will never be 
 fixed ;)


 https://issues.dlang.org/show_bug.cgi?id=16422 -- Andrei


https://github.com/dlang/dub/issues is full of existing issues, 
not sure whether it's best to split reports between there and 
issues.dlang.org
Aug 23 2016
parent Seb <seb wilzba.ch> writes:
On Tuesday, 23 August 2016 at 21:51:57 UTC, John Colvin wrote:

 On Tuesday, 23 August 2016 at 20:26:12 UTC, Andrei Alexandrescu 
 wrote:

 As Walter would say, if it's not in bugzilla it will never be 
 fixed ;)


 https://issues.dlang.org/show_bug.cgi?id=16422 -- Andrei


 https://github.com/dlang/dub/issues is full of existing issues, 
 not sure whether it's best to split reports between there and 
 issues.dlang.org


Yep, Sönke is highly overloaded with the amount of issues popping 
in.

Reminder: he also maintains our flagship web framework vibe.d, 
ddox, a proposal for std.data.json, the dub-registry itself, a 
couple of other projects and his full-time job. We really need 
more manpower behind DUB!
Aug 23 2016
prev sibling parent reply Edwin van Leeuwen <edder tkwsping.nl> writes:
On Tuesday, 23 August 2016 at 17:55:41 UTC, Andrei Alexandrescu 
wrote:

 Why does "dub test" rebuild the library? It should just re-run 
 unittests. -- Andrei


To be honest I prefer it to rebuild it. Why would you want to 
rerun the tests without a rebuild? It should give the same 
failure/success as last time.
Aug 23 2016
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-08-24 07:44, Edwin van Leeuwen wrote:


 To be honest I prefer it to rebuild it. Why would you want to rerun the
 tests without a rebuild? It should give the same failure/success as last
 time.


Sure, but one might not figure why a test is failing after only a single 
run. One might use "dub test" to run integration tests that depends on 
some external dependency that is causing the test to fail. There are 
many reasons to run the test without recompiling.

What is the point of recompiling when nothing has changed?

-- 
/Jacob Carlborg
Aug 23 2016
parent reply Edwin van Leeuwen <edder tkwsping.nl> writes:
On Wednesday, 24 August 2016 at 06:32:54 UTC, Jacob Carlborg 
wrote:

 Sure, but one might not figure why a test is failing after only 
 a single run. One might use "dub test" to run integration tests 
 that depends on some external dependency that is causing the 
 test to fail.


I always considered dub test to be only for running unittests and 
not really suited for integration tests, but if that is a use 
case then fair enough.


 There are many reasons to run the test without recompiling.


I might be dense, but the only other thing than integration tests 
that I can think of is if you use random data for testing, but 
that would be more correctly solved by using more random data 
during the unittests. Nothing is worse than tests that only 
sometimes fail.


 What is the point of recompiling when nothing has changed?


I'd be all for for dub to analyse whether there are any changes, 
but AFAIK dub's focus is as a packaging system, not as a build 
system. In that case I'd prefer it to ere on the side of caution 
and recompile more often in case there might be a change. Maybe 
the key would be to replace dub's limited build system by an 
external build system such as reggea.
Aug 24 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/24/2016 04:14 AM, Edwin van Leeuwen wrote:

 I might be dense, but the only other thing than integration tests that I
 can think of is if you use random data for testing, but that would be
 more correctly solved by using more random data during the unittests.
 Nothing is worse than tests that only sometimes fail.


Randomized unit testing is a respected approach to testing: 
https://hackage.haskell.org/package/QuickCheck, 
https://github.com/rickynils/scalacheck, 
https://blogs.msdn.microsoft.com/dsyme/2008/08/08/fscheck-0-2/, etc. 
Some of the Phobos tests I wrote use it (and virtually all of my current 
code e.g. on median computation), and when they fail I just print the 
seed of the RNG and then hardcode it to reproduce the test. -- Andrei
Aug 24 2016
next sibling parent reply Seb <seb wilzba.ch> writes:
On Wednesday, 24 August 2016 at 12:00:32 UTC, Andrei Alexandrescu 
wrote:

 On 08/24/2016 04:14 AM, Edwin van Leeuwen wrote:

 I might be dense, but the only other thing than integration 
 tests that I
 can think of is if you use random data for testing, but that 
 would be
 more correctly solved by using more random data during the 
 unittests.
 Nothing is worse than tests that only sometimes fail.


 Randomized unit testing is a respected approach to testing: 
 https://hackage.haskell.org/package/QuickCheck, 
 https://github.com/rickynils/scalacheck, 
 https://blogs.msdn.microsoft.com/dsyme/2008/08/08/fscheck-0-2/, 
 etc. Some of the Phobos tests I wrote use it (and virtually all 
 of my current code e.g. on median computation), and when they 
 fail I just print the seed of the RNG and then hardcode it to 
 reproduce the test. -- Andrei


Do you remember that there is a pending PR for this at Phobos 
that is blocked because of missing feedback / acceptance?

https://github.com/dlang/phobos/pull/2995
Aug 24 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/24/2016 09:25 AM, Seb wrote:

 On Wednesday, 24 August 2016 at 12:00:32 UTC, Andrei Alexandrescu wrote:

 On 08/24/2016 04:14 AM, Edwin van Leeuwen wrote:

 I might be dense, but the only other thing than integration tests that I
 can think of is if you use random data for testing, but that would be
 more correctly solved by using more random data during the unittests.
 Nothing is worse than tests that only sometimes fail.


 Randomized unit testing is a respected approach to testing:
 https://hackage.haskell.org/package/QuickCheck,
 https://github.com/rickynils/scalacheck,
 https://blogs.msdn.microsoft.com/dsyme/2008/08/08/fscheck-0-2/, etc.
 Some of the Phobos tests I wrote use it (and virtually all of my
 current code e.g. on median computation), and when they fail I just
 print the seed of the RNG and then hardcode it to reproduce the test.
 -- Andrei


 Do you remember that there is a pending PR for this at Phobos that is
 blocked because of missing feedback / acceptance?

 https://github.com/dlang/phobos/pull/2995


I know. What I need to write:

unittest
{
     void theFunctionToTest(Gen!(int, 1, 5) a, Gen!(float, 0.0, 10.0) b)
     {
         // This will always be true
         assert(a >= 1 && a <= 5);
         assert(a >= 0.0 && a <= 10.0);

         // super expensive operation
         auto rslt = (a + b);
         doNotOptimizeAway(rslt);

         debug
         {
             assert(rslt > 1.0);
         }
     }

     benchmark!theFunctionToTest();
}

What I want to write:

 Benchmark(ParamDomain!1(1, 5), ParamDomain!2(0, 10), Aggregate.median)
void theFunctionToTest(int a, float b)
{
     // This will always be true
     assert(a >= 1 && a <= 5);
     assert(a >= 0.0 && a <= 10.0);

     // super expensive operation
     auto rslt = (a + b);
     doNotOptimizeAway(rslt);
     debug
     {
         assert(rslt > 1.0);
     }
}

Attributes are a very attractive (if not the best) way to set up 
benchmarks. There are a variety of Java benchmarking frameworks we could 
draw inspiration from, see list at 
http://stackoverflow.com/questions/7146207/what-is-the-best-macro-benchmarking-tool-framework-to-measu
e-a-single-threade. 
Take a look e.g. at 
https://github.com/google/caliper/blob/master/examples/src/main/java/examples/B
tSetBenchmark.java, 
which is so clear and simple.


Andrei
Aug 24 2016
next sibling parent reply Robert burner Schadek <rburners gmail.com> writes:
Lets move that back to https://github.com/dlang/phobos/pull/2995
Aug 24 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/24/2016 10:40 AM, Robert burner Schadek wrote:

 Lets move that back to https://github.com/dlang/phobos/pull/2995


D'accordo. -- Andrei
Aug 24 2016
prev sibling parent Lurker <lurker gmail.com> writes:
On Wednesday, 24 August 2016 at 13:52:43 UTC, Andrei Alexandrescu 
wrote:

     // This will always be true
     assert(a >= 1 && a <= 5);
     assert(a >= 0.0 && a <= 10.0);


Pretty sure you have a bug here :)
Aug 31 2016
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Aug 24, 2016 at 08:00:32AM -0400, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]

 Some of the Phobos tests I wrote use it (and virtually all of my
 current code e.g. on median computation), and when they fail I just
 print the seed of the RNG and then hardcode it to reproduce the test.
 -- Andrei


Just a side note: this depends on the assumption that the PRNG is
explicitly chosen, or that the default PRNG will never change; and that
the implementation of the RNG is independent of environmental factors
(such as different CPU's resulting in different answers).

Otherwise, if you find a bug with a specific seed and then hardcode that
seed in a unittest to prevent regression, a regression may go undetected
once somebody changes the default RNG, or tweaks the implementation
(e.g. to fix a RNG weakness due to security concerns) yielding different
results for the same seed, or is run in an environment other than the
one you tested it in.


T

-- 
An elephant: A mouse built to government specifications. -- Robert Heinlein
Aug 24 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/24/2016 10:41 AM, H. S. Teoh via Digitalmars-d wrote:

 Just a side note: this depends on the assumption that the PRNG is
 explicitly chosen, or that the default PRNG will never change;


Correctamundo, it should be explicitly chosen. -- Andrei
Aug 24 2016
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Wednesday, 24 August 2016 at 12:00:32 UTC, Andrei Alexandrescu 
wrote:

 On 08/24/2016 04:14 AM, Edwin van Leeuwen wrote:

 I might be dense, but the only other thing than integration 
 tests that I
 can think of is if you use random data for testing, but that 
 would be
 more correctly solved by using more random data during the 
 unittests.
 Nothing is worse than tests that only sometimes fail.


 Randomized unit testing is a respected approach to testing: 
 https://hackage.haskell.org/package/QuickCheck, 
 https://github.com/rickynils/scalacheck, 
 https://blogs.msdn.microsoft.com/dsyme/2008/08/08/fscheck-0-2/, 
 etc. Some of the Phobos tests I wrote use it (and virtually all 
 of my current code e.g. on median computation), and when they 
 fail I just print the seed of the RNG and then hardcode it to 
 reproduce the test. -- Andrei


unit-threaded has had QuickCheck-like property-based testing 
features for a while now. As mentioned previously in the forum, I 
wrote this test in cerealed:


 Types!(bool, byte, ubyte, short, ushort, int, uint, long, ulong,
         float, double,
         char, wchar, dchar,
         ubyte[], ushort[], int[], long[], float[], double[])
void testEncodeDecodeProperty(T)() {
     check!((T val) {
         auto enc = Cerealiser();
         enc ~= val;
         auto dec = Decerealiser(enc.bytes);
         return dec.value!T == val;
     });
}


Checks that for every type in the list that given a random value 
of that type, serialising then deserialising should yield the 
same value back.


Atila
Aug 30 2016
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 08/24/2016 01:44 AM, Edwin van Leeuwen wrote:

 On Tuesday, 23 August 2016 at 17:55:41 UTC, Andrei Alexandrescu wrote:

 Why does "dub test" rebuild the library? It should just re-run
 unittests. -- Andrei


 To be honest I prefer it to rebuild it. Why would you want to rerun the
 tests without a rebuild? It should give the same failure/success as last
 time.


Randomized testing. -- Andrei
Aug 24 2016