www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Parameterized unit testing and benchmarking of phobos

reply "Robert burner Schadek" <rburners gmail.com> writes:
Lately, I find myself wondering, if I should add parameterized 
unit tests to std.string, because the last few bugs I fixed where 
not caught by tests, as the test-data was not good enough. I know 
random data is not perfect either, but it would be good addition 
IMO.

Additionally, I thought these unit tests could be used to 
benchmark the performance of the functions and the compilers. 
Thus allowing continues monitoring of performance changes.

I'm thinking of something like:

version(unittest_benchmark) {
unittest {
     auto ben = Benchmark("bench_result_file", numberOfRuns);
     auto values = ValueGen!(StringGen(0, 12), IntGen(2, 24))(ben);

     foreach(string rStr, int rInt; values) {
         auto rslt = functionToTest(rStr, rInt);
         // some asserts
     }
}
}

The bench_result_file would be a csv with e.g. date,runtime,...

ideas, suggestions?
Oct 08 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/8/14, 2:37 PM, Robert burner Schadek wrote:

 version(unittest_benchmark) {
 unittest {


No need for the outer braces :o). -- Andrei
Oct 08 2014
parent reply "Robert burner Schadek" <rburners gmail.com> writes:
On Wednesday, 8 October 2014 at 23:31:59 UTC, Andrei Alexandrescu 
wrote:

 On 10/8/14, 2:37 PM, Robert burner Schadek wrote:

 version(unittest_benchmark) {
 unittest {


 No need for the outer braces :o). -- Andrei


I just love my braces.
If that's gone be the most negative comment, I will have a PR 
ready before next week.
Oct 08 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/8/14, 4:44 PM, Robert burner Schadek wrote:

 On Wednesday, 8 October 2014 at 23:31:59 UTC, Andrei Alexandrescu wrote:

 On 10/8/14, 2:37 PM, Robert burner Schadek wrote:

 version(unittest_benchmark) {
 unittest {


 No need for the outer braces :o). -- Andrei


 I just love my braces.


If you love your braces you gotta love your indentation. They come 
together... -- Andrei
Oct 08 2014
parent reply "Robert burner Schadek" <rburners gmail.com> writes:
On Thursday, 9 October 2014 at 00:13:25 UTC, Andrei Alexandrescu 
wrote:


 If you love your braces you gotta love your indentation. They 
 come together... -- Andrei


I guess you're right, but back to the topic.
Has anyone tried something similar for phobos? (I couldn't find 
anything)
Does anyone else consider the idea worth pursuing?
Oct 09 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/9/14, 1:01 AM, Robert burner Schadek wrote:

 On Thursday, 9 October 2014 at 00:13:25 UTC, Andrei Alexandrescu wrote:


 If you love your braces you gotta love your indentation. They come
 together... -- Andrei


 I guess you're right, but back to the topic.
 Has anyone tried something similar for phobos? (I couldn't find anything)
 Does anyone else consider the idea worth pursuing?


Not yet but I think we need to create a task and a plan for adding the 
old std.benchmark to std.experimental.benchmark, and use it internally 
for Phobos. -- Andrei
Oct 09 2014
parent reply Johannes Pfau <nospam example.com> writes:
Am Thu, 09 Oct 2014 08:11:33 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 On 10/9/14, 1:01 AM, Robert burner Schadek wrote:

 On Thursday, 9 October 2014 at 00:13:25 UTC, Andrei Alexandrescu
 wrote:


 If you love your braces you gotta love your indentation. They come
 together... -- Andrei


 I guess you're right, but back to the topic.
 Has anyone tried something similar for phobos? (I couldn't find
 anything) Does anyone else consider the idea worth pursuing?


 
 Not yet but I think we need to create a task and a plan for adding
 the old std.benchmark to std.experimental.benchmark, and use it
 internally for Phobos. -- Andrei


+1, std.benchmark should be revived. There's still the constructor
cycles issue but we can probably find workarounds. In the worst
case we do not generate constructors but runBenchmark functions and
call these manually via some runAllBenchmarks("std.xyz", "std.abc")
template magic)
Oct 09 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/9/14, 9:36 AM, Johannes Pfau wrote:

 Am Thu, 09 Oct 2014 08:11:33 -0700
 schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:


 On 10/9/14, 1:01 AM, Robert burner Schadek wrote:

 On Thursday, 9 October 2014 at 00:13:25 UTC, Andrei Alexandrescu
 wrote:


 If you love your braces you gotta love your indentation. They come
 together... -- Andrei


 I guess you're right, but back to the topic.
 Has anyone tried something similar for phobos? (I couldn't find
 anything) Does anyone else consider the idea worth pursuing?


 Not yet but I think we need to create a task and a plan for adding
 the old std.benchmark to std.experimental.benchmark, and use it
 internally for Phobos. -- Andrei


 +1, std.benchmark should be revived. There's still the constructor
 cycles issue but we can probably find workarounds. In the worst
 case we do not generate constructors but runBenchmark functions and
 call these manually via some runAllBenchmarks("std.xyz", "std.abc")
 template magic)


My suggestion is to first use it in std.internal for a couple of months, 
and only then propose it for inclusion. -- Andrei
Oct 09 2014
prev sibling parent reply "Robert burner Schadek" <rburners gmail.com> writes:
On Thursday, 9 October 2014 at 16:36:35 UTC, Johannes Pfau wrote:>

 +1, std.benchmark should be revived.



-1, This really does not solve my problem. I need a parameterized 
unit test facility aka. Haskell QuickCheck with the additional 
benchmarking feature that allows me/us to track performance over 
time aka. benchmark result dump to file in csv style manner.
Oct 09 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 10/9/14, 9:57 AM, Robert burner Schadek wrote:

 On Thursday, 9 October 2014 at 16:36:35 UTC, Johannes Pfau wrote:>

 +1, std.benchmark should be revived.



 -1, This really does not solve my problem. I need a parameterized unit
 test facility aka. Haskell QuickCheck with the additional benchmarking
 feature that allows me/us to track performance over time aka. benchmark
 result dump to file in csv style manner.


version (xyz) unittest
{
    ...
}

is a good idiom. I noticed you feel strongly about bracing but I 
unrecommend it (and we should disallow it for phobos) as it forces you 
to either double-indent everything (net loss) or create an exception 
with regard to indentation. Also the opening extra brace has a "garden 
path sentence" liability, it suggests more stuff will be encompassed by 
the brace but then only at the closing brace there's the realization 
there's only encompassed entity.

The idiom is already used, see e.g. 
https://github.com/D-Programming-Language/phobos/blob/master/std/file.d#L322.


Andrei
Oct 09 2014
parent "Robert burner Schadek" <rburners gmail.com> writes:
I get the bracing point, but how does this translate to my unit 
testing (quickcheck) problem?
Oct 09 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 08/10/14 23:37, Robert burner Schadek wrote:

 Lately, I find myself wondering, if I should add parameterized unit
 tests to std.string, because the last few bugs I fixed where not caught
 by tests, as the test-data was not good enough. I know random data is
 not perfect either, but it would be good addition IMO.

 Additionally, I thought these unit tests could be used to benchmark the
 performance of the functions and the compilers. Thus allowing continues
 monitoring of performance changes.

 I'm thinking of something like:

 version(unittest_benchmark) {
 unittest {
      auto ben = Benchmark("bench_result_file", numberOfRuns);
      auto values = ValueGen!(StringGen(0, 12), IntGen(2, 24))(ben);

      foreach(string rStr, int rInt; values) {
          auto rslt = functionToTest(rStr, rInt);
          // some asserts
      }
 }
 }

 The bench_result_file would be a csv with e.g. date,runtime,...

 ideas, suggestions?


I think you should use a UDA's instead. The a unit test framework can, 
hopefully, handle this automatically.

-- 
/Jacob Carlborg
Oct 09 2014
parent reply "Robert burner Schadek" <rburners gmail.com> writes:
On Friday, 10 October 2014 at 06:39:25 UTC, Jacob Carlborg wrote:


 I think you should use a UDA's instead. The a unit test 
 framework can, hopefully, handle this automatically.


This will work for me private project, but I want this in 
std.string.
Oct 10 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 10/10/14 10:09, Robert burner Schadek wrote:


 This will work for me private project, but I want this in std.string.


Why wouldn't this working std.string?

-- 
/Jacob Carlborg
Oct 10 2014
parent "Robert burner Schadek" <rburners gmail.com> writes:
On Friday, 10 October 2014 at 14:20:39 UTC, Jacob Carlborg wrote:

 On 10/10/14 10:09, Robert burner Schadek wrote:


 This will work for me private project, but I want this in 
 std.string.


 Why wouldn't this working std.string?


because, this would require changing the phobos unittester, doing 
it non intrusive inside the unittest is way more likely to 
succeed. add is more flexible IMO.
Oct 12 2014
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 10/08/2014 11:37 PM, Robert burner Schadek wrote:

 Lately, I find myself wondering, if I should add parameterized unit
 tests to std.string, because the last few bugs I fixed where not caught
 by tests, as the test-data was not good enough. I know random data is
 not perfect either, but it would be good addition IMO.


https://github.com/MartinNowak/qcheck
Haven't used it for quite a while, file a bug report if anything doesn't 
work.
Oct 13 2014
parent reply "Robert burner Schadek" <rburners gmail.com> writes:
On Tuesday, 14 October 2014 at 06:54:42 UTC, Martin Nowak wrote:

 On 10/08/2014 11:37 PM, Robert burner Schadek wrote:

 Lately, I find myself wondering, if I should add parameterized 
 unit
 tests to std.string, because the last few bugs I fixed where 
 not caught
 by tests, as the test-data was not good enough. I know random 
 data is
 not perfect either, but it would be good addition IMO.


 https://github.com/MartinNowak/qcheck
 Haven't used it for quite a while, file a bug report if 
 anything doesn't work.


well quite a nice and big library. You add the benchmark feature, 
get a merged into phobos and I will gladly use it to test 
std.string.
Oct 14 2014
parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 10/14/2014 10:38 AM, Robert burner Schadek wrote:

 well quite a nice and big library. You add the benchmark feature, get a
 merged into phobos and I will gladly use it to test std.string.


Not sure whether a random testing library belongs into phobos.
Oct 14 2014
parent "Robert burner Schadek" <rburners gmail.com> writes:
On Tuesday, 14 October 2014 at 12:17:05 UTC, Martin Nowak wrote:

 On 10/14/2014 10:38 AM, Robert burner Schadek wrote:

 well quite a nice and big library. You add the benchmark 
 feature, get a
 merged into phobos and I will gladly use it to test std.string.


 Not sure whether a random testing library belongs into phobos.


After taking a look at the bug I found in std.string, I think 
that this is a good first step.
Oct 14 2014