www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Benchmark

reply Robert burner Schadek <rburners gmail.com> writes:
At this years DConf I gave a very short overview of the 
benchmarking library I'm working on [1]. The source can be found 
here [2]

The general idea goes as following.
Unit tests in D help us to find bugs and avoid regressions, but 
they do not cover performance regressions. That is bad, we are 
basically, currently just hoping that performance diminishes.

So my idea is to eventually get this library into phobos, write 
benchmarks for all functions in phobos, execute the benchmarks 
for every merge into master, use gnuplot to display the results 
on the dlang webpage, profit.

Right now I need people to give feedback on the library. You can 
find it on code.dlang.org [3]



[1] 
https://www.youtube.com/watch?v=QELK73JSpFk&list=PL3jwVPmk_PRxo23yyoc0Ip_cP3-rCm7eB&index=30
[2] https://github.com/burner/benchmark
[3] http://code.dlang.org/packages/std_benchmark
Jun 01 2017
next sibling parent Robert burner Schadek <rburners gmail.com> writes:
On Thursday, 1 June 2017 at 21:08:43 UTC, Robert burner Schadek 
wrote:

 they do not cover performance regressions. That is bad, we are 
 basically, currently just hoping that performance diminishes.


diminishes -> does not diminishes
Jun 01 2017
prev sibling next sibling parent reply Jack Stouffer <jack jackstouffer.com> writes:
On Thursday, 1 June 2017 at 21:08:43 UTC, Robert burner Schadek 
wrote:

 So my idea is to eventually get this library into phobos, write 
 benchmarks for all functions in phobos, execute the benchmarks 
 for every merge into master, use gnuplot to display the results 
 on the dlang webpage


And reject out of hand any PR which reduces performance in order 
to "ratchet" performance over time.

I would also do this to a lesser degree to DMD as well.
Jun 02 2017
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 06/02/2017 02:49 PM, Jack Stouffer wrote:

 On Thursday, 1 June 2017 at 21:08:43 UTC, Robert burner Schadek wrote:

 So my idea is to eventually get this library into phobos, write 
 benchmarks for all functions in phobos, execute the benchmarks for 
 every merge into master, use gnuplot to display the results on the 
 dlang webpage


 
 And reject out of hand any PR which reduces performance in order to 
 "ratchet" performance over time.
 
 I would also do this to a lesser degree to DMD as well.


That's indeed worth it. Robert, we discussed first deploying the library 
and a driver for dmd/druntime/phobos on dub, is this your plan? Thanks! 
-- Andrei
Jun 02 2017
parent reply Robert burner Schadek <rburners gmail.com> writes:
On Friday, 2 June 2017 at 18:51:24 UTC, Andrei Alexandrescu wrote:

 That's indeed worth it. Robert, we discussed first deploying 
 the library and a driver for dmd/druntime/phobos on dub, is 
 this your plan? Thanks! -- Andrei


I think a practical approach is to create another dub project 
that solely consists of benchmarks for functions of phobos, and 
prints pretty pictures of their performance.
This will properly help to find shortcomings of the benchmark 
library, and help to develop idioms for benchmarking and random 
testing.
If that works out, the next step is properly to integrate that in 
the phobos/druntime/dmd CI. Which properly means putting the 
benchmark library through the experimental integration process, 
and finding a place for the benchmarks to life.
Jun 02 2017
parent David Nadlinger <code klickverbot.at> writes:
On Friday, 2 June 2017 at 21:56:47 UTC, Robert burner Schadek 
wrote:

 If that works out, the next step is properly to integrate that 
 in the phobos/druntime/dmd CI. Which properly means putting the 
 benchmark library through the experimental integration process, 
 and finding a place for the benchmarks to life.


As I've pointed out before, having the benchmark support code be 
part of the public standard library is not required for the 
Phobos project to make use of it internally.

  — David
Jun 03 2017
prev sibling parent Robert burner Schadek <rburners gmail.com> writes:
On Friday, 2 June 2017 at 18:49:49 UTC, Jack Stouffer wrote:

 And reject out of hand any PR which reduces performance in 
 order to "ratchet" performance over time.

 I would also do this to a lesser degree to DMD as well.


I think being binary about it is the wrong approach.
Lets say we fix a bug, and the function is slower afterwards. Do 
we merge the bugfix? I say yes.
Lets say we add a feature to a function that makes more useful. 
Do we merge it? I say yes.

This is just another hammer, it does not mean everything is a 
nail now.
Jun 02 2017
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2017-06-01 23:08, Robert burner Schadek wrote:


 So my idea is to eventually get this library into phobos, write
 benchmarks for all functions in phobos, execute the benchmarks for every
 merge into master, use gnuplot to display the results on the dlang
 webpage, profit.


In Xcode you can actually have benchmarks that behave as tests. That is, 
you write your benchmark. Define a boundary for a set of platforms, if 
the benchmark takes longer to run then the boundary it will fail the 
test. Then you don't need anyone to constantly look at the result. Of 
course, this only work if you have a limited set of platforms with a 
consistent performance.

-- 
/Jacob Carlborg
Jun 03 2017