www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Benchmarking suite

reply "qznc" <qznc web.de> writes:
Since issue 13487 [0] seems to rot away, I started something on 
my own.
Made a benchmark script and inserted C/C++/D programs for 
comparison.

However, various programs are broken, as you see in the example 
report [1].
The D code is at least 7 years old. I only fixed compile errors.
The C/C++ programs were selected quite randomly.

It should be easy to checkout the repo [2] and run the benchmarks 
yourself
as long as you run on Linux:

   git clone git github.com:qznc/d-shootout.git
   cd d-shootout
   ./benchmark.d --quickly
   xdg-open index.html

Maybe somebody has already fixed or improved benchmark programs?



[0] https://issues.dlang.org/show_bug.cgi?id=13487
[1] https://qznc.github.io/d-shootout/
[2] https://github.com/qznc/d-shootout
Aug 29 2015
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 29-Aug-2015 15:05, qznc wrote:

 Since issue 13487 [0] seems to rot away, I started something on my own.
 Made a benchmark script and inserted C/C++/D programs for comparison.

 However, various programs are broken, as you see in the example report [1].
 The D code is at least 7 years old. I only fixed compile errors.
 The C/C++ programs were selected quite randomly.

 It should be easy to checkout the repo [2] and run the benchmarks yourself
 as long as you run on Linux:

    git clone git github.com:qznc/d-shootout.git
    cd d-shootout
    ./benchmark.d --quickly
    xdg-open index.html

 Maybe somebody has already fixed or improved benchmark programs?


Well, here is the regex-dna one with 3 versions including C-T regex:

https://github.com/DmitryOlshansky/FReD/blob/master/bench/regex-dna/d_dna.d

Could be trivially parallelized with std.parallelism.

-- 
Dmitry Olshansky
Aug 29 2015
parent reply "qznc" <qznc web.de> writes:
On Saturday, 29 August 2015 at 12:35:14 UTC, Dmitry Olshansky 
wrote:

 Well, here is the regex-dna one with 3 versions including C-T 
 regex:

 https://github.com/DmitryOlshansky/FReD/blob/master/bench/regex-dna/d_dna.d


Thanks Dmitry!

Which version should be used?
Aug 29 2015
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 29-Aug-2015 21:14, qznc wrote:

 On Saturday, 29 August 2015 at 12:35:14 UTC, Dmitry Olshansky wrote:

 Well, here is the regex-dna one with 3 versions including C-T regex:

 https://github.com/DmitryOlshansky/FReD/blob/master/bench/regex-dna/d_dna.d


 Thanks Dmitry!

 Which version should be used?


I'd try all of them, I think C-T was the fastest (as it should).

-- 
Dmitry Olshansky
Aug 29 2015
parent reply "qznc" <qznc web.de> writes:
On Saturday, 29 August 2015 at 19:17:47 UTC, Dmitry Olshansky 
wrote:

 On 29-Aug-2015 21:14, qznc wrote:

 On Saturday, 29 August 2015 at 12:35:14 UTC, Dmitry Olshansky 
 wrote:

 Well, here is the regex-dna one with 3 versions including C-T 
 regex:

 https://github.com/DmitryOlshansky/FReD/blob/master/bench/regex-dna/d_dna.d


 Thanks Dmitry!

 Which version should be used?


 I'd try all of them, I think C-T was the fastest (as it should).


Yes, C-T is fastest. Even dmd is faster than C/C++ now. :)
Aug 30 2015
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 30-Aug-2015 16:21, qznc wrote:

 On Saturday, 29 August 2015 at 19:17:47 UTC, Dmitry Olshansky wrote:

 On 29-Aug-2015 21:14, qznc wrote:

 On Saturday, 29 August 2015 at 12:35:14 UTC, Dmitry Olshansky wrote:

 Well, here is the regex-dna one with 3 versions including C-T regex:

 https://github.com/DmitryOlshansky/FReD/blob/master/bench/regex-dna/d_dna.d


 Thanks Dmitry!

 Which version should be used?


 I'd try all of them, I think C-T was the fastest (as it should).


 Yes, C-T is fastest. Even dmd is faster than C/C++ now. :)


Was one of the first benchmarks where std.regex destroyed the 
competition. It may still do so ;)

-- 
Dmitry Olshansky
Aug 30 2015
parent reply "qznc" <qznc web.de> writes:
On Sunday, 30 August 2015 at 14:56:34 UTC, Dmitry Olshansky wrote:

 Was one of the first benchmarks where std.regex destroyed the 
 competition. It may still do so ;)


Rust has compile-time regex as well now.

http://doc.rust-lang.org/regex/regex/index.html#the-regex!-macro
Aug 30 2015
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 30-Aug-2015 19:57, qznc wrote:

 On Sunday, 30 August 2015 at 14:56:34 UTC, Dmitry Olshansky wrote:

 Was one of the first benchmarks where std.regex destroyed the
 competition. It may still do so ;)


 Rust has compile-time regex as well now.

 http://doc.rust-lang.org/regex/regex/index.html#the-regex!-macro


Yeahm I've seen that. Last year they were just catching up, they may 
have production quality stuff by now.


-- 
Dmitry Olshansky
Aug 30 2015
prev sibling parent reply "qznc" <qznc web.de> writes:
On Sunday, 30 August 2015 at 13:21:42 UTC, qznc wrote:

 On Saturday, 29 August 2015 at 19:17:47 UTC, Dmitry Olshansky 
 wrote:

 On 29-Aug-2015 21:14, qznc wrote:

 On Saturday, 29 August 2015 at 12:35:14 UTC, Dmitry Olshansky 
 wrote:

 Well, here is the regex-dna one with 3 versions including 
 C-T regex:

 https://github.com/DmitryOlshansky/FReD/blob/master/bench/regex-dna/d_dna.d


 Thanks Dmitry!

 Which version should be used?


 I'd try all of them, I think C-T was the fastest (as it 
 should).


 Yes, C-T is fastest. Even dmd is faster than C/C++ now. :)


Unfortunately, I have to take that back. C is faster than D even 
with compile-time regexes. I used the short running benchmarks 
first, where compile-time regex wins, probably because it saves 
some startup time. For large data, C is faster. It uses the regex 
engine from TCL. Maybe std.regex has just space for optimization?

I updated the benchmark results: 
https://qznc.github.io/d-shootout/
Sep 07 2015
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 07-Sep-2015 11:29, qznc wrote:

 Maybe std.regex has just space for optimization?


Sure thing, see WIP here (~25% faster but not yet complete):
https://github.com/D-Programming-Language/phobos/pull/3314

-- 
Dmitry Olshansky
Sep 07 2015
next sibling parent "Suliman" <evermind live.ru> writes:
On Monday, 7 September 2015 at 08:33:33 UTC, Dmitry Olshansky 
wrote:

 On 07-Sep-2015 11:29, qznc wrote:

 Maybe std.regex has just space for optimization?


 Sure thing, see WIP here (~25% faster but not yet complete):
 https://github.com/D-Programming-Language/phobos/pull/3314
Sep 07 2015
prev sibling parent "Jack Stouffer" <jack jackstouffer.com> writes:
On Monday, 7 September 2015 at 08:33:33 UTC, Dmitry Olshansky 
wrote:

 On 07-Sep-2015 11:29, qznc wrote:

 Maybe std.regex has just space for optimization?


 Sure thing, see WIP here (~25% faster but not yet complete):
 https://github.com/D-Programming-Language/phobos/pull/3314


It's been over a year since the last commit on that. Any updates?
Sep 08 2015
prev sibling next sibling parent "qznc" <qznc web.de> writes:
On Saturday, 29 August 2015 at 12:05:18 UTC, qznc wrote:

 Maybe somebody has already fixed or improved benchmark programs?


As of now, most things work.

Only meteor.d is broken. Crashes at runtime.

Ldc and gdc sometimes fail, because they are behind dmd.

regexdna.cpp fails, because re2 is not available via Ubuntu apt.

Many benchmarks need some performance tuning, though. We should 
not lose to C/C++.
Some benchmarks are suspiciously fast, which means probably wrong.
Aug 30 2015
prev sibling next sibling parent reply "Robert burner Schadek" <rburners gmail.com> writes:
Why not go really big. aka:

http://forum.dlang.org/post/vzcvwrbqpeamtnopmtoa forum.dlang.org
Sep 08 2015
parent reply "qznc" <qznc web.de> writes:
On Tuesday, 8 September 2015 at 08:24:43 UTC, Robert burner 
Schadek wrote:

 Why not go really big. aka:

 http://forum.dlang.org/post/vzcvwrbqpeamtnopmtoa forum.dlang.org


You suggest to create a benchmark suite from all the unittests in 
Phobos?

I don't think this is a good idea. Most programs don't make good 
performance tests. Even the Benchmarks Game / Shootout benchmarks 
are partially stupid. For example, threadring measures context 
switching. The best strategy is "use pthreads and pthread mutex 
and restrict to one core". It only shows how good your language 
can access the pthread API. The context switching is done by 
Linux. The pidigits programs basically measures libGMP 
performance.

I'm all for adding more programs into the benchmark suite, but 
they should be carefully selected to measure different aspects. I 
don't understand all the programs enough to know what is lacking. 
Probably some memory management aspects. Maybe some concurrency 
stuff.
Sep 08 2015
next sibling parent "qznc" <qznc web.de> writes:
On Tuesday, 8 September 2015 at 09:27:13 UTC, qznc wrote:

 On Tuesday, 8 September 2015 at 08:24:43 UTC, Robert burner 
 Schadek wrote:

 Why not go really big. aka:

 http://forum.dlang.org/post/vzcvwrbqpeamtnopmtoa forum.dlang.org


 You suggest to create a benchmark suite from all the unittests 
 in Phobos?

 I don't think this is a good idea. Most programs don't make 
 good performance tests.


Read your PR more closely now. We already agree on that. :P

Still a difference: You propose a method to measure Phobos 
performance and detect regressions. My shootout wants to compare 
D and C/C++ performance.
Sep 08 2015
prev sibling parent reply "Isaac Gouy" <igouy2 yahoo.com> writes:
On Tuesday, 8 September 2015 at 09:27:13 UTC, qznc wrote:


 For example, threadring measures context switching.


thread-ring has aged badly. It was added when the measurements 
were only made on single-core hardware, and Erlang's huge number 
of lightweight processes seemed interesting ;-)

It's been many years since the thread-ring measurements were 
included in the summary charts.



 The pidigits programs basically measures libGMP performance.


And arbitrary precision arithmetic without libGMP :-)
Sep 08 2015
parent "qznc" <qznc web.de> writes:
On Tuesday, 8 September 2015 at 18:53:02 UTC, Isaac Gouy wrote:

 On Tuesday, 8 September 2015 at 09:27:13 UTC, qznc wrote:


 For example, threadring measures context switching.


 thread-ring has aged badly. It was added when the measurements 
 were only made on single-core hardware, and Erlang's huge 
 number of lightweight processes seemed interesting ;-)

 It's been many years since the thread-ring measurements were 
 included in the summary charts.


It is interesting that Erlang and others are considered 
"preemptive" threads. Afaik the Erlang runtime does not interrupt 
processes.


 The pidigits programs basically measures libGMP performance.


 And arbitrary precision arithmetic without libGMP :-)


In this comparison it is actually interesting, because D has its 
own bignum implementation in the standard library. It holds well 
against libGMP.
Sep 08 2015
prev sibling parent reply "Isaac Gouy" <igouy2 yahoo.com> writes:
On Saturday, 29 August 2015 at 12:05:18 UTC, qznc wrote:


 I started something on my own.


Kudos to qznc!



 The C/C++ programs were selected quite randomly.


Note: There are separate C and C++ programs shown on the 
benchmarks game -- so for something like regex-dna there's a C 
program using the C library written for Tcl and there's a C++ 
program using the re library.

fwiw Doing both would make your comparison a little broader.
Sep 08 2015
parent reply "qznc" <qznc web.de> writes:
On Tuesday, 8 September 2015 at 18:41:10 UTC, Isaac Gouy wrote:

 On Saturday, 29 August 2015 at 12:05:18 UTC, qznc wrote:


 I started something on my own.


 Kudos to qznc!



 The C/C++ programs were selected quite randomly.


 Note: There are separate C and C++ programs shown on the 
 benchmarks game -- so for something like regex-dna there's a C 
 program using the C library written for Tcl and there's a C++ 
 program using the re library.

 fwiw Doing both would make your comparison a little broader.


Yes. I'm not sure how to structure this whole suite. The general 
goal is "D claims that it can match C/C++ in performance, let's 
have some actual numbers". So far D mostly disappoints in terms 
of performance.

There are at least three interesting variations "fastest parallel 
programs", "fastest sequential programs" and "short idiomatic 
programs". Probably all of them should be compared.
Sep 08 2015
next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 8 September 2015 at 21:11:15 UTC, qznc wrote:

 Yes. I'm not sure how to structure this whole suite. The 
 general goal is "D claims that it can match C/C++ in 
 performance, let's have some actual numbers". So far D mostly 
 disappoints in terms of performance.


The most interesting thing to test is how they fare with high 
level optimization, not low level optimization. So make sure the 
implementation is similar...
Sep 08 2015
parent reply "qznc" <qznc web.de> writes:
On Tuesday, 8 September 2015 at 23:20:05 UTC, Ola Fosheim Grøstad 
wrote:

 On Tuesday, 8 September 2015 at 21:11:15 UTC, qznc wrote:

 Yes. I'm not sure how to structure this whole suite. The 
 general goal is "D claims that it can match C/C++ in 
 performance, let's have some actual numbers". So far D mostly 
 disappoints in terms of performance.


 The most interesting thing to test is how they fare with high 
 level optimization, not low level optimization. So make sure 
 the implementation is similar...


I'm not sure if I understand you correctly. What is "high level" 
and "low level" optimization?

What I want to know is a) how fast is "idiomatic" D code (using 
ranges etc) compared to "idiomatic" C/C++ and b) how do they 
compare if you push performance to the limits (code beauty be 
damned).

For a) you want a similar implementation although C/C++ will most 
certainly always loose in terms of length and convenience.

For b) we don't care. C/C++ is free to use builtins, pragmas, and 
whatnot. If for loops are faster than ranges in D, then we will 
use for loops here.
Sep 09 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 9 September 2015 at 07:59:48 UTC, qznc wrote:

 I'm not sure if I understand you correctly. What is "high 
 level" and "low level" optimization?


Low level are local optimizations, which basically will be the 
same if you use the same backend (like LLVM). It would just 
measure the programmer's approach and not the compiler.


 What I want to know is a) how fast is "idiomatic" D code (using 
 ranges etc) compared to "idiomatic" C/C++ and b) how do they 
 compare if you push performance to the limits (code beauty be 
 damned).

 For a) you want a similar implementation although C/C++ will 
 most certainly always loose in terms of length and convenience.

 For b) we don't care. C/C++ is free to use builtins, pragmas, 
 and whatnot. If for loops are faster than ranges in D, then we 
 will use for loops here.


The problem with a) is that in C++ there are many libraries and 
you'll have a hard time finding comparable alternatives on both 
sides... People don't stick to the standard libraries in C++.

For b) C++ will be slightly faster because of things like modular 
arithmetics and OpenMP support.

I think the better approach is to write up the same algorithms in 
a high level fashion (using generic templates on both sides) from 
the ground up using the same constructs and measure the ability 
to optimize.

Otherwise you end up comparing apples and oranges in a rather 
subjective manner.
Sep 09 2015
parent reply "qznc" <qznc web.de> writes:
On Wednesday, 9 September 2015 at 09:56:10 UTC, Ola Fosheim 
Grøstad wrote:

 I think the better approach is to write up the same algorithms 
 in a high level fashion (using generic templates on both sides) 
 from the ground up using the same constructs and measure the 
 ability to optimize.


That is a good idea, if you want to measure compiler 
optimizations. Ideally g++ and gdc should always yield the same 
performance then?

However, it does answer the wrong question imho.

Suppose you consider using D with C/C++ as the stable 
alternative. D lures you with its high level features. However, 
you know that you will have to really optimize some hot spots 
sooner or later. Will D impose a penalty on you and C/C++ could 
have provided better performance?

Walter argues that there is no technical reason why D should be 
slower than C/C++. My experience with the benchmarks says, there 
seem to be such penalties. For example, there is no 
__builtin_ia32_cmplepd or __builtin_ia32_movmskpd like gcc has.
Sep 09 2015
next sibling parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9 September 2015 at 16:00, qznc via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Wednesday, 9 September 2015 at 09:56:10 UTC, Ola Fosheim Gr=C3=B8stad =


wrote:

 I think the better approach is to write up the same algorithms in a high
 level fashion (using generic templates on both sides) from the ground up
 using the same constructs and measure the ability to optimize.


 That is a good idea, if you want to measure compiler optimizations.
 Ideally g++ and gdc should always yield the same performance then?

 However, it does answer the wrong question imho.

 Suppose you consider using D with C/C++ as the stable alternative. D lure=


s

 you with its high level features. However, you know that you will have to
 really optimize some hot spots sooner or later. Will D impose a penalty o=


n

 you and C/C++ could have provided better performance?

 Walter argues that there is no technical reason why D should be slower
 than C/C++. My experience with the benchmarks says, there seem to be such
 penalties. For example, there is no __builtin_ia32_cmplepd or
 __builtin_ia32_movmskpd like gcc has.


import gcc.builtins;  // OK, cheating. :-)
Sep 09 2015
parent "qznc" <qznc web.de> writes:
On Wednesday, 9 September 2015 at 14:09:36 UTC, Iain Buclaw wrote:

 import gcc.builtins;  // OK, cheating. :-)


Thanks, I did not know this. :)

I would not consider it cheating. Using builtins in C is not 
portable C11 either. It also shows off how D does versions.
Sep 09 2015
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 9 September 2015 at 14:00:07 UTC, qznc wrote:

 That is a good idea, if you want to measure compiler 
 optimizations. Ideally g++ and gdc should always yield the same 
 performance then?


Hopefully, as I understand GCC uses a highlevel IR, but if 
performance is equal that is a pretty strong argument to get 
people to adopt, if the rest of the language is polished.

And you could measure regressions.


 Suppose you consider using D with C/C++ as the stable 
 alternative. D lures you with its high level features. However, 
 you know that you will have to really optimize some hot spots 
 sooner or later. Will D impose a penalty on you and C/C++ could 
 have provided better performance?

 Walter argues that there is no technical reason why D should be 
 slower than C/C++. My experience with the benchmarks says, 
 there seem to be such penalties. For example, there is no 
 __builtin_ia32_cmplepd or __builtin_ia32_movmskpd like gcc has.


Ok, I see your point. You want to measure maximum throughput for 
critical applications that might benefit from language specific 
intrinsics.

Multithreaded applications could probably show some differences 
too, due to TLS/shared...

Maybe some kind of actor based benchmark. Essentially running 
thousands of fibers with lots of intercommunication.
Sep 09 2015
prev sibling next sibling parent "Isaac Gouy" <igouy2 yahoo.com> writes:
On Tuesday, 8 September 2015 at 21:06:26 UTC, qznc wrote:


 Afaik the Erlang runtime does not interrupt processes.


Depends what you mean by "processes" :-)


 In this comparison it is actually interesting, because D has 
 its own bignum implementation in the standard library.


There you go!



On Tuesday, 8 September 2015 at 21:11:15 UTC, qznc wrote:


 The general goal is "D claims that it can match C/C++ in 
 performance, let's have some actual numbers".


- You're only dealing with 3 programming languages, although more 
than 3 language implementations

- Those programming languages are intended to be used for similar 
tasks.

- You'll correctly be seen as a D language advocate, so your 
presentation needs to show that you accept advice on how to 
improve the C and C++ programs.

- "short idiomatic programs" is difficult because the tradeoff 
between performance and "idiomatic" is so subjective, and you 
will correctly be seen as a D language advocate :-)

    When asked, one of the C++ program contributors to the 
benchmarks game did try to write some "shorter" C++ programs, see:

    
http://benchmarksgame.alioth.debian.org/u64/code-used-time-used-shapes.php#shortest
Sep 09 2015
prev sibling parent reply Russel Winder via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Tue, 2015-09-08 at 21:11 +0000, qznc via Digitalmars-d wrote:

=20


[=E2=80=A6]

 Yes. I'm not sure how to structure this whole suite. The general=20
 goal is "D claims that it can match C/C++ in performance, let's=20
 have some actual numbers". So far D mostly disappoints in terms=20
 of performance.
=20
 [=E2=80=A6]


Which D compilers are you testing with?

I have found DMD to be OK but often a bit slow compared to C++ codes
using GCC and Clang. However, LDC and GDC generally create executables
from D codes that are at least as fast as C++ with GCC and Clang.
=20
--=20
Russel.
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D
Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder ekiga.n=
et
41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel winder.org.uk
London SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder
Sep 09 2015
parent "qznc" <qznc web.de> writes:
On Wednesday, 9 September 2015 at 16:58:41 UTC, Russel Winder 
wrote:

 On Tue, 2015-09-08 at 21:11 +0000, qznc via Digitalmars-d wrote:

 


 […]

 Yes. I'm not sure how to structure this whole suite. The 
 general goal is "D claims that it can match C/C++ in 
 performance, let's have some actual numbers". So far D mostly 
 disappoints in terms of performance.
 
 […]


 Which D compilers are you testing with?


The benchmark records all versions: 
https://qznc.github.io/d-shootout/


 I have found DMD to be OK but often a bit slow compared to C++ 
 codes
 using GCC and Clang. However, LDC and GDC generally create 
 executables
 from D codes that are at least as fast as C++ with GCC and 
 Clang.


Yes, that is the general opinion. However, I have a hard time to 
reach the obscenely tuned C++ programs.
Sep 09 2015