www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Language performance benchmark be updated 2019/11/09

reply zoujiaqing <zoujiaqing gmail.com> writes:
| Language        | Time, s | Memory, MiB |
| --------------- | ------- | ----------- |
| Kotlin          | 2.01    | 37.6        |
| Nim Gcc         | 2.17    | 0.7         |
| C++ Gcc         | 2.41    | 1.7         |
| OCaml           | 2.50    | 4.4         |
| Go              | 2.94    | 1.5         |
| Java            | 3.05    | 37.2        |
| Crystal         | 3.06    | 2.7         |
| ML MLton        | 3.22    | 0.7         |
| Go Gcc          | 3.30    | 19.2        |
| Rust            | 3.43    | 0.8         |
| Nim Clang       | 3.43    | 1.0         |
| D Ldc           | 3.57    | 1.4         |
| D Gdc           | 3.72    | 5.8         |
| C# .NET Core    | 4.30    | 24.3        |
| Scala           | 4.30    | 136.3       |
| F# .NET Core    | 4.35    | 24.5        |
| D Dmd           | 4.74    | 3.3         |
| Haskell (MArray)| 6.88    | 3.5         |
| C# Mono         | 6.88    | 17.6        |
| Javascript Node | 6.97    | 31.5        |
| V Gcc           | 7.30    | 0.8         |
| V Clang         | 9.06    | 1.0         |
| Racket          | 10.49   | 77.4        |
| LuaJIT          | 10.99   | 2.1         |
| Python PyPy     | 21.51   | 95.4        |
| Chez Scheme     | 24.72   | 29.2        |
| Haskell         | 29.14   | 3.4         |
| Ruby truffle    | 32.52   | 613.3       |
| Ruby JRuby      | 180.65  | 241.7       |
| Ruby            | 191.36  | 13.1        |
| Lua 5.3         | 201.26  | 1.4         |
| Elixir          | 279.03  | 48.9        |
| Python3         | 388.22  | 7.8         |
| Python          | 399.75  | 6.2         |
| Tcl (FP)        | 494.78  | 4.3         |
| Perl            | 769.17  | 5.2         |
| Tcl (OO)        | 1000.55 | 4.3         |

https://github.com/kostya/benchmarks
Nov 14
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 15 November 2019 at 03:31:24 UTC, zoujiaqing wrote:
 https://github.com/kostya/benchmarks
Sadly, the benchmark entries appears to use different algorithms... despite the site claiming otherwise. As far as I can tell...
Nov 15
parent reply aliak <something something.com> writes:
On Friday, 15 November 2019 at 08:25:26 UTC, Ola Fosheim Grøstad 
wrote:
 On Friday, 15 November 2019 at 03:31:24 UTC, zoujiaqing wrote:
 https://github.com/kostya/benchmarks
Sadly, the benchmark entries appears to use different algorithms... despite the site claiming otherwise. As far as I can tell...
You mean for sorting one uses quick sort while another uses bubble or something to that affect? Did you check a fair amount and found them all different? (I haven't looked yet obviously, and trying to avoid it depending on how much you peaked :) )
Nov 15
next sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 15 November 2019 at 18:05:35 UTC, aliak wrote:
 You mean for sorting one uses quick sort while another uses 
 bubble or something to that affect? Did you check a fair amount 
 and found them all different?
The testset are very limited. Matrix multiplication for one... If you benchmark a language that calls into a C-implementation library... and get twice as good results as the C-benchmark... then you know something is not right! :-D
Nov 15
prev sibling parent Gregor =?UTF-8?B?TcO8Y2ts?= <gregormueckl gmx.de> writes:
On Friday, 15 November 2019 at 18:05:35 UTC, aliak wrote:
 On Friday, 15 November 2019 at 08:25:26 UTC, Ola Fosheim 
 Grøstad wrote:
 On Friday, 15 November 2019 at 03:31:24 UTC, zoujiaqing wrote:
 https://github.com/kostya/benchmarks
Sadly, the benchmark entries appears to use different algorithms... despite the site claiming otherwise. As far as I can tell...
You mean for sorting one uses quick sort while another uses bubble or something to that affect? Did you check a fair amount and found them all different? (I haven't looked yet obviously, and trying to avoid it depending on how much you peaked :) )
The Json test uses very different parser implementations. There are even multiple implementations for D. One of them uses the fast I/O library while the other one ises std.json, I think. I haven't checked the others, but I expect them to have a similar spread.
Nov 15
prev sibling parent reply Jacob Shtokolov <jacob.100205 gmail.com> writes:
On Friday, 15 November 2019 at 03:31:24 UTC, zoujiaqing wrote:
 https://github.com/kostya/benchmarks
Sorry, but have you tried it by yourself? I'm running this benchmark outside of Docker, and the numbers I see are very interesting. First of all, the time measurement script is quite questionable. It's a Ruby script, and I found that the running time depends on the shell and environment. For example, when I run it under VSCode's console window, I get 20% worse results in time for all binaries than when I run it under the regular terminal emulator window. Second, the numbers are, hmmm, how to say... bullshit? Pardon my French! Here is what I get for the Brainfuck2 mandelbrot benchmark (a simple Brainfuck interpreter implemented in different languages): C++ gcc version 7.4.0 (g++ -flto -O3 -o bin_cpp bf.cpp): ``` $ ../xtime.rb ./bin_cpp mandel.b # ... here is the Mandelbrot set ASCII art printed ... 18.05s, 3.6Mb ``` D LDC2 1.18.0 (ldc2 -ofbin_d_ldc -O5 -release -boundscheck=off bf.d) ``` $ ../xtime.rb ./bin_d_ldc mandel.b # ... here is the Mandelbrot set ASCII art printed ... 19.53s, 3.6Mb ``` Nim 1.0.2 (nim c -o:bin_nim_gcc -d:danger --cc:gcc --verbosity:0 bf.nim) ``` $ ../xtime.rb ./bin_nim_gcc mandel.b # ... here is the Mandelbrot set ASCII art printed ... 25.07s, 2.2Mb ``` Kotlin kotlinc-jvm 1.3.50 (JRE 1.8.0_201-b09) (kotlinc bf2.kt -include-runtime -d bf2-kt.jar) ``` $ ../xtime.rb java -jar bf2-kt.jar mandel.b JIT warming up time: 1.25s run # ... here is the Mandelbrot set ASCII art printed ... 26.81s, 36.6Mb ``` Golang go1.13.4 linux/amd64 (go build -o bin_go bf.go) ``` $ ../xtime.rb ./bin_go mandel.b # ... here is the Mandelbrot set ASCII art printed ... 38.73s, 2.9Mb ``` ============================================ So the results for Braunfuck2 mandel.b are: ``` C++ gcc: 18.05s, 3.6Mb D LDC2: 19.53s, 3.6Mb Nim: 25.07s, 2.2Mb Kotlin: 26.81s, 36.6Mb ``` Please note that I've added `boundscheck=off` to LDC2 command line there. Also, Kotlin is always printing `JIT warming up` and takes about 1 to 2 seconds to warm up, so the results for Brainfuck2 `bench.b` are VERY different. Kotlin is not the first one obviously. I'm running everything on my laptop with Intel(R) Core(TM) i7-8550U CPU 1.80GHz As I mentioned, no Docker containers, just using the included scripts to build the binaries. Haven't tried other tests, but I feel like I'll get very interesting results for them also. Would be great if someone else is able to confirm these results because these benchmarks look very manipulative.
Nov 16
parent reply Jacob Shtokolov <jacob.100205 gmail.com> writes:
On Saturday, 16 November 2019 at 16:07:29 UTC, Jacob Shtokolov 
wrote:
 Haven't tried other tests
Just tried to compile and run Base64 The results are: ``` C: 1.46s, 1.9Mb Rust: 1.49s, 2.4Mb D LDC2: 1.98s, 4.2Mb Golang: 2.89s, 10.9Mb C++: 3.18s, 4.8Mb ``` This test is closer to the author's numbers, but Rust implementation isn't faster than the C implementation on my machine. Golang here is faster than C++. The D version was built with bounds checks this time.
Nov 16
next sibling parent reply Jacob Shtokolov <jacob.100205 gmail.com> writes:
On Saturday, 16 November 2019 at 16:34:58 UTC, Jacob Shtokolov 
wrote:
 Just tried to compile and run Base64
The Havlak test is closer to reality: ``` Nim: 12.24s, 477.8Mb C++: 17.33s, 179.3Mb Golang: 21.58s, 358.0Mb D LDC2: 23.55s, 460.4Mb D DMD: 29.04s, 461.9Mb ``` Nim is the winner. But here I would look into the code: what makes LDC produce such poorly optimized binary.
Nov 16
next sibling parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Sat, Nov 16, 2019 at 5:50 PM Jacob Shtokolov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 16 November 2019 at 16:34:58 UTC, Jacob Shtokolov
 wrote:
 Just tried to compile and run Base64
The Havlak test is closer to reality: ``` Nim: 12.24s, 477.8Mb C++: 17.33s, 179.3Mb Golang: 21.58s, 358.0Mb D LDC2: 23.55s, 460.4Mb D DMD: 29.04s, 461.9Mb ``` Nim is the winner. But here I would look into the code: what makes LDC produce such poorly optimized binary.
LDC binary is ok, this is about GC, I was able to make it lamost as twice fast for ldc with some improvments
Nov 17
parent reply Jacob Shtokolov <jacob.100205 gmail.com> writes:
On Sunday, 17 November 2019 at 10:36:41 UTC, Daniel Kozak wrote:
 LDC binary is ok, this is about GC, I was able to make it 
 almost as twice fast for ldc with some improvements
Just checked the code and found that they're using allocations with `new` in loops. But that's very interesting to see what changes you made to make it run so much faster! Could you please share it somewhere?
Nov 17
parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Sun, Nov 17, 2019 at 2:50 PM Jacob Shtokolov via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 17 November 2019 at 10:36:41 UTC, Daniel Kozak wrote:
 LDC binary is ok, this is about GC, I was able to make it
 almost as twice fast for ldc with some improvements
Just checked the code and found that they're using allocations with `new` in loops. But that's very interesting to see what changes you made to make it run so much faster! Could you please share it somewhere?
Sorry I missed insert the link. It is on my github: https://github.com/Kozzi11/benchmarks/tree/improve_d
Nov 17
parent reply Jacob Shtokolov <jacob.100205 gmail.com> writes:
On Sunday, 17 November 2019 at 14:15:00 UTC, Daniel Kozak wrote:
 Sorry I missed insert the link. It is on my github: 
 https://github.com/Kozzi11/benchmarks/tree/improve_d
Now it's faster than the C++ version on my machine: ``` Nim: 12.01s, 478.1Mb D LDC2: 13.48s, 428.1Mb C++: 19.97s, 179.3Mb Golang: 21.90s, 364.7Mb ``` So basically the only critical change was to replace the built-in associative arrays with Appender types? That's really amazing!
Nov 17
parent reply Daniel Kozak <kozzi11 gmail.com> writes:
 So basically the only critical change was to replace the built-in
 associative arrays with Appender types?

 That's really amazing!
Not only, other change is not filling number AA with UNVISITED, the other change is to disable parallel GC, because it is cause performance decrease
Nov 17
parent reply Jon Degenhardt <jond noreply.com> writes:
On Sunday, 17 November 2019 at 16:25:52 UTC, Daniel Kozak wrote:
 So basically the only critical change was to replace the 
 built-in associative arrays with Appender types?

 That's really amazing!
Not only, other change is not filling number AA with UNVISITED, the other change is to disable parallel GC, because it is cause performance decrease
Regarding the benefits seen from switching from AAs to Appenders - This is a nice performance improvement. Also a nice example of often available performance improvements in D programs. At a high level, I feel I've seen this pattern a number of times. When people starting with D run benchmarks as part of their initial experiments, they naturally start with the simplest and most straightforward programming approaches. Nothing wrong with this. It's a strength of D that quality code can be written quickly. However, in many cases these simple approaches allocate a fair bit of GC memory, memory that becomes unused quickly and needs to be GC collected. Again, nothing wrong with this. But, I have the impression that many times there is an expectation that such code will perform similarly to code using manually managed memory in other native compiled languages. And often this expectation is not met, as memory allocation and use patterns are a major performance driver. What often gets missed in these assessments is that D has quite a few mechanisms available to enable better memory management use, without needing to drop GC paradigms entirely and move to fully manually managed memory. Modifying performance sensitive programs to use these mechanisms is often not hard. The switch here from AAs to Appenders is an example. Being able to improve program performance in this way is a strength of D. One consideration is that until one has some experience with the language, it may not be obvious that these options exist, and the specific changes and approaches that can be used. This can lead to perception issues if nothing else. --Jon
Nov 17
parent reply JN <666total wp.pl> writes:
On Sunday, 17 November 2019 at 21:42:37 UTC, Jon Degenhardt wrote:
 At a high level, I feel I've seen this pattern a number of 
 times. When people starting with D run benchmarks as part of 
 their initial experiments, they naturally start with the 
 simplest and most straightforward programming approaches. 
 Nothing wrong with this. It's a strength of D that quality code 
 can be written quickly.
I think it signifies a deeper problem with these kind of benchmarks. Most people would expect these benchmarks to measure idiomatic code, "every day" kind of code. Most people would write their code with associative arrays in this case. Sure, you can optimize it later, but just as well you can just drop into asm {} block and write hand optimized code. Same with Java, you can write a lot of the code in a very C-like way for a large speedup, but the code will be completely foreign for most Java programmers and not very representative for the language.
Nov 18
parent reply bachmeier <no spam.net> writes:
On Monday, 18 November 2019 at 21:35:08 UTC, JN wrote:

 I think it signifies a deeper problem with these kind of 
 benchmarks. Most people would expect these benchmarks to 
 measure idiomatic code, "every day" kind of code. Most people 
 would write their code with associative arrays in this case. 
 Sure, you can optimize it later, but just as well you can just 
 drop into asm {} block and write hand optimized code.
If you're in a position where you care about "fast as possible" code, how fast your "every day" code runs isn't really helpful. Now, I do understand that you might want to measure the performance of a piece of code written when you aren't optimizing for execution speed. Someone in that position is going to care about speed of execution and speed of development, among other things. The problem is that you can't learn anything useful in that case from a benchmark that reports execution time and nothing else.
Nov 18
parent Jon Degenhardt <jond noreply.com> writes:
On Monday, 18 November 2019 at 21:50:04 UTC, bachmeier wrote:
 On Monday, 18 November 2019 at 21:35:08 UTC, JN wrote:

 I think it signifies a deeper problem with these kind of 
 benchmarks. Most people would expect these benchmarks to 
 measure idiomatic code, "every day" kind of code. Most people 
 would write their code with associative arrays in this case. 
 Sure, you can optimize it later, but just as well you can just 
 drop into asm {} block and write hand optimized code.
If you're in a position where you care about "fast as possible" code, how fast your "every day" code runs isn't really helpful. Now, I do understand that you might want to measure the performance of a piece of code written when you aren't optimizing for execution speed. Someone in that position is going to care about speed of execution and speed of development, among other things. The problem is that you can't learn anything useful in that case from a benchmark that reports execution time and nothing else.
Yes, there are often multiple goals behind a benchmark like this, goals that may not be explicitly identified. There is also the question of what "idiomatic" means. This is can be quite subjective, especially in multi-paradigm languages. And, what "idiomatic" means to an individual may change as familiarity with the language grows. For D performance studies, an example is that it can take time to learn how to use lazy, range-based programming facilities. This is certainly one idiomatic D coding style. And, it often results in much better memory management and performance improvements. Code can move further from the most common paradigms of course, including all the way to inline assembly blocks. Makes it difficult to say when versions of a program in different languages are similarly idiomatic.
Nov 18
prev sibling next sibling parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Sun, Nov 17, 2019 at 11:36 AM Daniel Kozak <kozzi11 gmail.com> wrote:
 Nim is the winner.

 But here I would look into the code: what makes LDC produce such
 poorly optimized binary.
LDC binary is ok, this is about GC, I was able to make it lamost as twice fast for ldc with some improvements
original code Golang: 22.74s, 364.1Mb D LDC2: 29.55s, 463.9Mb D DMD: 29.42s, 462.5Mb D GDC: 25.28s, 415.3Mb Nim: 14.26s, 468,9Mb with small changes: Golang: 22.74s, 364.1Mb D LDC2: 15.90s, 389.8Mb D DMD: 16.86s, 387.3Mb D GDC: 19.48s, 403.8Mb Nim: 14.26s, 468,9Mb
Nov 17
next sibling parent James Blachly <james.blachly gmail.com> writes:
On 11/17/19 6:04 AM, Daniel Kozak wrote:
 On Sun, Nov 17, 2019 at 11:36 AM Daniel Kozak <kozzi11 gmail.com> wrote:
 LDC binary is ok, this is about GC, I was able to make it lamost as
 twice fast for ldc with some improvements
Can you summarize or share the changes for learning purposes?
Nov 17
prev sibling parent kinke <noone nowhere.com> writes:
On Sunday, 17 November 2019 at 11:04:55 UTC, Daniel Kozak wrote:
 On Sun, Nov 17, 2019 at 11:36 AM Daniel Kozak 
 <kozzi11 gmail.com> wrote:
 Nim is the winner.

 But here I would look into the code: what makes LDC produce 
 such poorly optimized binary.
LDC binary is ok, this is about GC, I was able to make it lamost as twice fast for ldc with some improvements
original code Golang: 22.74s, 364.1Mb D LDC2: 29.55s, 463.9Mb D DMD: 29.42s, 462.5Mb D GDC: 25.28s, 415.3Mb Nim: 14.26s, 468,9Mb with small changes: Golang: 22.74s, 364.1Mb D LDC2: 15.90s, 389.8Mb D DMD: 16.86s, 387.3Mb D GDC: 19.48s, 403.8Mb Nim: 14.26s, 468,9Mb
With full LTO, I'm seeing an additional 5% boost on Windows (-flto=full -defaultlib=phobos2-ldc-lto,druntime-ldc-lto). As they are using gcc LTO for the brainfuck2 benchmark too (https://github.com/kostya/benchmarks/blob/2777925c4e64987e83e9a53478910de080408057/brai fuck2/build.sh#L5), I wouldn't consider it to be cheating.
Nov 17
prev sibling parent IGotD- <nise nise.com> writes:
On Saturday, 16 November 2019 at 16:45:02 UTC, Jacob Shtokolov 
wrote:
 On Saturday, 16 November 2019 at 16:34:58 UTC, Jacob Shtokolov 
 wrote:
 Just tried to compile and run Base64
The Havlak test is closer to reality: ``` Nim: 12.24s, 477.8Mb C++: 17.33s, 179.3Mb Golang: 21.58s, 358.0Mb D LDC2: 23.55s, 460.4Mb D DMD: 29.04s, 461.9Mb ``` Nim is the winner. But here I would look into the code: what makes LDC produce such poorly optimized binary.
C++ memory consumption is way lower than the rest. Is this because of the tracing GC penalty? It would have been interesting to see Rust here as it doesn't use GC and if it would get close to the C++ memory consumption.
Nov 17
prev sibling parent reply IGotD- <nise nise.com> writes:
On Saturday, 16 November 2019 at 16:34:58 UTC, Jacob Shtokolov 
wrote:
 On Saturday, 16 November 2019 at 16:07:29 UTC, Jacob Shtokolov 
 wrote:
 Haven't tried other tests
Just tried to compile and run Base64 The results are: ``` C: 1.46s, 1.9Mb Rust: 1.49s, 2.4Mb D LDC2: 1.98s, 4.2Mb Golang: 2.89s, 10.9Mb C++: 3.18s, 4.8Mb ``` This test is closer to the author's numbers, but Rust implementation isn't faster than the C implementation on my machine. Golang here is faster than C++. The D version was built with bounds checks this time.
Why is C++ doing so badly? Is it because of inefficient usage of buffers and that it doesn't natively support slices?
Nov 16
parent Jacob Shtokolov <jacob.100205 gmail.com> writes:
On Saturday, 16 November 2019 at 16:47:40 UTC, IGotD- wrote:
 Why is C++ doing so badly?
Looks like that's because they're using some libcrypto APIs (like the BIO). Also, my C++ compiler is not the latest one - 7.4.0. In the benchmark, it's claimed as GCC 9.2.1. But the rest compilers are up to date and actually the same versions as in the benchmark README file
Nov 16