www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Re: The Computer Languages Shootout Game

reply Isaac Gouy <igouy2 yahoo.com> writes:
--- On Mon, 11/1/10, Walter Bright <newshound2 digitalmars.com> wrote:

 From: Walter Bright <newshound2 digitalmars.com>
 Subject: Re: The Computer Languages Shootout Game
 To: digitalmars-d puremagic.com
 Date: Monday, November 1, 2010, 1:39 PM
 Isaac Gouy wrote:
 Nobody would believe benchmarks on the D web site.


 any benchmarks published by the developers of any


 When you publish the source code of the programs, and

 logs, and the compiler and linker versions, and the OS

 made on, ... others don't have to just "believe"

 confirm the measurements for themselves.

It's true, they can, but they don't. I have nearly 30 years of experience with this.

It doesn't matter whether they try to confirm the measurements for themselves or not - what matters is that they are provided with the all the information required to do so. I only have 5 years experience publishing the measurements for the benchmarks game - and I've come across a handful of people who did try to confirm the measurements for themselves. (The most interesting example compared a couple of language implementations on one particular task but measured at 2 dozen different input values. That nicely demonstrated that the same language implementation wasn't always faster across all the input values. The 3 different input values shown on the benchmarks game isn't usually enough to demonstrate that kind of thing.)
Nov 01 2010
parent sybrandy <sybrandy gmail.com> writes:
 It doesn't matter whether they try to confirm the measurements for themselves
or not - what matters is that they are provided with the all the information
required to do so.


 I only have 5 years experience publishing the measurements for the benchmarks
game - and I've come across a handful of people who did try to confirm the
measurements for themselves.

 (The most interesting example compared a couple of language implementations on
one particular task but measured at 2 dozen different input values. That nicely
demonstrated that the same language implementation wasn't always faster across
all the input values. The 3 different input values shown on the benchmarks game
isn't usually enough to demonstrate that kind of thing.)

That's an interesting observation. I didn't even think of that before, but it does make sense. I was debating on posting this, but I figured it couldn't hurt: the biggest problem I have with the benchmarks they use is that, at least from my perspective, they're not all very common algorithms. Some things I'd love to see are B-Trees, which are common in databases, encryption, compression, etc. as they are very common and therefore provide more useful comparisons. Even MapReduce would be good since that's becoming very popular. Taking it a step further, there needs to be well-defined standard implementations and alternative implementations. The standard implementations would be designed to be straight-forward designs that don't use any trickery so that we can actually compare language implementations. The alternative ones would then show how you can make the implementations faster. I mention this because a buddy of mine submitted a C version of one benchmark, but implemented his own thread pooling code. It was rejected even though the C++ version used Boost, which also, from what I'm told, uses thread pooling. A standard implementation could be used to define if things like thread pooling can/should be used. I'd argue not in this case as not every language supports and/or requires it. E.g. Erlang. Of course, this is all just some ideas that I'm not going to try to implement as it's just going to be too much work to do and I don't have the resources to do it right. Even then, how do we make it truly fair and accurate? Based on what I've seen in this thread, it's a pretty hard problem if even the data can affect a languages performance. Casey
Nov 02 2010