• Isaac Gouy <igouy2 yahoo.com> Nov 01 2010
• sybrandy <sybrandy gmail.com> Nov 02 2010

Isaac Gouy <igouy2 yahoo.com> writes:

--- On Mon, 11/1/10, Walter Bright <newshound2 digitalmars.com> wrote:

 From: Walter Bright <newshound2 digitalmars.com>
 Subject: Re: The Computer Languages Shootout Game
 To: digitalmars-d puremagic.com
 Date: Monday, November 1, 2010, 1:39 PM
 Isaac Gouy wrote:
 Nobody would believe benchmarks on the D web site.




 any benchmarks published by the developers of any




 When you publish the source code of the programs, and


 logs, and the compiler and linker versions, and the OS


 made on, ... others don't have to just "believe"


 confirm the measurements for themselves.


 
 It's true, they can, but they don't. I have nearly 30 years
 of experience with this.


It doesn't matter whether they try to confirm the measurements for themselves
or not - what matters is that they are provided with the all the information
required to do so.


I only have 5 years experience publishing the measurements for the benchmarks
game - and I've come across a handful of people who did try to confirm the
measurements for themselves.

(The most interesting example compared a couple of language implementations on
one particular task but measured at 2 dozen different input values. That nicely
demonstrated that the same language implementation wasn't always faster across
all the input values. The 3 different input values shown on the benchmarks game
isn't usually enough to demonstrate that kind of thing.)


      

sybrandy <sybrandy gmail.com> writes:

 It doesn't matter whether they try to confirm the measurements for themselves
or not - what matters is that they are provided with the all the information
required to do so.


 I only have 5 years experience publishing the measurements for the benchmarks
game - and I've come across a handful of people who did try to confirm the
measurements for themselves.

 (The most interesting example compared a couple of language implementations on
one particular task but measured at 2 dozen different input values. That nicely
demonstrated that the same language implementation wasn't always faster across
all the input values. The 3 different input values shown on the benchmarks game
isn't usually enough to demonstrate that kind of thing.)


That's an interesting observation.  I didn't even think of that before, 
but it does make sense.

I was debating on posting this, but I figured it couldn't hurt: the 
biggest problem I have with the benchmarks they use is that, at least 
from my perspective, they're not all very common algorithms.  Some 
things I'd love to see are B-Trees, which are common in databases, 
encryption, compression, etc. as they are very common and therefore 
provide more useful comparisons.  Even MapReduce would be good since 
that's becoming very popular.

Taking it a step further, there needs to be well-defined standard 
implementations and alternative implementations.  The standard 
implementations would be designed to be straight-forward designs that 
don't use any trickery so that we can actually compare language 
implementations.  The alternative ones would then show how you can make 
the implementations faster.  I mention this because a buddy of mine 
submitted a C version of one benchmark, but implemented his own thread 
pooling code.  It was rejected even though the C++ version used Boost, 
which also, from what I'm told, uses thread pooling.  A standard 
implementation could be used to define if things like thread pooling 
can/should be used.  I'd argue not in this case as not every language 
supports and/or requires it.  E.g. Erlang.

Of course, this is all just some ideas that I'm not going to try to 
implement as it's just going to be too much work to do and I don't have 
the resources to do it right.  Even then, how do we make it truly fair 
and accurate?  Based on what I've seen in this thread, it's a pretty 
hard problem if even the data can affect a languages performance.

Casey