www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - The Computer Languages Shootout Game

reply Isaac Gouy <igouy2 yahoo.com> writes:
On 10/31/2010 14:41 PM, Eric Poggel wrote:


 What can we do to get it back on there?



1) What you can do is - make your own measurements and publish them.

To make that easy I packaged the Python script used to make the benchmarks game
measurements.

A) Notes and download (Alioth issue their own security certificate, so your
browser will complain) -

https://alioth.debian.org/frs/shownotes.php?release_id=1484

B) Download (from the benchmarks game website Help page)

http://shootout.alioth.debian.org/help.php#languagex


2) Per Bothner used the script to make and publish measurements of his
Kawa-Scheme implementation

http://per.bothner.com/blog/2010/Kawa-in-shootout/
Nov 01 2010
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 11/1/10 12:23 PM, Isaac Gouy wrote:

 On 10/31/2010 14:41 PM, Eric Poggel wrote:


 What can we do to get it back on there?



 1) What you can do is - make your own measurements and publish them.

 To make that easy I packaged the Python script used to make the benchmarks
game measurements.

 A) Notes and download (Alioth issue their own security certificate, so your
browser will complain) -

 https://alioth.debian.org/frs/shownotes.php?release_id=1484

 B) Download (from the benchmarks game website Help page)

 http://shootout.alioth.debian.org/help.php#languagex


 2) Per Bothner used the script to make and publish measurements of his
Kawa-Scheme implementation

 http://per.bothner.com/blog/2010/Kawa-in-shootout/


Thanks Isaac for weighing in.

Well we now have a good answer for "Re: What can the community do to 
help D?" thread.


Andrei
Nov 01 2010
prev sibling next sibling parent reply Isaac Gouy <igouy2 yahoo.com> writes:
--- On Mon, 11/1/10, Walter Bright <newshound2 digitalmars.com> wrote:


 From: Walter Bright <newshound2 digitalmars.com>
 Subject: Re: The Computer Languages Shootout Game
 To: digitalmars-d puremagic.com
 Date: Monday, November 1, 2010, 11:02 AM
 Isaac Gouy wrote:

 On 10/31/2010 5:13 PM, Walter Bright wrote:

 The thing about that shootout is D used to be on




 there, until the

 maintainer of the site removed it. 


 
 D still is on there. The maintainer of the site did


 not remove it.

 
 http://shootout.alioth.debian.org/gp4/measurements.php?lang=dlang


 
 Thanks, but I don't find any links to that from the home
 page.



The home page shows the current benchmarks game.

The link to the out-of-date measurements is on the help page

http://shootout.alioth.debian.org/help.php#history

D is one of the 30 or so language implementations not measured on Q6600 that
were measured on Pentium 4.


(Incidentally if you would prefer D not to be shown in the out-of-date
measurements - because you feel it's unrepresentative of the current state of D
language implementations - just ask for it to be removed.)


 

 He refuses to include D on the benchmarks.


 
 Being included in the current benchmarks game isn't an


 entitlement.

 What I refuse to do is let you guys dictate how I


 spend my free time.
 
 I'm not happy with your choice, but I don't dictate to you
 or anyone else. It's your site and you can do as you please
 with it.


Maybe someone in the D community will make the effort and produce comparison
performance measurements, and then you can choose to publish them (or not) on
the D website.
Nov 01 2010
parent Walter Bright <newshound2 digitalmars.com> writes:
Isaac Gouy wrote:

 I'm not happy with your choice, but I don't dictate to you or anyone else.
 It's your site and you can do as you please with it.


 Maybe someone in the D community will make the effort and produce comparison
 performance measurements, and then you can choose to publish them (or not) on
 the D website.



Nobody would believe benchmarks on the D web site. Heck, I don't believe any 
benchmarks published by the developers of any language.

I was even once accused of literally sabotaging DMC++ in order to make DMD look 
better.
Nov 01 2010
prev sibling next sibling parent reply Isaac Gouy <igouy2 yahoo.com> writes:
--- On Mon, 11/1/10, Walter Bright <newshound2 digitalmars.com> wrote:


 From: Walter Bright <newshound2 digitalmars.com>
 Subject: Re: The Computer Languages Shootout Game
 To: digitalmars-d puremagic.com
 Date: Monday, November 1, 2010, 11:54 AM
 Isaac Gouy wrote:

 I'm not happy with your choice, but I don't




 dictate to you or anyone else.

 It's your site and you can do as you please with




 it.

 Maybe someone in the D community will make the effort


 and produce comparison

 performance measurements, and then you can choose to


 publish them (or not) on

 the D website.


 
 
 Nobody would believe benchmarks on the D web site. Heck, I
 don't believe any benchmarks published by the developers of
 any language.



When you publish the source code of the programs, and the compile and build
logs, and the compiler and linker versions, and the OS the measurements were
made on, ... others don't have to just "believe" because they can try to
confirm the measurements for themselves.

Which is why the benchmarks game measurement scripts markup source code and log
the build -

http://shootout.alioth.debian.org/u32q/program.php?test=meteor&lang=gpp&id=4#log
Nov 01 2010
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
Isaac Gouy wrote:

 Nobody would believe benchmarks on the D web site. Heck, I don't believe
 any benchmarks published by the developers of any language.


 When you publish the source code of the programs, and the compile and build
 logs, and the compiler and linker versions, and the OS the measurements were
 made on, ... others don't have to just "believe" because they can try to
 confirm the measurements for themselves.



It's true, they can, but they don't. I have nearly 30 years of experience with
this.
Nov 01 2010
parent retard <re tard.com.invalid> writes:
Mon, 01 Nov 2010 13:39:07 -0700, Walter Bright wrote:


 Isaac Gouy wrote:

 Nobody would believe benchmarks on the D web site. Heck, I don't
 believe any benchmarks published by the developers of any language.


 When you publish the source code of the programs, and the compile and
 build logs, and the compiler and linker versions, and the OS the
 measurements were made on, ... others don't have to just "believe"
 because they can try to confirm the measurements for themselves.


 
 
 It's true, they can, but they don't. I have nearly 30 years of
 experience with this.


Instead of this useless bikeshedding someone could provide the required 
help to Isaac here. I'm guessing he also accepts money and other kinds of 
gifts as a dedication of gratitude. I personally thank him now, the test 
has revealed many useful languages to me and I've found it very 
informative.

What some language distributions have done is they include the language 
shootout tests in the standard distribution. The language community can 
do all sorts of things to make the up to date test results a reality: up 
to date open source compiler for all required platforms, debian packages, 
help with the scripts to make the tests run and so on. The site is a 
great advertisement - I'm pretty sure the Go developers have realized 
that already.
Nov 01 2010
prev sibling parent Rainer Deyke <rainerd eldwood.com> writes:
On 11/1/2010 13:04, Isaac Gouy wrote:

 --- On Mon, 11/1/10, Walter Bright <newshound2 digitalmars.com>
 wrote:

 Nobody would believe benchmarks on the D web site. Heck, I don't
 believe any benchmarks published by the developers of any
 language.


 
 When you publish the source code of the programs, and the compile and
 build logs, and the compiler and linker versions, and the OS the
 measurements were made on, ... others don't have to just "believe"
 because they can try to confirm the measurements for themselves.


I tend to assume that benchmarks posted by the developers of the
language are correct, but deliberately chosen to make the language look
good and the competition look bad.  Reproducing the tests locally would
only catch the most blatant forms of cheating.


-- 
Rainer Deyke - rainerd eldwood.com
Nov 01 2010
prev sibling next sibling parent reply Isaac Gouy <igouy2 yahoo.com> writes:
--- On Mon, 11/1/10, Walter Bright <newshound2 digitalmars.com> wrote:


 From: Walter Bright <newshound2 digitalmars.com>
 Subject: Re: The Computer Languages Shootout Game
 To: digitalmars-d puremagic.com
 Date: Monday, November 1, 2010, 1:39 PM
 Isaac Gouy wrote:

 Nobody would believe benchmarks on the D web site.




 Heck, I don't believe

 any benchmarks published by the developers of any




 language.

 When you publish the source code of the programs, and


 the compile and build

 logs, and the compiler and linker versions, and the OS


 the measurements were

 made on, ... others don't have to just "believe"


 because they can try to

 confirm the measurements for themselves.


 
 
 It's true, they can, but they don't. I have nearly 30 years
 of experience with this.


It doesn't matter whether they try to confirm the measurements for themselves
or not - what matters is that they are provided with the all the information
required to do so.


I only have 5 years experience publishing the measurements for the benchmarks
game - and I've come across a handful of people who did try to confirm the
measurements for themselves.

(The most interesting example compared a couple of language implementations on
one particular task but measured at 2 dozen different input values. That nicely
demonstrated that the same language implementation wasn't always faster across
all the input values. The 3 different input values shown on the benchmarks game
isn't usually enough to demonstrate that kind of thing.)
Nov 01 2010
parent sybrandy <sybrandy gmail.com> writes:
 It doesn't matter whether they try to confirm the measurements for themselves
or not - what matters is that they are provided with the all the information
required to do so.


 I only have 5 years experience publishing the measurements for the benchmarks
game - and I've come across a handful of people who did try to confirm the
measurements for themselves.

 (The most interesting example compared a couple of language implementations on
one particular task but measured at 2 dozen different input values. That nicely
demonstrated that the same language implementation wasn't always faster across
all the input values. The 3 different input values shown on the benchmarks game
isn't usually enough to demonstrate that kind of thing.)


That's an interesting observation.  I didn't even think of that before, 
but it does make sense.

I was debating on posting this, but I figured it couldn't hurt: the 
biggest problem I have with the benchmarks they use is that, at least 
from my perspective, they're not all very common algorithms.  Some 
things I'd love to see are B-Trees, which are common in databases, 
encryption, compression, etc. as they are very common and therefore 
provide more useful comparisons.  Even MapReduce would be good since 
that's becoming very popular.

Taking it a step further, there needs to be well-defined standard 
implementations and alternative implementations.  The standard 
implementations would be designed to be straight-forward designs that 
don't use any trickery so that we can actually compare language 
implementations.  The alternative ones would then show how you can make 
the implementations faster.  I mention this because a buddy of mine 
submitted a C version of one benchmark, but implemented his own thread 
pooling code.  It was rejected even though the C++ version used Boost, 
which also, from what I'm told, uses thread pooling.  A standard 
implementation could be used to define if things like thread pooling 
can/should be used.  I'd argue not in this case as not every language 
supports and/or requires it.  E.g. Erlang.

Of course, this is all just some ideas that I'm not going to try to 
implement as it's just going to be too much work to do and I don't have 
the resources to do it right.  Even then, how do we make it truly fair 
and accurate?  Based on what I've seen in this thread, it's a pretty 
hard problem if even the data can affect a languages performance.

Casey
Nov 02 2010
prev sibling next sibling parent Isaac Gouy <igouy2 yahoo.com> writes:
=0A=0A--- On Mon, 11/1/10, Rainer Deyke <rainerd eldwood.com> wrote:=0A=0A>=
 From: Rainer Deyke <rainerd eldwood.com>=0A> Subject: Re: The Computer Lan=
guages Shootout Game=0A> To: digitalmars-d puremagic.com=0A> Date: Monday, =
November 1, 2010, 2:01 PM=0A> On 11/1/2010 13:04, Isaac Gouy=0A> wrote:=0A>=

 --- On Mon, 11/1/10, Walter Bright <newshound2 digitalmars.com>=0A> > wr=


ote:=0A> >> Nobody would believe benchmarks on the D web site.=0A> Heck, I =
don't=0A> >> believe any benchmarks published by the developers=0A> of any=
=0A> >> language.=0A> > =0A> > When you publish the source code of the prog=
rams, and=0A> the compile and=0A> > build logs, and the compiler and linker=
 versions, and=0A> the OS the=0A> > measurements were made on, ... others d=
on't have to=0A> just "believe"=0A> > because they can try to confirm the m=
easurements for=0A> themselves.=0A> =0A> I tend to assume that benchmarks p=
osted by the developers=0A> of the=0A> language are correct, but deliberate=
ly chosen to make the=0A> language look=0A> good and the competition look b=
ad.=A0 Reproducing the=0A> tests locally would=0A> only catch the most blat=
ant forms of cheating.=0A=0A=0AGiven this discussion was about The Computer=
 Language Benchmarks Game I had assumed those were the benchmarks under dis=
cussion - in which case I don't really see how your concern about benchmark=
s "deliberately chosen to make the language look good" would be relevant.=
=0A=0A=0A
Nov 01 2010
prev sibling parent Isaac Gouy <igouy2 yahoo.com> writes:
--- On Mon, 11/1/10, retard <re tard.com.invalid> wrote:


 From: retard <re tard.com.invalid>
 Subject: Re: The Computer Languages Shootout Game
 To: digitalmars-d puremagic.com
 Date: Monday, November 1, 2010, 5:23 PM
 Mon, 01 Nov 2010 13:39:07 -0700,



 I'm guessing he also accepts money and
 other kinds of gifts as a dedication of gratitude. 


No.
Nov 01 2010