www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D vs Go in real life, part 2. Also, Erlang.

reply "Atila Neves" <atila.neves gmail.com> writes:
It all started here:

http://forum.dlang.org/thread/lmkvmfpysiokpqgsynar forum.dlang.org

Not wanting to be outdone, my buddy Jeff (the Go guy) went and 
wrote a new benchmark. My implementation didn't do as well on 
that one so I went back to the code and optimised it. Also, our 
other buddy Patrick from work decided to get in on this and use 
it as an excuse to learn Erlang. So now 4 different 
implementations in 4 different languages are duking it out.

There are two benchmarks, both written in Go. The first is 
loadtest and it measures throughput. I varied the number of 
client connections to the server and measured how many messages 
were sent per second.

The second benchmark is pingtest and it measures latency. The 
quantity I varied here was the number of wildcard subscribers 
(it'd take a while to explain what they are to anyone who isn't 
familiar with MQTT). It has a bunch of client connections talk to 
each other, and instead of trying to send as many messages as 
computerly possible they wait for the acknowledgement from the 
partner connection. The results are as follows (I tried making 
charts but LibreOffice wasn't helping and then I got bored), with 
the numbers being thousands of messages per second:

loadtest
Connections:   100            500            750            1k
D + vibe.d:    121.7 +/- 1.5  166.9 +/- 1.5  171.1 +/- 3.3  167.9 
+/- 1.3
C (mosquitto): 106.1 +/- 0.8  122.4 +/- 0.4   95.2 +/- 1.3   74.7 
+/- 0.4
Erlang:        104.1 +/- 2.2  124.2 +/- 5.9  117.6 +/- 4.6  117.7 
+/- 3.2
Go:             90.9 +/- 11   100.1 +/- 0.1   99.3 +/- 0.2   98.8 
+/- 0.3

pingtest
wsubs:         20             200            400
D + vibe.d:    50.9 +/- 0.3   38.3 +/- 0.2   20.1 +/- 0.1
C (mosquitto): 65.4 +/- 4.4   45.2 +/- 0.2   20.0 +/- 0.0
Erlang:        49.1 +/- 0.8   30.9 +/- 0.3   15.6 +/- 0.1
Go:            45.2 +/- 0.2   27.5 +/- 0.1   16.0 +/- 0.1

So, D was faster than the other contenders by far in throughput, 
2nd place losing to the C implementation on latency. I'm still 
not sure why that is. Profiling in this case is tricky. I'm 
pretty sure the profiler is still ticking away when a fiber 
yields - the top function is the one that reads from the network, 
which I can't do much about.

GC was pretty much a non-issue, which I'm both surprised at and 
happy about. I allocated as much memory as I wanted at first, and 
by the time I tried to optmise memory usage by allocating less 
all I managed to eke out in performance terms was a paltry extra 
8% or so.

All in all though, so much for his "nothing is going to be faster 
than Go because this is what it's good at". If he'd measured it 
against mosquitto first and realised it was already losing to C I 
probably wouldn't have written an MQTT broker in D, so I guess 
that was a good thing. :)

Atila

P.S. vibe.d is awesome, although I wish it'd compile with ldc or 
gdc
Dec 04 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:

 GC was pretty much a non-issue, which I'm both surprised at and 
 happy about. I allocated as much memory as I wanted at first, 
 and by the time I tried to optmise memory usage by allocating 
 less all I managed to eke out in performance terms was a paltry 
 extra 8% or so.


GC should not impact general performance in such scenario but is 
likely to hinder latency which pretty much matches what you are 
observing. Can you possibly provide information about network 
layout, h/w and server code used for testing? There are lot of 
possible oversights that can make results less legible, would 
have been nice to verify those ;)
Dec 04 2013
parent reply "Atila Neves" <atila.neves gmail.com> writes:
 GC should not impact general performance in such scenario but 
 is likely to hinder latency which pretty much matches what you 
 are observing. Can you possibly provide information about 
 network layout, h/w and server code used for testing? There are 
 lot of possible oversights that can make results less legible, 
 would have been nice to verify those ;)


I'd have to dig in and see data to be convinced it's the GC 
making the latency worse than the C implementation, but from what 
I know you've got more experience with this so who knows :)

Doing this on a real network was waaay more work than I was 
willing to put up with for something a colleague said during 
lunch break, so it was done on localhost. The h/w is a Core i7 
Lenovo laptop (W530) with 8GB of RAM running Arch Linux. My 
server code can be found at https://github.com/atilaneves/mqtt, 
the C implementation has its own website: http://mosquitto.org/.

Atila
Dec 04 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 13:50:52 UTC, Atila Neves wrote:

 I'd have to dig in and see data to be convinced it's the GC 
 making the latency worse than the C implementation, but from 
 what I know you've got more experience with this so who knows :)


I am also not proficient enough with GC stuff to actually prove 
that but it fitted observations during my own tests pretty well. 
But I am not 100% sure.


 Doing this on a real network was waaay more work than I was 
 willing to put up with for something a colleague said during 
 lunch break, so it was done on localhost. The h/w is a Core i7 
 Lenovo laptop (W530) with 8GB of RAM running Arch Linux. My 
 server code can be found at https://github.com/atilaneves/mqtt, 
 the C implementation has its own website: http://mosquitto.org/.


Ah this is rather sad. It makes tests results somewhat unstable 
because client load interferes with server load. Ideally client 
should be separate machine and has more powerful h/w than server. 
However, that also requires ~gigabit network in between as well 
as matching network cards to hit the limits of top server 
performance - this makes such tests rather cumbersome to execute.

At the very least you should use process affinity to separate 
resources between client and server in more predictable manner.
Dec 04 2013
next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
 Ah this is rather sad. It makes tests results somewhat unstable


They're not ideal, but they're not unstable with respect to time 
- I did and redid all those measurements a few times, the error 
bars are the standard deviation taken to be the systematic error.

I got challenged so I decided to beat his implementation, even 
this level of detail got called going overboard ;)

Atila
Dec 04 2013
parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 18:02:18 UTC, Atila Neves wrote:

 Ah this is rather sad. It makes tests results somewhat unstable


 They're not ideal, but they're not unstable with respect to 
 time - I did and redid all those measurements a few times, the 
 error bars are the standard deviation taken to be the 
 systematic error.

 I got challenged so I decided to beat his implementation, even 
 this level of detail got called going overboard ;)

 Atila


By "stable" here I mean "reproducible on similar h/w and within 
similar (but not same) environment and meaningfully scaling to 
high load factors". It is simply impossible in general case when 
single PC is used for testing.

That does not make results less interesting though :)
Dec 04 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-12-04 17:20, Dicebot wrote:


 Ah this is rather sad. It makes tests results somewhat unstable because
 client load interferes with server load. Ideally client should be
 separate machine and has more powerful h/w than server. However, that
 also requires ~gigabit network in between as well as matching network
 cards to hit the limits of top server performance - this makes such
 tests rather cumbersome to execute.

 At the very least you should use process affinity to separate resources
 between client and server in more predictable manner.


Isn't it most important that all languages were tested in the same way?

-- 
/Jacob Carlborg
Dec 05 2013
next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
 Isn't it most important that all languages were tested in the 
 same way?


Maybe. I actually thought that if there might be a lesson here if 
it turns out that the quality of the implementation and 
appropriate use of data structures and algorithms mattered more 
than the choice of programming language, but in the end competing 
with each other just made all of the implementations in the same 
order of magnitude of performance.

This was never meant to be scientific, it's my Physics background 
showing up when I can't make measurements without calculating 
error bars.

This started as a lunch joke. I came here to gloat, then you guys 
go and ask intelligent questions, what's up with that? :P

Atila
Dec 05 2013
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 5 December 2013 at 08:21:07 UTC, Jacob Carlborg 
wrote:

 On 2013-12-04 17:20, Dicebot wrote:


 Ah this is rather sad. It makes tests results somewhat 
 unstable because
 client load interferes with server load. Ideally client should 
 be
 separate machine and has more powerful h/w than server. 
 However, that
 also requires ~gigabit network in between as well as matching 
 network
 cards to hit the limits of top server performance - this makes 
 such
 tests rather cumbersome to execute.

 At the very least you should use process affinity to separate 
 resources
 between client and server in more predictable manner.


 Isn't it most important that all languages were tested in the 
 same way?


Depends on your goals :) If you want just to say "hah, your 
language is nothing compared to my language" it is enough. If you 
want to make some observations/conclusions about specific 
implementation performance and how it scale for different 
conditions it becomes important to remove as many side impact as 
possible. And of course at high load/concurrency levels client 
competing with server for connections does make notable impact. 
In other words, it is good enough for comparison but not good 
enough for actual performance analysis.
Dec 05 2013
parent reply "Jeff R. Allen" <jra nella.org> writes:
I am the buddy who's always going on and on about Go.

Here's the blog posting I made explaining why/how I made the MQTT 
server in Go and what I learned:

   http://blog.nella.org/mqtt-code-golf/

I was a bit surprised and interested to see that MQTT with small 
transactions is not easy to optimize because kernel/user context 
switching dominates the work that's done under control of the 
programmer and which can be optimized via data structures and/or 
reducing GC overhead.

Something that both Atila and I verified in this exercise is that 
writing a fast network server that scales is so much easier with 
a very good networking library that takes advantage of 
lightweight threads, and with a runtime that provides GC. This 
frees up the programmer to focus on the protocol logic, and leave 
concurrency and bookkeeping in the capable hands of the machine.

And, as has been mentioned on other Go versus D threads, a lot of 
this is a matter of taste and team dynamics. I've worked on many 
teams in my career where there was not a critical mass of people 
who could reason correctly about C++ memory management, and who 
could use generic programming techniques reliably. And, in line 
with discoveries from psychological research, it's common that 
people who are not competent at something do not recognize that 
fact. Perhaps I'm above average in this regard: I KNOW I'm not 
smart enough to write correct code using templating techniques. :)

   -jeff
Dec 05 2013
next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 5 December 2013 at 14:55:53 UTC, Jeff R. Allen wrote:

 Perhaps I'm above average in this regard: I KNOW I'm not smart 
 enough to write correct code using templating techniques. :)

   -jeff


I bet you're smart enough to do it in D. It's a different world 
compared to C++ in this regard.
Dec 05 2013
prev sibling next sibling parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 5 December 2013 at 14:55:53 UTC, Jeff R. Allen wrote:

 I am the buddy who's always going on and on about Go.

 Here's the blog posting I made explaining why/how I made the 
 MQTT server in Go and what I learned:

   http://blog.nella.org/mqtt-code-golf/

 I was a bit surprised and interested to see that MQTT with 
 small transactions is not easy to optimize because kernel/user 
 context switching dominates the work that's done under control 
 of the programmer and which can be optimized via data 
 structures and/or reducing GC overhead.

 Something that both Atila and I verified in this exercise is 
 that writing a fast network server that scales is so much 
 easier with a very good networking library that takes advantage 
 of lightweight threads, and with a runtime that provides GC. 
 This frees up the programmer to focus on the protocol logic, 
 and leave concurrency and bookkeeping in the capable hands of 
 the machine.

 And, as has been mentioned on other Go versus D threads, a lot 
 of this is a matter of taste and team dynamics. I've worked on 
 many teams in my career where there was not a critical mass of 
 people who could reason correctly about C++ memory management, 
 and who could use generic programming techniques reliably. And, 
 in line with discoveries from psychological research, it's 
 common that people who are not competent at something do not 
 recognize that fact. Perhaps I'm above average in this regard: 
 I KNOW I'm not smart enough to write correct code using 
 templating techniques. :)

   -jeff




This is why we already did quite a few consultancy projects that
replaced C++ coded servers by JVM/.NET ones with comparable
performance.

Despite what many anti-GC naysayers say, if you code your data
strucutures and algorithms to be GC friendly, coupled with the
current state of art JIT compilers, one achieves similar
performance coupled with safety.

It doesn't need to extract every ms out of the hardware as C and
C++ developers always try to do, but to be fast enough to be able
to fulfill the task.

--
Paulo
Dec 05 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/5/2013 7:27 AM, Paulo Pinto wrote:

 It doesn't need to extract every ms out of the hardware as C and
 C++ developers always try to do, but to be fast enough to be able
 to fulfill the task.


Although C and C++ are capable of extracting every ms out of the hardware as
far 
as any compiled language goes, the number of programmers who are actually able 
to get those results are small. Just because someone writes a program in C or 
C++ doesn't at all mean that they're getting the most out of the machine.

Reminds me of when I worked in a metal shop at Caltech fabricating parts. I'd 
produce crummy, ugly things. The resident machinist, with the same milling 
machine, would tweak it here and there and produce things of beauty. (That old 
bridgeport milling machine had more knobs, levers, and wheels than a steam 
locomotive cockpit, and none of them were labelled.)
Dec 05 2013
parent reply "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 5 December 2013 at 19:58:17 UTC, Walter Bright wrote:

 On 12/5/2013 7:27 AM, Paulo Pinto wrote:

 It doesn't need to extract every ms out of the hardware as C 
 and
 C++ developers always try to do, but to be fast enough to be 
 able
 to fulfill the task.


 Although C and C++ are capable of extracting every ms out of 
 the hardware as far as any compiled language goes, the number 
 of programmers who are actually able to get those results are 
 small. Just because someone writes a program in C or C++ 
 doesn't at all mean that they're getting the most out of the 
 machine.


Fully agree.

Additionally there seems to be a contiguous disease when using
those languages, where one tries to micro-optimize every code
line as it gets written.


--
Paulo
Dec 05 2013
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/12/13 08:58, Paulo Pinto wrote:

 Additionally there seems to be a contiguous disease when using
 those languages, where one tries to micro-optimize every code
 line as it gets written.


In a way I think it's a disease that one has to go through with those
languages, 
because in the end it usually leads to a few optimization insights (or "don't
do 
this" insights) combined with the realization that it's much more productive to 
focus on algorithms and to let the compiler get on with its job, which it 
usually does better than any human.
Dec 06 2013
prev sibling parent reply "eles" <eles eles.com> writes:
On Friday, 6 December 2013 at 07:58:46 UTC, Paulo Pinto wrote:

 On Thursday, 5 December 2013 at 19:58:17 UTC, Walter Bright 
 wrote:

 On 12/5/2013 7:27 AM, Paulo Pinto wrote:





 Additionally there seems to be a contiguous disease when using
 those languages, where one tries to micro-optimize every code
 line as it gets written.


add to that the optimization of number of keyboard strokes :D
Dec 06 2013
next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 06/12/13 16:52, eles wrote:

 add to that the optimization of number of keyboard strokes :D


Hah, I got burned by that once in a very amusing way.  I wrote a blog post 
comparing an existing C implementation of some data structures and algorithms 
(not mine) to my new D implementation.  One reader on Reddit assumed the terse 
variable names of the C code extracts (not mine!) must be representative of all 
the code in the blog post and refused to read past the first paragraph.

He was very nice when he realized his mistake, though. :-)
Dec 06 2013
prev sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 06.12.2013 16:52, schrieb eles:

 On Friday, 6 December 2013 at 07:58:46 UTC, Paulo Pinto wrote:

 On Thursday, 5 December 2013 at 19:58:17 UTC, Walter Bright wrote:

 On 12/5/2013 7:27 AM, Paulo Pinto wrote:





 Additionally there seems to be a contiguous disease when using
 those languages, where one tries to micro-optimize every code
 line as it gets written.


 add to that the optimization of number of keyboard strokes :D


Canonical Go seems to suffer from that as well, in terms of variable names.
Dec 06 2013
prev sibling parent "Max Samukha" <maxsamukha gmail.com> writes:
On Thursday, 5 December 2013 at 14:55:53 UTC, Jeff R. Allen wrote:

 I am the buddy who's always going on and on about Go.

 Here's the blog posting I made explaining why/how I made the 
 MQTT server in Go and what I learned:

   http://blog.nella.org/mqtt-code-golf/

 I was a bit surprised and interested to see that MQTT with 
 small transactions is not easy to optimize because kernel/user 
 context switching dominates the work that's done under control 
 of the programmer and which can be optimized via data 
 structures and/or reducing GC overhead.

 Something that both Atila and I verified in this exercise is 
 that writing a fast network server that scales is so much 
 easier with a very good networking library that takes advantage 
 of lightweight threads, and with a runtime that provides GC. 
 This frees up the programmer to focus on the protocol logic, 
 and leave concurrency and bookkeeping in the capable hands of 
 the machine.

 And, as has been mentioned on other Go versus D threads, a lot 
 of this is a matter of taste and team dynamics. I've worked on 
 many teams in my career where there was not a critical mass of 
 people who could reason correctly about C++ memory management, 
 and who could use generic programming techniques reliably. And, 
 in line with discoveries from psychological research, it's 
 common that people who are not competent at something do not 
 recognize that fact. Perhaps I'm above average in this regard: 
 I KNOW I'm not smart enough to write correct code using 
 templating techniques. :)

   -jeff


Templates are easy. They are pure functions. You just need to 
overcome idiosyncrasies introduced by people who thought they 
were something else. ;)
Dec 05 2013
prev sibling next sibling parent reply "Craig Dillabaugh" <craig.dillabaugh gmail.com> writes:
On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:

 It all started here:

 http://forum.dlang.org/thread/lmkvmfpysiokpqgsynar forum.dlang.org

 Not wanting to be outdone, my buddy Jeff (the Go guy) went and 
 wrote a new benchmark. My implementation didn't do as well on 
 that one so I went back to the code and optimised it. Also, our 
 other buddy Patrick from work decided to get in on this and use 
 it as an excuse to learn Erlang. So now 4 different 
 implementations in 4 different languages are duking it out.

 There are two benchmarks, both written in Go. The first is 
 loadtest and it measures throughput. I varied the number of 
 client connections to the server and measured how many messages 
 were sent per second.

 The second benchmark is pingtest and it measures latency. The 
 quantity I varied here was the number of wildcard subscribers 
 (it'd take a while to explain what they are to anyone who isn't 
 familiar with MQTT). It has a bunch of client connections talk 
 to each other, and instead of trying to send as many messages 
 as computerly possible they wait for the acknowledgement from 
 the partner connection. The results are as follows (I tried 
 making charts but LibreOffice wasn't helping and then I got 
 bored), with the numbers being thousands of messages per second:

 loadtest
 Connections:   100            500            750            1k
 D + vibe.d:    121.7 +/- 1.5  166.9 +/- 1.5  171.1 +/- 3.3  
 167.9 +/- 1.3
 C (mosquitto): 106.1 +/- 0.8  122.4 +/- 0.4   95.2 +/- 1.3   
 74.7 +/- 0.4
 Erlang:        104.1 +/- 2.2  124.2 +/- 5.9  117.6 +/- 4.6  
 117.7 +/- 3.2
 Go:             90.9 +/- 11   100.1 +/- 0.1   99.3 +/- 0.2   
 98.8 +/- 0.3

 pingtest
 wsubs:         20             200            400
 D + vibe.d:    50.9 +/- 0.3   38.3 +/- 0.2   20.1 +/- 0.1
 C (mosquitto): 65.4 +/- 4.4   45.2 +/- 0.2   20.0 +/- 0.0
 Erlang:        49.1 +/- 0.8   30.9 +/- 0.3   15.6 +/- 0.1
 Go:            45.2 +/- 0.2   27.5 +/- 0.1   16.0 +/- 0.1

 So, D was faster than the other contenders by far in 
 throughput, 2nd place losing to the C implementation on 
 latency. I'm still not sure why that is. Profiling in this case 
 is tricky. I'm pretty sure the profiler is still ticking away 
 when a fiber yields - the top function is the one that reads 
 from the network, which I can't do much about.

 GC was pretty much a non-issue, which I'm both surprised at and 
 happy about. I allocated as much memory as I wanted at first, 
 and by the time I tried to optmise memory usage by allocating 
 less all I managed to eke out in performance terms was a paltry 
 extra 8% or so.

 All in all though, so much for his "nothing is going to be 
 faster than Go because this is what it's good at". If he'd 
 measured it against mosquitto first and realised it was already 
 losing to C I probably wouldn't have written an MQTT broker in 
 D, so I guess that was a good thing. :)

 Atila

 P.S. vibe.d is awesome, although I wish it'd compile with ldc 
 or gdc


So either D is faster or you are a better coder than your buddies!
Dec 04 2013
parent "Atila Neves" <atila.neves gmail.com> writes:
 So either D is faster or you are a better coder than your 
 buddies!


I'd say we're all the same skill level. One of them sits next to 
me and I heard about his Erlang implementation as it was 
happening. It's probable that certain techniques could be reused 
from another program but it's hard to tell. The easiest way would 
be to have another person write a different implementation in the 
same language and compare the two. It might happen, another 
colleague has been working on another Erlang implementation as 
well, but he's taking a while.
Dec 04 2013
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 04/12/13 13:49, Atila Neves wrote:

 So, D was faster than the other contenders by far in throughput, 2nd place
 losing to the C implementation on latency. I'm still not sure why that is.
 Profiling in this case is tricky. I'm pretty sure the profiler is still ticking
 away when a fiber yields - the top function is the one that reads from the
 network, which I can't do much about.


What about the relative elegance/maintainability/ease of comprehension of the 
different solutions?  Playing devil's advocate for a moment, I can well 
understand if a preference for one language over another was decided on the 
basis of its performance being good _enough_ and the code being really easy to 
work with, rather than simply the best performer.
Dec 04 2013
parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 4 December 2013 at 15:02:44 UTC, Joseph Rushton 
Wakeling wrote:

 On 04/12/13 13:49, Atila Neves wrote:

 So, D was faster than the other contenders by far in 
 throughput, 2nd place
 losing to the C implementation on latency. I'm still not sure 
 why that is.
 Profiling in this case is tricky. I'm pretty sure the profiler 
 is still ticking
 away when a fiber yields - the top function is the one that 
 reads from the
 network, which I can't do much about.


 What about the relative elegance/maintainability/ease of 
 comprehension of the different solutions?  Playing devil's 
 advocate for a moment, I can well understand if a preference 
 for one language over another was decided on the basis of its 
 performance being good _enough_ and the code being really easy 
 to work with, rather than simply the best performer.


I can't read Erlang (yet) so I don't know about that one. The C 
code is... well C code so not great. I have a profound dislike 
for Go but I'd say the Go and D implementations are just as 
readable. Then again, I wrote the D code so if I didn't think 
that was readable... :P

Atila
Dec 04 2013
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/4/13 7:41 AM, Atila Neves wrote:

 On Wednesday, 4 December 2013 at 15:02:44 UTC, Joseph Rushton Wakeling
 wrote:

 On 04/12/13 13:49, Atila Neves wrote:

 So, D was faster than the other contenders by far in throughput, 2nd
 place
 losing to the C implementation on latency. I'm still not sure why
 that is.
 Profiling in this case is tricky. I'm pretty sure the profiler is
 still ticking
 away when a fiber yields - the top function is the one that reads
 from the
 network, which I can't do much about.


 What about the relative elegance/maintainability/ease of comprehension
 of the different solutions?  Playing devil's advocate for a moment, I
 can well understand if a preference for one language over another was
 decided on the basis of its performance being good _enough_ and the
 code being really easy to work with, rather than simply the best
 performer.


 I can't read Erlang (yet) so I don't know about that one. The C code
 is... well C code so not great. I have a profound dislike for Go but I'd
 say the Go and D implementations are just as readable. Then again, I
 wrote the D code so if I didn't think that was readable... :P


How do they compare in lines of code?

Andrei
Dec 04 2013
parent "Atila Neves" <atila.neves gmail.com> writes:
 What about the relative elegance/maintainability/ease of 
 comprehension
 of the different solutions?  Playing devil's advocate for a 
 moment, I
 can well understand if a preference for one language over 
 another was
 decided on the basis of its performance being good _enough_ 
 and the
 code being really easy to work with, rather than simply the 
 best
 performer.


 I can't read Erlang (yet) so I don't know about that one. The 
 C code
 is... well C code so not great. I have a profound dislike for 
 Go but I'd
 say the Go and D implementations are just as readable. Then 
 again, I
 wrote the D code so if I didn't think that was readable... :P


 How do they compare in lines of code?


That's hard to measure. Not only because of the standard fare 
about not counting empty lines and comments, but because of some 
other issues as well. The Erlang implementation has the unit 
tests in the same files so it's all intertwined. The Go 
implementation makes use of a repository on github, should that 
be counted? In my own case I used my D serialisation library and 
also vibe.d. I would say neither should count towards the final 
tally, but others may beg to differ.

All in all the MQTT specific parts in all 3 projects weigh in at 
around 500 - 700 total lines as measured by wc -l, if memory 
serves me correctly. I'm on another computer now and pressed for 
time.

I wouldn't be surprised if the Erlang version was the shortest at 
the end of the day. I know that the Go and D implementations are 
comparable.

Atila
Dec 04 2013
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/4/13 4:49 AM, Atila Neves wrote:

 It all started here:

 http://forum.dlang.org/thread/lmkvmfpysiokpqgsynar forum.dlang.org

 Not wanting to be outdone, my buddy Jeff (the Go guy) went and wrote a
 new benchmark. My implementation didn't do as well on that one so I went
 back to the code and optimised it. Also, our other buddy Patrick from
 work decided to get in on this and use it as an excuse to learn Erlang.
 So now 4 different implementations in 4 different languages are duking
 it out.


[snip]

Interesting. Care to convert this post (only a little adjustment needed) 
to a blog and publish with source code? Would make a great article. Ask 
your friends to contribute with descriptions of their implementations, too.

Andrei
Dec 04 2013
next sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
 Interesting. Care to convert this post (only a little 
 adjustment needed) to a blog and publish with source code? 
 Would make a great article. Ask your friends to contribute with 
 descriptions of their implementations, too.


If I convert this post into a blog it's going to take more than a 
little adjustment for me to be happy with it. I guess having had 
to write a PhD thesis has now scarred me for life.

I'll see if I get around to it in the next few days and I'll ask 
the guys about their implementations as well. I guess I should go 
set up wordpress.

Atila
Dec 04 2013
next sibling parent reply Etienne <etcimon globecsys.com> writes:
On 2013-12-04 12:59 PM, Atila Neves wrote:

 Interesting. Care to convert this post (only a little adjustment
 needed) to a blog and publish with source code? Would make a great
 article. Ask your friends to contribute with descriptions of their
 implementations, too.


 If I convert this post into a blog it's going to take more than a little
 adjustment for me to be happy with it. I guess having had to write a PhD
 thesis has now scarred me for life.

 I'll see if I get around to it in the next few days and I'll ask the
 guys about their implementations as well. I guess I should go set up
 wordpress.

 Atila


It would be interesting to have a GitHub repo with the scripts for each 
language, vibe.d also has a multithreaded version that is off by default 
(fibers created from new tcp connections are distributed in a thread 
pool where communication is done through Isolated!()). So I'd love to be 
able to fork it and rerun this with some adjustments.
Dec 04 2013
parent "Atila Neves" <atila.neves gmail.com> writes:
 It would be interesting to have a GitHub repo with the scripts 
 for each language,


I might set this up after / because of the blog post. We'll see 
how it goes.


vibe.d also has a multithreaded version that
 is off by default (fibers created from new tcp connections are 
 distributed in a thread pool where communication is done 
 through Isolated!()).


Oh, I know. I tried it. The threads spent more time sending 
messages to each other than doing actual work. And that was 
before I changed the data structure for the subscriptions so it 
did even less work.

The problem (at least with those 2 benchmarks) just isn't 
amenable to be made faster by throwing threads at it. It wasn't 
just the D/vibe.d version. The Go and Erlang versions were tried 
with multiple threads as well. They either stayed at the same 
speed or ran slower.

Atila
Dec 04 2013
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/4/13 9:59 AM, Atila Neves wrote:

 Interesting. Care to convert this post (only a little adjustment
 needed) to a blog and publish with source code? Would make a great
 article. Ask your friends to contribute with descriptions of their
 implementations, too.


 If I convert this post into a blog it's going to take more than a little
 adjustment for me to be happy with it. I guess having had to write a PhD
 thesis has now scarred me for life.


Yah, I was just trying to lure you into starting on it until it's too 
late to change your mind :o).


 I'll see if I get around to it in the next few days and I'll ask the
 guys about their implementations as well. I guess I should go set up
 wordpress.

 Atila


This is terrific. Would make for a sure hit and a lot of interesting 
discussion on reddit/hackernews.


Thanks,

Andrei
Dec 04 2013
prev sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
 Interesting. Care to convert this post (only a little 
 adjustment needed) to a blog and publish with source code? 
 Would make a great article. Ask your friends to contribute with 
 descriptions of their implementations, too.


Ask, and you shall receive:

https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/
Dec 05 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 5 December 2013 at 15:24:31 UTC, Atila Neves wrote:

 Interesting. Care to convert this post (only a little 
 adjustment needed) to a blog and publish with source code? 
 Would make a great article. Ask your friends to contribute 
 with descriptions of their implementations, too.


 Ask, and you shall receive:

 https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/


_very_ well-written. I can only dream every other benchmarking 
article out there would have been done with same attention to 
details :)

http://www.reddit.com/r/programming/comments/1s5ze3/benchmarking_d_vs_go_vs_erlang_vs_c_for_mqtt/
Dec 05 2013
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/5/13 7:46 AM, Dicebot wrote:

 On Thursday, 5 December 2013 at 15:24:31 UTC, Atila Neves wrote:

 Interesting. Care to convert this post (only a little adjustment
 needed) to a blog and publish with source code? Would make a great
 article. Ask your friends to contribute with descriptions of their
 implementations, too.


 Ask, and you shall receive:

 https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/


 _very_ well-written. I can only dream every other benchmarking article
 out there would have been done with same attention to details :)

 http://www.reddit.com/r/programming/comments/1s5ze3/benchmarking_d_vs_go_vs_erlang_vs_c_for_mqtt/


Fasten your seatbelts, it's gonna be a bumpy ride!

Andrei
Dec 05 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 12/5/13 7:46 AM, Dicebot wrote:

 On Thursday, 5 December 2013 at 15:24:31 UTC, Atila Neves wrote:

 Interesting. Care to convert this post (only a little adjustment
 needed) to a blog and publish with source code? Would make a great
 article. Ask your friends to contribute with descriptions of their
 implementations, too.


 Ask, and you shall receive:

 https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/


 _very_ well-written. I can only dream every other benchmarking article
 out there would have been done with same attention to details :)

 http://www.reddit.com/r/programming/comments/1s5ze3/benchmarking_d_vs_go_vs_erlang_vs_c_for_mqtt/


https://news.ycombinator.com/item?id=6855278

https://twitter.com/D_Programming/status/408634007052492800


Andrei
Dec 05 2013
next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
 https://news.ycombinator.com/item?id=6855278

 https://twitter.com/D_Programming/status/408634007052492800


Yeah, I thought that would happen. :) Thanks!

Anyone care to guess how many comments it will take to get to 
name calling?

Atila
Dec 05 2013
prev sibling parent "qznc" <qznc web.de> writes:
On Thursday, 5 December 2013 at 16:40:06 UTC, Andrei Alexandrescu 
wrote:

 On 12/5/13 7:46 AM, Dicebot wrote:

 On Thursday, 5 December 2013 at 15:24:31 UTC, Atila Neves 
 http://www.reddit.com/r/programming/comments/1s5ze3/benchmarking_d_vs_go_vs_erlang_vs_c_for_mqtt/


 https://news.ycombinator.com/item?id=6855278


This might be the first time that I consider the reddit comments 
of higher quality than HN. I completely left reddit for HN a few 
years ago, but in the last months I see myself drifting back.

Oh, and thanks for the great article. :)
Dec 06 2013
prev sibling next sibling parent Justin Whear <justin economicmodeling.com> writes:
On Thu, 05 Dec 2013 16:24:29 +0100, Atila Neves wrote:


 Interesting. Care to convert this post (only a little adjustment
 needed) to a blog and publish with source code? Would make a great
 article. Ask your friends to contribute with descriptions of their
 implementations, too.


 
 Ask, and you shall receive:
 
 https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-


c-in-real-life-mqtt-broker-implementation-shootout/

Thanks, excellent write-up.
Dec 05 2013
prev sibling next sibling parent reply "Chris Cain" <clcain uncg.edu> writes:
On Thursday, 5 December 2013 at 15:24:31 UTC, Atila Neves wrote:

 Interesting. Care to convert this post (only a little 
 adjustment needed) to a blog and publish with source code? 
 Would make a great article. Ask your friends to contribute 
 with descriptions of their implementations, too.


 Ask, and you shall receive:

 https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/


Very nice article! I'm pretty happy to see D doing so well on 
this.

"pingtest (latency - bigger is better)"

Did you mean "lower is better"? Or is this not measuring 
time-taken?
Dec 05 2013
parent reply "Atila Neves" <atila.neves gmail.com> writes:
 Very nice article! I'm pretty happy to see D doing so well on 
 this.

 "pingtest (latency - bigger is better)"

 Did you mean "lower is better"? Or is this not measuring 
 time-taken?


No, it's correct. Instead of measuring the round-trip time I 
measured how many messages per second were sent. They're 
equivalent measures, I just preferred this one, which also makes 
it easy to compare with the other benchmark.

Atila
Dec 05 2013
parent "Chris Cain" <clcain uncg.edu> writes:
On Thursday, 5 December 2013 at 16:34:11 UTC, Atila Neves wrote:

 Very nice article! I'm pretty happy to see D doing so well on 
 this.

 "pingtest (latency - bigger is better)"

 Did you mean "lower is better"? Or is this not measuring 
 time-taken?


 No, it's correct. Instead of measuring the round-trip time I 
 measured how many messages per second were sent. They're 
 equivalent measures, I just preferred this one, which also 
 makes it easy to compare with the other benchmark.

 Atila


Oh, okay, I see. Thanks :)
Dec 05 2013
prev sibling parent "Chris" <wendlec tcd.ie> writes:
On Thursday, 5 December 2013 at 15:24:31 UTC, Atila Neves wrote:

 Interesting. Care to convert this post (only a little 
 adjustment needed) to a blog and publish with source code? 
 Would make a great article. Ask your friends to contribute 
 with descriptions of their implementations, too.


 Ask, and you shall receive:

 https://atilanevesoncode.wordpress.com/2013/12/05/go-vs-d-vs-erlang-vs-c-in-real-life-mqtt-broker-implementation-shootout/


"Go is an opinionated language"

It's funny, but I got the same impression after reading about it 
and looking at the syntax. One of D's advantages is - as I've 
said before - the freedom you have. When I first started with D, 
I did Java/Objective-C style things (classes etc.), and it was 
possible and worked well. As I went along I learned more about 
structs, templates and everything and I started to "think outside 
the box", because D is not opinionated. I enjoy coding much more 
now and it's good when a language gives you freedom instead of 
telling you what to do.

In Java:

1. Write a class
2. Now think ...

Objective-C:

1. Think, but write a class anyway.
2. Think how to bypass classes.

In D:

1. Think
2. Write a struct, class, template ... whatever is appropriate.

"In the end the choice of algorithm and data structures matter 
more than the programming language so my personal advice is to 
choose the language that makes you productive."

And D actually gives you a choice.
Dec 06 2013
prev sibling next sibling parent reply "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:

 P.S. vibe.d is awesome, although I wish it'd compile with ldc 
 or gdc


What are the issues that are blocking you from using it with LDC? 
The 2.064 frontend not being officially yet? I couldn't find any 
mention of vibe.d in the open bugs on our GitHub tracker (besides 

agree that vibe.d would be a very interesting target for LDC, 
there is little chance of the situation improving.

David
Dec 04 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 18:05:11 UTC, David Nadlinger 
wrote:

 What are the issues that are blocking you from using it with 
 LDC? The 2.064 frontend not being officially yet? I couldn't 
 find any mention of vibe.d in the open bugs on our GitHub 

 So even though I agree that vibe.d would be a very interesting 
 target for LDC, there is little chance of the situation 
 improving.


Quick search has found these issues:

dub:
https://github.com/rejectedsoftware/dub/issues/172

vibe.d:
https://github.com/rejectedsoftware/vibe.d/issues/351

I remember also someone mentioning crash during fiber switch when 
using LDC but can't find that place right now.
Dec 04 2013
parent reply Iain Buclaw <ibuclaw gdcproject.org> writes:
On 4 December 2013 18:09, Dicebot <public dicebot.lv> wrote:

 On Wednesday, 4 December 2013 at 18:05:11 UTC, David Nadlinger wrote:

 What are the issues that are blocking you from using it with LDC? The
 2.064 frontend not being officially yet? I couldn't find any mention of

 the unstable Win64 port). So even though I agree that vibe.d would be a very
 interesting target for LDC, there is little chance of the situation
 improving.



 Quick search has found these issues:

 dub:
 https://github.com/rejectedsoftware/dub/issues/172

 vibe.d:
 https://github.com/rejectedsoftware/vibe.d/issues/351

 I remember also someone mentioning crash during fiber switch when using LDC
 but can't find that place right now.


Ditto with GDC too... I did a quick search on those two project and
found that people had reported supposed bugs in GDC, but again none of
which have been raised in GDC itself.

Regards
Iain.
Dec 04 2013
parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 19:05:12 UTC, Iain Buclaw wrote:

 Ditto with GDC too... I did a quick search on those two project 
 and
 found that people had reported supposed bugs in GDC, but again 
 none of
 which have been raised in GDC itself.


I think main issue is that no one was able to reduce those low 
enough for reporting. And yeah, that fiber switch issues is 
actually one for GDC : 
https://github.com/rejectedsoftware/vibe.d/issues/101
Dec 04 2013
prev sibling parent reply Etienne <etcimon globecsys.com> writes:
On 2013-12-04 1:05 PM, David Nadlinger wrote:

 On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:

 P.S. vibe.d is awesome, although I wish it'd compile with ldc or gdc


 What are the issues that are blocking you from using it with LDC? The
 2.064 frontend not being officially yet? I couldn't find any mention of

 concerns the unstable Win64 port). So even though I agree that vibe.d
 would be a very interesting target for LDC, there is little chance of
 the situation improving.

 David


I tried on linux x64 about 2 weeks ago and it's not vibe.d itself but 
the subpackages like vibelog, userman, or vibed.org, the issues were 
with std.range I think. I can start sending bug reports in, it would be 
great to see this working.
Dec 04 2013
parent "David Nadlinger" <code klickverbot.at> writes:
On Wednesday, 4 December 2013 at 18:13:54 UTC, Etienne wrote:

 I can start sending bug reports in, it would be great to see 
 this working.


Yes, please!

David
Dec 04 2013
prev sibling parent reply "qznc" <qznc web.de> writes:
On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:

 It all started here:

 http://forum.dlang.org/thread/lmkvmfpysiokpqgsynar forum.dlang.org

 Not wanting to be outdone, my buddy Jeff (the Go guy) went and 
 wrote a new benchmark. My implementation didn't do as well on 
 that one so I went back to the code and optimised it. Also, our 
 other buddy Patrick from work decided to get in on this and use 
 it as an excuse to learn Erlang. So now 4 different 
 implementations in 4 different languages are duking it out.

 There are two benchmarks, both written in Go. The first is 
 loadtest and it measures throughput. I varied the number of 
 client connections to the server and measured how many messages 
 were sent per second.

 The second benchmark is pingtest and it measures latency. The 
 quantity I varied here was the number of wildcard subscribers 
 (it'd take a while to explain what they are to anyone who isn't 
 familiar with MQTT). It has a bunch of client connections talk 
 to each other, and instead of trying to send as many messages 
 as computerly possible they wait for the acknowledgement from 
 the partner connection. The results are as follows (I tried 
 making charts but LibreOffice wasn't helping and then I got 
 bored), with the numbers being thousands of messages per second:

 loadtest
 Connections:   100            500            750            1k
 D + vibe.d:    121.7 +/- 1.5  166.9 +/- 1.5  171.1 +/- 3.3  
 167.9 +/- 1.3
 C (mosquitto): 106.1 +/- 0.8  122.4 +/- 0.4   95.2 +/- 1.3   
 74.7 +/- 0.4
 Erlang:        104.1 +/- 2.2  124.2 +/- 5.9  117.6 +/- 4.6  
 117.7 +/- 3.2
 Go:             90.9 +/- 11   100.1 +/- 0.1   99.3 +/- 0.2   
 98.8 +/- 0.3

 pingtest
 wsubs:         20             200            400
 D + vibe.d:    50.9 +/- 0.3   38.3 +/- 0.2   20.1 +/- 0.1
 C (mosquitto): 65.4 +/- 4.4   45.2 +/- 0.2   20.0 +/- 0.0
 Erlang:        49.1 +/- 0.8   30.9 +/- 0.3   15.6 +/- 0.1
 Go:            45.2 +/- 0.2   27.5 +/- 0.1   16.0 +/- 0.1

 So, D was faster than the other contenders by far in 
 throughput, 2nd place losing to the C implementation on 
 latency. I'm still not sure why that is. Profiling in this case 
 is tricky. I'm pretty sure the profiler is still ticking away 
 when a fiber yields - the top function is the one that reads 
 from the network, which I can't do much about.

 GC was pretty much a non-issue, which I'm both surprised at and 
 happy about. I allocated as much memory as I wanted at first, 
 and by the time I tried to optmise memory usage by allocating 
 less all I managed to eke out in performance terms was a paltry 
 extra 8% or so.

 All in all though, so much for his "nothing is going to be 
 faster than Go because this is what it's good at". If he'd 
 measured it against mosquitto first and realised it was already 
 losing to C I probably wouldn't have written an MQTT broker in 
 D, so I guess that was a good thing. :)

 Atila

 P.S. vibe.d is awesome, although I wish it'd compile with ldc 
 or gdc


Nice benchmark! :)

So, a DMD-compiled D program vs GCC-compiled (?) C program 
fighting for latency? That might be a reason. Interesting we 
still win with throughput.

The variance/standard deviation of pingtest-C-20 with 4.4 stands 
out. What happens there?
Dec 05 2013
parent "Atila Neves" <atila.neves gmail.com> writes:
 So, a DMD-compiled D program vs GCC-compiled (?) C program 
 fighting for latency? That might be a reason. Interesting we 
 still win with throughput.


Yeah, the C version was compiled with gcc 4.8.2. I'd've used 
either ldc or gdc with my version but:

. vibe.d didn't even compile with ldc.
. The gdc in the Arch Linux package couldn't compile anything 
that included one of the files in std, I can't remember which 
now, for most of the time I worked on this.
. When the gdc package got fixed, I managed to compile my app 
with vibe.d. It crashed as soon as I tried using it.


 The variance/standard deviation of pingtest-C-20 with 4.4 
 stands out. What happens there?


Can't remember.

Atila
Dec 05 2013