www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Why is vibe.d still performing bad in benchmarks?

reply bauss <jj_1337 live.dk> writes:
As seen in: 
https://www.techempower.com/benchmarks/#section=data-r18

vibe.d once again ranks terrible on the list.

With the power of D etc. it should be able to perform at the same 
level as the top frameworks but it just isn't so what are the 
reasons, where can it be improved and what areas do we as 
community need to look into and improve upon?
Jul 10 2019
next sibling parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Wed, Jul 10, 2019 at 9:20 PM bauss via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 As seen in:
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the same
 level as the top frameworks but it just isn't so what are the
 reasons, where can it be improved and what areas do we as
 community need to look into and improve upon?


Terrible? Really? I would say not perfect but still good. Main problem is
with vibe-d:http package. I have try to use vibe-d without it, just use
vibe-core and speed was simmilar to others frameworks on top.
Jul 10 2019
parent reply sanjayss <dummy dummy.dummy> writes:
On Wednesday, 10 July 2019 at 20:47:30 UTC, Daniel Kozak wrote:

 On Wed, Jul 10, 2019 at 9:20 PM bauss via Digitalmars-d < 
 digitalmars-d puremagic.com> wrote:


 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?


 Terrible? Really? I would say not perfect but still good. Main 
 problem is with vibe-d:http package. I have try to use vibe-d 
 without it, just use vibe-core and speed was simmilar to others 
 frameworks on top.


It is terrible! Whatever the excuse, there is no other way to 
look at it than to say that based on the tech-empower tests, you 
wouldn't really consider vibe-d for a web-framework.
Jul 10 2019
parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Thu, Jul 11, 2019 at 12:10 AM sanjayss via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 It is terrible! Whatever the excuse, there is no other way to
 look at it than to say that based on the tech-empower tests, you
 wouldn't really consider vibe-d for a web-framework.


I disagree, there are many other more well known frameworks use all over
the world which has much  terrible performance and there are still
considered.
Jul 11 2019
parent reply bauss <jj_1337 live.dk> writes:
On Thursday, 11 July 2019 at 08:04:03 UTC, Daniel Kozak wrote:

 On Thu, Jul 11, 2019 at 12:10 AM sanjayss via Digitalmars-d < 
 digitalmars-d puremagic.com> wrote:


 It is terrible! Whatever the excuse, there is no other way to 
 look at it than to say that based on the tech-empower tests, 
 you wouldn't really consider vibe-d for a web-framework.


 I disagree, there are many other more well known frameworks use 
 all over the world which has much  terrible performance and 
 there are still considered.


It's still terrible compared to what it should rank.

vibe.d should be ranking near the top since that's where D's 
usual performance is ranked.

Look at the difference between D's performance and then look at 
Rust,C and C++. D is not even near those. Rust ranks in the top.

Even .NET performs better than D in those benchmarks.

We need to just realize that D did terrible in this benchmark and 
perhaps the problem is something internally in vibe.d and then we 
need to focus on what exactly the issue is, locate it and fix 
that.

vibe.d can indeed rank just as high but it doesn't.
Jul 11 2019
next sibling parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Thu, Jul 11, 2019 at 10:30 AM bauss via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 ...
 We need to just realize that D did terrible in this benchmark and
 perhaps the problem is something internally in vibe.d and then we
 need to focus on what exactly the issue is, locate it and fix
 that.


This is what I am trying to do for some time now (few months), but vibe.d
code is not so easy to read and debug. So far I have found out that
vibe-core seems to not be the main problem (there are still some
potentional speed improvment), but it seems the issue is somewhere in
vibe-d:http.
There is some work on new version of this package (
https://github.com/vibe-d/vibe-http) , so maybe the performance of this new
attempt will be better. I am goint to try this new vibe-http, but I am not
sure if it is already in usable state.
Jul 11 2019
parent Boris-Barboris <ismailsiege gmail.com> writes:
On Thursday, 11 July 2019 at 09:50:52 UTC, Daniel Kozak wrote:

 This is what I am trying to do for some time now (few months), 
 but vibe.d
 code is not so easy to read and debug. So far I have found out 
 that
 vibe-core seems to not be the main problem (there are still some
 potentional speed improvment), but it seems the issue is 
 somewhere in
 vibe-d:http.
 There is some work on new version of this package (
 https://github.com/vibe-d/vibe-http) , so maybe the performance 
 of this new
 attempt will be better. I am goint to try this new vibe-http, 
 but I am not
 sure if it is already in usable state.


I remember finding vibe-core's reference-counting facility, aka 
InterfaceProxy, very taxing on the upper layers. There is a lot 
of passing a round and the lack of move semantics makes each 
function call that involves some kind of stream or managed 
reference an excercise in integer mutation. In pure http 
throughput tests it was very noticeable, around 10-30% drop in 
rps.

Eventcore is another arcane library very deep on the stack, I 
don't think anyone ever profiled it in isolation (correct me if 
I'm wrong), yet it's the heart of the event loop.

On the topic itself, well, the answer is always the same: 
everything's in it's right place.
In the usual hierarchy of merits:

productivity > correctness > flexibility and domain coverage > 
performance

vibe-d stack is still wrestling with the first three components, 
and rightfully so.
Jul 11 2019
prev sibling parent lagfra <me fragal.eu> writes:
 There is some work on new version of this package (
 https://github.com/vibe-d/vibe-http) , so maybe the performance of this new
 attempt will be better. I am goint to try this new vibe-http, but I am not
 sure if it is already in usable state.


The vibe-http repository is where HTTP/2 support is being worked on but it still
lacks some of the modules to allow for a complete vibe-d:http replacement.
The idea behind this repository is to rewrite parts of the vibe-d:http module
to follow these guidelines:

https://github.com/vibe-d/vibe-http/wiki

I'm currently having little spare time to finish the modifications, but I'm
working on some benchmarks which will be focused on HTTP/1 and HTTP/2 as soon as
it is ready. At the moment the server supports HTTP/1 and HTTP/2 request
handling and response delivery, some functionalities are still missing (e.g.
the vibe-d:http fileserver module).
Jul 11 2019
prev sibling next sibling parent reply Exil <Exil gmall.com> writes:
On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:

 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?


There's more to performance than the language you choose.
Jul 10 2019
next sibling parent reply JN <666total wp.pl> writes:
On Wednesday, 10 July 2019 at 20:50:16 UTC, Exil wrote:

 On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:

 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?


 There's more to performance than the language you choose.


Buut fast code fast!
Jul 10 2019
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jul 10, 2019 at 08:54:56PM +0000, JN via Digitalmars-d wrote:

 On Wednesday, 10 July 2019 at 20:50:16 UTC, Exil wrote:

 On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:

 As seen in: https://www.techempower.com/benchmarks/#section=data-r18
 
 vibe.d once again ranks terrible on the list.
 
 With the power of D etc. it should be able to perform at the same
 level as the top frameworks but it just isn't so what are the
 reasons, where can it be improved and what areas do we as
 community need to look into and improve upon?


 
 There's more to performance than the language you choose.


 
 Buut fast code fast!


That motto makes me cringe. Every time.


T

-- 
It won't be covered in the book. The source code has to be useful for
something, after all. -- Larry Wall
Jul 10 2019
prev sibling parent reply bauss <jj_1337 live.dk> writes:
On Wednesday, 10 July 2019 at 20:50:16 UTC, Exil wrote:

 On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:

 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?


 There's more to performance than the language you choose.


Yes there is BUT for a web framework to properly scale you need 
performance.

D is supposed to be powerful, it's supposed to have performance 
on pair with C / C++, so there's no excuse for why it ranks even 
worse than .NET.

Of course the problem here is most likely not D but vibe.d's 
infrastructure.

My post is not about "bashing" D. I have been using D primarily 
for all my projects for the past 6 years.

My post is about highlighting an issue and figuring out what we 
can do as a community to fix it and have D rank higher.
Jul 11 2019
parent Exil <Exil gmall.com> writes:
On Thursday, 11 July 2019 at 08:29:12 UTC, bauss wrote:

 On Wednesday, 10 July 2019 at 20:50:16 UTC, Exil wrote:

 On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:

 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what 
 are the reasons, where can it be improved and what areas do 
 we as community need to look into and improve upon?


 There's more to performance than the language you choose.


 Yes there is BUT for a web framework to properly scale you need 
 performance.

 D is supposed to be powerful, it's supposed to have performance 
 on pair with C / C++, so there's no excuse for why it ranks 
 even worse than .NET.

 Of course the problem here is most likely not D but vibe.d's 
 infrastructure.

 My post is not about "bashing" D. I have been using D primarily 
 for all my projects for the past 6 years.

 My post is about highlighting an issue and figuring out what we 
 can do as a community to fix it and have D rank higher.


The problem is how much developer time there is, and what their 
focus is. It doesn't look like there are that many people working 
on the project. They are probably working on it in their free 
time too. I doubt they are focusing on optimizing it to the 
extreme but rather bug fixes, features, and maybe some obvious 
optimizations.
Jul 11 2019
prev sibling next sibling parent Newbie2019 <newbie2019 gmail.com> writes:
On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:

 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?


Compare to C/C++/Rust/GO/Dart/JAVA/JavaScript/Erlang,   the 
Github D trending is empty for most time.

D has long history but small community,   producible feature but 
poor ecosystem.

In this forum,  people talk about to use D  in mobile almost 
decade ago,   until now the LDC Android developer and IOS 
developer both give up.   GDC/DMD not support mobile.

D use for web also start almost decade ago,   but in much better 
status compare to mobile support.
Jul 11 2019
prev sibling next sibling parent Radu <void null.pt> writes:
On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:

 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?


I guess the issue is that there are not enough people working on 
vibe.d

A healthier eco-system tends to solve this, as people competent 
enough would have jumped in and measure why it performs so bad 
and come up with fixes.

Maybe D can have another challenger that is simpler in design and 
performs better than vibe.d in this benchmark?
Jul 11 2019
prev sibling parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
Am 10.07.2019 um 21:16 schrieb bauss:

 As seen in: https://www.techempower.com/benchmarks/#section=data-r18
 
 vibe.d once again ranks terrible on the list.
 
 With the power of D etc. it should be able to perform at the same level 
 as the top frameworks but it just isn't so what are the reasons, where 
 can it be improved and what areas do we as community need to look into 
 and improve upon?


The problem is that for some reason it doesn't scale properly with the 
number of cores in that benchmark. I have a dual-Xeon server that I can 
in theory use for profiling this, but I simply still lack the time to do so.

If you or anyone else who is interested has a bit of time to spare, I 
would suggest to change the benchmark setup to start one process per 
core instead of using multiple threads. This is the preferred approach 
anyway, for multiple reasons, but one of them being that it ensures 
proper scaling, no matter if the GC kicks in, or some other kind of lock 
contention happens.

This would require a change to the startup script to run multiple 
instances, as well as removing the runTaskDist call, while keeping 
HTTPServerOption.reusePort.
Jul 11 2019
next sibling parent bauss <jj_1337 live.dk> writes:
On Thursday, 11 July 2019 at 11:23:42 UTC, Sönke Ludwig wrote:

 Am 10.07.2019 um 21:16 schrieb bauss:

 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18
 
 vibe.d once again ranks terrible on the list.
 
 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?


 The problem is that for some reason it doesn't scale properly 
 with the number of cores in that benchmark. I have a dual-Xeon 
 server that I can in theory use for profiling this, but I 
 simply still lack the time to do so.

 If you or anyone else who is interested has a bit of time to 
 spare, I would suggest to change the benchmark setup to start 
 one process per core instead of using multiple threads. This is 
 the preferred approach anyway, for multiple reasons, but one of 
 them being that it ensures proper scaling, no matter if the GC 
 kicks in, or some other kind of lock contention happens.

 This would require a change to the startup script to run 
 multiple instances, as well as removing the runTaskDist call, 
 while keeping HTTPServerOption.reusePort.


Thank you for the insight! And of course thank you for vibe.d 
overall!

That was the kind of respond I was looking for, like some area to 
explore and if I find time myself I will check it out and see 
what I can work out.

Kind of on a limited time-basis right now but any spare time I 
might find in the next two weeks I will see if I can benchmark 
and figure out some things as well.
Jul 11 2019
prev sibling parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Thu, Jul 11, 2019 at 1:25 PM S=C3=B6nke Ludwig via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 The problem is that for some reason it doesn't scale properly with the
 number of cores in that benchmark. I have a dual-Xeon server that I can
 in theory use for profiling this, but I simply still lack the time to do
 so.


Unfortunately that is not true anymore. Some time ago it was true, on my 4
core system vibed generaly has been one of the most fastest.
But now it is not true. So I am not sure the main issue is with number of
cores.



 If you or anyone else who is interested has a bit of time to spare, I
 would suggest to change the benchmark setup to start one process per
 core instead of using multiple threads. This is the preferred approach
 anyway, for multiple reasons, but one of them being that it ensures
 proper scaling, no matter if the GC kicks in, or some other kind of lock
 contention happens.


I have already try this and there is no (countable) difference.

As I already said I have spent a lot of time to figure it out, but it seems
the main issue right now is with vibe-d:http. But as Boris-Barboris
mentioned maybe the main issue is with RC.
Jul 11 2019
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
Am 12.07.2019 um 00:33 schrieb Daniel Kozak:

 On Thu, Jul 11, 2019 at 1:25 PM Sönke Ludwig via Digitalmars-d 
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:
 
     The problem is that for some reason it doesn't scale properly with the
     number of cores in that benchmark. I have a dual-Xeon server that I can
     in theory use for profiling this, but I simply still lack the time
     to do so.
 
 
 Unfortunately that is not true anymore. Some time ago it was true, on my 
 4 core system vibed generaly has been one of the most fastest.
 But now it is not true. So I am not sure the main issue is with number 
 of cores.
 
     If you or anyone else who is interested has a bit of time to spare, I
     would suggest to change the benchmark setup to start one process per
     core instead of using multiple threads. This is the preferred approach
     anyway, for multiple reasons, but one of them being that it ensures
     proper scaling, no matter if the GC kicks in, or some other kind of
     lock
     contention happens.
 
 
 I have already try this and there is no (countable) difference.
 
 As I already said I have spent a lot of time to figure it out, but it 
 seems the main issue right now is with vibe-d:http. But as 
 Boris-Barboris mentioned maybe the main issue is with RC.


The issue with InterfaceProxy is definitely true, this has a 
considerable performance hit and the new vibe-http module will get rid 
of that. However, even with InterfaceProxy, the performance is still 
well within the same order of magnitude.

But the Techempower benchmark runs on a 14-core machine (plus HT) - even 
if a single core would only yield 15 kreq/s, that should give at least 

200 kreq/s and not just 60 kreq/s. So the main issue here is 


definitely scaling AFAICS.

It could of course still be that the GC is somehow involved and the 
system is RAM bandwidth limited due to constant scanning. The GC 
pressure should be pretty low, but who knows. Once I get a little bit 
more time available and have worked a bit on the backlog, I'll run some 
tests on my 2x8 core box, which should be able to reproduce the issue.
Jul 12 2019
parent reply Andre Pany <andre s-e-a-p.de> writes:
On Friday, 12 July 2019 at 07:47:23 UTC, Sönke Ludwig wrote:

 Am 12.07.2019 um 00:33 schrieb Daniel Kozak:

 [...]


 The issue with InterfaceProxy is definitely true, this has a 
 considerable performance hit and the new vibe-http module will 
 get rid of that. However, even with InterfaceProxy, the 
 performance is still well within the same order of magnitude.

 [...]


In recent dmd a bug was fixed which maybe is related to your 
scaling observation:

https://issues.dlang.org/show_bug.cgi?id=19861

Kind regards
Andre
Jul 20 2019
parent reply Daniel Kozak <kozzi11 gmail.com> writes:
How? AFAIK it is about wrong number of core and threads for AMD cpus. But
those tests are runining on intel cpus

On Sat, Jul 20, 2019 at 7:10 PM Andre Pany via Digitalmars-d <
digitalmars-d puremagic.com> wrote:


 On Friday, 12 July 2019 at 07:47:23 UTC, S=C3=B6nke Ludwig wrote:

 Am 12.07.2019 um 00:33 schrieb Daniel Kozak:

 [...]


 The issue with InterfaceProxy is definitely true, this has a
 considerable performance hit and the new vibe-http module will
 get rid of that. However, even with InterfaceProxy, the
 performance is still well within the same order of magnitude.

 [...]


 In recent dmd a bug was fixed which maybe is related to your
 scaling observation:

 https://issues.dlang.org/show_bug.cgi?id=3D19861

 Kind regards
 Andre
Jul 20 2019
parent Andre Pany <andre s-e-a-p.de> writes:
On Saturday, 20 July 2019 at 21:49:56 UTC, Daniel Kozak wrote:

 How? AFAIK it is about wrong number of core and threads for AMD 
 cpus. But those tests are runining on intel cpus

 On Sat, Jul 20, 2019 at 7:10 PM Andre Pany via Digitalmars-d < 
 digitalmars-d puremagic.com> wrote:


 On Friday, 12 July 2019 at 07:47:23 UTC, Sönke Ludwig wrote:

 Am 12.07.2019 um 00:33 schrieb Daniel Kozak:

 [...]


 The issue with InterfaceProxy is definitely true, this has a 
 considerable performance hit and the new vibe-http module 
 will get rid of that. However, even with InterfaceProxy, the 
 performance is still well within the same order of magnitude.

 [...]


 In recent dmd a bug was fixed which maybe is related to your 
 scaling observation:

 https://issues.dlang.org/show_bug.cgi?id=19861

 Kind regards
 Andre




I wasn't aware of this detail. You are right, it doesn't solve 
the issue.

Kind regards
Andre
Jul 21 2019