www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Why is vibe.d still performing bad in benchmarks?

reply bauss <jj_1337 live.dk> writes:
As seen in: 
https://www.techempower.com/benchmarks/#section=data-r18

vibe.d once again ranks terrible on the list.

With the power of D etc. it should be able to perform at the same 
level as the top frameworks but it just isn't so what are the 
reasons, where can it be improved and what areas do we as 
community need to look into and improve upon?
Jul 10
next sibling parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Wed, Jul 10, 2019 at 9:20 PM bauss via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 As seen in:
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the same
 level as the top frameworks but it just isn't so what are the
 reasons, where can it be improved and what areas do we as
 community need to look into and improve upon?
Terrible? Really? I would say not perfect but still good. Main problem is with vibe-d:http package. I have try to use vibe-d without it, just use vibe-core and speed was simmilar to others frameworks on top.
Jul 10
parent reply sanjayss <dummy dummy.dummy> writes:
On Wednesday, 10 July 2019 at 20:47:30 UTC, Daniel Kozak wrote:
 On Wed, Jul 10, 2019 at 9:20 PM bauss via Digitalmars-d < 
 digitalmars-d puremagic.com> wrote:

 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?
Terrible? Really? I would say not perfect but still good. Main problem is with vibe-d:http package. I have try to use vibe-d without it, just use vibe-core and speed was simmilar to others frameworks on top.
It is terrible! Whatever the excuse, there is no other way to look at it than to say that based on the tech-empower tests, you wouldn't really consider vibe-d for a web-framework.
Jul 10
parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Thu, Jul 11, 2019 at 12:10 AM sanjayss via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 It is terrible! Whatever the excuse, there is no other way to
 look at it than to say that based on the tech-empower tests, you
 wouldn't really consider vibe-d for a web-framework.
I disagree, there are many other more well known frameworks use all over the world which has much terrible performance and there are still considered.
Jul 11
parent reply bauss <jj_1337 live.dk> writes:
On Thursday, 11 July 2019 at 08:04:03 UTC, Daniel Kozak wrote:
 On Thu, Jul 11, 2019 at 12:10 AM sanjayss via Digitalmars-d < 
 digitalmars-d puremagic.com> wrote:

 It is terrible! Whatever the excuse, there is no other way to 
 look at it than to say that based on the tech-empower tests, 
 you wouldn't really consider vibe-d for a web-framework.
I disagree, there are many other more well known frameworks use all over the world which has much terrible performance and there are still considered.
It's still terrible compared to what it should rank. vibe.d should be ranking near the top since that's where D's usual performance is ranked. Look at the difference between D's performance and then look at Rust,C and C++. D is not even near those. Rust ranks in the top. Even .NET performs better than D in those benchmarks. We need to just realize that D did terrible in this benchmark and perhaps the problem is something internally in vibe.d and then we need to focus on what exactly the issue is, locate it and fix that. vibe.d can indeed rank just as high but it doesn't.
Jul 11
next sibling parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Thu, Jul 11, 2019 at 10:30 AM bauss via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 ...
 We need to just realize that D did terrible in this benchmark and
 perhaps the problem is something internally in vibe.d and then we
 need to focus on what exactly the issue is, locate it and fix
 that.
This is what I am trying to do for some time now (few months), but vibe.d code is not so easy to read and debug. So far I have found out that vibe-core seems to not be the main problem (there are still some potentional speed improvment), but it seems the issue is somewhere in vibe-d:http. There is some work on new version of this package ( https://github.com/vibe-d/vibe-http) , so maybe the performance of this new attempt will be better. I am goint to try this new vibe-http, but I am not sure if it is already in usable state.
Jul 11
parent Boris-Barboris <ismailsiege gmail.com> writes:
On Thursday, 11 July 2019 at 09:50:52 UTC, Daniel Kozak wrote:
 This is what I am trying to do for some time now (few months), 
 but vibe.d
 code is not so easy to read and debug. So far I have found out 
 that
 vibe-core seems to not be the main problem (there are still some
 potentional speed improvment), but it seems the issue is 
 somewhere in
 vibe-d:http.
 There is some work on new version of this package (
 https://github.com/vibe-d/vibe-http) , so maybe the performance 
 of this new
 attempt will be better. I am goint to try this new vibe-http, 
 but I am not
 sure if it is already in usable state.
I remember finding vibe-core's reference-counting facility, aka InterfaceProxy, very taxing on the upper layers. There is a lot of passing a round and the lack of move semantics makes each function call that involves some kind of stream or managed reference an excercise in integer mutation. In pure http throughput tests it was very noticeable, around 10-30% drop in rps. Eventcore is another arcane library very deep on the stack, I don't think anyone ever profiled it in isolation (correct me if I'm wrong), yet it's the heart of the event loop. On the topic itself, well, the answer is always the same: everything's in it's right place. In the usual hierarchy of merits: productivity > correctness > flexibility and domain coverage > performance vibe-d stack is still wrestling with the first three components, and rightfully so.
Jul 11
prev sibling parent lagfra <me fragal.eu> writes:
 There is some work on new version of this package (
 https://github.com/vibe-d/vibe-http) , so maybe the performance of this new
 attempt will be better. I am goint to try this new vibe-http, but I am not
 sure if it is already in usable state.
The vibe-http repository is where HTTP/2 support is being worked on but it still lacks some of the modules to allow for a complete vibe-d:http replacement. The idea behind this repository is to rewrite parts of the vibe-d:http module to follow these guidelines: https://github.com/vibe-d/vibe-http/wiki I'm currently having little spare time to finish the modifications, but I'm working on some benchmarks which will be focused on HTTP/1 and HTTP/2 as soon as it is ready. At the moment the server supports HTTP/1 and HTTP/2 request handling and response delivery, some functionalities are still missing (e.g. the vibe-d:http fileserver module).
Jul 11
prev sibling next sibling parent reply Exil <Exil gmall.com> writes:
On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:
 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?
There's more to performance than the language you choose.
Jul 10
next sibling parent reply JN <666total wp.pl> writes:
On Wednesday, 10 July 2019 at 20:50:16 UTC, Exil wrote:
 On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:
 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?
There's more to performance than the language you choose.
Buut fast code fast!
Jul 10
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jul 10, 2019 at 08:54:56PM +0000, JN via Digitalmars-d wrote:
 On Wednesday, 10 July 2019 at 20:50:16 UTC, Exil wrote:
 On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:
 As seen in: https://www.techempower.com/benchmarks/#section=data-r18
 
 vibe.d once again ranks terrible on the list.
 
 With the power of D etc. it should be able to perform at the same
 level as the top frameworks but it just isn't so what are the
 reasons, where can it be improved and what areas do we as
 community need to look into and improve upon?
There's more to performance than the language you choose.
Buut fast code fast!
That motto makes me cringe. Every time. T -- It won't be covered in the book. The source code has to be useful for something, after all. -- Larry Wall
Jul 10
prev sibling parent reply bauss <jj_1337 live.dk> writes:
On Wednesday, 10 July 2019 at 20:50:16 UTC, Exil wrote:
 On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:
 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?
There's more to performance than the language you choose.
Yes there is BUT for a web framework to properly scale you need performance. D is supposed to be powerful, it's supposed to have performance on pair with C / C++, so there's no excuse for why it ranks even worse than .NET. Of course the problem here is most likely not D but vibe.d's infrastructure. My post is not about "bashing" D. I have been using D primarily for all my projects for the past 6 years. My post is about highlighting an issue and figuring out what we can do as a community to fix it and have D rank higher.
Jul 11
parent Exil <Exil gmall.com> writes:
On Thursday, 11 July 2019 at 08:29:12 UTC, bauss wrote:
 On Wednesday, 10 July 2019 at 20:50:16 UTC, Exil wrote:
 On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:
 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what 
 are the reasons, where can it be improved and what areas do 
 we as community need to look into and improve upon?
There's more to performance than the language you choose.
Yes there is BUT for a web framework to properly scale you need performance. D is supposed to be powerful, it's supposed to have performance on pair with C / C++, so there's no excuse for why it ranks even worse than .NET. Of course the problem here is most likely not D but vibe.d's infrastructure. My post is not about "bashing" D. I have been using D primarily for all my projects for the past 6 years. My post is about highlighting an issue and figuring out what we can do as a community to fix it and have D rank higher.
The problem is how much developer time there is, and what their focus is. It doesn't look like there are that many people working on the project. They are probably working on it in their free time too. I doubt they are focusing on optimizing it to the extreme but rather bug fixes, features, and maybe some obvious optimizations.
Jul 11
prev sibling next sibling parent Newbie2019 <newbie2019 gmail.com> writes:
On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:
 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?
Compare to C/C++/Rust/GO/Dart/JAVA/JavaScript/Erlang, the Github D trending is empty for most time. D has long history but small community, producible feature but poor ecosystem. In this forum, people talk about to use D in mobile almost decade ago, until now the LDC Android developer and IOS developer both give up. GDC/DMD not support mobile. D use for web also start almost decade ago, but in much better status compare to mobile support.
Jul 11
prev sibling next sibling parent Radu <void null.pt> writes:
On Wednesday, 10 July 2019 at 19:16:11 UTC, bauss wrote:
 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18

 vibe.d once again ranks terrible on the list.

 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?
I guess the issue is that there are not enough people working on vibe.d A healthier eco-system tends to solve this, as people competent enough would have jumped in and measure why it performs so bad and come up with fixes. Maybe D can have another challenger that is simpler in design and performs better than vibe.d in this benchmark?
Jul 11
prev sibling parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
Am 10.07.2019 um 21:16 schrieb bauss:
 As seen in: https://www.techempower.com/benchmarks/#section=data-r18
 
 vibe.d once again ranks terrible on the list.
 
 With the power of D etc. it should be able to perform at the same level 
 as the top frameworks but it just isn't so what are the reasons, where 
 can it be improved and what areas do we as community need to look into 
 and improve upon?
The problem is that for some reason it doesn't scale properly with the number of cores in that benchmark. I have a dual-Xeon server that I can in theory use for profiling this, but I simply still lack the time to do so. If you or anyone else who is interested has a bit of time to spare, I would suggest to change the benchmark setup to start one process per core instead of using multiple threads. This is the preferred approach anyway, for multiple reasons, but one of them being that it ensures proper scaling, no matter if the GC kicks in, or some other kind of lock contention happens. This would require a change to the startup script to run multiple instances, as well as removing the runTaskDist call, while keeping HTTPServerOption.reusePort.
Jul 11
next sibling parent bauss <jj_1337 live.dk> writes:
On Thursday, 11 July 2019 at 11:23:42 UTC, Sönke Ludwig wrote:
 Am 10.07.2019 um 21:16 schrieb bauss:
 As seen in: 
 https://www.techempower.com/benchmarks/#section=data-r18
 
 vibe.d once again ranks terrible on the list.
 
 With the power of D etc. it should be able to perform at the 
 same level as the top frameworks but it just isn't so what are 
 the reasons, where can it be improved and what areas do we as 
 community need to look into and improve upon?
The problem is that for some reason it doesn't scale properly with the number of cores in that benchmark. I have a dual-Xeon server that I can in theory use for profiling this, but I simply still lack the time to do so. If you or anyone else who is interested has a bit of time to spare, I would suggest to change the benchmark setup to start one process per core instead of using multiple threads. This is the preferred approach anyway, for multiple reasons, but one of them being that it ensures proper scaling, no matter if the GC kicks in, or some other kind of lock contention happens. This would require a change to the startup script to run multiple instances, as well as removing the runTaskDist call, while keeping HTTPServerOption.reusePort.
Thank you for the insight! And of course thank you for vibe.d overall! That was the kind of respond I was looking for, like some area to explore and if I find time myself I will check it out and see what I can work out. Kind of on a limited time-basis right now but any spare time I might find in the next two weeks I will see if I can benchmark and figure out some things as well.
Jul 11
prev sibling parent reply Daniel Kozak <kozzi11 gmail.com> writes:
On Thu, Jul 11, 2019 at 1:25 PM S=C3=B6nke Ludwig via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 The problem is that for some reason it doesn't scale properly with the
 number of cores in that benchmark. I have a dual-Xeon server that I can
 in theory use for profiling this, but I simply still lack the time to do
 so.
Unfortunately that is not true anymore. Some time ago it was true, on my 4 core system vibed generaly has been one of the most fastest. But now it is not true. So I am not sure the main issue is with number of cores.
 If you or anyone else who is interested has a bit of time to spare, I
 would suggest to change the benchmark setup to start one process per
 core instead of using multiple threads. This is the preferred approach
 anyway, for multiple reasons, but one of them being that it ensures
 proper scaling, no matter if the GC kicks in, or some other kind of lock
 contention happens.
I have already try this and there is no (countable) difference. As I already said I have spent a lot of time to figure it out, but it seems the main issue right now is with vibe-d:http. But as Boris-Barboris mentioned maybe the main issue is with RC.
Jul 11
parent reply =?UTF-8?Q?S=c3=b6nke_Ludwig?= <sludwig+d outerproduct.org> writes:
Am 12.07.2019 um 00:33 schrieb Daniel Kozak:
 On Thu, Jul 11, 2019 at 1:25 PM Sönke Ludwig via Digitalmars-d 
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:
 
     The problem is that for some reason it doesn't scale properly with the
     number of cores in that benchmark. I have a dual-Xeon server that I can
     in theory use for profiling this, but I simply still lack the time
     to do so.
 
 
 Unfortunately that is not true anymore. Some time ago it was true, on my 
 4 core system vibed generaly has been one of the most fastest.
 But now it is not true. So I am not sure the main issue is with number 
 of cores.
 
     If you or anyone else who is interested has a bit of time to spare, I
     would suggest to change the benchmark setup to start one process per
     core instead of using multiple threads. This is the preferred approach
     anyway, for multiple reasons, but one of them being that it ensures
     proper scaling, no matter if the GC kicks in, or some other kind of
     lock
     contention happens.
 
 
 I have already try this and there is no (countable) difference.
 
 As I already said I have spent a lot of time to figure it out, but it 
 seems the main issue right now is with vibe-d:http. But as 
 Boris-Barboris mentioned maybe the main issue is with RC.
The issue with InterfaceProxy is definitely true, this has a considerable performance hit and the new vibe-http module will get rid of that. However, even with InterfaceProxy, the performance is still well within the same order of magnitude. But the Techempower benchmark runs on a 14-core machine (plus HT) - even if a single core would only yield 15 kreq/s, that should give at least
200 kreq/s and not just 60 kreq/s. So the main issue here is 
definitely scaling AFAICS. It could of course still be that the GC is somehow involved and the system is RAM bandwidth limited due to constant scanning. The GC pressure should be pretty low, but who knows. Once I get a little bit more time available and have worked a bit on the backlog, I'll run some tests on my 2x8 core box, which should be able to reproduce the issue.
Jul 12
parent reply Andre Pany <andre s-e-a-p.de> writes:
On Friday, 12 July 2019 at 07:47:23 UTC, Sönke Ludwig wrote:
 Am 12.07.2019 um 00:33 schrieb Daniel Kozak:
 [...]
The issue with InterfaceProxy is definitely true, this has a considerable performance hit and the new vibe-http module will get rid of that. However, even with InterfaceProxy, the performance is still well within the same order of magnitude. [...]
In recent dmd a bug was fixed which maybe is related to your scaling observation: https://issues.dlang.org/show_bug.cgi?id=19861 Kind regards Andre
Jul 20
parent reply Daniel Kozak <kozzi11 gmail.com> writes:
How? AFAIK it is about wrong number of core and threads for AMD cpus. But
those tests are runining on intel cpus

On Sat, Jul 20, 2019 at 7:10 PM Andre Pany via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 On Friday, 12 July 2019 at 07:47:23 UTC, S=C3=B6nke Ludwig wrote:
 Am 12.07.2019 um 00:33 schrieb Daniel Kozak:
 [...]
The issue with InterfaceProxy is definitely true, this has a considerable performance hit and the new vibe-http module will get rid of that. However, even with InterfaceProxy, the performance is still well within the same order of magnitude. [...]
In recent dmd a bug was fixed which maybe is related to your scaling observation: https://issues.dlang.org/show_bug.cgi?id=3D19861 Kind regards Andre
Jul 20
parent Andre Pany <andre s-e-a-p.de> writes:
On Saturday, 20 July 2019 at 21:49:56 UTC, Daniel Kozak wrote:
 How? AFAIK it is about wrong number of core and threads for AMD 
 cpus. But those tests are runining on intel cpus

 On Sat, Jul 20, 2019 at 7:10 PM Andre Pany via Digitalmars-d < 
 digitalmars-d puremagic.com> wrote:

 On Friday, 12 July 2019 at 07:47:23 UTC, Sönke Ludwig wrote:
 Am 12.07.2019 um 00:33 schrieb Daniel Kozak:
 [...]
The issue with InterfaceProxy is definitely true, this has a considerable performance hit and the new vibe-http module will get rid of that. However, even with InterfaceProxy, the performance is still well within the same order of magnitude. [...]
In recent dmd a bug was fixed which maybe is related to your scaling observation: https://issues.dlang.org/show_bug.cgi?id=19861 Kind regards Andre
I wasn't aware of this detail. You are right, it doesn't solve the issue. Kind regards Andre
Jul 21