www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Wanted: Review manager for std.data.json

reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Anyone up to this? The issues of the previous discussion [1] have all 
been addressed now more or less, so the package is ready for a more 
thorough review.

Code: https://github.com/s-ludwig/std_data_json
Docs: http://s-ludwig.github.io/std_data_json/

[1]: 
http://forum.dlang.org/thread/lt5s76$is$1 digitalmars.com#post-lt5s76:24is:241:40digitalmars.com
Apr 07 2015
next sibling parent reply "tcha" <chalucha gmail.com> writes:
On Tuesday, 7 April 2015 at 16:37:15 UTC, Sönke Ludwig wrote:

 Anyone up to this? The issues of the previous discussion [1] 
 have all been addressed now more or less, so the package is 
 ready for a more thorough review.

 Code: https://github.com/s-ludwig/std_data_json
 Docs: http://s-ludwig.github.io/std_data_json/

 [1]: 
 http://forum.dlang.org/thread/lt5s76$is$1 digitalmars.com#post-lt5s76:24is:241:40digitalmars.com


Out of curiosity I tried to use this lib in lately discussed 
benchmark [1]

Original values on my machine (dmd 2.067, gdc 4.8.4, ldc2 0.15.1 
llvm 3.5.1):
D - 10.22s, 1322.7Mb
D Gdc - 24.30s, 926.8Mb
D Ldc - 25.93s, 925.8Mb

With this lib it gets to [2]:
D - 7.48s, 1794.0Mb
Gdc and Ldc cannot build it with release (debug works) [3] and [4]

I also tried it with vibe.d's json [5]:
D - 44.77s, 2752.5Mb
D Gdc - 163.79s, 936.5Mb
D Ldc - 165.52s, 935.0Mb

So it seems that the new lib is a way to go.

As a newb to D I would like to know if there is a more elegant 
way to write:
auto coordinates = 
jval.get!(JSONValue[string])["coordinates"].get!(JSONValue[]);

and

x += val.opt("x").coerce!double;

I tried to fiddle with used JSONValue, Algebraic and VariantN 
things, but without success.
It's also weird (and undocumented in examples how to use the 
JSONValue other than in asserts) that this works:
JSONValue a = 12;
assert(a == 12.0);

and this is not:
long b = a;

I'd like if something like this can be possible:
auto coords = parseJson(text)["coordinates"];
foreach (val; coordinates) {
     x += val["x"].get!double;
     y += cast(double)val["y"];
     z += to!double(val["z"]);
   }

Also is there some way to use this lib for serialization and 
deserialization of normal objects? To use it like:
auto coords = 
parseJson(text)["coordinates"].map!(a=>cast(SomeStruct)a);


[1] https://github.com/kostya/benchmarks/tree/master/json
[2] http://pastebin.com/22xX6V0J
[3] http://pastebin.com/WshDpnQD
[4] http://pastebin.com/zj13Z7yG
[5] http://pastebin.com/FLLXBCe1
Apr 07 2015
parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8 Apr 2015 00:05, "tcha via Digitalmars-d" <digitalmars-d puremagic.com>
wrote:

 On Tuesday, 7 April 2015 at 16:37:15 UTC, S=C3=B6nke Ludwig wrote:

 Anyone up to this? The issues of the previous discussion [1] have all




been addressed now more or less, so the package is ready for a more
thorough review.

 Code: https://github.com/s-ludwig/std_data_json
 Docs: http://s-ludwig.github.io/std_data_json/

 [1]:




http://forum.dlang.org/thread/lt5s76$is$1 digitalmars.com#post-lt5s76:24is:=
241:40digitalmars.com

 Out of curiosity I tried to use this lib in lately discussed benchmark [1=


]

 Original values on my machine (dmd 2.067, gdc 4.8.4, ldc2 0.15.1 llvm


3.5.1):

 D - 10.22s, 1322.7Mb
 D Gdc - 24.30s, 926.8Mb
 D Ldc - 25.93s, 925.8Mb

 With this lib it gets to [2]:
 D - 7.48s, 1794.0Mb
 Gdc and Ldc cannot build it with release (debug works) [3] and [4]


I assume you cleared your dub cache and didn't try linking a dmd built
library to a gdc/ldc application.  :)

Iain.
Apr 08 2015
next sibling parent "Martin Nowak" <code dawg.eu> writes:
On Wednesday, 8 April 2015 at 07:14:32 UTC, Iain Buclaw wrote:

 With this lib it gets to [2]:
 D - 7.48s, 1794.0Mb
 Gdc and Ldc cannot build it with release (debug works) [3] and 
 [4]




Have you tried to use the pull/stream parser?
Apr 08 2015
prev sibling next sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 08.04.2015 um 09:14 schrieb Iain Buclaw via Digitalmars-d:

 On 8 Apr 2015 00:05, "tcha via Digitalmars-d"
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:
  >
  > Out of curiosity I tried to use this lib in lately discussed
 benchmark [1]
  >
  > Original values on my machine (dmd 2.067, gdc 4.8.4, ldc2 0.15.1 llvm
 3.5.1):
  > D - 10.22s, 1322.7Mb
  > D Gdc - 24.30s, 926.8Mb
  > D Ldc - 25.93s, 925.8Mb
  >
  > With this lib it gets to [2]:
  > D - 7.48s, 1794.0Mb
  > Gdc and Ldc cannot build it with release (debug works) [3] and [4]
  >

 I assume you cleared your dub cache and didn't try linking a dmd built
 library to a gdc/ldc application.  :)

 Iain.


DUB uses the path of the compiler binary, as well as the reported 
frontend version to distinguish builds of different compilers, so that 
clearing the cache shouldn't be necessary.

But there are definitely performance issues. My own tests are far from 
what I would have expected, including for the pull parser...
Apr 08 2015
prev sibling parent reply "tcha" <chalucha gmail.com> writes:
On Wednesday, 8 April 2015 at 07:14:32 UTC, Iain Buclaw wrote:

 I assume you cleared your dub cache and didn't try linking a 
 dmd built
 library to a gdc/ldc application.  :)

 Iain.


I tried it with dub clean, dub --force, even removed 
std_data_json package to clone it again, but no success.

Here is a full dub build -v output: http://pastebin.com/yHV3gYcX
Apr 08 2015
parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8 April 2015 at 12:39, tcha via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Wednesday, 8 April 2015 at 07:14:32 UTC, Iain Buclaw wrote:

 I assume you cleared your dub cache and didn't try linking a dmd built
 library to a gdc/ldc application.  :)

 Iain.



 I tried it with dub clean, dub --force, even removed std_data_json package
 to clone it again, but no success.

 Here is a full dub build -v output: http://pastebin.com/yHV3gYcX


I find these undefined references frankly bizarre  (undefined
reference to std.math.pow?) - can you reduce these with dustmite?

Iain.
Apr 08 2015
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 7 April 2015 at 16:37:15 UTC, Sönke Ludwig wrote:

 Anyone up to this? The issues of the previous discussion [1] 
 have all been addressed now more or less, so the package is 
 ready for a more thorough review.

 Code: https://github.com/s-ludwig/std_data_json
 Docs: http://s-ludwig.github.io/std_data_json/

 [1]: 
 http://forum.dlang.org/thread/lt5s76$is$1 digitalmars.com#post-lt5s76:24is:241:40digitalmars.com


If no one else steps up, I can do it after DConf.
Apr 07 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/7/15 5:43 PM, Dicebot wrote:

 On Tuesday, 7 April 2015 at 16:37:15 UTC, Sönke Ludwig wrote:

 Anyone up to this? The issues of the previous discussion [1] have all
 been addressed now more or less, so the package is ready for a more
 thorough review.

 Code: https://github.com/s-ludwig/std_data_json
 Docs: http://s-ludwig.github.io/std_data_json/

 [1]:
 http://forum.dlang.org/thread/lt5s76$is$1 digitalmars.com#post-lt5s76:24is:241:40digitalmars.com


 If no one else steps up, I can do it after DConf.


Thanks. I think it would be great if we moved faster with this - any 
chance that you or someone else starts this before DConf such that we 
get to vote during DConf? That'd be pretty cool. -- Andrei
Apr 08 2015
prev sibling next sibling parent reply "Andrea Fontana" <mail example.com> writes:
Any plan to support functions like these?

http://forum.dlang.org/thread/lrknjl$co7$1 digitalmars.com?page=4#post-bcszdbasnjzmbwzdgeqy:40forum.dlang.org

On Tuesday, 7 April 2015 at 16:37:15 UTC, Sönke Ludwig wrote:

 Anyone up to this? The issues of the previous discussion [1] 
 have all been addressed now more or less, so the package is 
 ready for a more thorough review.

 Code: https://github.com/s-ludwig/std_data_json
 Docs: http://s-ludwig.github.io/std_data_json/

 [1]: 
 http://forum.dlang.org/thread/lt5s76$is$1 digitalmars.com#post-lt5s76:24is:241:40digitalmars.com
Apr 08 2015
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 08.04.2015 um 10:24 schrieb Andrea Fontana:

 Any plan to support functions like these?

 http://forum.dlang.org/thread/lrknjl$co7$1 digitalmars.com?page=4#post-bcszdbasnjzmbwzdgeqy:40forum.dlang.org


There is opt() [1], which takes a path and returns a 
`Nullable!JSONValue`. "get" is unfortunately already taken by 
std.typecons.Algebraic, so we can't use it for its AA meaning.

Maybe another overload of opt?
JSONValue opt(JSONValue default_value, string[] path...);

[1]: 
http://s-ludwig.github.io/std_data_json/stdx/data/json/value/json_value.opt.html
Apr 08 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Wednesday, 8 April 2015 at 10:02:30 UTC, Sönke Ludwig wrote:

 Am 08.04.2015 um 10:24 schrieb Andrea Fontana:

 Any plan to support functions like these?

 http://forum.dlang.org/thread/lrknjl$co7$1 digitalmars.com?page=4#post-bcszdbasnjzmbwzdgeqy:40forum.dlang.org


 There is opt() [1], which takes a path and returns a 
 `Nullable!JSONValue`. "get" is unfortunately already taken by 
 std.typecons.Algebraic, so we can't use it for its AA meaning.

 Maybe another overload of opt?
 JSONValue opt(JSONValue default_value, string[] path...);

 [1]: 
 http://s-ludwig.github.io/std_data_json/stdx/data/json/value/json_value.opt.html


This seems a bit awkward to use. For example, it cannot handle 
numeric indices. Could we instead have an `undefined` value, that 
evaluates to `false` in a boolean context, and otherwise behaves 
as if it were absent? The two `opIndex` overloads will again 
return `undefined` for this value, as well as for non-objects and 
non-arrays, and for non-existing elements. Then you could access 
JSON values in an intuitive way without worrying about getting 
Exceptions or even Errors:

     writeln(json[42]["abc"][10 .. 20].get!int);

Only the `get` operation would throw if the type is wrong.

This also allows for easy and unambiguous testing whether an 
object contains a key:

     if(json["bla"] == JSONValue.undefined) { writeln("not 
found"); }

Anyway, it needs to be made clear what the behaviour on 
out-of-bounds or non-existing keys is: `undefined` value, 
exception, error?
Jul 28 2015
prev sibling next sibling parent reply "Robert burner Schadek" <rburners gmail.com> writes:
IMO this should be a PR for phobos so all comments to the code 
can be collected in one location.

Where is the benchmark against std.json and rapidjson?
Apr 08 2015
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 08.04.2015 um 11:05 schrieb Robert burner Schadek:

 IMO this should be a PR for phobos so all comments to the code can be
 collected in one location.

 Where is the benchmark against std.json and rapidjson?


That's what we have the review thread for. The library is now in a state 
that everyone can easily try out. If it were a Phobos PR, that would be 
much more difficult (or I'd have to maintain two versions in parallel).

I still have to look into running a proper benchmark. Initial numbers 
that I just collected have not been as good as expected - I'll have to 
take a closer look at the compiler output.
Apr 08 2015
next sibling parent "Robert burner Schadek" <rburners gmail.com> writes:
On Wednesday, 8 April 2015 at 09:58:31 UTC, Sönke Ludwig wrote:

 That's what we have the review thread for. The library is now 
 in a state that everyone can easily try out. If it were a 
 Phobos PR, that would be much more difficult (or I'd have to 
 maintain two versions in parallel).


from experience: Hardly anybody reads previous posts. comments on 
the source make it easier for you to get useful criticism and 
filter all the noise.


 I still have to look into running a proper benchmark. Initial 
 numbers that I just collected have not been as good as expected 
 - I'll have to take a closer look at the compiler output.


I thought more performance was one of the major reason for the 
json replacement. We need graphs!
Apr 08 2015
prev sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Wednesday, 8 April 2015 at 09:58:31 UTC, Sönke Ludwig wrote:

 Initial numbers that I just collected have not been as good as 
 expected. I'll have to take a closer look at the compiler 
 output.


I made a note, will see if I time to help with that. Algebraic 
might be a problem as it's based on typeinfo not tags, just a 
wild guess though.
Apr 08 2015
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 08.04.2015 um 12:34 schrieb Martin Nowak:

 On Wednesday, 8 April 2015 at 09:58:31 UTC, Sönke Ludwig wrote:

 Initial numbers that I just collected have not been as good as
 expected. I'll have to take a closer look at the compiler output.


 I made a note, will see if I time to help with that. Algebraic might be
 a problem as it's based on typeinfo not tags, just a wild guess though.


The problem is that even the pull parser alone is relatively slow. Also, 
for some reason the linker reports unresolved symbols as soon as I build 
without the -debug flag...
Apr 08 2015
next sibling parent reply "tcha" <chalucha gmail.com> writes:
On Wednesday, 8 April 2015 at 13:56:55 UTC, Sönke Ludwig wrote:

 The problem is that even the pull parser alone is relatively 
 slow. Also, for some reason the linker reports unresolved 
 symbols as soon as I build without the -debug flag...


Unfortunatelly I overlooked that I tested it with std_data_json v 
0.12.0 yesterday.
So tested it again today on master:
dmd and gdc failed to build in release
ldc2 works

I tried it also with ParseJSONStream and made fork from original 
benchmark with added tests 
(https://github.com/chalucha/benchmarks) so anyone can look (and 
PR)

Some results:
Ruby - 6.74s, 2106.2Mb
Go - 5.92s, 420.9Mb
D - 10.26s, 1322.5Mb
D Gdc - 24.76s, 926.1Mb
D Ldc - 26.31s, 925.0Mb
D new - debug - 14.98s, 1782.0Mb
D new Gdc - debug - 29.08s, 1663.9Mb
D new Ldc - 16.99s, 1663.0Mb
D new lazy - debug - 11.50s, 213.2Mb
D new lazy Gdc - debug - 13.66s, 206.1Mb
D new lazy Ldc - 3.59s, 205.4Mb
Python - 5.89s, 1063.4Mb
C++ Boost - 14.60s, 2884.8Mb
C++ Rapid - 0.80s, 678.9Mb


Also tried to dustmite the minimal failing version and here is a 
result: http://pastebin.com/YjdvT3G4

It's my first use of it so I hope it can help to solve this 
problem. It outputs less errors, but also compiles fine in debug 
and fails to link in release.
Apr 08 2015
next sibling parent reply Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8 April 2015 at 20:32, tcha via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Wednesday, 8 April 2015 at 13:56:55 UTC, Sönke Ludwig wrote:

 The problem is that even the pull parser alone is relatively slow. Also,
 for some reason the linker reports unresolved symbols as soon as I build
 without the -debug flag...



 Unfortunatelly I overlooked that I tested it with std_data_json v 0.12.0
 yesterday.
 So tested it again today on master:
 dmd and gdc failed to build in release
 ldc2 works

 I tried it also with ParseJSONStream and made fork from original benchmark
 with added tests (https://github.com/chalucha/benchmarks) so anyone can look
 (and PR)

 Some results:
 Ruby - 6.74s, 2106.2Mb
 Go - 5.92s, 420.9Mb
 D - 10.26s, 1322.5Mb
 D Gdc - 24.76s, 926.1Mb
 D Ldc - 26.31s, 925.0Mb
 D new - debug - 14.98s, 1782.0Mb
 D new Gdc - debug - 29.08s, 1663.9Mb
 D new Ldc - 16.99s, 1663.0Mb
 D new lazy - debug - 11.50s, 213.2Mb
 D new lazy Gdc - debug - 13.66s, 206.1Mb
 D new lazy Ldc - 3.59s, 205.4Mb
 Python - 5.89s, 1063.4Mb
 C++ Boost - 14.60s, 2884.8Mb
 C++ Rapid - 0.80s, 678.9Mb


 Also tried to dustmite the minimal failing version and here is a result:
 http://pastebin.com/YjdvT3G4

 It's my first use of it so I hope it can help to solve this problem. It
 outputs less errors, but also compiles fine in debug and fails to link in
 release.


Frankly, if we are not as fast (or elegant) as Python's json library,
it should be thrown out back to the drawing board.

Iain.
Apr 08 2015
next sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 08.04.2015 um 20:55 schrieb Iain Buclaw via Digitalmars-d:

 On 8 April 2015 at 20:32, tcha via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 (...)
 Also tried to dustmite the minimal failing version and here is a result:
 http://pastebin.com/YjdvT3G4

 It's my first use of it so I hope it can help to solve this problem. It
 outputs less errors, but also compiles fine in debug and fails to link in
 release.




I've filed two tickets for linker errors (and added a workaround for the 
first one):
https://issues.dlang.org/show_bug.cgi?id=14425
https://issues.dlang.org/show_bug.cgi?id=14429

I'll try to reduce the pastebin reduced sample further, too, as it looks 
like it has yet another root cause.


 Frankly, if we are not as fast (or elegant) as Python's json library,
 it should be thrown out back to the drawing board.

 Iain.


As far as the profiler results can be trusted, a good chunk of the time 
gets spent for reading individual bytes from memory, but there must be 
something else low-level going on that make things this bad. However, 
there is nothing fundamental in the structure/design that would cause 
this, so I think spending more time with the profiler is the only 
logical step now. Unfortunately my VTune license has expired and perf on 
Linux makes the task quite a bit more involved.

If we want to be really fast, though, we need to add optimized SIMD 
paths, but this is currently outside of the possibilities of my time budget.
Apr 09 2015
next sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 09.04.2015 um 10:59 schrieb Sönke Ludwig:

 Am 08.04.2015 um 20:55 schrieb Iain Buclaw via Digitalmars-d:

 On 8 April 2015 at 20:32, tcha via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 (...)
 Also tried to dustmite the minimal failing version and here is a result:
 http://pastebin.com/YjdvT3G4

 It's my first use of it so I hope it can help to solve this problem. It
 outputs less errors, but also compiles fine in debug and fails to
 link in
 release.




 I've filed two tickets for linker errors (and added a workaround for the
 first one):
 https://issues.dlang.org/show_bug.cgi?id=14425
 https://issues.dlang.org/show_bug.cgi?id=14429

 I'll try to reduce the pastebin reduced sample further, too, as it looks
 like it has yet another root cause.


Actually it seems to be the first issue (14425). I've already added a 
workaround for that on GIT master.
Apr 09 2015
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/09/2015 10:59 AM, Sönke Ludwig wrote:

 As far as the profiler results can be trusted, a good chunk of the time
 gets spent for reading individual bytes from memory, but there must be
 something else low-level going on that make things this bad. However,
 there is nothing fundamental in the structure/design that would cause
 this, so I think spending more time with the profiler is the only
 logical step now. Unfortunately my VTune license has expired and perf on
 Linux makes the task quite a bit more involved.


I didn't found too many issues.

Most of the time is spent in parseJSONValue (looks like there are some
expansive struct copies)
https://github.com/s-ludwig/std_data_json/blob/1da3f828ae6c4fd7cac7f7e13ae9e51ec93e6f02/source/stdx/data/json/parser.d#L148

and skipWhitespace. This function could really take some optimization,
e.g. avoid UTF decoding.

https://github.com/s-ludwig/std_data_json/blob/1da3f828ae6c4fd7cac7f7e13ae9e51ec93e6f02/source/stdx/data/json/lexer.d#L345

Libdparse has some optimized ASM function, might be useful.
https://github.com/Hackerpilot/libdparse/blob/51b7d9d321aac0fcc4a9be99bbbed5db3158326c/src/std/d/lexer.d#L2233
Apr 09 2015
parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 09.04.2015 um 15:20 schrieb Martin Nowak:

 On 04/09/2015 10:59 AM, Sönke Ludwig wrote:

 As far as the profiler results can be trusted, a good chunk of the time
 gets spent for reading individual bytes from memory, but there must be
 something else low-level going on that make things this bad. However,
 there is nothing fundamental in the structure/design that would cause
 this, so I think spending more time with the profiler is the only
 logical step now. Unfortunately my VTune license has expired and perf on
 Linux makes the task quite a bit more involved.


 I didn't found too many issues.

 Most of the time is spent in parseJSONValue (looks like there are some
 expansive struct copies)
 https://github.com/s-ludwig/std_data_json/blob/1da3f828ae6c4fd7cac7f7e13ae9e51ec93e6f02/source/stdx/data/json/parser.d#L148


I'll have a look at the DOM part later. At the moment I'm looking at low 
level issues of the lexer. There is a lot of potential there, but it's a 
lot of detail work.


 and skipWhitespace. This function could really take some optimization,
 e.g. avoid UTF decoding.


All of the lexing work happens on str.representation instead of the 
string directly, so no decoding should happen there.


 https://github.com/s-ludwig/std_data_json/blob/1da3f828ae6c4fd7cac7f7e13ae9e51ec93e6f02/source/stdx/data/json/lexer.d#L345

 Libdparse has some optimized ASM function, might be useful.
 https://github.com/Hackerpilot/libdparse/blob/51b7d9d321aac0fcc4a9be99bbbed5db3158326c/src/std/d/lexer.d#L2233
Apr 09 2015
prev sibling parent "w0rp" <devw0rp gmail.com> writes:
On Wednesday, 8 April 2015 at 18:56:00 UTC, Iain Buclaw wrote:

 Frankly, if we are not as fast (or elegant) as Python's json 
 library,
 it should be thrown out back to the drawing board.

 Iain.


I'll leave the speed aside, as more recent posts show 
improvements and I think Sönke will be able to take what he has, 
tweak it, and get it close to the fastest libraries. I think we 
can beat the competition.

For elegance, dynamic languages will always be able to deal with 
JSON with less syntax, as you get just the basic type back out of 
arrays and maps, without having to explictly request a string, 
integer, etc. Static languages will always have more verbose JSON 
libraries, but they might also catch bugs more easily. We just 
need the short properties for converting JSON values to basic 
types.
Apr 16 2015
prev sibling next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/08/2015 08:32 PM, tcha wrote:

Now with release numbers.


 D new - debug - 14.98s, 1782.0Mb


8.53s, 1786.8Mb

 D new Gdc - debug - 29.08s, 1663.9Mb


GDC still misses  nogc support.

 D new Ldc - 16.99s, 1663.0Mb


18.76s, 1664.1Mb

 D new lazy - debug - 11.50s, 213.2Mb


4.57s, 206Mb

 D new lazy Gdc - 13.66s, 206.1Mb


Can't compile stdx.data.json with gdc-4.9.0 which doesn't yet support  nogc.

 D new lazy Ldc - 3.59s, 205.4Mb


4.0s, 205.4Mb

LDC doesn't yet have the GC improvements, therefor is much slower for
the DOM parsing benchmarks.
Apr 09 2015
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 9 April 2015 at 11:49:00 UTC, Martin Nowak wrote:

 On 04/08/2015 08:32 PM, tcha wrote:

 Now with release numbers.


 D new - debug - 14.98s, 1782.0Mb


 8.53s, 1786.8Mb

 D new Gdc - debug - 29.08s, 1663.9Mb


 GDC still misses  nogc support.

 D new Ldc - 16.99s, 1663.0Mb


 18.76s, 1664.1Mb

 D new lazy - debug - 11.50s, 213.2Mb


 4.57s, 206Mb

 D new lazy Gdc - 13.66s, 206.1Mb


 Can't compile stdx.data.json with gdc-4.9.0 which doesn't yet 
 support  nogc.

 D new lazy Ldc - 3.59s, 205.4Mb


 4.0s, 205.4Mb

 LDC doesn't yet have the GC improvements, therefor is much 
 slower for
 the DOM parsing benchmarks.


Still getting trounced across the board by rapidjson.
Apr 09 2015
next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/09/2015 02:10 PM, John Colvin wrote:

 
 Still getting trounced across the board by rapidjson.


Yep, anyone knows why? They don't even use a lazy parser.
Apr 09 2015
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 9 April 2015 at 12:16:43 UTC, Martin Nowak wrote:

 On 04/09/2015 02:10 PM, John Colvin wrote:

 
 Still getting trounced across the board by rapidjson.


 Yep, anyone knows why? They don't even use a lazy parser.


simd optimized scanning and format-optimized inlined conversion
Apr 09 2015
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/9/15 5:10 AM, John Colvin wrote:

 On Thursday, 9 April 2015 at 11:49:00 UTC, Martin Nowak wrote:

 On 04/08/2015 08:32 PM, tcha wrote:

 Now with release numbers.


 D new - debug - 14.98s, 1782.0Mb


 8.53s, 1786.8Mb

 D new Gdc - debug - 29.08s, 1663.9Mb


 GDC still misses  nogc support.

 D new Ldc - 16.99s, 1663.0Mb


 18.76s, 1664.1Mb

 D new lazy - debug - 11.50s, 213.2Mb


 4.57s, 206Mb

 D new lazy Gdc - 13.66s, 206.1Mb


 Can't compile stdx.data.json with gdc-4.9.0 which doesn't yet support
  nogc.

 D new lazy Ldc - 3.59s, 205.4Mb


 4.0s, 205.4Mb

 LDC doesn't yet have the GC improvements, therefor is much slower for
 the DOM parsing benchmarks.


 Still getting trounced across the board by rapidjson.


Agreed. Rapidjson is the most important baseline. -- Andrei
Apr 09 2015
prev sibling next sibling parent Iain Buclaw via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9 April 2015 at 13:48, Martin Nowak via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 04/08/2015 08:32 PM, tcha wrote:

 Now with release numbers.


 D new - debug - 14.98s, 1782.0Mb


 8.53s, 1786.8Mb

 D new Gdc - debug - 29.08s, 1663.9Mb


 GDC still misses  nogc support.



Wasn't  nogc introduced in 2.066 ?

https://github.com/D-Programming-GDC/GDC/blob/master/gcc/d/VERSION
Apr 09 2015
prev sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Thursday, 9 April 2015 at 11:49:00 UTC, Martin Nowak wrote:

 On 04/08/2015 08:32 PM, tcha wrote:

 Now with release numbers.


 D new - debug - 14.98s, 1782.0Mb


 8.53s, 1786.8Mb

 D new Gdc - debug - 29.08s, 1663.9Mb


 GDC still misses  nogc support.

 D new Ldc - 16.99s, 1663.0Mb


 18.76s, 1664.1Mb

 D new lazy - debug - 11.50s, 213.2Mb


 4.57s, 206Mb

 D new lazy Gdc - 13.66s, 206.1Mb


 Can't compile stdx.data.json with gdc-4.9.0 which doesn't yet 
 support  nogc.

 D new lazy Ldc - 3.59s, 205.4Mb


 4.0s, 205.4Mb

 LDC doesn't yet have the GC improvements, therefor is much 
 slower for
 the DOM parsing benchmarks.


ldc -singleobj flag speeds it up dramatically

ldc2 -ofjson_d_new_lazy_ldc -O5 -release -enable-inlining -w -oq 
-Istd_data_json/source/ std_data_json/source/stdx/data/json/*.d 
test_new_lazy.d

./json_d_new_lazy_ldc  2.79s user 0.08s system 99% cpu 2.868 total

ldc2 -singleobj -ofjson_d_new_lazy_ldc -O5 -release 
-enable-inlining -w -oq -Istd_data_json/source/ 
std_data_json/source/stdx/data/json/*.d test_new_lazy.d

./json_d_new_lazy_ldc  1.78s user 0.08s system 99% cpu 1.869 total


for comparison:
./json_rapid_cpp  0.80s user 0.30s system 99% cpu 1.106 total


Bye.
Apr 09 2015
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 09.04.2015 um 21:06 schrieb weaselcat:

 On Thursday, 9 April 2015 at 11:49:00 UTC, Martin Nowak wrote:

 On 04/08/2015 08:32 PM, tcha wrote:

 Now with release numbers.


 D new - debug - 14.98s, 1782.0Mb


 8.53s, 1786.8Mb

 D new Gdc - debug - 29.08s, 1663.9Mb


 GDC still misses  nogc support.

 D new Ldc - 16.99s, 1663.0Mb


 18.76s, 1664.1Mb

 D new lazy - debug - 11.50s, 213.2Mb


 4.57s, 206Mb

 D new lazy Gdc - 13.66s, 206.1Mb


 Can't compile stdx.data.json with gdc-4.9.0 which doesn't yet support
  nogc.

 D new lazy Ldc - 3.59s, 205.4Mb


 4.0s, 205.4Mb

 LDC doesn't yet have the GC improvements, therefor is much slower for
 the DOM parsing benchmarks.


 ldc -singleobj flag speeds it up dramatically

 ldc2 -ofjson_d_new_lazy_ldc -O5 -release -enable-inlining -w -oq
 -Istd_data_json/source/ std_data_json/source/stdx/data/json/*.d
 test_new_lazy.d

 ./json_d_new_lazy_ldc  2.79s user 0.08s system 99% cpu 2.868 total

 ldc2 -singleobj -ofjson_d_new_lazy_ldc -O5 -release -enable-inlining -w
 -oq -Istd_data_json/source/ std_data_json/source/stdx/data/json/*.d
 test_new_lazy.d

 ./json_d_new_lazy_ldc  1.78s user 0.08s system 99% cpu 1.869 total


 for comparison:
 ./json_rapid_cpp  0.80s user 0.30s system 99% cpu 1.106 total


 Bye.


Not sure, but that may also have been my recent optimizations.
Apr 09 2015
parent reply "weaselcat" <weaselcat gmail.com> writes:
On Thursday, 9 April 2015 at 19:17:48 UTC, Sönke Ludwig wrote:


 Not sure, but that may also have been my recent optimizations.


Just tried it with your recent optimizations and it doesn't build 
with LDC 0.15.1.
Apr 09 2015
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 09.04.2015 um 21:26 schrieb weaselcat:

 On Thursday, 9 April 2015 at 19:17:48 UTC, Sönke Ludwig wrote:


 Not sure, but that may also have been my recent optimizations.


 Just tried it with your recent optimizations and it doesn't build with
 LDC 0.15.1.


Should work now. I just tested LDC with -singleobj and it's faster than 
RapidJSON! Now the DOM parser gets interesting.
Apr 09 2015
next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Thursday, 9 April 2015 at 19:35:24 UTC, Sönke Ludwig wrote:

 Am 09.04.2015 um 21:26 schrieb weaselcat:

 On Thursday, 9 April 2015 at 19:17:48 UTC, Sönke Ludwig wrote:


 Not sure, but that may also have been my recent optimizations.


 Just tried it with your recent optimizations and it doesn't 
 build with
 LDC 0.15.1.


 Should work now. I just tested LDC with -singleobj and it's 
 faster than RapidJSON! Now the DOM parser gets interesting.


What flags are you using? RapidJSON is still a lot faster for me.
also, I just realized the build script has the wrong inline flag 
for LDC(it's using -enable-inlining instead of -inline)
Apr 09 2015
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 09.04.2015 um 21:37 schrieb weaselcat:

 On Thursday, 9 April 2015 at 19:35:24 UTC, Sönke Ludwig wrote:

 Am 09.04.2015 um 21:26 schrieb weaselcat:

 On Thursday, 9 April 2015 at 19:17:48 UTC, Sönke Ludwig wrote:


 Not sure, but that may also have been my recent optimizations.


 Just tried it with your recent optimizations and it doesn't build with
 LDC 0.15.1.


 Should work now. I just tested LDC with -singleobj and it's faster
 than RapidJSON! Now the DOM parser gets interesting.


 What flags are you using? RapidJSON is still a lot faster for me.
 also, I just realized the build script has the wrong inline flag for
 LDC(it's using -enable-inlining instead of -inline)


I'll have to test with that exact version, because I've used my own 
main() function for the test. The flags were: -singleobj -release -O5 
-enable-inlining -disable-bondscheck (-disable-boundscheck is only a 
little faster).
Apr 09 2015
next sibling parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 09.04.2015 um 21:40 schrieb Sönke Ludwig:

 Am 09.04.2015 um 21:37 schrieb weaselcat:

 On Thursday, 9 April 2015 at 19:35:24 UTC, Sönke Ludwig wrote:

 Am 09.04.2015 um 21:26 schrieb weaselcat:

 On Thursday, 9 April 2015 at 19:17:48 UTC, Sönke Ludwig wrote:


 Not sure, but that may also have been my recent optimizations.


 Just tried it with your recent optimizations and it doesn't build with
 LDC 0.15.1.


 Should work now. I just tested LDC with -singleobj and it's faster
 than RapidJSON! Now the DOM parser gets interesting.


 What flags are you using? RapidJSON is still a lot faster for me.
 also, I just realized the build script has the wrong inline flag for
 LDC(it's using -enable-inlining instead of -inline)


 I'll have to test with that exact version, because I've used my own
 main() function for the test. The flags were: -singleobj -release -O5
 -enable-inlining -disable-bondscheck (-disable-boundscheck is only a
 little faster).


One thing about the D benchmark is that it uses readText(). To be a fair 
comparison with Rapid it would have to use cast(string)read() instead.
Apr 09 2015
parent reply "weaselcat" <weaselcat gmail.com> writes:
On Thursday, 9 April 2015 at 19:43:13 UTC, Sönke Ludwig wrote:

 Am 09.04.2015 um 21:40 schrieb Sönke Ludwig:

 Am 09.04.2015 um 21:37 schrieb weaselcat:

 On Thursday, 9 April 2015 at 19:35:24 UTC, Sönke Ludwig wrote:

 Am 09.04.2015 um 21:26 schrieb weaselcat:

 On Thursday, 9 April 2015 at 19:17:48 UTC, Sönke Ludwig 
 wrote:


 Not sure, but that may also have been my recent 
 optimizations.


 Just tried it with your recent optimizations and it doesn't 
 build with
 LDC 0.15.1.


 Should work now. I just tested LDC with -singleobj and it's 
 faster
 than RapidJSON! Now the DOM parser gets interesting.


 What flags are you using? RapidJSON is still a lot faster for 
 me.
 also, I just realized the build script has the wrong inline 
 flag for
 LDC(it's using -enable-inlining instead of -inline)


 I'll have to test with that exact version, because I've used 
 my own
 main() function for the test. The flags were: -singleobj 
 -release -O5
 -enable-inlining -disable-bondscheck (-disable-boundscheck is 
 only a
 little faster).


 One thing about the D benchmark is that it uses readText(). To 
 be a fair comparison with Rapid it would have to use 
 cast(string)read() instead.


I saw that commit to the benchmark and changed it locally.
They're about the same performance now comparing clang to LDC, 
with -inline -boundscheck=off -singleobj flags

Nice.
Apr 09 2015
parent reply "weaselcat" <weaselcat gmail.com> writes:
On Thursday, 9 April 2015 at 19:44:57 UTC, weaselcat wrote:

 I saw that commit to the benchmark and changed it locally.
 They're about the same performance now comparing clang to LDC, 
 with -inline -boundscheck=off -singleobj flags

 Nice.


Also, since an LDC dev might read this - is there a reason 
-singleobj isn't on by default when creating an executable?
Apr 09 2015
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 9 April 2015 at 20:01:10 UTC, weaselcat wrote:

 On Thursday, 9 April 2015 at 19:44:57 UTC, weaselcat wrote:

 I saw that commit to the benchmark and changed it locally.
 They're about the same performance now comparing clang to LDC, 
 with -inline -boundscheck=off -singleobj flags

 Nice.


 Also, since an LDC dev might read this - is there a reason 
 -singleobj isn't on by default when creating an executable?


I argued it should be the default a while ago and I vaguely 
remember agreement from at least one LDC dev. but I can't find 
the post now :/
Apr 10 2015
parent "David Nadlinger" <code klickverbot.at> writes:
On Friday, 10 April 2015 at 07:35:13 UTC, John Colvin wrote:

 On Thursday, 9 April 2015 at 20:01:10 UTC, weaselcat wrote:

 Also, since an LDC dev might read this - is there a reason 
 -singleobj isn't on by default when creating an executable?


 I argued it should be the default a while ago and I vaguely 
 remember agreement from at least one LDC dev. but I can't find 
 the post now :/


I still agree with you. ;)

The only reason why I didn't just commit the change so far is 
backwards compatibility, but as we might overhaul some of the 
command line options soon anyway, I think we should just go ahead 
with that change. As noted previously, mixing one-object-per-file 
and all-at-once compilation (e.g. in incremental builds) does not 
work correctly in all cases due to frontend restrictions anyway.

  — David
Apr 10 2015
prev sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 9 April 2015 at 19:40:31 UTC, Sönke Ludwig wrote:

 Am 09.04.2015 um 21:37 schrieb weaselcat:

 On Thursday, 9 April 2015 at 19:35:24 UTC, Sönke Ludwig wrote:

 Am 09.04.2015 um 21:26 schrieb weaselcat:

 On Thursday, 9 April 2015 at 19:17:48 UTC, Sönke Ludwig 
 wrote:


 Not sure, but that may also have been my recent 
 optimizations.


 Just tried it with your recent optimizations and it doesn't 
 build with
 LDC 0.15.1.


 Should work now. I just tested LDC with -singleobj and it's 
 faster
 than RapidJSON! Now the DOM parser gets interesting.


 What flags are you using? RapidJSON is still a lot faster for 
 me.
 also, I just realized the build script has the wrong inline 
 flag for
 LDC(it's using -enable-inlining instead of -inline)


 I'll have to test with that exact version, because I've used my 
 own main() function for the test. The flags were: -singleobj 
 -release -O5 -enable-inlining -disable-bondscheck 
 (-disable-boundscheck is only a little faster).


I can't remember which -O level inlining is enabled, but there's 
definitely no need to explicitly ask for it at -O5.
Apr 09 2015
parent reply "weaselcat" <weaselcat gmail.com> writes:
On Thursday, 9 April 2015 at 20:00:28 UTC, John Colvin wrote:

 I can't remember which -O level inlining is enabled, but 
 there's definitely no need to explicitly ask for it at -O5.


-enable-inlining(enabled at -O2) and -inline do different things 
and -inline isn't automatically enabled.
Apr 09 2015
parent reply "weaselcat" <weaselcat gmail.com> writes:
On Thursday, 9 April 2015 at 20:11:07 UTC, weaselcat wrote:

 On Thursday, 9 April 2015 at 20:00:28 UTC, John Colvin wrote:

 I can't remember which -O level inlining is enabled, but 
 there's definitely no need to explicitly ask for it at -O5.


 -enable-inlining(enabled at -O2) and -inline do different 
 things and -inline isn't automatically enabled.


Sorry Sönke for polluting your thread, but I did some reading of 
the LLVM manual - -inline is a specific LLVM pass and not a 
frontend flag. Using the -partial-inliner pass instead it(the 
benchmark) becomes faster than rapidJSON.

Bye.
Apr 09 2015
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 09.04.2015 um 23:14 schrieb weaselcat:

 On Thursday, 9 April 2015 at 20:11:07 UTC, weaselcat wrote:

 On Thursday, 9 April 2015 at 20:00:28 UTC, John Colvin wrote:

 I can't remember which -O level inlining is enabled, but there's
 definitely no need to explicitly ask for it at -O5.


 -enable-inlining(enabled at -O2) and -inline do different things and
 -inline isn't automatically enabled.


 Sorry Sönke for polluting your thread, but I did some reading of the
 LLVM manual - -inline is a specific LLVM pass and not a frontend flag.
 Using the -partial-inliner pass instead it(the benchmark) becomes faster
 than rapidJSON.

 Bye.


Hm interesting, but for me LDC unfortunately crashes when adding 
-partial-inliner. What else did you specify?
Apr 09 2015
parent reply "weaselcat" <weaselcat gmail.com> writes:
On Thursday, 9 April 2015 at 21:52:08 UTC, Sönke Ludwig wrote:

 Am 09.04.2015 um 23:14 schrieb weaselcat:

 On Thursday, 9 April 2015 at 20:11:07 UTC, weaselcat wrote:

 On Thursday, 9 April 2015 at 20:00:28 UTC, John Colvin wrote:

 I can't remember which -O level inlining is enabled, but 
 there's
 definitely no need to explicitly ask for it at -O5.


 -enable-inlining(enabled at -O2) and -inline do different 
 things and
 -inline isn't automatically enabled.


 Sorry Sönke for polluting your thread, but I did some reading 
 of the
 LLVM manual - -inline is a specific LLVM pass and not a 
 frontend flag.
 Using the -partial-inliner pass instead it(the benchmark) 
 becomes faster
 than rapidJSON.

 Bye.


 Hm interesting, but for me LDC unfortunately crashes when 
 adding -partial-inliner. What else did you specify?


ldc2 -partial-inliner -boundscheck=off -singleobj 
-ofjson_d_new_lazy_ldc -O5 -release -w -oq 
-Istd_data_json/source/ std_data_json/source/stdx/data/json/*.d 
test_new_lazy.d

it seems to segfault with -O0 optimization

./json_d_new_lazy_ldc  0.95s user 0.08s system 99% cpu 1.025 total
(clang)./json_rapid_cpp  0.98s user 0.30s system 99% cpu 1.282 
total
(gcc)./json_rapid_cpp  0.80s user 0.31s system 99% cpu 1.108 total
Apr 09 2015
parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 09.04.2015 um 23:56 schrieb weaselcat:

 On Thursday, 9 April 2015 at 21:52:08 UTC, Sönke Ludwig wrote:

 Hm interesting, but for me LDC unfortunately crashes when adding
 -partial-inliner. What else did you specify?


 ldc2 -partial-inliner -boundscheck=off -singleobj -ofjson_d_new_lazy_ldc
 -O5 -release -w -oq -Istd_data_json/source/
 std_data_json/source/stdx/data/json/*.d test_new_lazy.d

 it seems to segfault with -O0 optimization

 ./json_d_new_lazy_ldc  0.95s user 0.08s system 99% cpu 1.025 total
 (clang)./json_rapid_cpp  0.98s user 0.30s system 99% cpu 1.282 total
 (gcc)./json_rapid_cpp  0.80s user 0.31s system 99% cpu 1.108 total


Okay, turned out that this was a 0.15.2-beta.1 issue. I've downgraded to 
0.15.1 and now get a nice time of 1.30 seconds on my tablet PC!
Apr 09 2015
parent "y" <dlang srtnwz.com> writes:
what a difference. cool stuff Sönke!

below results from a 3.1 GHz Intel Core i7 Broadwell mbp13" on os 
x:
ldc2 0.15.1 from homebrew
dmd v2.067 from homebrew
optimized => " -partial-inliner -boundscheck=off -singleobj "

D new lazy Ldc optimized                  0.85            204.5Mb
Crystal Schema                            1.41            283.4Mb
Crystal Pull                              1.49            1.2Mb
C++ Rapid                                 1.92            946.5Mb
D new lazy dmd                            2               204.5Mb
D new lazy Ldc                            2.12            204.6Mb
D new Ldc optimized                       3.19            1640.5Mb
Crystal                                   3.23            1091.9Mb
Javascript Node                           3.48            905.3Mb
D new Ldc                                 3.98            1640.5Mb
Nim                                       4.38            1336.8Mb
Go                                        5.34            420.3Mb
D new - debug                             6.34            1793.6Mb
Python3                                   7.06            999.7Mb
Rust                                      7.46            3041.7Mb
Python Pypy 3                             9.27            2710.3Mb
Julia                                     10.09           3054.8Mb
D                                         12.47           1219.1Mb
Ruby                                      13.94           2167.2Mb
C++ Boost                                 24.64           2699.4Mb
D Ldc                                     27.38           928.1Mb

y
Apr 09 2015
prev sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 09.04.2015 um 21:35 schrieb Sönke Ludwig:

 Am 09.04.2015 um 21:26 schrieb weaselcat:

 On Thursday, 9 April 2015 at 19:17:48 UTC, Sönke Ludwig wrote:


 Not sure, but that may also have been my recent optimizations.


 Just tried it with your recent optimizations and it doesn't build with
 LDC 0.15.1.


 Should work now. I just tested LDC with -singleobj and it's faster than
 RapidJSON! Now the DOM parser gets interesting.


Okay let's say about the same speed, the numbers fluctuate quite a bit.
Apr 09 2015
prev sibling next sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com> writes:
Am 08.04.2015 um 20:32 schrieb tcha:

 On Wednesday, 8 April 2015 at 13:56:55 UTC, Sönke Ludwig wrote:

 The problem is that even the pull parser alone is relatively slow.
 Also, for some reason the linker reports unresolved symbols as soon as
 I build without the -debug flag...


 Unfortunatelly I overlooked that I tested it with std_data_json v 0.12.0
 yesterday.
 So tested it again today on master:
 dmd and gdc failed to build in release
 ldc2 works

 I tried it also with ParseJSONStream and made fork from original
 benchmark with added tests (https://github.com/chalucha/benchmarks) so
 anyone can look (and PR)

 Some results:
 Ruby - 6.74s, 2106.2Mb
 Go - 5.92s, 420.9Mb
 D - 10.26s, 1322.5Mb
 D Gdc - 24.76s, 926.1Mb
 D Ldc - 26.31s, 925.0Mb
 D new - debug - 14.98s, 1782.0Mb
 D new Gdc - debug - 29.08s, 1663.9Mb
 D new Ldc - 16.99s, 1663.0Mb
 D new lazy - debug - 11.50s, 213.2Mb
 D new lazy Gdc - debug - 13.66s, 206.1Mb
 D new lazy Ldc - 3.59s, 205.4Mb
 Python - 5.89s, 1063.4Mb
 C++ Boost - 14.60s, 2884.8Mb
 C++ Rapid - 0.80s, 678.9Mb


 Also tried to dustmite the minimal failing version and here is a result:
 http://pastebin.com/YjdvT3G4

 It's my first use of it so I hope it can help to solve this problem. It
 outputs less errors, but also compiles fine in debug and fails to link
 in release.


I got the latest version using the pull parser down to the Crystal Pull 
parser speed*. This is about twice as slow as Rapid JSON.

* Using DMD to compile. LDC is for some reason much slower, but I didn't 
try the -singleobj switch yet.
Apr 09 2015
prev sibling parent =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig outerproduct.org> writes:
These are my results (a little bit flaky timings because it's a tablet 
PC which seems to throttle the turbo boost to not exceed the TDP):

std.json (DMD): 27.7
std.json (LDC): 72.4
stdx.data.json DOM (DMD): 8.2
stdx.data.json DOM (LDC): 3.96
stdx.data.json PULL (DMD): 2.96
stdx.data.json PULL (LDC): 1.61
RapidJSON: 1.21

I'd say these start to look good enough for a first fitness test.
Apr 09 2015
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/08/2015 03:56 PM, Sönke Ludwig wrote:


 
 The problem is that even the pull parser alone is relatively slow. Also,
 for some reason the linker reports unresolved symbols as soon as I build
 without the -debug flag...


The review hasn't yet started and I'm already against the "stream"
parser, because it hardly deserves the names parser, it's more like a lexer.

Because the benchmark code by tcha was a very specific hack for the used
data structure, I tried to write a proper stream parser to have a fair
comparison. This is where I stopped (it doesn't work).

http://dpaste.dzfl.pl/8282d70a1254

The biggest problem with that interface is, that you have to count
matching start/end markers for objects and arrays in order to skip an
entry, not much fun and definitely needs a dedicated skip value function.

There are 2 very nice alternative approaches in the benchmark repo.

https://github.com/kostya/benchmarks/blob/master/json/test_pull.cr
https://github.com/kostya/benchmarks/blob/master/json/test_schema.cr



 Apr 09 2015



  parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com>  writes:


Am 09.04.2015 um 14:25 schrieb Martin Nowak:

 On 04/08/2015 03:56 PM, Sönke Ludwig wrote:


 The problem is that even the pull parser alone is relatively slow. Also,
 for some reason the linker reports unresolved symbols as soon as I build
 without the -debug flag...


 The review hasn't yet started and I'm already against the "stream"
 parser, because it hardly deserves the names parser, it's more like a lexer.

 Because the benchmark code by tcha was a very specific hack for the used
 data structure, I tried to write a proper stream parser to have a fair
 comparison. This is where I stopped (it doesn't work).

 http://dpaste.dzfl.pl/8282d70a1254

 The biggest problem with that interface is, that you have to count
 matching start/end markers for objects and arrays in order to skip an
 entry, not much fun and definitely needs a dedicated skip value function.

 There are 2 very nice alternative approaches in the benchmark repo.

 https://github.com/kostya/benchmarks/blob/master/json/test_pull.cr
 https://github.com/kostya/benchmarks/blob/master/json/test_schema.cr


That would be a nice intermediate level parser. However, the range based 
parser as it is now is also useful for things like automated 
deserialization (which I don't want to include at this point) where you 
don't know in which order fields come in and where you have to use that 
style of filtering through the data anyway.

But if inlining works properly, it should be no problem to implement 
other APIs on top of it.



 Apr 09 2015



  parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com>  writes:


Am 09.04.2015 um 15:11 schrieb Sönke Ludwig:

 Am 09.04.2015 um 14:25 schrieb Martin Nowak:

 (...)

 There are 2 very nice alternative approaches in the benchmark repo.

 https://github.com/kostya/benchmarks/blob/master/json/test_pull.cr
 https://github.com/kostya/benchmarks/blob/master/json/test_schema.cr


 That would be a nice intermediate level parser. However, the range based
 parser as it is now is also useful for things like automated
 deserialization (which I don't want to include at this point) where you
 don't know in which order fields come in and where you have to use that
 style of filtering through the data anyway.

 But if inlining works properly, it should be no problem to implement
 other APIs on top of it.


https://github.com/s-ludwig/benchmarks/blob/master/json/test_new_lazy.d



 Apr 09 2015



  parent  Martin Nowak <code+news.digitalmars dawg.eu>  writes:


On 04/09/2015 11:13 PM, Sönke Ludwig wrote:

 Am 09.04.2015 um 15:11 schrieb Sönke Ludwig:
 https://github.com/s-ludwig/benchmarks/blob/master/json/test_new_lazy.d


That was fast :).



 Apr 09 2015



prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>  writes:


On Tuesday, 7 April 2015 at 16:37:15 UTC, Sönke Ludwig wrote:

 Anyone up to this? The issues of the previous discussion [1] 
 have all been addressed now more or less, so the package is 
 ready for a more thorough review.

 Code: https://github.com/s-ludwig/std_data_json
 Docs: http://s-ludwig.github.io/std_data_json/

 [1]: 
 http://forum.dlang.org/thread/lt5s76$is$1 digitalmars.com#post-lt5s76:24is:241:40digitalmars.com


It seems the `in` operator (or an equivalent method) isn't 
supported...



 Apr 08 2015



  parent  =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com>  writes:


Am 08.04.2015 um 12:52 schrieb "Marc =?UTF-8?B?U2Now7x0eiI=?= 
<schuetzm gmx.net>":

 On Tuesday, 7 April 2015 at 16:37:15 UTC, Sönke Ludwig wrote:

 Anyone up to this? The issues of the previous discussion [1] have all
 been addressed now more or less, so the package is ready for a more
 thorough review.

 Code: https://github.com/s-ludwig/std_data_json
 Docs: http://s-ludwig.github.io/std_data_json/

 [1]:
 http://forum.dlang.org/thread/lt5s76$is$1 digitalmars.com#post-lt5s76:24is:241:40digitalmars.com


 It seems the `in` operator (or an equivalent method) isn't supported...


I didn't try it, but the idea is that all operations on JSONValue are 
supported by std.variant.Algebraic, so anything that doesn't work, but 
should, should be fixed with a PR against Algebraic.



 Apr 08 2015



prev sibling next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2015-04-07 18:37, Sönke Ludwig wrote:

 Anyone up to this? The issues of the previous discussion [1] have all
 been addressed now more or less, so the package is ready for a more
 thorough review.


Is it possible to use "toJSON" or a similar method to generate JSON from 
a primitive type without going through JSONValue?

-- 
/Jacob Carlborg



 Apr 16 2015



  parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com>  writes:


Am 16.04.2015 um 11:17 schrieb Jacob Carlborg:

 On 2015-04-07 18:37, Sönke Ludwig wrote:

 Anyone up to this? The issues of the previous discussion [1] have all
 been addressed now more or less, so the package is ready for a more
 thorough review.


 Is it possible to use "toJSON" or a similar method to generate JSON from
 a primitive type without going through JSONValue?


I'd like to let that be part of a more general serialization framework 
in top of this package instead of integrating a simplistic custom 
solution that will then later be obsoleted. Having said that, you can go 
through JSONToken instead of JSONValue for lower (computational) overhead.



 Apr 16 2015



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2015-04-16 11:29, Sönke Ludwig wrote:


 I'd like to let that be part of a more general serialization framework
 in top of this package instead of integrating a simplistic custom
 solution that will then later be obsoleted.


I was thinking about some low level primitives that a serialization 
library could use. Types that cannot be broken in to smaller parts, 
integer, bool and so on.


 Having said that, you can go through JSONToken instead of JSONValue for lower
(computational) overhead.


Something like?

JSONToken token;
token.number = 3;
token.kind = JSONToken.Kind.number;
toJSON(token);

Or what would you recommend a serialization library used?

-- 
/Jacob Carlborg



 Apr 16 2015



  parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com>  writes:


Am 16.04.2015 um 13:03 schrieb Jacob Carlborg:

 On 2015-04-16 11:29, Sönke Ludwig wrote:


 I'd like to let that be part of a more general serialization framework
 in top of this package instead of integrating a simplistic custom
 solution that will then later be obsoleted.


 I was thinking about some low level primitives that a serialization
 library could use. Types that cannot be broken in to smaller parts,
 integer, bool and so on.


 Having said that, you can go through JSONToken instead of JSONValue
 for lower (computational) overhead.


 Something like?

 JSONToken token;
 token.number = 3;
 token.kind = JSONToken.Kind.number;
 toJSON(token);


Yes, just without setting the kind explicitly (it's set automatically by 
assigning to the number property). Instead of toJSON(), writeJSON() 
would probably be more appropriate to avoid additional allocations.


 Or what would you recommend a serialization library used?


The simplest target for a serialization library would be to generate a 
stream of JSONParserNodes. That way the serializer doesn't have to keep 
track of nesting levels and can reuse the pretty printing functionality 
of stdx.data.generator.

However, this currently requires an input range of JSONParserNodes, 
which looks like it is suboptimal for a serializer (either requires an 
explicitly allocated node array, or a strange serializer architecture 
that directly works as an input range). For that reason I'm thinking 
about providing an output range interface like so:

	auto dst = appender!string;
	auto rng = jsonOutputRange(dst);

	JSONParserNode n;
	// start an object
	n.kind = JSONParserNode.Kind.objectStart;
	rng.put(n);

	// write a single entry "foo": 10.5
	n.key = "foo";
	rng.put(n);
	rng.put(10.5);

	// write another entry "bar": true
	n.key = "bar";
	rng.put(n);
	rng.put(true);

	// finish the object
	n.kind = JSONParserNode.Kind.objectEnd;
	rng.put(n);

	writefln("JSON: %s", dst.data);

What do you think?

BTW, looks like JSONToken could use a set of constructors for more 
convenient in-place construction.



 Apr 16 2015



 next sibling parent reply "w0rp" <devw0rp gmail.com>  writes:


On Thursday, 16 April 2015 at 12:17:55 UTC, Sönke Ludwig wrote:

 Am 16.04.2015 um 13:03 schrieb Jacob Carlborg:

 On 2015-04-16 11:29, Sönke Ludwig wrote:


 I'd like to let that be part of a more general serialization 
 framework
 in top of this package instead of integrating a simplistic 
 custom
 solution that will then later be obsoleted.


 I was thinking about some low level primitives that a 
 serialization
 library could use. Types that cannot be broken in to smaller 
 parts,
 integer, bool and so on.


 Having said that, you can go through JSONToken instead of 
 JSONValue
 for lower (computational) overhead.


 Something like?

 JSONToken token;
 token.number = 3;
 token.kind = JSONToken.Kind.number;
 toJSON(token);


 Yes, just without setting the kind explicitly (it's set 
 automatically by assigning to the number property). Instead of 
 toJSON(), writeJSON() would probably be more appropriate to 
 avoid additional allocations.


 Or what would you recommend a serialization library used?


 The simplest target for a serialization library would be to 
 generate a stream of JSONParserNodes. That way the serializer 
 doesn't have to keep track of nesting levels and can reuse the 
 pretty printing functionality of stdx.data.generator.

 However, this currently requires an input range of 
 JSONParserNodes, which looks like it is suboptimal for a 
 serializer (either requires an explicitly allocated node array, 
 or a strange serializer architecture that directly works as an 
 input range). For that reason I'm thinking about providing an 
 output range interface like so:

 	auto dst = appender!string;
 	auto rng = jsonOutputRange(dst);

 	JSONParserNode n;
 	// start an object
 	n.kind = JSONParserNode.Kind.objectStart;
 	rng.put(n);

 	// write a single entry "foo": 10.5
 	n.key = "foo";
 	rng.put(n);
 	rng.put(10.5);

 	// write another entry "bar": true
 	n.key = "bar";
 	rng.put(n);
 	rng.put(true);

 	// finish the object
 	n.kind = JSONParserNode.Kind.objectEnd;
 	rng.put(n);

 	writefln("JSON: %s", dst.data);

 What do you think?

 BTW, looks like JSONToken could use a set of constructors for 
 more convenient in-place construction.


I think serialiastion for this JSON library should probably be 
considered out of scope until we have a general serisalisation 
API. Then once we have both, we can marry the two together. So as 
you say, the support from your end seems to be there. There just 
needs to be a serialiser to hook into it.



 Apr 16 2015



  parent  Jacob Carlborg <doob me.com>  writes:


On 2015-04-16 14:28, w0rp wrote:


 I think serialiastion for this JSON library should probably be
 considered out of scope until we have a general serisalisation API. Then
 once we have both, we can marry the two together. So as you say, the
 support from your end seems to be there. There just needs to be a
 serialiser to hook into it.


I'm working on the general serialization API. I'm working on rewriting 
Orange [1] to get it into Phobos. I'm asking about an API for converting 
basic types, that cannot be further taken apart, to JSON.

[1] https://github.com/jacob-carlborg/orange

-- 
/Jacob Carlborg



 Apr 16 2015



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 2015-04-16 14:17, Sönke Ludwig wrote:


 The simplest target for a serialization library would be to generate a
 stream of JSONParserNodes. That way the serializer doesn't have to keep
 track of nesting levels and can reuse the pretty printing functionality
 of stdx.data.generator.

 However, this currently requires an input range of JSONParserNodes,
 which looks like it is suboptimal for a serializer (either requires an
 explicitly allocated node array, or a strange serializer architecture
 that directly works as an input range).


The serialization library I'm working on has an output range interface.


 For that reason I'm thinking about providing an output range interface like so:

      auto dst = appender!string;
      auto rng = jsonOutputRange(dst);

      JSONParserNode n;
      // start an object
      n.kind = JSONParserNode.Kind.objectStart;
      rng.put(n);

      // write a single entry "foo": 10.5
      n.key = "foo";
      rng.put(n);
      rng.put(10.5);

      // write another entry "bar": true
      n.key = "bar";
      rng.put(n);
      rng.put(true);

      // finish the object
      n.kind = JSONParserNode.Kind.objectEnd;
      rng.put(n);

      writefln("JSON: %s", dst.data);

 What do you think?


That looks very handy, that would be great to have.

-- 
/Jacob Carlborg



 Apr 16 2015



prev sibling  parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com>  writes:


Am 07.04.2015 um 18:37 schrieb Sönke Ludwig:

 Anyone up to this? The issues of the previous discussion [1] have all
 been addressed now more or less, so the package is ready for a more
 thorough review.

 Code: https://github.com/s-ludwig/std_data_json
 Docs: http://s-ludwig.github.io/std_data_json/

 [1]:
 http://forum.dlang.org/thread/lt5s76$is$1 digitalmars.com#post-lt5s76:24is:241:40digitalmars.com


So, since Dicebot has stepped down from his review manager role, is 
there anyone else who would take that over? It's a little bit of a shame 
to have that code just lying around for so long.



 Jul 26 2015



  parent reply "Atila Neves" <atila.neves gmail.com>  writes:


On Sunday, 26 July 2015 at 09:09:51 UTC, Sönke Ludwig wrote:

 Am 07.04.2015 um 18:37 schrieb Sönke Ludwig:

 Anyone up to this? The issues of the previous discussion [1] 
 have all
 been addressed now more or less, so the package is ready for a 
 more
 thorough review.

 Code: https://github.com/s-ludwig/std_data_json
 Docs: http://s-ludwig.github.io/std_data_json/

 [1]:
 http://forum.dlang.org/thread/lt5s76$is$1 digitalmars.com#post-lt5s76:24is:241:40digitalmars.com


 So, since Dicebot has stepped down from his review manager 
 role, is there anyone else who would take that over? It's a 
 little bit of a shame to have that code just lying around for 
 so long.


I can do it. It's just announcing and counting votes, right? Is 
the formal review now?

Atila



 Jul 27 2015



  parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com>  writes:


Am 27.07.2015 um 22:00 schrieb Atila Neves:

 On Sunday, 26 July 2015 at 09:09:51 UTC, Sönke Ludwig wrote:

 Am 07.04.2015 um 18:37 schrieb Sönke Ludwig:

 Anyone up to this? The issues of the previous discussion [1] have all
 been addressed now more or less, so the package is ready for a more
 thorough review.

 Code: https://github.com/s-ludwig/std_data_json
 Docs: http://s-ludwig.github.io/std_data_json/

 [1]:
 http://forum.dlang.org/thread/lt5s76$is$1 digitalmars.com#post-lt5s76:24is:241:40digitalmars.com


 So, since Dicebot has stepped down from his review manager role, is
 there anyone else who would take that over? It's a little bit of a
 shame to have that code just lying around for so long.


 I can do it. It's just announcing and counting votes, right? Is the
 formal review now?

 Atila


That'd be great, thanks! I *think* that's all (maybe also updating the 
wiki) and since there is no other review going on, it should be possible 
to start a new one now. But I also don't know for sure...



 Jul 27 2015



  parent reply "extrawurst" <stephan extrawurst.org>  writes:


On Monday, 27 July 2015 at 20:30:23 UTC, Sönke Ludwig wrote:

 Am 27.07.2015 um 22:00 schrieb Atila Neves:

 On Sunday, 26 July 2015 at 09:09:51 UTC, Sönke Ludwig wrote:

 Am 07.04.2015 um 18:37 schrieb Sönke Ludwig:

[...]


 So, since Dicebot has stepped down from his review manager 
 role, is
 there anyone else who would take that over? It's a little bit 
 of a
 shame to have that code just lying around for so long.


 I can do it. It's just announcing and counting votes, right? 
 Is the
 formal review now?

 Atila


 That'd be great, thanks! I *think* that's all (maybe also 
 updating the wiki) and since there is no other review going on, 
 it should be possible to start a new one now. But I also don't 
 know for sure...


I guess folks would be much more willing to do this if it was 
defined somewhere what it exactly means ;)

--Stephan



 Jul 28 2015



  parent reply =?UTF-8?B?U8O2bmtlIEx1ZHdpZw==?= <sludwig rejectedsoftware.com>  writes:


Am 28.07.2015 um 10:38 schrieb extrawurst:

 On Monday, 27 July 2015 at 20:30:23 UTC, Sönke Ludwig wrote:

 (...)
 That'd be great, thanks! I *think* that's all (maybe also updating the
 wiki) and since there is no other review going on, it should be
 possible to start a new one now. But I also don't know for sure...


 I guess folks would be much more willing to do this if it was defined
 somewhere what it exactly means ;)

 --Stephan


Just found something:
http://wiki.dlang.org/Review/Process#Review_Manager



 Jul 28 2015



  parent  Jacob Carlborg <doob me.com>  writes:


On 2015-07-28 11:23, Sönke Ludwig wrote:


 Just found something:
 http://wiki.dlang.org/Review/Process#Review_Manager


I would add "making sure the review queue is updated".

-- 
/Jacob Carlborg



 Jul 28 2015