www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Optimising JSON parsing leads to wierd execution timings

reply xtreak <tir.karthi gmail.com> writes:
Hi,

Sorry for the double post. I have asked a question at 
Stackoverflow regarding this : 
https://stackoverflow.com/questions/42992507/get-float-value-out-o
-jsonvalue-in-dlang . I have a `rating` field that might have 3 which parses to
JSONValue.integer or 3.4 which parses to JSONValue.floating and I need to use
float. I was using exceptions that was causing the program to be slow as per
the profile information. Now I have used an if clause as suggested by Adam in
the answer. Though the execution time for the relevant function that used
exceptions earlier reduced the program was still taking the same time.

So I profiled again and now the time for JSON parsing has 
suddenly increased as per the profile. I thought it was due to 
using .type and I returned a hardcoded number for the function 
get_number and the time to write to terminal increased. I don't 
know why reducing the time on one function leads to increase in 
another function.

JSON file : 
gist.github.com/tirkarthi/c4b2ea745f15dbc6a01327b86c1d059f
Compiler : ldc2 with -O3 and -release flags
LLVM : LLVM 4.0
Operating System : Mac OS (Early 2015 model) with 4GB RAM

Original program with exceptions :

import std.net.curl;
import std.exception;
import std.json;
import std.stdio;
import std.file;
import std.algorithm;
import std.array;
import std.conv;
import std.string;

auto fmt = "%35s | %10s | %10s | %10s | %15s";
auto value_fmt = "%35s | %10.2f | %10.0f | %10.2f | %15s";

void print_header() {
   
writeln("-----------------------------------------------------------------------------------------------");
   writefln(fmt, "Name", "Price", "Window", "Rating", "Type");
   
writeln("-----------------------------------------------------------------------------------------------");
}

float get_number(T)(T item) {
   float output = 0;
   try {
     output = to!float(item.integer);
   } catch(Exception e) {
     output = to!float(item.floating);
   }
   return to!float(output);
}

void print_item(T)(T item) {
   writefln(value_fmt, item["Tvs"].str, 
item["MinFare"].get_number(), item["WnSt"].get_number(), 
item["Rtg"]["totRt"].get_number(), 
item["BusCategory"]["IsSleeper"]);
}

void main() {
   auto content = readText("sample.json");
   auto parsed = parseJSON(content);
   auto raw = parsed["SRD"][0]["RIN"][0]["InvList"].array;
   auto filtered = raw.filter!(item => 
to!bool(to!string(item["BusCategory"]["IsAc"])));

   print_header();
   foreach(item; filtered) {
     print_item(item);
   }
}
Mar 24 2017
parent reply Seb <seb wilzba.ch> writes:
On Saturday, 25 March 2017 at 06:53:58 UTC, xtreak wrote:

 Hi,

 Sorry for the double post. I have asked a question at 
 Stackoverflow regarding this : 
 https://stackoverflow.com/questions/42992507/get-float-value-out-o
-jsonvalue-in-dlang . I have a `rating` field that might have 3 which parses to
JSONValue.integer or 3.4 which parses to JSONValue.floating and I need to use
float. I was using exceptions that was causing the program to be slow as per
the profile information. Now I have used an if clause as suggested by Adam in
the answer. Though the execution time for the relevant function that used
exceptions earlier reduced the program was still taking the same time.

 [...]


If you need a fast JSON library, have a look at 
https://github.com/tamediadigital/asdf or 
https://github.com/s-ludwig/std_data_json
Mar 27 2017
parent Laeeth Isharc <laeeth nospamlaeeth.com> writes:
On Monday, 27 March 2017 at 09:05:00 UTC, Seb wrote:

 On Saturday, 25 March 2017 at 06:53:58 UTC, xtreak wrote:

 Hi,

 Sorry for the double post. I have asked a question at 
 Stackoverflow regarding this : 
 https://stackoverflow.com/questions/42992507/get-float-value-out-o
-jsonvalue-in-dlang . I have a `rating` field that might have 3 which parses to
JSONValue.integer or 3.4 which parses to JSONValue.floating and I need to use
float. I was using exceptions that was causing the program to be slow as per
the profile information. Now I have used an if clause as suggested by Adam in
the answer. Though the execution time for the relevant function that used
exceptions earlier reduced the program was still taking the same time.

 [...]


 If you need a fast JSON library, have a look at 
 https://github.com/tamediadigital/asdf or 
 https://github.com/s-ludwig/std_data_json


Or search forum for fastest Json parser in the world.  I use asdf 
though, which is fast enough and has a nice interface.
Mar 28 2017