www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - stdx.data.json

reply "Laeeth Isharc" <nospamlaeeth nospam.laeeth.com> writes:
Hi.

What's the best way to pass the contents of a file to the stream 
parser without reading the whole thing into memory first?  I get 
an error if using byLine because the kind of range this function 
returns is not what the stream parser is expecting.

There is an optional filename argument to parseJSONStream, but I 
am not sure what this is for, since it still requires the first 
argument for the input if the filename is passed.


Thanks.


Laeeth.
Apr 29 2015
parent reply "Laeeth Isharc" <nospamlaeeth nospam.laeeth.com> writes:
On Wednesday, 29 April 2015 at 18:48:22 UTC, Laeeth Isharc wrote:

 Hi.

 What's the best way to pass the contents of a file to the 
 stream parser without reading the whole thing into memory 
 first?  I get an error if using byLine because the kind of 
 range this function returns is not what the stream parser is 
 expecting.

 There is an optional filename argument to parseJSONStream, but 
 I am not sure what this is for, since it still requires the 
 first argument for the input if the filename is passed.


 Thanks.


 Laeeth.


some more suggestions for enhancements here:
https://github.com/s-ludwig/std_data_json/issues/12

a big advantage of languages like python is that there are 
end-to-end examples of using the language+libraries to accomplish 
real work.  (they don't need to be more than a screenful of code, 
but for the newcomer having it all set out makes a big 
difference).  since I believe a significant reason for people to 
switch to D will be a realisation that actually CPU time isn't 
free, it makes sense to be accommodating to people who previously 
used scripting types of languages for these jobs.



Laeeth.
May 01 2015
parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
Current std.stdio is deprecated. This ad-hoc should works.

	auto json = File("fileName")
		.byChunk(1024 * 1024) //1 MB. Data cluster equals 1024 * 4
		.map!(ch => ch.idup)
		.joiner
		.map!(b => cast(char)b)
		.parseJSON;


On Friday, 1 May 2015 at 14:19:26 UTC, Laeeth Isharc wrote:

 On Wednesday, 29 April 2015 at 18:48:22 UTC, Laeeth Isharc 
 wrote:

 Hi.

 What's the best way to pass the contents of a file to the 
 stream parser without reading the whole thing into memory 
 first?  I get an error if using byLine because the kind of 
 range this function returns is not what the stream parser is 
 expecting.

 There is an optional filename argument to parseJSONStream, but 
 I am not sure what this is for, since it still requires the 
 first argument for the input if the filename is passed.


 Thanks.


 Laeeth.


 some more suggestions for enhancements here:
 https://github.com/s-ludwig/std_data_json/issues/12

 a big advantage of languages like python is that there are 
 end-to-end examples of using the language+libraries to 
 accomplish real work.  (they don't need to be more than a 
 screenful of code, but for the newcomer having it all set out 
 makes a big difference).  since I believe a significant reason 
 for people to switch to D will be a realisation that actually 
 CPU time isn't free, it makes sense to be accommodating to 
 people who previously used scripting types of languages for 
 these jobs.



 Laeeth.
May 01 2015
parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
This line can be removed:
                 .map!(ch => ch.idup)


On Friday, 1 May 2015 at 20:02:46 UTC, Ilya Yaroshenko wrote:

 Current std.stdio is deprecated. This ad-hoc should works.

 	auto json = File("fileName")
 		.byChunk(1024 * 1024) //1 MB. Data cluster equals 1024 * 4
 		.map!(ch => ch.idup)
 		.joiner
 		.map!(b => cast(char)b)
 		.parseJSON;
May 01 2015
parent reply "Laeeth Isharc" <laeeth nospamlaeeth.com> writes:
On Friday, 1 May 2015 at 20:04:58 UTC, Ilya Yaroshenko wrote:

 This line can be removed:
                 .map!(ch => ch.idup)


 On Friday, 1 May 2015 at 20:02:46 UTC, Ilya Yaroshenko wrote:

 Current std.stdio is deprecated. This ad-hoc should works.

 	auto json = File("fileName")
 		.byChunk(1024 * 1024) //1 MB. Data cluster equals 1024 * 4
 		.map!(ch => ch.idup)
 		.joiner
 		.map!(b => cast(char)b)
 		.parseJSON;




Thanks for this.  I am trying to call parseJSONStream, so I will 
see if that works.

(Sample code here:
https://github.com/s-ludwig/std_data_json/blob/master/source/stdx/data/json/parser.d
)
May 02 2015
parent reply "Laeeth Isharc" <laeeth nospamlaeeth.com> writes:
It doesn't like it.  Any thoughts ?

lexer.d(257): Error: safe function 
'stdx.data.json.parser.JSONLexerRange!(MapResult!(__lambda3, 
Result), cast(LexOptions)0, __lambda31).JSONLexerRange.empty' 
cannot call system function 
'app.lookupTickers.MapResult!(__lambda3, Result).MapResult.empty'

string[2][] lookupTickers(string dataSource,string[] searchItems)
{
	import stdx.data.json;
	import std.conv:to;
	import std.algorithm:canFind,countUntil,joiner,map;
	import std.string:toLower;
	bool found=false;
	bool checkedCode=false;
	bool checkedName=false;
	string[2][] ret;
	string buf;
	auto filename="../importquandl/"~dataSource~".json";
	//auto data=cast(string)std.file.read(filename);
	auto data = File("fileName")
		.byChunk(100*1024 * 1024) //1 MB. Data cluster equals 1024 * 4
		// .map!(ch => ch.idup)
		.joiner
		.map!(b => cast(char)b);
	auto range1=parseJSONStream(data);
}
May 02 2015
parent "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
You can use std.json or create TrustedInputRangeShell template 
with  trasted methods:


struct TrustedInputRangeShell(Range)
{
     Range* data;

     auto front()  property  trusted { return (*data).front; }

     //etc
}

But I am not sure about other parseJSONStream bugs.
May 02 2015