www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - D styled data format, Json failed

reply "Saaa" <empty needmail.com> writes:
I looked at the JSON format and it seems very inefficient at loading arrays 
as it isn't limited to one type per array.
This is nice when you want to save a small array with different typed 
elements but for my purposes this is kind of a performance problem.

This is why I will try and get suggestions again about the D-styled format I 
tried to suggest a few threads ago :)

Let me suggest a simple example:
(Please tell me when something isn't obvious :)

---file.dat
//comment
int number = 10;

float [3][2]  simpleArray = [
[0.1, 0.2, 0.3],
[0.4, 0.5, 0.6]];
--

---main.d
static import ddata;
void main()
{
char[][] file = read (`file.dat`);
int i;
char c;

ddata.get(file,'number',i);
ddata.get(file,'number',c); //type fail, thus returns -1 or throws an 
exception

float [3][2] a;

ddata.get(file,'simpleArray',a);
ddata.write(file,'simpleArrayCopy', a);

a[0][0] =3.0;

ddata.write(file,'simpleArray', a);

write('file.dat', file);
}
--

resulting data file:

---file.dat
//comment
int number = 10;

float [3][2]  simpleArray = [
[3.0, 0.2, 0.3],
[0.4, 0.5, 0.6]];

float [3][2]  simpleArrayCopy = [
[0.1, 0.2, 0.3],
[0.4, 0.5, 0.6]];
--
May 03 2009
next sibling parent "nobody" <somebody somewhere.com> writes:
"Saaa" <empty needmail.com> wrote in message 
news:gtlrs3$1b9s$1 digitalmars.com...

 I looked at the JSON format and it seems very inefficient at loading 
 arrays as it isn't limited to one type per array.
 This is nice when you want to save a small array with different typed 
 elements but for my purposes this is kind of a performance problem.

 This is why I will try and get suggestions again about the D-styled format 
 I tried to suggest a few threads ago :)

 Let me suggest a simple example:
 (Please tell me when something isn't obvious :)

 ---file.dat
 //comment
 int number = 10;

 float [3][2]  simpleArray = [
 [0.1, 0.2, 0.3],
 [0.4, 0.5, 0.6]];
 --

 ---main.d
 static import ddata;
 void main()
 {
 char[][] file = read (`file.dat`);
 int i;
 char c;

 ddata.get(file,'number',i);
 ddata.get(file,'number',c); //type fail, thus returns -1 or throws an 
 exception

 float [3][2] a;

 ddata.get(file,'simpleArray',a);
 ddata.write(file,'simpleArrayCopy', a);

 a[0][0] =3.0;

 ddata.write(file,'simpleArray', a);

 write('file.dat', file);
 }
 --

 resulting data file:

 ---file.dat
 //comment
 int number = 10;

 float [3][2]  simpleArray = [
 [3.0, 0.2, 0.3],
 [0.4, 0.5, 0.6]];

 float [3][2]  simpleArrayCopy = [
 [0.1, 0.2, 0.3],
 [0.4, 0.5, 0.6]];
 -- 


I've run into similar problems when I wanted to save lots of stuff to a 
file, and be able to load them again.
Huge multi-dimensional arrays seem to be an issue with JSON and similar..

Such a file writer as you propose would be very handy to have.
May 04 2009
prev sibling next sibling parent reply Lutger <lutger.blijdestijn gmail.com> writes:
Saaa wrote:


 
 I looked at the JSON format and it seems very inefficient at loading 


arrays 

 as it isn't limited to one type per array.
 This is nice when you want to save a small array with different typed 
 elements but for my purposes this is kind of a performance problem.
 
 This is why I will try and get suggestions again about the D-styled format 


I 

 tried to suggest a few threads ago :)
 
 Let me suggest a simple example:
 (Please tell me when something isn't obvious :)
 


I guess you will have to write this one yourself, it will be to D what JSON 
is to javascript ;)

I wonder how much of an performance improvement you will get though when 
loading data at runtime. You still have to parse and check everything since 
it remains a text based format. 

I know of one other solution but that's again not available for D: google 
protobuf: http://code.google.com/p/protobuf/
This uses a text-based format for describing the structure of the data, but 
the actual data can be in an optimized binary format.
May 04 2009
parent "Saaa" <empty needmail.com> writes:
 I guess you will have to write this one yourself, it will be to D what 
 JSON
 is to javascript ;)


:-)


 I wonder how much of an performance improvement you will get though when
 loading data at runtime.


As I couldn't even figure out how to save an multidimensional array in JSON
we may never know :D


 You still have to parse and check everything since
 it remains a text based format.


I will be focussed on (safe) speed and not flexibility because as you 
mention
otherwise it might be useless for the large arrays I wish to get/write



 I know of one other solution but that's again not available for D: google
 protobuf: http://code.google.com/p/protobuf/
 This uses a text-based format for describing the structure of the data, 
 but
 the actual data can be in an optimized binary format.


For me the data should be readable.
You may write the binary extension if you like :)
May 05 2009
prev sibling parent reply "Saaa" <empty needmail.com> writes:
My first stab at the get function.
As you might see, I need help :D
Thanks!

How do I make the function take a variadic argument and get its type?

//I guess the void pointer isn't the correct way, but I don't know any other 
way
void get(in char[][] file, in char[] indentifier, void* var)
{

TypeInfo type = typeof(var);
int row;

// Is it possible to check it is an array, no matter the depth?
if ( !( type ==  typeid(int) || type == typeid(int[][])  ) ) // || etc
{
  throw new exception;
  //which kind of exception should I throw?
}

if (file.length == 0 || file.length >int.max)
{
  throw new exception;
  //ditto
}

//gets the row number of the first correct occurence for which the following 
is true
//( file[row][0..len] ==  type.stringof ~ ' ' ~ identifier )
row = getRow(file, type, identifier);
if (row < 0 || row >=line.length)
{
  throw new exception;
  //same question :)
}
//parses the file from row
//what is the prototype of a function which has a variadic return type?
var = parse(file, row, type, identifier);
//can all types hold null?
if (var == null)
{
  throw new exception;
  //again
}

return;
}
May 05 2009
parent reply Christopher Wright <dhasenan gmail.com> writes:
Saaa wrote:

 My first stab at the get function.
 As you might see, I need help :D
 Thanks!
 
 How do I make the function take a variadic argument and get its type?


void get(in char[][] file, in char[] identifier, ...)
{
	TypeInfo type = _arguments[0];
	void* var = _argptr;
	// etc
}


 void get(in char[][] file, in char[] indentifier, void* var)
 {
 
 TypeInfo type = typeof(var);
 int row;
 
 // Is it possible to check it is an array, no matter the depth?


If you use tango:

import tango.core.RuntimeTraits;
if (isArray(type)) {}

Otherwise, just copy the code from RuntimeTraits. The functions you 
would need are: realType, isStaticArray, isDynamicArray, isArray


 if ( !( type ==  typeid(int) || type == typeid(int[][])  ) ) // || etc
 {
   throw new exception;
   //which kind of exception should I throw?


Define your own exception type.


 //what is the prototype of a function which has a variadic return type?


"Variadic" means that the function can take any number of arguments.

Use Variant, from std.variant or tango.core.Variant. It's typesafe. You 
could also return a void*, but that will require allocation.


 var = parse(file, row, type, identifier);
 //can all types hold null?


If you use Variant, you can check isEmpty / hasValue or some such.
May 06 2009
parent "Saaa" <empty needmail.com> writes:
Thanks!
With this I can go on for a bit :)

"Christopher Wright" <dhasenan gmail.com> wrote in message 
news:gtrpo7$2h7j$1 digitalmars.com...

 Saaa wrote:

 My first stab at the get function.
 As you might see, I need help :D
 Thanks!

 How do I make the function take a variadic argument and get its type?


 void get(in char[][] file, in char[] identifier, ...)
 {
 TypeInfo type = _arguments[0];
 void* var = _argptr;
 // etc
 }


 void get(in char[][] file, in char[] indentifier, void* var)
 {

 TypeInfo type = typeof(var);
 int row;

 // Is it possible to check it is an array, no matter the depth?


 If you use tango:

 import tango.core.RuntimeTraits;
 if (isArray(type)) {}

 Otherwise, just copy the code from RuntimeTraits. The functions you would 
 need are: realType, isStaticArray, isDynamicArray, isArray


 if ( !( type ==  typeid(int) || type == typeid(int[][])  ) ) // || etc
 {
   throw new exception;
   //which kind of exception should I throw?


 Define your own exception type.


 //what is the prototype of a function which has a variadic return type?


 "Variadic" means that the function can take any number of arguments.

 Use Variant, from std.variant or tango.core.Variant. It's typesafe. You 
 could also return a void*, but that will require allocation.


 var = parse(file, row, type, identifier);
 //can all types hold null?


 If you use Variant, you can check isEmpty / hasValue or some such.
May 06 2009