www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Cerealed v0.2.0 - a(nother) D serialisation library

reply "Atila Neves" <atila.neves gmail.com> writes:
http://code.dlang.org/packages/cerealed
https://github.com/atilaneves/cerealed

But what about Orange / std.serialization? Well,
1. I started this before I knew about Orange. This is actually 
based on a library I wrote in C++ 
(https://bitbucket.org/atilaneves/cereal) for the game I'm trying 
to finish, and it dawned on me it would probably be easier in D. 
It was.
2. Orange serialises to XML, I need/want binary. I could've 
written a binary archive, I guess, but I'd rather do this.
3. I would've written this for fun anyway.

The deal here is binary serialisation with minimal to no 
boilerplate. As mentioned in the README, I used this to implement 
a MQTT broker with minimal fuss. Serialising classes, structs, 
and normal D types requires writing nothing at all. Custom 
serialisation can be handled in two different ways and both of 
them require writing only one function for both struct/class -> 
bytes and vice-versa. Since a code sample is worth a 1000 words:

     import cerealed; //uses package.d from 2.064

     struct MyStruct {
         ubyte mybyte1;
          NoCereal uint nocereal1; //won't be serialised
         //the next 3 members will all take up one byte
          Bits!4 ubyte nibble; //gets packed into 4 bits
          Bits!1 ubyte bit; //gets packed into 1 bit
          Bits!3 ubyte bits3; //gets packed into 3 bits
         ubyte mybyte2;
     }

     auto enc = new Cerealiser();
     enc ~= MyStruct(3, 123, 14, 1, 2, 42);
     assert(enc.bytes == [ 3, 0xea /*1110 1 010*/, 42]);

     auto dec = new Decerealizer([ 3, 0xea, 42]); //US spelling 
works too
     //the 2nd value is 0 and not 123 since that value
     //doesn't get serialised/deserialised
     assert(dec.value!MyStruct == MyStruct(3, 0, 14, 1, 2, 42));

Custom serialisation is explained in the README. I had to use it 
for my MQTT project and it worked quite well.

I still need to add a bunch of negative tests and accompanying 
code but this is good enough to fool around with for the moment.

Atila
Nov 18 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-18 17:49, Atila Neves wrote:

 http://code.dlang.org/packages/cerealed
 https://github.com/atilaneves/cerealed


How does it handle:

* Slices - are they restored?
* Pointers - are they restored?
* Objects - is the same object only serialized once or duplicated?
* Serializing through base class reference - possible?

-- 
/Jacob Carlborg
Nov 18 2013
next sibling parent reply Rory McGuire <rjmcguire gmail.com> writes:
On 18 Nov 2013 22:45, "Jacob Carlborg" <doob me.com> wrote:

 How does it handle:

 * Slices - are they restored?
 * Pointers - are they restored?
 * Objects - is the same object only serialized once or duplicated?
 * Serializing through base class reference - possible?

 --
 /Jacob Carlborg


Hi Jacob,

Is there an example somewhere of how to use Orange to serialise data that
will not be going to a file?
When I looked at it appeared as though Orange's archives are designed for
file serialising only.

-Rory
Nov 18 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-11-19 06:49, Rory McGuire wrote:


 Hi Jacob,

 Is there an example somewhere of how to use Orange to serialise data
 that will not be going to a file?
 When I looked at it appeared as though Orange's archives are designed
 for file serialising only.


Orange doesn't have any data transport layer or similar. But you can do 
whatever you want with the data you get from the archiver.

Check the usage example in the readme[1]. Use "archive.data" to get the 
typed data after serialization and do whatever you want with it.

https://github.com/jacob-carlborg/orange#simple-usage-example

-- 
/Jacob Carlborg
Nov 18 2013
parent reply Rory McGuire <rjmcguire gmail.com> writes:
On Tue, Nov 19, 2013 at 9:18 AM, Jacob Carlborg <doob me.com> wrote:


 On 2013-11-19 06:49, Rory McGuire wrote:
 https://github.com/jacob-carlborg/orange#simple-usage-example

 --
 /Jacob Carlborg


Thanks. I believe that is the example that made me think that its main
purpose was for archiving.

So is an Archive actually a serialization driver? Can one make a protobuf
driver or a hessian driver?
Are the serializers able to stream data, with something like a range
interface?
Nov 19 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-11-19 09:19, Rory McGuire wrote:


 Thanks. I believe that is the example that made me think that its main
 purpose was for archiving.

 So is an Archive actually a serialization driver? Can one make a
 protobuf driver or a hessian driver?


Yes, that should be possible. It depends on how the data looks like. The 
upcoming std.serialization version will relax quite a lot of the 
requirements of the archiver. That means more archivers can be implemented.


 Are the serializers able to stream data, with something like a range
 interface?


No, unfortunately not. I'm working on including Orange into Phobos as 
std.serialization. That version will have a range API.

-- 
/Jacob Carlborg
Nov 19 2013
prev sibling parent reply "Atila Neves" <atila.neves gmail.com> writes:
On Monday, 18 November 2013 at 20:43:51 UTC, Jacob Carlborg wrote:

 On 2013-11-18 17:49, Atila Neves wrote:

 http://code.dlang.org/packages/cerealed
 https://github.com/atilaneves/cerealed


 How does it handle:

 * Slices - are they restored?


It serialises the slice and restores it as it was, not the data 
behind it. I should add some tests for that.


 * Pointers - are they restored?


No, good point. I keep forgetting there are legitimate uses of 
pointers in D. I was looking at the SList implementation the 
other day and was surprised to see "new" used to create a struct 
on the heap, something that would seem totally normal to me in 
C++.

It should be easy enough to extend to pointers, I'll add that in 
the next version.


 * Objects - is the same object only serialized once or 
 duplicated?


I'm not sure I understand the question.


 * Serializing through base class reference - possible?


No. I can't see how that would be possible without implementing 
an interface (I may well be wrong), and that was the kind of 
thing I really didn't want to do. Not only that, but for the kind 
of code I write, I either want to send something to the network, 
in which case I know which class to instantiate and don't need a 
base class reference, or I want to unmarshall bytes that came in. 
In the latter case there's usually an identifier in the header to 
tell the factory method which class to instantiante. So that 
doesn't usually come up.

Atila
Nov 19 2013
next sibling parent reply Rory McGuire <rjmcguire gmail.com> writes:
On Tue, Nov 19, 2013 at 3:24 PM, Atila Neves <atila.neves gmail.com> wrote:


  * Objects - is the same object only serialized once or duplicated?


 I'm not sure I understand the question.


...

Atila

 Perhaps Jacob refers to a lot of serialization formats being capable of


storing references to an object, this makes it possible to store circular
lists for example.



 Nov 19 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-19 14:30, Rory McGuire wrote:


 Perhaps Jacob refers to a lot of serialization formats being capable of
 storing references to an object, this makes it possible to store
 circular lists for example.


Yes, exactly.

-- 
/Jacob Carlborg



 Nov 19 2013



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-19 14:24, Atila Neves wrote:


 It serialises the slice and restores it as it was, not the data behind
 it. I should add some tests for that.


class Foo
{
     int[] a;
     int[] b;
}

auto foo = new Foo;
foo.a = [1, 2, 3, 4];
foo.b = foo.a[1 .. 3];

auto cereal = new Cerealiser();
cereal ~= foo;

auto decerealiser = new Decerealiser(cereal.bytes);
auto foo2 = decerealiser.value!(Foo);
foo2.b[0] = 10;
assert(foo2.a[1] == 10); // will this pass?


 I'm not sure I understand the question.


auto cereal = new Cerealiser();
cereal ~= foo;
cereal ~= foo; // will this be serialized a second time or will a 
reference to it be serialized?


 No. I can't see how that would be possible without implementing an
 interface (I may well be wrong), and that was the kind of thing I really
 didn't want to do. Not only that, but for the kind of code I write, I
 either want to send something to the network, in which case I know which
 class to instantiate and don't need a base class reference, or I want to
 unmarshall bytes that came in. In the latter case there's usually an
 identifier in the header to tell the factory method which class to
 instantiante. So that doesn't usually come up.


Even if you know the identifier in the header file you need a static 
type when deserializing:

decerealiser.value!(Foo)

Even if you can instantiate the right subclass you need to have the 
static type information to deserialize it.

It is possible to do, as I have done in Orange. The requirement is to 
register the subclasses that need to be serialized through a base class 
reference. See 
https://github.com/jacob-carlborg/orange/blob/master/orange/serialization/Ser
alizer.d#L241..L262 
and 
https://github.com/jacob-carlborg/orange/blob/master/orange/serialization/Serializer.d#L787..L788

-- 
/Jacob Carlborg



 Nov 19 2013



  parent reply "Atila Neves" <atila.neves gmail.com>  writes:


 assert(foo2.a[1] == 10); // will this pass?


Ah, I see. No, it won't. Which answers the other question about 
serialising references to objects. I'm told this is a common 
problem, but this being my first bash at a serialisation 
library...

I was certain that Orange would be more mature and handle more 
corner cases than this library (or else it wouldn't be considered 
for inclusion in the std lib), but I wanted to share it since it 
works for what I'm doing, does things a bit differently and does 
what I need. Also, it was fun coding it.

If anything, if you think some of these ideas are worth it, feel 
free to use them. I'm just glad to not have to write the same old 
bit twiddling code for networking headers again!


 Even if you know the identifier in the header file you need a 
 static type when deserializing:

 decerealiser.value!(Foo)

 Even if you can instantiate the right subclass you need to have 
 the static type information to deserialize it.

 It is possible to do, as I have done in Orange. The requirement 
 is to register the subclasses that need to be serialized 
 through a base class reference. See 
 https://github.com/jacob-carlborg/orange/blob/master/orange/serialization/Ser
alizer.d#L241..L262 
 and 
 https://github.com/jacob-carlborg/orange/blob/master/orange/serialization/Serializer.d#L787..L788


True, I thought of doing something like it to simplify my MQTT 
broker implementation but ended up deciding against it.



 Nov 20 2013



 next sibling parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-20 18:26, Atila Neves wrote:


 Ah, I see. No, it won't. Which answers the other question about
 serialising references to objects. I'm told this is a common problem,
 but this being my first bash at a serialisation library...


It's a lot easier to add support for references than slices. The reason 
is that you always know when there is a reference type. With arrays you 
don't know if it's a slice of some array or an actual array. The type 
system treats both slices and arrays the same.

-- 
/Jacob Carlborg



 Nov 20 2013



prev sibling  parent reply "Vicente" <dontwantspam example.com>  writes:


Hello Atila,
I've seen that cerealed has made much progress, it's now at v0.5.0
This post is for a feature request, I would like to use it with a 
struct like this:

import cerealed;

struct nested_associative_array { nested_associative_array[int] 
x; }

struct some_struct {
	string[] x;
	int[][] y;
	nested_associative_array[] z;
};

void main(char[][] args) {
	some_struct original, restored;

	auto enc = Cerealiser();
	enc ~= original;
	auto dec = Decerealiser(enc.bytes);
	restored = dec.value!(some_struct);

	assert(original==restored);
}

The serialization process compiles fine, but the restoring part 
fails to compile.
I've tried to figure out how to do it, but I am still a beginner 
on D.
How difficult do you think it is?
The alternative, orange, works fine, but the struct I'm trying to 
export to disk is a few hundreds of MB in size and orange takes 
hours to complete and may also run out of memory. Your 
implementation, instead, is really fast and efficient!



 Sep 08 2014



 next sibling parent  "Atila Neves" <atila.neves gmail.com>  writes:


I'll take a look. It's easier to post this on github though.

Atila

On Monday, 8 September 2014 at 19:09:30 UTC, Vicente wrote:

 Hello Atila,
 I've seen that cerealed has made much progress, it's now at 
 v0.5.0
 This post is for a feature request, I would like to use it with 
 a struct like this:

 import cerealed;

 struct nested_associative_array { nested_associative_array[int] 
 x; }

 struct some_struct {
 	string[] x;
 	int[][] y;
 	nested_associative_array[] z;
 };

 void main(char[][] args) {
 	some_struct original, restored;

 	auto enc = Cerealiser();
 	enc ~= original;
 	auto dec = Decerealiser(enc.bytes);
 	restored = dec.value!(some_struct);

 	assert(original==restored);
 }

 The serialization process compiles fine, but the restoring part 
 fails to compile.
 I've tried to figure out how to do it, but I am still a 
 beginner on D.
 How difficult do you think it is?
 The alternative, orange, works fine, but the struct I'm trying 
 to export to disk is a few hundreds of MB in size and orange 
 takes hours to complete and may also run out of memory. Your 
 implementation, instead, is really fast and efficient!





 Sep 09 2014



prev sibling  parent  "Atila Neves" <atila.neves gmail.com>  writes:


Done.

Thanks for the feature request and using Cerealed! I hope it's 
useful to you.

Atila

On Monday, 8 September 2014 at 19:09:30 UTC, Vicente wrote:

 Hello Atila,
 I've seen that cerealed has made much progress, it's now at 
 v0.5.0
 This post is for a feature request, I would like to use it with 
 a struct like this:

 import cerealed;

 struct nested_associative_array { nested_associative_array[int] 
 x; }

 struct some_struct {
 	string[] x;
 	int[][] y;
 	nested_associative_array[] z;
 };

 void main(char[][] args) {
 	some_struct original, restored;

 	auto enc = Cerealiser();
 	enc ~= original;
 	auto dec = Decerealiser(enc.bytes);
 	restored = dec.value!(some_struct);

 	assert(original==restored);
 }

 The serialization process compiles fine, but the restoring part 
 fails to compile.
 I've tried to figure out how to do it, but I am still a 
 beginner on D.
 How difficult do you think it is?
 The alternative, orange, works fine, but the struct I'm trying 
 to export to disk is a few hundreds of MB in size and orange 
 takes hours to complete and may also run out of memory. Your 
 implementation, instead, is really fast and efficient!





 Sep 11 2014