digitalmars.D - Working on new binary serialization module for phobos (hopefully)
- Sean Campbell (5/5) Jun 01 2015 I've been working on a new serialization module for Phobos and
- Jacob Carlborg (20/25) Jun 01 2015 I had a quick look at the code. Based on that and how short the code is
- CraigDillabaugh (4/31) Jun 01 2015 I noticed there hasn't been any activity on the Github repo for 8
- Jacob Carlborg (5/8) Jun 01 2015 I just haven't worked on it for a while. Working on the , at least,
- lobo (8/16) Jun 01 2015 +1 for orange.
- Atila Neves (5/21) Jun 02 2015 I only set out to write a binary backend, it didn't occur to me
- Jonas Drewsen (4/12) Jun 03 2015 Couldn't be put into std.experimental. That would probably be
- Sean Campbell (6/14) Jun 05 2015 I hope you can get the third review process done because Phobos
- Atila Neves (4/8) Jun 05 2015 Does phobos really _need_ it? It'd be nice, sure, but given
- Atila Neves (15/20) Jun 01 2015 I'm biased since I wrote this:
- weaselcat (4/13) Jun 01 2015 I've used cerealed a bunch, I would be in favor of it being the
- Sean Campbell (9/30) Jun 01 2015 I changed this earlier but forgot to push changes from my laptop
- Atila Neves (4/15) Jun 05 2015 Yes. And sometimes mixed endianess depending on the architecture.
- Sean Campbell (6/21) Jun 05 2015 if you mean bit numbering then it dosen't matter because its
- Basile Burg (18/23) Jun 02 2015 We are many on this segment. Your one is "under-featured". One
- Jacob Carlborg (5/17) Jun 02 2015 Orange supports custom serialization [1].
- Atila Neves (5/26) Jun 02 2015 So does cerealed, in one of two ways, postBlit being the _much_
- Jacob Carlborg (6/7) Jun 03 2015 I forgot to mention that Orange also supports pre and post
- Atila Neves (4/11) Jun 03 2015 From the unit tests, that doesn't seem to do the same thing, but
- Jacob Carlborg (5/7) Jun 03 2015 No, it does not. But it can still be useful. Like if you want to do any
- Atila Neves (9/15) Jun 03 2015 Right. In my case, which is totally and completely motivated by
- Sean Campbell (15/39) Jun 05 2015 using property setters and getters doesn't seem like the best idea
- Basile Burg (8/28) Jun 05 2015 No, you don't get my point with setters: if a during the
- Sean Campbell (3/34) Jun 05 2015 just add @customSerialized to _width and make deserialize_width
- Basile Burg (2/15) Jun 05 2015 thank you, it's good to know.
- Jacob Carlborg (8/14) Jun 06 2015 If the whole code is written in D then that's not necessary. The
I've been working on a new serialization module for Phobos and its only reliant on 4 Phobos modules it is available at https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d I would like some feedback on it
Jun 01 2015
On 2015-06-01 14:52, Sean Campbell wrote:I've been working on a new serialization module for Phobos and its only reliant on 4 Phobos modules it is available at https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d I would like some feedback on itI had a quick look at the code. Based on that and how short the code is I'm guessing it lacks some features. I've been working on a serialization package for Phobos for a long time. Basically moving Orange [1] in to Phobos. My serialization library supports: * Fully automatic serialization * Serializing non-public fields * Serializing without registering types (except when serialized through a base class reference) * Serializing third party types * Custom serialization, both intrusive and no-intrusive * Tracking of references, pointers and arrays to only serialize the data once * Properly restores arrays and slices * Separate front end (serializer) from back end (archiver/format) * Deserializing of classes without default constructor [1] https://github.com/jacob-carlborg/orange -- /Jacob Carlborg
Jun 01 2015
On Monday, 1 June 2015 at 14:45:08 UTC, Jacob Carlborg wrote:On 2015-06-01 14:52, Sean Campbell wrote:I noticed there hasn't been any activity on the Github repo for 8 months. Why is that? Do you consider this a completely finished product, or are you held up by the PHobos review process?I've been working on a new serialization module for Phobos and its only reliant on 4 Phobos modules it is available at https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d I would like some feedback on itI had a quick look at the code. Based on that and how short the code is I'm guessing it lacks some features. I've been working on a serialization package for Phobos for a long time. Basically moving Orange [1] in to Phobos. My serialization library supports: * Fully automatic serialization * Serializing non-public fields * Serializing without registering types (except when serialized through a base class reference) * Serializing third party types * Custom serialization, both intrusive and no-intrusive * Tracking of references, pointers and arrays to only serialize the data once * Properly restores arrays and slices * Separate front end (serializer) from back end (archiver/format) * Deserializing of classes without default constructor [1] https://github.com/jacob-carlborg/orange
Jun 01 2015
On 2015-06-01 21:22, CraigDillabaugh wrote:I noticed there hasn't been any activity on the Github repo for 8 months. Why is that? Do you consider this a completely finished product, or are you held up by the PHobos review process?I just haven't worked on it for a while. Working on the , at least, third review process isn't really motivating. -- /Jacob Carlborg
Jun 01 2015
On Monday, 1 June 2015 at 19:41:58 UTC, Jacob Carlborg wrote:On 2015-06-01 21:22, CraigDillabaugh wrote:+1 for orange. I have used cerealed as well and liked it but with it except orange it's easy to add and switch between custom archiver formats. I couldn't figure out a way to do this with cereraled without hacking it. bye, loboI noticed there hasn't been any activity on the Github repo for 8 months. Why is that? Do you consider this a completely finished product, or are you held up by the PHobos review process?I just haven't worked on it for a while. Working on the , at least, third review process isn't really motivating.
Jun 01 2015
On Tuesday, 2 June 2015 at 05:07:29 UTC, lobo wrote:On Monday, 1 June 2015 at 19:41:58 UTC, Jacob Carlborg wrote:I only set out to write a binary backend, it didn't occur to me to support anything else. I've considered it but nobody seemed to ask for it, so... AtilaOn 2015-06-01 21:22, CraigDillabaugh wrote:+1 for orange. I have used cerealed as well and liked it but with it except orange it's easy to add and switch between custom archiver formats. I couldn't figure out a way to do this with cereraled without hacking it.I noticed there hasn't been any activity on the Github repo for 8 months. Why is that? Do you consider this a completely finished product, or are you held up by the PHobos review process?I just haven't worked on it for a while. Working on the , at least, third review process isn't really motivating.
Jun 02 2015
On Monday, 1 June 2015 at 19:41:58 UTC, Jacob Carlborg wrote:On 2015-06-01 21:22, CraigDillabaugh wrote:Couldn't be put into std.experimental. That would probably be both motivating and increase testing of the API. /JonasI noticed there hasn't been any activity on the Github repo for 8 months. Why is that? Do you consider this a completely finished product, or are you held up by the PHobos review process?I just haven't worked on it for a while. Working on the , at least, third review process isn't really motivating.
Jun 03 2015
On Monday, 1 June 2015 at 19:41:58 UTC, Jacob Carlborg wrote:On 2015-06-01 21:22, CraigDillabaugh wrote:I hope you can get the third review process done because Phobos needs serialization and it would be much better to use tried and tested code like orange rather than something I've pulled together over a couple of weeks sorry for the double-postI noticed there hasn't been any activity on the Github repo for 8 months. Why is that? Do you consider this a completely finished product, or are you held up by the PHobos review process?I just haven't worked on it for a while. Working on the , at least, third review process isn't really motivating.
Jun 05 2015
I hope you can get the third review process done because Phobos needs serialization and it would be much better to use tried and tested code like orange rather than something I've pulled together over a couple of weeksDoes phobos really _need_ it? It'd be nice, sure, but given there's more than one option available on dub right now I don't think it's a massive deal. Atila
Jun 05 2015
On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:I've been working on a new serialization module for Phobos and its only reliant on 4 Phobos modules it is available at https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d I would like some feedback on itI'm biased since I wrote this: https://github.com/atilaneves/cerealed. At a glance, I don't like at all that types have to opt-in to be serialised. Why the limitation? You don't need reverseOf, just use std.range.retro. Cerealed has more features than this as well. I'd struggle to write code as short as I did when using it to implement networking protocols. I also only encoded bytes as big-endian since binary serialization is usually followed by sending those bytes over the wire. Given you check the endianess of the system here, how would that work? I liked the union trick, I wonder why I didn't think of that. Well, there's the endianness problem I guess. Atila
Jun 01 2015
On Monday, 1 June 2015 at 15:11:54 UTC, Atila Neves wrote:On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:I've used cerealed a bunch, I would be in favor of it being the basis of a phobos library. It's very easy to use. But I have not used Orange, so I can't comment on how it compares.I've been working on a new serialization module for Phobos and its only reliant on 4 Phobos modules it is available at https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d I would like some feedback on itI'm biased since I wrote this: https://github.com/atilaneves/cerealed.
Jun 01 2015
On Monday, 1 June 2015 at 15:11:54 UTC, Atila Neves wrote:On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:I changed this earlier but forgot to push changes from my laptop if you check it now it will be fixedI've been working on a new serialization module for Phobos and its only reliant on 4 Phobos modules it is available at https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d I would like some feedback on itI'm biased since I wrote this: https://github.com/atilaneves/cerealed. At a glance, I don't like at all that types have to opt-in to be serialised. Why the limitation?You don't need reverseOf, just use std.range.retro.thank you, I have a bad habit of writing already existing algorithmsCerealed has more features than this as well. I'd struggle to write code as short as I did when using it to implement networking protocols. I also only encoded bytes as big-endian since binary serialization is usually followed by sending those bytes over the wire. Given you check the endianess of the system here, how would that work?I don't know what you mean bytes by themselves don't have an endianness, do they? I thought is was only anything that was larger than a byte had endiannessI liked the union trick, I wonder why I didn't think of that. Well, there's the endianness problem I guess. Atila
Jun 01 2015
Yes. And sometimes mixed endianess depending on the architecture. I might be wrong, but it seems to me that it'd be hard to send the bytes over the wire with this. AtilaCerealed has more features than this as well. I'd struggle to write code as short as I did when using it to implement networking protocols. I also only encoded bytes as big-endian since binary serialization is usually followed by sending those bytes over the wire. Given you check the endianess of the system here, how would that work?I don't know what you mean bytes by themselves don't have an endianness, do they? I thought is was only anything that was larger than a byte had endianness
Jun 05 2015
On Friday, 5 June 2015 at 14:55:24 UTC, Atila Neves wrote:if you mean bit numbering then it dosen't matter because its managed by the hardware anyway but just to be sure i'll do some testsYes. And sometimes mixed endianess depending on the architecture. I might be wrong, but it seems to me that it'd be hard to send the bytes over the wire with this. AtilaCerealed has more features than this as well. I'd struggle to write code as short as I did when using it to implement networking protocols. I also only encoded bytes as big-endian since binary serialization is usually followed by sending those bytes over the wire. Given you check the endianess of the system here, how would that work?I don't know what you mean bytes by themselves don't have an endianness, do they? I thought is was only anything that was larger than a byte had endianness
Jun 05 2015
On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:I've been working on a new serialization module for Phobos and its only reliant on 4 Phobos modules it is available at https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d I would like some feedback on itWe are many on this segment. Your one is "under-featured". One thing i need in serialization is the use of prop getter to serialize and prop setter to deserialize (*) and not only a dump of the fields that __traits() can find. That's why i also done mine: https://github.com/BBasile/iz/blob/master/import/iz/serializer.d based on the concept of property descriptor, UDA anotations: https://github.com/BBasile/iz/blob/master/import/iz/properties.d However still not happy with it... Maybe with every other examples that people gave you, you'll get a better idea of how to design the thing. One plus is that your code is very clean. --- (*): an obvious reason is that when you change a piece in an engine you don't just drop it under the hood and it's done. You have to demount and remount a lot of pieces...that's what the setters do silently.
Jun 02 2015
On 2015-06-02 12:32, Basile Burg wrote:On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:Orange supports custom serialization [1]. [1] https://github.com/jacob-carlborg/orange/blob/master/tests/Custom.d -- /Jacob CarlborgI've been working on a new serialization module for Phobos and its only reliant on 4 Phobos modules it is available at https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d I would like some feedback on itWe are many on this segment. Your one is "under-featured". One thing i need in serialization is the use of prop getter to serialize and prop setter to deserialize (*) and not only a dump of the fields that __traits() can find.
Jun 02 2015
On Tuesday, 2 June 2015 at 19:49:39 UTC, Jacob Carlborg wrote:On 2015-06-02 12:32, Basile Burg wrote:So does cerealed, in one of two ways, postBlit being the _much_ better one: https://github.com/atilaneves/cerealed/blob/master/README.md AtilaOn Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:Orange supports custom serialization [1]. [1] https://github.com/jacob-carlborg/orange/blob/master/tests/Custom.dI've been working on a new serialization module for Phobos and its only reliant on 4 Phobos modules it is available at https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d I would like some feedback on itWe are many on this segment. Your one is "under-featured". One thing i need in serialization is the use of prop getter to serialize and prop setter to deserialize (*) and not only a dump of the fields that __traits() can find.
Jun 02 2015
On 2015-06-02 22:02, Atila Neves wrote:So does cerealed, in one of two ways, postBlit being the _much_ better one:I forgot to mention that Orange also supports pre and post actions/callbacks [1]. [1] https://github.com/jacob-carlborg/orange/blob/master/tests/Events.d -- /Jacob Carlborg
Jun 03 2015
On Wednesday, 3 June 2015 at 07:00:23 UTC, Jacob Carlborg wrote:On 2015-06-02 22:02, Atila Neves wrote:From the unit tests, that doesn't seem to do the same thing, but I could be wrong. AtilaSo does cerealed, in one of two ways, postBlit being the _much_ better one:I forgot to mention that Orange also supports pre and post actions/callbacks [1]. [1] https://github.com/jacob-carlborg/orange/blob/master/tests/Events.d
Jun 03 2015
On 2015-06-03 09:26, Atila Neves wrote:From the unit tests, that doesn't seem to do the same thing, but I could be wrong.No, it does not. But it can still be useful. Like if you want to do any post processing after deserialization. -- /Jacob Carlborg
Jun 03 2015
On Wednesday, 3 June 2015 at 12:15:43 UTC, Jacob Carlborg wrote:On 2015-06-03 09:26, Atila Neves wrote:Right. In my case, which is totally and completely motivated by writing networking protocols, what I really want is to assemble packets with the least amount of code possible. I'm already getting annoyed at having to write the amount of code that's necessary now; I'm currently working on improving cerealed to be smart enough to know which bytes in the packet correspond to length and do even more automatically. AtilaFrom the unit tests, that doesn't seem to do the same thing, but I could be wrong.No, it does not. But it can still be useful. Like if you want to do any post processing after deserialization.
Jun 03 2015
On Tuesday, 2 June 2015 at 10:32:25 UTC, Basile Burg wrote:On Monday, 1 June 2015 at 12:52:45 UTC, Sean Campbell wrote:using property setters and getters doesn't seem like the best idea as they may either: they are used to get live data e.g. setter calls hashing function for an object that isn't part of the class/struct property may only have setter or getter not both. but in case you need to store anything that __traits() can find I added support for custom serialized fields and types. just annotate a type with customSerialized and define the static methods serialize[membername] and deserialize[membername] where [membername] is the name of the field or have full custom serialization by defining the static methods serializeThis and deserializeThis.I've been working on a new serialization module for Phobos and its only reliant on 4 Phobos modules it is available at https://github.com/sycam0inc/phobos/blob/master/std/experimental/serialization.d I would like some feedback on itWe are many on this segment. Your one is "under-featured". One thing i need in serialization is the use of prop getter to serialize and prop setter to deserialize (*) and not only a dump of the fields that __traits() can find.That's why i also done mine: https://github.com/BBasile/iz/blob/master/import/iz/serializer.d based on the concept of property descriptor, UDA anotations: https://github.com/BBasile/iz/blob/master/import/iz/properties.d However still not happy with it... Maybe with every other examples that people gave you, you'll get a better idea of how to design the thing. One plus is that your code is very clean.thank you, it's good to know--- (*): an obvious reason is that when you change a piece in an engine you don't just drop it under the hood and it's done. You have to demount and remount a lot of pieces...that's what the setters do silently.
Jun 05 2015
On Friday, 5 June 2015 at 12:21:19 UTC, Sean Campbell wrote:On Tuesday, 2 June 2015 at 10:32:25 UTC, Basile Burg wrote:No, you don't get my point with setters: if a during the deserialization you restore, let's say, the _width field and that 12 children controls rely on this field then they won't be aware of the change. But if the deserializer restores using the width(int value) setter, the children can be resized if the the setter contain a sub method like updateChildren()...[...]using property setters and getters doesn't seem like the best idea as they may either: they are used to get live data e.g. setter calls hashing function for an object that isn't part of the class/struct property may only have setter or getter not both. but in case you need to store anything that __traits() can find I added support for custom serialized fields and types. just annotate a type with customSerialized and define the static methods serialize[membername] and deserialize[membername] where [membername] is the name of the field or have full custom serialization by defining the static methods serializeThis and deserializeThis.[...]thank you, it's good to know[...]
Jun 05 2015
On Friday, 5 June 2015 at 12:53:45 UTC, Basile Burg wrote:On Friday, 5 June 2015 at 12:21:19 UTC, Sean Campbell wrote:just add customSerialized to _width and make deserialize_width call updateChildren() or whateverOn Tuesday, 2 June 2015 at 10:32:25 UTC, Basile Burg wrote:No, you don't get my point with setters: if a during the deserialization you restore, let's say, the _width field and that 12 children controls rely on this field then they won't be aware of the change. But if the deserializer restores using the width(int value) setter, the children can be resized if the the setter contain a sub method like updateChildren()...[...]using property setters and getters doesn't seem like the best idea as they may either: they are used to get live data e.g. setter calls hashing function for an object that isn't part of the class/struct property may only have setter or getter not both. but in case you need to store anything that __traits() can find I added support for custom serialized fields and types. just annotate a type with customSerialized and define the static methods serialize[membername] and deserialize[membername] where [membername] is the name of the field or have full custom serialization by defining the static methods serializeThis and deserializeThis.[...]thank you, it's good to know[...]
Jun 05 2015
On Friday, 5 June 2015 at 13:05:55 UTC, Sean Campbell wrote:On Friday, 5 June 2015 at 12:53:45 UTC, Basile Burg wrote:thank you, it's good to know.On Friday, 5 June 2015 at 12:21:19 UTC, Sean Campbell wrote:just add customSerialized to _width and make deserialize_width call updateChildren() or whateverNo, you don't get my point with setters: if a during the deserialization you restore, let's say, the _width field and that 12 children controls rely on this field then they won't be aware of the change. But if the deserializer restores using the width(int value) setter, the children can be resized if the the setter contain a sub method like updateChildren()...[...]
Jun 05 2015
On 2015-06-05 14:53, Basile Burg wrote:No, you don't get my point with setters: if a during the deserialization you restore, let's say, the _width field and that 12 children controls rely on this field then they won't be aware of the change. But if the deserializer restores using the width(int value) setter, the children can be resized if the the setter contain a sub method like updateChildren()...If the whole code is written in D then that's not necessary. The children will be restored properly as well. But if you have a D class that wraps a native GUI control with a setter that calls some method on the native control then it would be necessary to call the setter when deserializing. -- /Jacob Carlborg
Jun 06 2015