www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Orange serializer/deserializer

reply IntegratedDimensions <IntegratedDimensions gmail.com> writes:
How can I modify the pre serialization and post serialization 
values? I need to transform some variables that are stored but I 
would like to do this easily "inline"(would be cool to be able to 
provide a delegate to do the transformations at the site of 
definition of the fields).


Also, how does orange handle properties? Seems it just deals with 
fields and ignores all functions(does not use getter and setter 
of properties). This is valid, of course, just want to make sure. 
I still need to be able to transform values pre and post though.
Jun 01 2018
parent reply Jacob Carlborg <doob me.com> writes:
On 2018-06-02 03:30, IntegratedDimensions wrote:

 How can I modify the pre serialization and post serialization values? I 
 need to transform some variables that are stored but I would like to do 
 this easily "inline"(would be cool to be able to provide a delegate to 
 do the transformations at the site of definition of the fields).


Use the "onSerializing" and "onSerialized" UDAs on a method. 
"onSerializing" will be called before serializing and "onSerialized" 
after serializing. Have a look at the unit tests [1].


 Also, how does orange handle properties? Seems it just deals with fields 
 and ignores all functions(does not use getter and setter of properties). 
 This is valid, of course, just want to make sure. I still need to be 
 able to transform values pre and post though.


That is correct, it only (de)serializes fields. If you want to 
(de)serialize proprieties, implement the "toData" and "fromData". See 
the example in the wiki [2]. Note, by implementing these methods none of 
the standard serialization will occur. If you want to serialize the 
fields as well, you need to do that as well when implementing "toData" 
and "fromData".

It's also possible to implement these "methods" in a non-intrusive way, 
i.e. for customizing serialization of third party type [3].

[1] 
https://github.com/jacob-carlborg/orange/blob/master/tests/Events.d#L39-L54

[2] https://github.com/jacob-carlborg/orange/wiki/Custom-Serialization

[3] 
https://github.com/jacob-carlborg/orange/blob/master/tests/NonIntrusive.d

-- 
/Jacob Carlborg
Jun 02 2018
parent reply InfiniteDimensional <IntegratedDimensions gmail.com> writes:
On Saturday, 2 June 2018 at 20:11:17 UTC, Jacob Carlborg wrote:

 On 2018-06-02 03:30, IntegratedDimensions wrote:

 How can I modify the pre serialization and post serialization 
 values? I need to transform some variables that are stored but 
 I would like to do this easily "inline"(would be cool to be 
 able to provide a delegate to do the transformations at the 
 site of definition of the fields).


 Use the "onSerializing" and "onSerialized" UDAs on a method. 
 "onSerializing" will be called before serializing and 
 "onSerialized" after serializing. Have a look at the unit tests 
 [1].


 Also, how does orange handle properties? Seems it just deals 
 with fields and ignores all functions(does not use getter and 
 setter of properties). This is valid, of course, just want to 
 make sure. I still need to be able to transform values pre and 
 post though.


 That is correct, it only (de)serializes fields. If you want to 
 (de)serialize proprieties, implement the "toData" and 
 "fromData". See the example in the wiki [2]. Note, by 
 implementing these methods none of the standard serialization 
 will occur. If you want to serialize the fields as well, you 
 need to do that as well when implementing "toData" and 
 "fromData".

 It's also possible to implement these "methods" in a 
 non-intrusive way, i.e. for customizing serialization of third 
 party type [3].

 [1] 
 https://github.com/jacob-carlborg/orange/blob/master/tests/Events.d#L39-L54

 [2] 
 https://github.com/jacob-carlborg/orange/wiki/Custom-Serialization

 [3] 
 https://github.com/jacob-carlborg/orange/blob/master/tests/NonIntrusive.d



Thanks.

I'm having problems preventing void* pointers from not being 
serialized

..\..\..\orange\serialization\Serializer.d(975): Error: 
expression `*value` is `void` and has no value

..\..\..\orange\serialization\Serializer.d(1491): Error: new can 
only create structs, dynamic arrays or class objects, not `void`'s


and all I've added to my class is

 nonSerialized void* ptr;

It seems that the (de)serializer should just ignore all void's no 
matter what. They can't be serialized to any meaningful thing. 
Maybe spit a warning out if the uda is not added. Usually pointer 
values are not meant to be serialized anyways.
Jun 05 2018
next sibling parent reply InfiniteDimensional <IntegratedDimensions gmail.com> writes:
I'm also having some issue now when I changed a type from using a 
class to using it's base interface

Unhandled exception: 
orange.serialization.SerializationException.SerializationException The object
of the static type "const(ItemInterface)" have a different runtime type (Item)
and therefore needs to either register its type or register a serializer for
its type "Item". at ..\..\..\orange\serialization\SerializationException.d(25)

Item inherits from ItemInterface.

I was storing a list of Items and changed it to store 
ItemInterface

Item[] -> ItemInterface[]

and this is when the error happened.

Of course, I'd expect the interface not being 
serializable(although, maybe  properties should be?) it would be 
nice if it would store the actual type in it's place(an Item). 
Else, this prevents me from using interfaces.
Jun 05 2018
parent reply Jacob Carlborg <doob me.com> writes:
On 2018-06-05 20:14, InfiniteDimensional wrote:

 I'm also having some issue now when I changed a type from using a class 
 to using it's base interface
 
 Unhandled exception: 
 orange.serialization.SerializationException.SerializationException The 
 object of the static type "const(ItemInterface)" have a different 
 runtime type (Item) and therefore needs to either register its type or 
 register a serializer for its type "Item". at 
 ..\..\..\orange\serialization\SerializationException.d(25)
 
 Item inherits from ItemInterface.
 
 I was storing a list of Items and changed it to store ItemInterface
 
 Item[] -> ItemInterface[]
 
 and this is when the error happened.
 
 Of course, I'd expect the interface not being serializable(although, 
 maybe  properties should be?) it would be nice if it would store the 
 actual type in it's place(an Item). Else, this prevents me from using 
 interfaces.


D doesn't support any runtime reflection (or very little). All 
reflection is done at compile time, i.e. getting all the fields. If the 
static type and the dynamic type differs Orange cannot properly 
serialize the dynamic type. For that to work you need to register all 
dynamic types that are expected to be serialized through a base class or 
interface. See this example [1].

[1] 
https://github.com/jacob-carlborg/orange/blob/master/tests/BaseClass.d#L73

-- 
/Jacob Carlborg
Jun 06 2018
parent reply InfiniteDimensional <InfiniteDimensional Gmail.com> writes:
On Wednesday, 6 June 2018 at 16:34:52 UTC, Jacob Carlborg wrote:

 On 2018-06-05 20:14, InfiniteDimensional wrote:

 I'm also having some issue now when I changed a type from 
 using a class to using it's base interface
 
 Unhandled exception: 
 orange.serialization.SerializationException.SerializationException The object
of the static type "const(ItemInterface)" have a different runtime type (Item)
and therefore needs to either register its type or register a serializer for
its type "Item". at ..\..\..\orange\serialization\SerializationException.d(25)
 
 Item inherits from ItemInterface.
 
 I was storing a list of Items and changed it to store 
 ItemInterface
 
 Item[] -> ItemInterface[]
 
 and this is when the error happened.
 
 Of course, I'd expect the interface not being 
 serializable(although, maybe  properties should be?) it would 
 be nice if it would store the actual type in it's place(an 
 Item). Else, this prevents me from using interfaces.


 D doesn't support any runtime reflection (or very little). All 
 reflection is done at compile time, i.e. getting all the 
 fields. If the static type and the dynamic type differs Orange 
 cannot properly serialize the dynamic type. For that to work 
 you need to register all dynamic types that are expected to be 
 serialized through a base class or interface. See this example 
 [1].

 [1] 
 https://github.com/jacob-carlborg/orange/blob/master/tests/BaseClass.d#L73



I did register the main derived type and everything seems to 
work. Why do I have to reset the registered types?

I still can't have a void* in my class though ;/ My exact code is

 nonSerialized void* ptr;


commenting out passes, else I get the errors

\orange\serialization\Serializer.d(975): Error: expression 
`*value` is `void` and has no value
\orange\serialization\Serializer.d(1605): Error: template 
instance 
`orange.serialization.Serializer.Serializer.serializePointer!(inout(void)*)`
error instantiating
\orange\serialization\Serializer.d(1698):        instantiated 
from here: `objectStructSerializeHelper!(inout(Item))`
\orange\serialization\Serializer.d(1616):        instantiated 
from here: `serializeBaseTypes!(item)`
\orange\serialization\Serializer.d(261):        instantiated from 
here: `objectStructSerializeHelper!(item)`
\orange\serialization\Serializer.d(247):        instantiated from 
here: `downcastSerialize!(item)`
main.d(50):        instantiated from here: `register!(item)`
\orange\serialization\Serializer.d(1609): Error: template 
instance 
`orange.serialization.Serializer.Serializer.serializeInternal!(inout(void*))`
error instantiating
\orange\serialization\Serializer.d(1698):        instantiated 
from here: `objectStructSerializeHelper!(inout(Item))`
\orange\serialization\Serializer.d(1616):        instantiated 
from here: `serializeBaseTypes!(item)`
\orange\serialization\Serializer.d(261):        instantiated from 
here: `objectStructSerializeHelper!(item)`
\orange\serialization\Serializer.d(247):        instantiated from 
here: `downcastSerialize!(item)`
main.d(50):        instantiated from here: `register!(item)`
\orange\serialization\Serializer.d(1491): Error: new can only 
create structs, dynamic arrays or class objects, not `void`'s
\orange\serialization\Serializer.d(1653): Error: template 
instance 
`orange.serialization.Serializer.Serializer.deserializePointer!(void*)` error
instantiating
\orange\serialization\Serializer.d(1709):        instantiated 
from here: `objectStructDeserializeHelper!(Item)`
\orange\serialization\Serializer.d(1688):        instantiated 
from here: `deserializeBaseTypes!(item)`
\orange\serialization\Serializer.d(264):        instantiated from 
here: `objectStructDeserializeHelper!(item)`
\orange\serialization\Serializer.d(247):        instantiated from 
here: `downcastSerialize!(item)`
main.d(50):        instantiated from here: `register!(item)`


changing void* ptr to int* ptr and everything works so this is an 
issue in orange.

when changing to int it serializes to

<null type="inout(int)*" key="ptr"/>
Jun 06 2018
parent Jacob Carlborg <doob me.com> writes:
On Wednesday, 6 June 2018 at 20:46:22 UTC, InfiniteDimensional 
wrote:


 I did register the main derived type and everything seems to 
 work. Why do I have to reset the registered types?


Do you have to reset the registered types? What happens otherwise?


 I still can't have a void* in my class though ;/ My exact code 
 is

  nonSerialized void* ptr;


 commenting out passes, else I get the errors


That's a bug, please report an issue on GitHub [1] so it's not 
forgotten.

[1] https://github.com/jacob-carlborg/orange/issues
Jun 06 2018
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2018-06-05 19:47, InfiniteDimensional wrote:


 Thanks.
 
 I'm having problems preventing void* pointers from not being serialized
 
 ..\..\..\orange\serialization\Serializer.d(975): Error: expression 
 `*value` is `void` and has no value
 
 ..\..\..\orange\serialization\Serializer.d(1491): Error: new can only 
 create structs, dynamic arrays or class objects, not `void`'s
 
 
 and all I've added to my class is
 
  nonSerialized void* ptr;
 
 It seems that the (de)serializer should just ignore all void's no matter 
 what. They can't be serialized to any meaningful thing. Maybe spit a 
 warning out if the uda is not added. Usually pointer values are not 
 meant to be serialized anyways.


Adding " nonSerialized" did not help?

-- 
/Jacob Carlborg
Jun 06 2018