www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Another serialization topic, reduced scope

reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
https://youtu.be/Y-cgh-rwoC8

I was watching Steven's 2019 talk. I'd like to put together a DIP 
to specify a collection of attributes which target this very 

at runtime, and we see it with different D libraries.

Adding and replacing serialization to the standard library has 
had its delays due to expectations of how serialization should be 
designed.

I think it is clear that attributes on structures is a common 
approach placed on top of the actual implementation.

Phobos updates are not traditionally going through a DIP, but I 
think this is the right avenue for review.

We see similar attributes used in things like argument parsing. 
But here is a short list of what I am thinking from an attribute 
list.

 ignore
 optional
 name("field")

Seeing this work through will be hard with a 3y and 3m old, but I 
feel this is the type of library support std can provide even 
with the benefits of dub.

Destroy.
Dec 07 2019
next sibling parent reply Tobias Pankrath <tobias+dlang pankrath.net> writes:
On Saturday, 7 December 2019 at 18:53:51 UTC, Jesse Phillips 
wrote:

  ignore
  optional
  name("field")

 Seeing this work through will be hard with a 3y and 3m old, but 
 I feel this is the type of library support std can provide even 
 with the benefits of dub.

 Destroy.


This should have a place in phobos, because it makes 
interoperability across libraries easier.
Dec 08 2019
parent reply Jacob Carlborg <doob me.com> writes:
On 2019-12-08 09:22, Tobias Pankrath wrote:


 This should have a place in phobos, because it makes interoperability 
 across libraries easier.


No, it should be in druntime. Then it's possible to attach the UDA's to 
symbols both in Phobos and druntime. For example, adding  ignore to Thread.

-- 
/Jacob Carlborg
Dec 11 2019
parent Gregor =?UTF-8?B?TcO8Y2ts?= <gregormueckl gmx.de> writes:
On Wednesday, 11 December 2019 at 18:53:56 UTC, Jacob Carlborg 
wrote:

 On 2019-12-08 09:22, Tobias Pankrath wrote:


 This should have a place in phobos, because it makes 
 interoperability across libraries easier.


 No, it should be in druntime. Then it's possible to attach the 
 UDA's to symbols both in Phobos and druntime. For example, 
 adding  ignore to Thread.


This points me in a completely different direction: you can't 
annotate 3rd party libraries with annotations that these 
libraries are unaware of.


GUI control was extracting info about presentation of properties 
from attributes via reflection. The classes I had to feed into it 
were data model classes that couldn't depend on the GUI. The 
resulting code bent over backwards to inject the required 
attributes at runtime based on a separately maintained database 
that covered thousands of properties. I'm still amazed that this 
worked. The main issue was keeping the database in sync with the 
model code. Compile time checks on the consistency of these 
external attributes would have been great.
Dec 11 2019
prev sibling next sibling parent Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Saturday, 7 December 2019 at 18:53:51 UTC, Jesse Phillips 
wrote:


 Destroy.


WIP 
https://github.com/JesseKPhillips/DIPs/blob/serialize/attribute/DIPs/1NNN-jkp.md
Dec 08 2019
prev sibling next sibling parent reply Tobias Pankrath <tobias+dlang pankrath.net> writes:
On Saturday, 7 December 2019 at 18:53:51 UTC, Jesse Phillips 
wrote:

  optional


If we look at e.g. json-schema.org or openapis.org they seem to 
have optional by default. So we should at least consider this as 
well, e.g.

1. have optional by default and introduce  required
2. make it configurable on the containing struct / class

enum IfMissing {
     setDefault,
     keep
}

 missing(IfMissing.SetDefault)
struct S
{
     // all fields by default optional due to  missing
      required
     int x;
     int a = 4; // set to 0 if not provided during deserialization
}

3. something else ..
Dec 08 2019
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 12/8/19 4:19 PM, Tobias Pankrath wrote:

 On Saturday, 7 December 2019 at 18:53:51 UTC, Jesse Phillips wrote:

  optional


 
 If we look at e.g. json-schema.org or openapis.org they seem to have 
 optional by default. So we should at least consider this as well, e.g.
 
 1. have optional by default and introduce  required
 2. make it configurable on the containing struct / class
 
 enum IfMissing {
      setDefault,
      keep
 }
 
  missing(IfMissing.SetDefault)
 struct S
 {
      // all fields by default optional due to  missing
       required
      int x;
      int a = 4; // set to 0 if not provided during deserialization
 }
 
 3. something else ..


First, I think the initiative is a good one. But in certain ways it is 
not enough.

I will tell you from first hand experience. I have types in my 
proprietary back end which are serialized in 2 ways -- one to the 
database, and one to a REST route as JSON. Specifically for one method 
of serialization, some things are ignored. For the other method of 
serialization, other things are ignored. For example, in the database, I 
may care very much that the id is serialized. But over Json, not at all.

So, the conclusion I came to (and actually, I use), is that I need to 
tag my members with BOTH, and have some way to distinguish them. That 
is, vibe.data.serialization.optional is a DIFFERENT attribute than 
db.DB.optional. And in my case, I use a DB struct, so all my attributes 
look like  (DB.optional) or  (DB.name!"class")

Decisions like whether the default is required or optional is very 
implementation dependent. Even the same mechanisms that are used in one 
project (i.e. Json serialization) may require a different default mode 
in another project.

And types may be defined that are used in both.

So how do we sort this out with a "universal" UDA system? I don't really 
know the answer. So far in my travels, I haven't needed to serialize 
types other than my own with serialization markers on it. So I control 
everything.

But that isn't always going to be the case. Perhaps in some cases, we 
need something other than UDA-based markers. Even setting the default 
may not be enough, maybe you need to generate a "dummy" struct which has 
the right UDAs on it. I go over something similar in this d meetup talk 
I did a while back: https://www.youtube.com/watch?v=ZxzczSDaobw

Part of me really likes the idea of a universal UDA system for 
serialization, to make things more general. But the other part of me 
feels like it's not a problem that can be solved universally -- it needs 
a local aspect. Is there a way to make it work both ways?

-Steve
Dec 08 2019
next sibling parent JN <666total wp.pl> writes:
On Sunday, 8 December 2019 at 22:00:53 UTC, Steven Schveighoffer 
wrote:

 Part of me really likes the idea of a universal UDA system for 
 serialization, to make things more general. But the other part 
 of me feels like it's not a problem that can be solved 
 universally -- it needs a local aspect. Is there a way to make 
 it work both ways?

 -Steve


I think serde in Rust is a good reference to look around. It's 
the de facto way of doing serialization in Rust and they have a 
list of attributes that are available: 
https://serde.rs/attributes.html
Dec 08 2019
prev sibling parent reply Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Sunday, 8 December 2019 at 22:00:53 UTC, Steven Schveighoffer 
wrote:


 So, the conclusion I came to (and actually, I use), is that I 
 need to tag my members with BOTH, and have some way to 
 distinguish them. That is, vibe.data.serialization.optional is 
 a DIFFERENT attribute than db.DB.optional. And in my case, I 
 use a DB struct, so all my attributes look like  (DB.optional) 
 or  (DB.name!"class")


I figured this would be likely. I took a look and it would be 
possible to specify a template argument to the attribute, the 
libraries could supply a type, we could supply some generic ones 
(DB, Web, Disk, Unspecified)

I'll watch the video.
Dec 08 2019
parent Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
On Monday, 9 December 2019 at 03:21:47 UTC, Jesse Phillips wrote:

 On Sunday, 8 December 2019 at 22:00:53 UTC, Steven 
 Schveighoffer wrote:


 So, the conclusion I came to (and actually, I use), is that I 
 need to tag my members with BOTH, and have some way to 
 distinguish them. That is, vibe.data.serialization.optional is 
 a DIFFERENT attribute than db.DB.optional. And in my case, I 
 use a DB struct, so all my attributes look like  (DB.optional) 
 or  (DB.name!"class")


 I figured this would be likely. I took a look and it would be 
 possible to specify a template argument to the attribute, the 
 libraries could supply a type, we could supply some generic 
 ones (DB, Web, Disk, Unspecified)

 I'll watch the video.


I took some more time to expand on the idea. While I didn't get 
my custom getUDA to work perfectly, it looks promising.

https://github.com/JesseKPhillips/DIPs/blob/serialize/attribute/DIPs/1NNN-jkp/serialattr.d

Basically the implementation will provide an attribute which 
takes an attribute to match on.

 SerializerDB!DB
struct Example {
      Ignore!DB

The standard library will need to help search these attributes.
Dec 14 2019
prev sibling next sibling parent Basile B. <b2.temp gmx.com> writes:
On Saturday, 7 December 2019 at 18:53:51 UTC, Jesse Phillips 
wrote:

 https://youtu.be/Y-cgh-rwoC8

 I was watching Steven's 2019 talk. I'd like to put together a 
 DIP to specify a collection of attributes which target this 

 and Java at runtime, and we see it with different D libraries.

 Adding and replacing serialization to the standard library has 
 had its delays due to expectations of how serialization should 
 be designed.


Then your DIP will not address the problem. As far as I can see 
it proposes conventions about the attributes to use on fields 
that have to be read from and written to. Is the lack of 
attributes really the reason why there's no std.serializer ?

A few remarks however. You want to collect the different 
attributes while it would be possible to use a single 
attribute... a class. This would allow to store functions 
pointers, tables or I don't know what else could be useful for a 
finely grained control of the serialization.
Dec 08 2019
prev sibling parent Jesse Phillips <Jesse.K.Phillips+D gmail.com> writes:
Ethan,

https://youtu.be/rSY78Hu8DqI

In your talk you mention needing

OnPreSerialize
OnPostSerialize
OnPreDeserialize
OnPostDeserialize

Could you explain the seed semantics for these?

Right now OnPostDeserialize on the class itself (not field) can 
be used for filling internal fields.
Dec 14 2019