www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Serialization for D. Comments, please!

reply BCS <none anon.com> writes:
I'm planning on taking a crack at a Serialization template library and I'm 
looking for feed back. My thinking so far is up on my blog here:

http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html

Please comment! (here or there, doesn't matter, I think I'll see both)
May 19 2009
next sibling parent reply "Tim Matthews" <tim.matthews7 gmail.com> writes:
On Wed, 20 May 2009 15:43:16 +1200, BCS <none anon.com> wrote:


 I'm planning on taking a crack at a Serialization template library and  
 I'm looking for feed back. My thinking so far is up on my blog here:

 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html

 Please comment! (here or there, doesn't matter, I think I'll see both)



Have you thought about making a OODB so have the full oo programming with  
data and functions but the objects can be saved and retrieved in a  
transactional way?
May 19 2009
parent BCS <none anon.com> writes:
Hello Tim,


 On Wed, 20 May 2009 15:43:16 +1200, BCS <none anon.com> wrote:
 

 I'm planning on taking a crack at a Serialization template library
 and  I'm looking for feed back. My thinking so far is up on my blog
 here:
 
 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part
 -1-of-n.html
 
 Please comment! (here or there, doesn't matter, I think I'll see
 both)
 


 Have you thought about making a OODB so have the full oo programming
 with  data and functions but the objects can be saved and retrieved in
 a  transactional way?
 


The target usage is to stuff data structures into file or network streams. 
Eventually, I'd like to build a web services lib from this that can serve 
proxy an interface with very little code:

/////common:

interface I { ... }
interface J { ... }

/////Server side

class CI : I { ... }
class CJ : J { ... }

auto server = new Server();

server.Serve!(I)(new CI(), 8080); //proxy that object on port 8080
server.Serve!(J)({return new CJ;}, 8081); // create a objects to proxy on 
port 8081

server.start();

//// client side

I i = new Proxy!(I)("somehost.net", 8080);
J j = new Proxy!(J)("somehost.net", 8081);
May 19 2009
prev sibling next sibling parent BCS <none anon.com> writes:
It turned out that the first pass was easy. No unexpected problems. However 
I noticed a few more things I'm going to have to deal with at some point.

http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-2-of-n.html

Again, please comment! (here or there, doesn't matter, I think I'll see both)

p.s. Sorry, no code yet (it's still messy) but fixing that is on the list 
as well.
May 23 2009
prev sibling next sibling parent BCS <none anon.com> writes:
Now with Code!

http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-3-of-n-with.html
May 25 2009
prev sibling next sibling parent reply Qian Xu <quian.xu stud.tu-ilmenau.de> writes:
BCS wrote:


 I'm planning on taking a crack at a Serialization template library and I'm
 looking for feed back. My thinking so far is up on my blog here:
 


http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html

 
 Please comment! (here or there, doesn't matter, I think I'll see both)


A question: 
Should every object contain "mixin Serializable!()" in its declaration?
It is easy to add this in own classes, but not easy to 3rd-party libraries.
for instance: tango


--Qian
May 26 2009
next sibling parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Tue, 26 May 2009 13:29:39 +0400, Qian Xu <quian.xu stud.tu-ilmenau.de> wrote:


 BCS wrote:


 I'm planning on taking a crack at a Serialization template library and  
 I'm
 looking for feed back. My thinking so far is up on my blog here:


 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html

 Please comment! (here or there, doesn't matter, I think I'll see both)


 A question:
 Should every object contain "mixin Serializable!()" in its declaration?
 It is easy to add this in own classes, but not easy to 3rd-party  
 libraries.
 for instance: tango


 --Qian


Good serialization library supports external serialization via template
specialization (or similar tricks) :)
May 26 2009
parent reply Bill Baxter <wbaxter gmail.com> writes:
On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com> wrote:

 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu <quian.xu stud.tu-ilmenau.de>
 wrote:


 BCS wrote:


 I'm planning on taking a crack at a Serialization template library and
 I'm
 looking for feed back. My thinking so far is up on my blog here:


 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html

 Please comment! (here or there, doesn't matter, I think I'll see both)


 A question:
 Should every object contain "mixin Serializable!()" in its declaration?
 It is easy to add this in own classes, but not easy to 3rd-party
 libraries.
 for instance: tango


 --Qian


 Good serialization library supports external serialization via template
 specialization (or similar tricks) :)


I'll also add that you should be able to properly
serialize/deserialize a BaseClass pointer that actually points to a
DerivedClass.
This is pretty tricky to get working seamlessly when combined with the
external serialization requirement.
H3r3tic's xpose library has this working IIRC.

--bb
May 26 2009
next sibling parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Tue, 26 May 2009 21:44:49 +0400, Bill Baxter <wbaxter gmail.com> wrote:


 On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com>  
 wrote:

 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu  
 <quian.xu stud.tu-ilmenau.de>
 wrote:


 BCS wrote:


 I'm planning on taking a crack at a Serialization template library and
 I'm
 looking for feed back. My thinking so far is up on my blog here:


 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html

 Please comment! (here or there, doesn't matter, I think I'll see both)


 A question:
 Should every object contain "mixin Serializable!()" in its declaration?
 It is easy to add this in own classes, but not easy to 3rd-party
 libraries.
 for instance: tango


 --Qian


 Good serialization library supports external serialization via template
 specialization (or similar tricks) :)


 I'll also add that you should be able to properly
 serialize/deserialize a BaseClass pointer that actually points to a
 DerivedClass.
 This is pretty tricky to get working seamlessly when combined with the
 external serialization requirement.
 H3r3tic's xpose library has this working IIRC.

 --bb


Sure! How about the following test:

struct A
{
    mixin Serializable!();

    B[] b;
}

struct B
{
    mixin Serializable!();

    A a;
    int i;
}

B[] b = new B[1];
b[0].a.b = b;
b[0].i = 42;

A* a = &b[0].a;

data = serialize(a); // should serialize outer struct (B), too, because it is
accessible through a

a = deserialize(data);
assert(a.b.length == 1);
assert(a.b[0].i == 42);

BTW, thanks for reminding about xpose. I gotta compare it, doost.serialize and
mine own one against each other in terms of efficiency and size (for binary
output) someday.
May 26 2009
parent reply Bill Baxter <wbaxter gmail.com> writes:
On Tue, May 26, 2009 at 12:19 PM, Denis Koroskin <2korden gmail.com> wrote:

 On Tue, 26 May 2009 21:44:49 +0400, Bill Baxter <wbaxter gmail.com> wrote=


:

 On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com>
 wrote:

 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu
 <quian.xu stud.tu-ilmenau.de>
 wrote:


 BCS wrote:


 I'm planning on taking a crack at a Serialization template library an=










d

 I'm
 looking for feed back. My thinking so far is up on my blog here:


 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-=








1-of-n.html

 Please comment! (here or there, doesn't matter, I think I'll see both=










)

 A question:
 Should every object contain "mixin Serializable!()" in its declaration=








?

 It is easy to add this in own classes, but not easy to 3rd-party
 libraries.
 for instance: tango


 --Qian


 Good serialization library supports external serialization via template
 specialization (or similar tricks) :)


 I'll also add that you should be able to properly
 serialize/deserialize a BaseClass pointer that actually points to a
 DerivedClass.
 This is pretty tricky to get working seamlessly when combined with the
 external serialization requirement.
 H3r3tic's xpose library has this working IIRC.

 --bb


 Sure! How about the following test:

 struct A
 {
 =A0 =A0mixin Serializable!();

 =A0 =A0B[] b;
 }

 struct B
 {
 =A0 =A0mixin Serializable!();

 =A0 =A0A a;
 =A0 =A0int i;
 }

 B[] b =3D new B[1];
 b[0].a.b =3D b;
 b[0].i =3D 42;

 A* a =3D &b[0].a;

 data =3D serialize(a); // should serialize outer struct (B), too, because=


 it is accessible through a

 a =3D deserialize(data);
 assert(a.b.length =3D=3D 1);
 assert(a.b[0].i =3D=3D 42);

 BTW, thanks for reminding about xpose. I gotta compare it, doost.serializ=


e and mine own one against each other in terms of efficiency and size (for =
binary output) someday.

That's not quite what I meant.

I mean this:

/// 3RD PARTY LIB
class BaseClass
{
     string toString() { return "base"; }
}

class DerivedClass
{
     string toString() { return "derived"; }
}


/// MY CODE

[...some kind of class registration here....]

BaseClass b =3D new DerivedClass;

data =3D serialize(b);
b2 =3D deserialize(data);

assert(b2.toString() =3D=3D "derived");
------------------

The data stream should contain enough info about the instance to
reconstruct the exact derived type.

--bb
May 26 2009
parent reply "Denis Koroskin" <2korden gmail.com> writes:
On Wed, 27 May 2009 01:23:58 +0400, Bill Baxter <wbaxter gmail.com> wrote:


 On Tue, May 26, 2009 at 12:19 PM, Denis Koroskin <2korden gmail.com>  
 wrote:

 On Tue, 26 May 2009 21:44:49 +0400, Bill Baxter <wbaxter gmail.com>  
 wrote:


 On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com>
 wrote:

 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu
 <quian.xu stud.tu-ilmenau.de>
 wrote:


 BCS wrote:


 I'm planning on taking a crack at a Serialization template library  
 and
 I'm
 looking for feed back. My thinking so far is up on my blog here:


 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-1-of-n.html

 Please comment! (here or there, doesn't matter, I think I'll see  
 both)


 A question:
 Should every object contain "mixin Serializable!()" in its  
 declaration?
 It is easy to add this in own classes, but not easy to 3rd-party
 libraries.
 for instance: tango


 --Qian


 Good serialization library supports external serialization via  
 template
 specialization (or similar tricks) :)


 I'll also add that you should be able to properly
 serialize/deserialize a BaseClass pointer that actually points to a
 DerivedClass.
 This is pretty tricky to get working seamlessly when combined with the
 external serialization requirement.
 H3r3tic's xpose library has this working IIRC.

 --bb


 Sure! How about the following test:

 struct A
 {
    mixin Serializable!();

    B[] b;
 }

 struct B
 {
    mixin Serializable!();

    A a;
    int i;
 }

 B[] b = new B[1];
 b[0].a.b = b;
 b[0].i = 42;

 A* a = &b[0].a;

 data = serialize(a); // should serialize outer struct (B), too, because  
 it is accessible through a

 a = deserialize(data);
 assert(a.b.length == 1);
 assert(a.b[0].i == 42);

 BTW, thanks for reminding about xpose. I gotta compare it,  
 doost.serialize and mine own one against each other in terms of  
 efficiency and size (for binary output) someday.


 That's not quite what I meant.

 I mean this:

 /// 3RD PARTY LIB
 class BaseClass
 {
      string toString() { return "base"; }
 }

 class DerivedClass
 {
      string toString() { return "derived"; }
 }


 /// MY CODE

 [...some kind of class registration here....]

 BaseClass b = new DerivedClass;

 data = serialize(b);
 b2 = deserialize(data);

 assert(b2.toString() == "derived");
 ------------------

 The data stream should contain enough info about the instance to
 reconstruct the exact derived type.

 --bb


Yes, I understood and showed yet another feature which is quite hard to
implement.
May 26 2009
parent Bill Baxter <wbaxter gmail.com> writes:
On Tue, May 26, 2009 at 2:26 PM, Denis Koroskin <2korden gmail.com> wrote:

 On Wed, 27 May 2009 01:23:58 +0400, Bill Baxter <wbaxter gmail.com> wrote=


:

 On Tue, May 26, 2009 at 12:19 PM, Denis Koroskin <2korden gmail.com>
 wrote:

 On Tue, 26 May 2009 21:44:49 +0400, Bill Baxter <wbaxter gmail.com>
 wrote:


 On Tue, May 26, 2009 at 2:35 AM, Denis Koroskin <2korden gmail.com>
 wrote:

 On Tue, 26 May 2009 13:29:39 +0400, Qian Xu
 <quian.xu stud.tu-ilmenau.de>
 wrote:


 BCS wrote:


 I'm planning on taking a crack at a Serialization template library
 and
 I'm
 looking for feed back. My thinking so far is up on my blog here:



 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-par=












t-1-of-n.html

 Please comment! (here or there, doesn't matter, I think I'll see
 both)


 A question:
 Should every object contain "mixin Serializable!()" in its
 declaration?
 It is easy to add this in own classes, but not easy to 3rd-party
 libraries.
 for instance: tango


 --Qian


 Good serialization library supports external serialization via templa=










te

 specialization (or similar tricks) :)


 I'll also add that you should be able to properly
 serialize/deserialize a BaseClass pointer that actually points to a
 DerivedClass.
 This is pretty tricky to get working seamlessly when combined with the
 external serialization requirement.
 H3r3tic's xpose library has this working IIRC.

 --bb


 Sure! How about the following test:

 struct A
 {
 =A0 mixin Serializable!();

 =A0 B[] b;
 }

 struct B
 {
 =A0 mixin Serializable!();

 =A0 A a;
 =A0 int i;
 }

 B[] b =3D new B[1];
 b[0].a.b =3D b;
 b[0].i =3D 42;

 A* a =3D &b[0].a;

 data =3D serialize(a); // should serialize outer struct (B), too, becau=






se

 it is accessible through a

 a =3D deserialize(data);
 assert(a.b.length =3D=3D 1);
 assert(a.b[0].i =3D=3D 42);

 BTW, thanks for reminding about xpose. I gotta compare it,
 doost.serialize and mine own one against each other in terms of efficie=






ncy

 and size (for binary output) someday.


 That's not quite what I meant.

 I mean this:

 /// 3RD PARTY LIB
 class BaseClass
 {
 =A0 =A0 string toString() { return "base"; }
 }

 class DerivedClass
 {
 =A0 =A0 string toString() { return "derived"; }
 }


 /// MY CODE

 [...some kind of class registration here....]

 BaseClass b =3D new DerivedClass;

 data =3D serialize(b);
 b2 =3D deserialize(data);

 assert(b2.toString() =3D=3D "derived");
 ------------------

 The data stream should contain enough info about the instance to
 reconstruct the exact derived type.

 --bb


 Yes, I understood and showed yet another feature which is quite hard to
 implement.


Ok.  I interpreted that as "Sure! How about the following test (of the
feature you just mentioned)".
Some segue words like "in addition" or "also" probably would have
tipped me off that you were changing the subject. :-)

--bb
May 26 2009
prev sibling parent BCS <none anon.com> writes:
Hello Bill,


 I'll also add that you should be able to properly
 serialize/deserialize a BaseClass pointer that actually points to a
 DerivedClass.
 This is pretty tricky to get working seamlessly when combined with the
 external serialization requirement.
 H3r3tic's xpose library has this working IIRC.
 --bb



The current version of the code has this working. It was fairly high on my 
feature list.
May 26 2009
prev sibling parent BCS <none anon.com> writes:
Hello Qian,


 BCS wrote:
 

 I'm planning on taking a crack at a Serialization template library
 and I'm looking for feed back. My thinking so far is up on my blog
 here:
 


 http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-
 1-of-n.html
 

 Please comment! (here or there, doesn't matter, I think I'll see
 both)
 


 A question:
 Should every object contain "mixin Serializable!()" in its
 declaration? It is easy to add this in own classes, but
 not easy to 3rd-party libraries. for instance: tango
 --Qian
 


I've been mentally dodging that question because it's going to be ugly. If 
anyone has some ideas on how to make it work, I'm interested.

Right now the system has a hard coded list of basic types (byte, real, etc.) 
that it can work with and failing that, it assume you have added "mixin
Serializable!()". 
The main problem is getting the third case is overload. Overloading doesn't 
work across mixins or modules last I checked. It would be a hack, but named 
function called via string mixins (mixin("Marshal"~ typeof(a).stringof ~ 
"(sink,a);")) might work.
May 26 2009
prev sibling next sibling parent aarti_pl <aarti interia.pl> writes:
BCS pisze:

 I'm planning on taking a crack at a Serialization template library and 
 I'm looking for feed back. My thinking so far is up on my blog here:
 
 http://arrayboundserror.blogspot.com/2009/05/serialization-fo
-d-part-1-of-n.html 
 
 
 Please comment! (here or there, doesn't matter, I think I'll see both)
 
 


Hello!

I would like to mention that serialization library for D already exists:

http://www.dsource.org/projects/doost

It supports most of common requirements for serialization library:
- easy to use API
- most built-in types and user defined types (UDT) are already serializable
- out of class serialization
- versioning of UDT; importing old versions to new versions
- thread/exception safe (well... AFAIK :-) )
- large unit test suite
- configurable serialization backend (archive): JSon, Binary, SimpleText
- serialization of classes with user defined constructors

I don't want to discourage you from implementing your own library, but 
on the other hand you may consider contributing to this, already 
existing library. It took me few months to get where it is right now, so 
it is really a LOT of work; Unfortunately I don't have time to develop 
it further right now...

There are still few things which should be implemented:
- serialization from base class (difficult, when we don't want to demand 
additional coding work from user)
- Tango support (should be relatively easy)
- XML archive support (foundation framework already there)
- serialization for complex types
- cleanups, improvements, porting to D 2.0
- better documentation

If you would be interested just drop me an e-mail:

aarti -at- interia -dot- pl

I am very open to suggestions and changes :-)

Best Regards
Marcin Kuszczak
(aarti_pl)
May 26 2009
prev sibling next sibling parent BCS <none anon.com> writes:
And yet more progress. This time I've got "hard-links" working and I'm thinking 
about how to do 3rd party types.

I don't have any useable ideas I really /like/, but I have one I'm sure I 
can make work. I do have one idea down at the bottom that I'd love to use 
but it would need some new language features.

http://arrayboundserror.blogspot.com/2009/05/serialization-for-d-part-5-of-n.html

As always: Comments, Please! (particularly on the "feature request")
May 31 2009
prev sibling parent reply BCS <none anon.com> writes:
the latest and greatest:

http://arrayboundserror.blogspot.com/2009/06/serialization-for-d-part-6-of-n.html

This time I'm hoping for some feedback on how people want to interface with 
3rd party types.
Jun 11 2009
parent reply grauzone <none example.net> writes:
BCS wrote:

 the latest and greatest:
 
 http://arrayboundserror.blogspot.com/2009/06/serialization-fo
-d-part-6-of-n.html 
 
 
 This time I'm hoping for some feedback on how people want to interface 
 with 3rd party types.
 
 


Is there any real reason for all those mixins?
Jun 11 2009
parent reply BCS <ao pathlink.com> writes:
Reply to grauzone,


 BCS wrote:
 

 the latest and greatest:
 
 http://arrayboundserror.blogspot.com/2009/06/serialization-for-d-part
 -6-of-n.html
 
 This time I'm hoping for some feedback on how people want to
 interface with 3rd party types.
 


 Is there any real reason for all those mixins?
 


which ones?
Jun 11 2009
parent reply grauzone <none example.net> writes:
 Is there any real reason for all those mixins?


 
 which ones?


All used by the user. That would be Serializable and SerializableRecuring.
Jun 11 2009
next sibling parent reply BCS <ao pathlink.com> writes:
Reply to grauzone,


 Is there any real reason for all those mixins?
 


 which ones?
 


 All used by the user. That would be Serializable and
 SerializableRecuring.
 


What else would you use? I guess if I really wanted to I could use the same 
device as for 3rd party types, but that just pushes the mixin to global scope 
(the functions have to be defined in the same module as the type or they 
won't be able to access private members) and introduces unneeded performance 
issues from using function pointers.

If you can spot a better solution, I'm all for it, but I don't see one and 
don't think this solution is to bad.
Jun 11 2009
parent reply grauzone <none example.net> writes:
BCS wrote:

 Reply to grauzone,
 

 Is there any real reason for all those mixins?


 which ones?


 All used by the user. That would be Serializable and
 SerializableRecuring.


 
 What else would you use? I guess if I really wanted to I could use the 
 same device as for 3rd party types, but that just pushes the mixin to 
 global scope (the functions have to be defined in the same module as the 
 type or they won't be able to access private members) and introduces 
 unneeded performance issues from using function pointers.
 
 If you can spot a better solution, I'm all for it, but I don't see one 
 and don't think this solution is to bad.


tupleof _can_ access private members, even if the type is from a 
different module than the function that uses the tupleof. This changed 
somewhere between dmd 1.030 and 1.040, I think.
Jun 11 2009
parent reply BCS <ao pathlink.com> writes:
Reply to grauzone,


 BCS wrote:
 

 Reply to grauzone,
 

 Is there any real reason for all those mixins?
 


 which ones?
 


 All used by the user. That would be Serializable and
 SerializableRecuring.
 


 What else would you use? I guess if I really wanted to I could use
 the same device as for 3rd party types, but that just pushes the
 mixin to global scope (the functions have to be defined in the same
 module as the type or they won't be able to access private members)
 and introduces unneeded performance issues from using function
 pointers.
 
 If you can spot a better solution, I'm all for it, but I don't see
 one and don't think this solution is to bad.
 


 tupleof _can_ access private members, even if the type is from a
 different module than the function that uses the tupleof. This changed
 somewhere between dmd 1.030 and 1.040, I think.


If it can, I would consider that a bug.

And even if Walter sanctions it, I wont use it because I'm very leery of 
generating a system that automaticly reaches in from the outside and mucks 
with private members. Whoever makes a type serializeable needs to put at 
least some thought into how to serialize it. If it is the type's author, 
I'm comfortable with them saying "just do everything", but if someone else 
is doing it, that is likely to get feet shot off. If I made a generic "any 
type" function, it would recur into any referenced types and either spew 
a big list of type that it's dealing with (that people won't look at) or 
give no warning that it's walking into new territory and that it may not 
be doing valid operations.
Jun 11 2009
parent reply grauzone <none example.net> writes:
 tupleof _can_ access private members, even if the type is from a
 different module than the function that uses the tupleof. This changed
 somewhere between dmd 1.030 and 1.040, I think.


 
 If it can, I would consider that a bug.


It would be nice to get an official statement.


 And even if Walter sanctions it, I wont use it because I'm very leery of 
 generating a system that automaticly reaches in from the outside and 
 mucks with private members. Whoever makes a type serializeable needs to 
 put at least some thought into how to serialize it. If it is the type's 


That's why I'd still require types to be marked as serializeable by the 
programmer. But using mixins and adding unknown members from the 
serialization machinery isn't that clean either. At leasr for all those 
structs, it's truly annoying and unneeded. Having to use different 
mixins for structs/classes sucks even more (wouldn't be required if you 
wanted).

By the way, how are you going to solve the other problems when 
serializing D types? Here are some troublesome types:
- pointers?
- function pointers?
- delegates?
- unions?
- deserializing members that are const/immutable?
- referential integrity across arrays and slices?

Also, it seems I just spotted a bug: floating point numbers loose 
precision as they are formatted with "%s".


 author, I'm comfortable with them saying "just do everything", but if 
 someone else is doing it, that is likely to get feet shot off. If I made 
 a generic "any type" function, it would recur into any referenced types 
 and either spew a big list of type that it's dealing with (that people 
 won't look at) or give no warning that it's walking into new territory 
 and that it may not be doing valid operations.


I didn't understand the last sentence.
Jun 13 2009
parent reply BCS <none anon.com> writes:
Hello grauzone,



 And even if Walter sanctions it, I wont use it because I'm very leery
 of generating a system that automaticly reaches in from the outside
 and mucks with private members. Whoever makes a type serializeable
 needs to put at least some thought into how to serialize it. If it is
 the type's
 


 That's why I'd still require types to be marked as serializeable by
 the programmer.


How would you do that aside from mixins?


 But using mixins and adding unknown members from the
 serialization machinery isn't that clean either.


I'm not going to fix it now, but I have a few ideas how I can only add the 
public API to the types. That would be one non static serialization function 
and a static deserialization function.


 At least for all
 those structs, it's truly annoying and unneeded. Having to use
 different mixins for structs/classes sucks even more (wouldn't be
 required if you wanted).


It isn't required, the difference is not struct vs. class but "do you care 
about repeated references to instances of this type?"


 
 By the way, how are you going to solve the other problems when
 serializing D types? Here are some troublesome types:
 - pointers?


class references are pointers, for struct and what not, I'll just use a similar 
approach


 - function pointers?
 - delegates?


I won't


 - unions?


for those I'll provide a way for the user to define the serialization
themselves.


 - deserializing members that are const/immutable?


D1, when I get to that; the same way you generated them anywhere else? I 
might even go with "unsafe" casts as I can be sure to get that correct.


 - referential integrity across arrays and slices?


I won't


 Also, it seems I just spotted a bug: floating point numbers loose
 precision as they are formatted with "%s".


I suppose that should be "%A" I'll get to that at some point.


 

 author, I'm comfortable with them saying "just do everything", but if
 someone else is doing it, that is likely to get feet shot off. If I
 made a generic "any type" function, it would recur into any
 referenced types and either spew a big list of type that it's dealing
 with (that people won't look at) or give no warning that it's walking
 into new territory and that it may not be doing valid operations.
 


 I didn't understand the last sentence.
 


First off there is not enough information to correctly generate sterilizers 
for types. So the user has to do some thinking for most types. How do I make 
the user thinks about how and for what types code is being generated?

Option 1) have the template run a paramg(msg,T.stringof) for each type and 
hope the user don't just ignore the list.
Option 2) do nothing and be sure the user won't see anything.
Option 3) Make the user write per type code and error when they don't.
Jun 13 2009
parent reply grauzone <none example.net> writes:
BCS wrote:

 That's why I'd still require types to be marked as serializeable by
 the programmer.


 
 How would you do that aside from mixins?


Make the user implement a marker interface, or let him provide a 
(single) special class member which fulfill the same function, or 
introduce annotations into the language. As far as marking goes, a mixin 
would be OK too, but as I said, I don't like adding arbitrary members 
into the user's scope without the user knowing.


 At least for all
 those structs, it's truly annoying and unneeded. Having to use
 different mixins for structs/classes sucks even more (wouldn't be
 required if you wanted).


 
 It isn't required, the difference is not struct vs. class but "do you 
 care about repeated references to instances of this type?"


IMHO a relatively useless optimization, that will only confuse the user. 
It will introduce subtle bugs if objects accidentally are "repeated" 
instead of serialized only once.


 By the way, how are you going to solve the other problems when
 serializing D types? Here are some troublesome types:
 - pointers?


 
 class references are pointers, for struct and what not, I'll just use a 
 similar approach


That's not really the problem here. You have no idea where a pointer 
points to. Is it a pointer to a new'ed memory block? Does it point into 
an object? Does it point into an array? Into the data segment?

The GC provides no API to find out. You may be able to handle some cases 
above, but not all.


 - function pointers?
 - delegates?


 
 I won't


Forces the user to use interfaces instead of delegates.


 - unions?


 
 for those I'll provide a way for the user to define the serialization 
 themselves.
 

 - deserializing members that are const/immutable?


 
 D1, when I get to that; the same way you generated them anywhere else? I 
 might even go with "unsafe" casts as I can be sure to get that correct.


In D1, it comes for free: you simply don't need to handle it.
But yes, it looks like you have to use dirty tricks in D2.


 - referential integrity across arrays and slices?


 
 I won't


I wonder if anyone (I mean, especially user of a serialization library) 
would disagree with that choice. Sure, there are valid D programs that 
would break with this, but is relying on this really clean?


 author, I'm comfortable with them saying "just do everything", but if
 someone else is doing it, that is likely to get feet shot off. If I
 made a generic "any type" function, it would recur into any
 referenced types and either spew a big list of type that it's dealing
 with (that people won't look at) or give no warning that it's walking
 into new territory and that it may not be doing valid operations.


 I didn't understand the last sentence.


 
 First off there is not enough information to correctly generate 
 sterilizers for types. So the user has to do some thinking for most 
 types. How do I make the user thinks about how and for what types code 
 is being generated?
 
 Option 1) have the template run a paramg(msg,T.stringof) for each type 
 and hope the user don't just ignore the list.
 Option 2) do nothing and be sure the user won't see anything.
 Option 3) Make the user write per type code and error when they don't.


Oh, you mean if there are types in the object graphs the serializer 
can't deal with it? But how would option 2) work?
Jun 14 2009
parent reply BCS <none anon.com> writes:
Hello grauzone,


 BCS wrote:
 

 That's why I'd still require types to be marked as serializeable by
 the programmer.
 


 How would you do that aside from mixins?
 


 Make the user implement a marker interface, 


that would work (for classes) but IMHO the side effects of that are more 
invasive and it still ends up mucking with internal state from the outside


 or let him provide a
 (single) special class member which fulfill the same function,


What's the difference between one and two?


 or
 introduce annotations into the language.


NO, not an option.


 As far as marking goes, a
 mixin would be OK too, but as I said, I don't like adding arbitrary
 members into the user's scope without the user knowing.
 


how about if what the mixins add is documented?


 At least for all
 those structs, it's truly annoying and unneeded. Having to use
 different mixins for structs/classes sucks even more (wouldn't be
 required if you wanted).


 It isn't required, the difference is not struct vs. class but "do you
 care about repeated references to instances of this type?"
 


 IMHO a relatively useless optimization, that will only confuse the
 user. It will introduce subtle bugs if objects accidentally are
 "repeated" instead of serialized only once.
 


Well, I can switch the default but, in my experience, most of the time
repetition 
doesn't matter. I also dissagree on the "relatively useless optimization" 
bit, it adds some not exactly trivial overhead in about 3 or 4 different 
places.


 By the way, how are you going to solve the other problems when
 serializing D types? Here are some troublesome types:
 - pointers?


 class references are pointers, for struct and what not, I'll just use
 a similar approach
 


 That's not really the problem here. You have no idea where a pointer
 points to. Is it a pointer to a new'ed memory block? Does it point
 into an object? Does it point into an array? Into the data segment?
 
 The GC provides no API to find out. You may be able to handle some
 cases above, but not all.
 


What I can't handle will be documented as unsupported.


 - function pointers?
 - delegates?


 I won't
 


 Forces the user to use interfaces instead of delegates.


interfaces are not supported either.


 - referential integrity across arrays and slices?
 


 I won't
 


 I wonder if anyone (I mean, especially user of a serialization
 library) would disagree with that choice. Sure, there are valid D
 programs that would break with this, but is relying on this really
 clean?
 


That's what I'm hoping for. D is to low level for setting up serialization 
to be completely automatic. The programmer will have to be careful how things 
are set up. My go is to make that as easy as I can.


 First off there is not enough information to correctly generate
 sterilizers for types. So the user has to do some thinking for most
 types. How do I make the user thinks about how and for what types
 code is being generated?
 
 Option 1) have the template run a paramg(msg,T.stringof) for each
 type
 and hope the user don't just ignore the list.
 Option 2) do nothing and be sure the user won't see anything.
 Option 3) Make the user write per type code and error when they
 don't.


 Oh, you mean if there are types in the object graphs the serializer
 can't deal with it?


Almost, I think. I'm looking at the case where a new types is /added/ to 
the the object graph at some point removed form the use of the sterilizer. 
Someone needs to be informed that new types are being serialized and that 
they need to make sure it's working correctly.


 But how would option 2) work?


It dosn't
Jun 14 2009
parent reply grauzone <none example.net> writes:
BCS wrote:

 introduce annotations into the language.


 
 NO, not an option.


What, why? Sure, this is not a realistic option.


 Well, I can switch the default but, in my experience, most of the time 
 repetition doesn't matter. I also dissagree on the "relatively useless 


Oh really?


 optimization" bit, it adds some not exactly trivial overhead in about 3 
 or 4 different places.


Maybe it costs a hash table lookup, but apart from that, you're saving 
space and time for marshaling additional instances. Of course, this is 
different with structs. But structs are value types.


 - function pointers?
 - delegates?


 I won't


 Forces the user to use interfaces instead of delegates.


 
 interfaces are not supported either.


But supporting interfaces would be very simple.
Jun 14 2009
parent reply BCS <none anon.com> writes:
Hello grauzone,


 BCS wrote:
 

 introduce annotations into the language.
 


 NO, not an option.
 


 What, why? Sure, this is not a realistic option.
 


D1 is fixed, and D2 will be in the next few months. I'm not going to even 
think of targeting D3 at this point. I'm writing this to be used, not as 
a theoretical construct.


 Well, I can switch the default but, in my experience, most of the
 time repetition doesn't matter. I also dissagree on the "relatively
 useless
 


 Oh really?
 


I haven't used a graph data structure in some time. Most of them have been 
trees. And the cases I can think of, the repeated reference bit has been 
central the the structure so the chances of getting it wrong (or of missing 
it under test) are about nil.


 optimization" bit, it adds some not exactly trivial overhead in about
 3 or 4 different places.
 


 Maybe it costs a hash table lookup, but apart from that, you're saving
 space and time for marshaling additional instances. Of course, this is
 different with structs. But structs are value types.
 


which side are you arguing there? 

OTOH pointers to struct are not value types...


 interfaces are not supported either.
 


 But supporting interfaces would be very simple.
 


It wouldn't be hard in the current form (you would add a mixin to the interface 
as well) but the non-mixin, outside in approach would have all sorts of
interesting 
issues like how to get the correct sterilizer function.
Jun 14 2009
parent reply grauzone <none example.net> writes:
BCS wrote:

 Well, I can switch the default but, in my experience, most of the
 time repetition doesn't matter. I also dissagree on the "relatively
 useless


 Oh really?


 
 I haven't used a graph data structure in some time. Most of them have 
 been trees. And the cases I can think of, the repeated reference bit has 
 been central the the structure so the chances of getting it wrong (or of 
 missing it under test) are about nil.


IMO, most tree-like structures are still full graphs in memory, because 
they often contain "parent" pointers, or point back to a parent 
indirectly (e.g. even if a generic tree data structure is implemented 
without parent pointers, the data element itself might contain such 
pointers).


 OTOH pointers to struct are not value types...


Pointers are a whole different thing. A pointer can still point to a 
"value" type, because that struct might be embedded within an object (a 
class member that's a struct).


 interfaces are not supported either.


 But supporting interfaces would be very simple.


 
 It wouldn't be hard in the current form (you would add a mixin to the 
 interface as well) but the non-mixin, outside in approach would have all 
 sorts of interesting issues like how to get the correct sterilizer 
 function.


Huh? You can simple cast the interface to an object. And then cast the 
object to the serializeable type. You need to be able to do that anyway, 
because object references can be of the type "Object", and there's no 
way you'd add your serialize mixin to Object.

Also, is you writing "sterilizer" a typo or not?
Jun 16 2009
parent reply BCS <none anon.com> writes:
Hello grauzone,


 BCS wrote:
 

 Well, I can switch the default but, in my experience, most of the
 time repetition doesn't matter. I also dissagree on the "relatively
 useless
 


 Oh really?
 


 I haven't used a graph data structure in some time. Most of them have
 been trees. And the cases I can think of, the repeated reference bit
 has been central the the structure so the chances of getting it wrong
 (or of missing it under test) are about nil.
 


 IMO, most tree-like structures are still full graphs in memory,
 because they often contain "parent" pointers, or point back to a
 parent indirectly (e.g. even if a generic tree data structure is
 implemented without parent pointers, the data element itself might
 contain such pointers).


I'm referring to data structure that I could add serialization to, e.i. ones 
where I would know of they have parent references. I still stand by my
assertion.


 

 OTOH pointers to struct are not value types...
 


 Pointers are a whole different thing. A pointer can still point to a
 "value" type, because that struct might be embedded within an object
 (a class member that's a struct).
 


pointers to members won't be supported any time soon.


 interfaces are not supported either.
 


 But supporting interfaces would be very simple.
 


 It wouldn't be hard in the current form (you would add a mixin to the
 interface as well) but the non-mixin, outside in approach would have
 all sorts of interesting issues like how to get the correct
 sterilizer function.
 


 Huh? You can simple cast the interface to an object.


That is not safe. not all interface instances are D objects.


 And then cast the
 object to the serializeable type.


Cast only works if you know /at compile time/ what type to cast to so I don't 
think that's going to work. 


 You need to be able to do that
 anyway, because object references can be of the type "Object", and
 there's no way you'd add your serialize mixin to Object.


And that just brought up another issue: how do you serialize a class that 
only ever shows up as a base class reference? The lib has no way to /find/ 
the type at compile time so it has no way to generate code to deal with it.


 
 Also, is you writing "sterilizer" a typo or not?


typo (is it in the lib or just this thread?) I'd be even worse without a 
spellchecker :(
Jun 16 2009
parent reply grauzone <none example.net> writes:
 Huh? You can simple cast the interface to an object.


 
 That is not safe. not all interface instances are D objects.


There are people who care for COM and C++ interfaces? COM is Windows 
specific, and C++ vtables are... uh, I don't know, 
platform/architecture/compiler vendor specific?

In any case, serializable objects shouldn't contain references to such 
interfaces in the first place.


 And that just brought up another issue: how do you serialize a class 
 that only ever shows up as a base class reference? The lib has no way to 
 /find/ the type at compile time so it has no way to generate code to 
 deal with it.


But you already handle this. One of your mixins contains a static this 
ctor (which, btw., makes it impossible to use serializable types in 
cyclic dependent modules). It seems right now this ctor is only for 
registering the demarshaller function, but the same can be done with the 
marshaller function.
Jun 16 2009
parent reply BCS <none anon.com> writes:
Hello grauzone,


 Huh? You can simple cast the interface to an object.
 


 That is not safe. not all interface instances are D objects.
 


 There are people who care for COM and C++ interfaces? COM is Windows
 specific, and C++ vtables are... uh, I don't know,
 platform/architecture/compiler vendor specific?
 
 In any case, serializable objects shouldn't contain references to such
 interfaces in the first place.
 


I think there are ways to have a D interface implemented by a non D object.


 And that just brought up another issue: how do you serialize a class
 that only ever shows up as a base class reference? The lib has no way
 to /find/ the type at compile time so it has no way to generate code
 to deal with it.
 


 But you already handle this. One of your mixins contains a static this
 ctor (which, btw., makes it impossible to use serializable types in
 cyclic dependent modules). It seems right now this ctor is only for
 registering the demarshaller function, but the same can be done with
 the marshaller function.
 


The demarshaller function is indexed via a string derived from the original 
object. What would the marshaller function key on? The best I can think of 
right now is the typeinfo and as of now, that's broken under DLLs

---

The solution I have now works. Unless someone can show an intractable problem 
with it I'm keeping it. The only significant issue so far voiced is the concern 
about what all that mixin is adding and I think I can kill that one with 
some minor refactoring and documentation.

p.s. could you please not delete the citation line from the quote, it makes 
it easier (at least with good NG clients) for people to find replies to there 
posts.
Jun 16 2009
next sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Tue, Jun 16, 2009 at 8:50 PM, BCS<none anon.com> wrote:

 Hello grauzone,


 Huh? You can simple cast the interface to an object.


 That is not safe. not all interface instances are D objects.


 There are people who care for COM and C++ interfaces? COM is Windows
 specific, and C++ vtables are... uh, I don't know,
 platform/architecture/compiler vendor specific?

 In any case, serializable objects shouldn't contain references to such
 interfaces in the first place.


 I think there are ways to have a D interface implemented by a non D object.


He listed the only two possibilities: COM objects and extern(C++)
interfaces in D2.

typeid(Interface).classinfo.flags & 1 if an interface is COM at least.
 I don't know if there is any runtime info to indicate whether an
interface is C++ or not.
Jun 16 2009
prev sibling parent reply grauzone <none example.net> writes:
BCS wrote:


 The demarshaller function is indexed via a string derived from the 
 original object. What would the marshaller function key on? The best I 
 can think of right now is the typeinfo and as of now, that's broken 
 under DLLs


DLLs are broken in general. There are many more problems associated with 
them, and you won't be happy with them. I write all my code with the 
assumption in mind, that TypeInfos/ClassInfos for the same type always 
are the same instance.
Jun 18 2009
parent reply BCS <ao pathlink.com> writes:
Reply to grauzone,


 BCS wrote:
 

 The demarshaller function is indexed via a string derived from the
 original object. What would the marshaller function key on? The best
 I can think of right now is the typeinfo and as of now, that's broken
 under DLLs
 


 DLLs are broken in general. There are many more problems associated
 with them, and you won't be happy with them. I write all my code with
 the assumption in mind, that TypeInfos/ClassInfos for the same type
 always are the same instance.
 


On second pass, even putting DLLs aside, I can't count on typeinfo being 
the same in both sides because I can't even count on them being in the same 
process, exe or even under the same compiler, OS or CPU.

Can you get the mangled name of an object instance at runtime via typeinfo?
Jun 18 2009
parent reply grauzone <none example.net> writes:
BCS wrote:

 Reply to grauzone,
 

 BCS wrote:


 The demarshaller function is indexed via a string derived from the
 original object. What would the marshaller function key on? The best
 I can think of right now is the typeinfo and as of now, that's broken
 under DLLs


 DLLs are broken in general. There are many more problems associated
 with them, and you won't be happy with them. I write all my code with
 the assumption in mind, that TypeInfos/ClassInfos for the same type
 always are the same instance.


 
 On second pass, even putting DLLs aside, I can't count on typeinfo being 
 the same in both sides because I can't even count on them being in the 
 same process, exe or even under the same compiler, OS or CPU.


For that, you had to write the TypeInfo pointer into the serialized data 
stream, which obviously is not going to work. Obviously, you need some 
type identification, which is independent from the compiled program binary.

But in-memory, using TypeInfo/ClassInfo as unique identifier is no 
problem. Except if you somehow got 2 or more D runtimes in your process. 
That's only possible when using the broken DLL support. Use DDL instead.


 Can you get the mangled name of an object instance at runtime via typeinfo?


Not that I know of. IMHO, ClassInfo.name() is good enough.

But if you don't like it, just keep using mangleof. You obviously have 
compile time access to the serializeable type, e.g.:

char[][ClassInfo] TypeMangledNames;

template SerializeMixin() {
    static this() {
       TypeMangledNames[typeof(this).classinfo] = typeof(this).mangleof;
    }
}
Jun 18 2009
parent reply BCS <ao pathlink.com> writes:
Reply to grauzone,


 Can you get the mangled name of an object instance at runtime via
 typeinfo?
 


 Not that I know of. IMHO, ClassInfo.name() is good enough.
 
 But if you don't like it, just keep using mangleof. You obviously have
 compile time access to the serializeable type, e.g.:
 
 char[][ClassInfo] TypeMangledNames;
 
 template SerializeMixin() {
 static this() { TypeMangledNames[typeof(this).classinfo] =
typeof(this).mangleof; 


}

 }
 


I'm looking at what it would take to get by without a mixin in the types 
(someone objected to that) without loosing anything I want. with a mixin 
in the class, I've already got a solution.
Jun 18 2009
parent grauzone <none example.net> writes:
 I'm looking at what it would take to get by without a mixin in the types 
 (someone objected to that) without loosing anything I want. with a mixin 
 in the class, I've already got a solution.


Though I never questioned that you do need compile time access to the 
serialized code, and that (at least if you don't want to force the user 
to implement interfaces) some code knowing the static type must be run 
at initialization.
Jun 18 2009
prev sibling parent reply Christopher Wright <dhasenan gmail.com> writes:
grauzone wrote:

 Is there any real reason for all those mixins?


 which ones?


 
 All used by the user. That would be Serializable and SerializableRecuring.


Also, what is curing in this context, and why would you need to do it 
multiple times?
Jun 11 2009
parent BCS <none anon.com> writes:
Hello Christopher,


 SerializableRecuring.


 
 Also, what is curing in this context, and why would you need to do it
 multiple times?
 


My spelling sucks: s/Recuring/Recurring/
Jun 11 2009