www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Chances of D getting proper runtime reflection?

reply Jacob Carlborg <doob me.com> writes:
What are the chances of D getting proper runtime reflection? Something 
like this:

class Foo
{
     private int a;

     private void bar (int i)
     {
         a = i;
     }
}

auto foo = new Foo;
Object o = foo;
o.setInstanceVariable("a", 3);
assert(foo.a == 3);

This would be useful as well:

o.send("bar", 5);
assert(foo.a == 5);
assert(foo.a == o.getInstanceVariable("a"));

Currently I have most use of the first example. These example should 
work regardless if the fields/methods are private, protected or public.

As far as I know this wouldn't require any changes to the language, only 
the compiler and runtime would need to be modified. To me, it doesn't 
look that complicated to implement, the runtime just needs to store an 
associative arrays that maps names of instance variables and methods to 
their addresses. This could probably be stored in the ClassInfo. 
Something like this:

Foo.classinfo.setInstanceVariable(foo, "a", 3);

And this could be shorted to:

o.setInstanceVariable("a", 3);

"setInstanceVariable" could just forward the call to the class info.

-- 
/Jacob Carlborg
Aug 21 2011
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
Jacob Carlborg:


 class Foo
 {
      private int a;
 
      private void bar (int i)
      {
          a = i;
      }
 }
 
 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);


Is this related to the Delphi feature I've recently shown?
http://blog.barrkel.com/2010/09/virtual-method-interception.html

I presume you will have to write:

 reflection class Foo
{
    private int a;
...

To be sure to not pay for this dictionary when you don't need it. In theory
this annotation is unnecessary.

Bye,
bearophile
Aug 21 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-21 13:41, bearophile wrote:

 Jacob Carlborg:


 class Foo
 {
       private int a;

       private void bar (int i)
       {
           a = i;
       }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);


 Is this related to the Delphi feature I've recently shown?
 http://blog.barrkel.com/2010/09/virtual-method-interception.html


I don't think so, but it might help.


 I presume you will have to write:

  reflection class Foo
 {
      private int a;
 ...

 To be sure to not pay for this dictionary when you don't need it. In theory
this annotation is unnecessary.

 Bye,
 bearophile


That's not an option because it needs to work on third party types as 
well. BTW, you're already paying for the class info.

-- 
/Jacob Carlborg
Aug 21 2011
parent reply "Marco Leise" <Marco.Leise gmx.de> writes:
Am 21.08.2011, 14:11 Uhr, schrieb Jacob Carlborg <doob me.com>:


 BTW, you're already paying for the class info.


That could be a necessity of D's OOP implementation to fulfill the  
specification and while it is there that data was exported. Delphi used to  
have a compiler switch to include RTTI. (If I remember correctly without  
the compiler switch the pointer to RTTI was simply null.) Do you think  
that is an option? I figure if a third party library wanted it's classes  
to be reflected at runtime they would compile their code with that switch  
on. I don't quite know what your use-case is. For object serialization in  
Java there is an interface that has to be implemented. This would be  
similar to the  reflected annotation bearophile mentioned, because it is  
mostly a tag. RTTI for specific classes is not useless at all. I have a  
module with a lot of classes in a flat hierarchy. They are the typical  
case of several "commands" inheriting from a base class/interface  
declaring an "execute" method and such. Since I have the complete list of  
commands I could annotate them all with  reflected to set their properties  
 from some XML definition.
The usefulness of an annotation based RTTI depends solely on how many use  
cases it can cover. It doesn't hurt anyone who doesn't want to use it. The  
same goes for the compiler switch. Full reflection like in Java ... oh  
wait. Do you plan on writing a debugger? That's the only use case I can  
come up with right now?

- Marco
Aug 21 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-21 14:55, Marco Leise wrote:

 Am 21.08.2011, 14:11 Uhr, schrieb Jacob Carlborg <doob me.com>:


 BTW, you're already paying for the class info.


 That could be a necessity of D's OOP implementation to fulfill the
 specification and while it is there that data was exported. Delphi used
 to have a compiler switch to include RTTI. (If I remember correctly
 without the compiler switch the pointer to RTTI was simply null.) Do you
 think that is an option? I figure if a third party library wanted it's
 classes to be reflected at runtime they would compile their code with
 that switch on. I don't quite know what your use-case is. For object
 serialization in Java there is an interface that has to be implemented.
 This would be similar to the  reflected annotation bearophile mentioned,
 because it is mostly a tag. RTTI for specific classes is not useless at
 all. I have a module with a lot of classes in a flat hierarchy. They are
 the typical case of several "commands" inheriting from a base
 class/interface declaring an "execute" method and such. Since I have the
 complete list of commands I could annotate them all with  reflected to
 set their properties from some XML definition.
 The usefulness of an annotation based RTTI depends solely on how many
 use cases it can cover. It doesn't hurt anyone who doesn't want to use
 it. The same goes for the compiler switch. Full reflection like in Java
 ... oh wait. Do you plan on writing a debugger? That's the only use case
 I can come up with right now?

 - Marco


It would help for my serialization library. Currently I can only get/set 
fields based on compile time strings via .tupleof. Currently I'm looping 
over the tuple and calling the archive, like: "given me the value of 
field 'foo'". Instead the archive could run through the data and to 
something this: "ok, now I'm at field 'foo', I'm just setting the value 
of the field".

-- 
/Jacob Carlborg
Aug 21 2011
prev sibling next sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 21/08/2011 11:55, Jacob Carlborg wrote:

 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
 private int a;

 private void bar (int i)
 {
 a = i;
 }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language, only
 the compiler and runtime would need to be modified. To me, it doesn't
 look that complicated to implement, the runtime just needs to store an
 associative arrays that maps names of instance variables and methods to
 their addresses. This could probably be stored in the ClassInfo.
 Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.


You can actually do all of this now (albeit using a mixin), there's just 
no standard way of doing it. All you need to do is use compile time 
reflection to pull out all the fields/methods then store them for use at 
runtime. It would be good to get this functionality into phobos so 
there's a standard way of doing it though. The plus side to having to 
use a mixin is you don't get loads of bloat from classes you never want 
to reflect at runtime.

-- 
Robert
http://octarineparrot.com/
Aug 21 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-21 13:42, Robert Clipsham wrote:

 On 21/08/2011 11:55, Jacob Carlborg wrote:

 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
 private int a;

 private void bar (int i)
 {
 a = i;
 }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language, only
 the compiler and runtime would need to be modified. To me, it doesn't
 look that complicated to implement, the runtime just needs to store an
 associative arrays that maps names of instance variables and methods to
 their addresses. This could probably be stored in the ClassInfo.
 Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.


 You can actually do all of this now (albeit using a mixin), there's just
 no standard way of doing it. All you need to do is use compile time
 reflection to pull out all the fields/methods then store them for use at
 runtime. It would be good to get this functionality into phobos so
 there's a standard way of doing it though. The plus side to having to
 use a mixin is you don't get loads of bloat from classes you never want
 to reflect at runtime.


Yes, I know. But the point is to be able to do it without the use of 
mixins. It needs to work with third party types, otherwise there no use. 
It also needs to work when the static type is Object.

You're already paying for the class info, which already contains 
getMembers. But this methods always returns an empty array and as far as 
I can see, it's not possible to set/get fields or call methods via these 
member instances.

-- 
/Jacob Carlborg
Aug 21 2011
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 21 Aug 2011 15:16:11 +0300, Jacob Carlborg <doob me.com> wrote:


 Yes, I know. But the point is to be able to do it without the use of  
 mixins. It needs to work with third party types, otherwise there no use.  
 It also needs to work when the static type is Object.

 You're already paying for the class info, which already contains  
 getMembers. But this methods always returns an empty array and as far as  
 I can see, it's not possible to set/get fields or call methods via these  
 member instances.


I think the best way to do this is to improve compile-time reflection to  
the point where it'd be possible to enumerate all declarations in a module  
and create run-time reflection information for them - then you'd just need  
to specify the modules for which you'd like run-time reflection. One thing  
to note is that there are use cases where more run-time information about  
a program's internals may not be desired, e.g. with commercial  
closed-source applications, so this would need to be optional.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Aug 21 2011
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 21-08-2011 15:13, Vladimir Panteleev wrote:

 On Sun, 21 Aug 2011 15:16:11 +0300, Jacob Carlborg <doob me.com> wrote:


 Yes, I know. But the point is to be able to do it without the use of
 mixins. It needs to work with third party types, otherwise there no
 use. It also needs to work when the static type is Object.

 You're already paying for the class info, which already contains
 getMembers. But this methods always returns an empty array and as far
 as I can see, it's not possible to set/get fields or call methods via
 these member instances.


 I think the best way to do this is to improve compile-time reflection to
 the point where it'd be possible to enumerate all declarations in a
 module and create run-time reflection information for them - then you'd
 just need to specify the modules for which you'd like run-time
 reflection. One thing to note is that there are use cases where more
 run-time information about a program's internals may not be desired,
 e.g. with commercial closed-source applications, so this would need to
 be optional.


I really don't think this is a valid cause of concern in this day and 
age. Look at Java and .NET. They're doing just fine even with all 
metadata being available. Plenty of commercial C++ products ship with 
RTTI these days, too (AAA games, even).

What I mean to say is, I don't think we should let this "concern" hamper 
runtime reflection in D, because it evidently is not as big of an issue 
as people have made it out to be in the past. Security through obscurity 
is bad, as the saying goes.

Regards,
Alex
Aug 21 2011
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 21 Aug 2011 16:31:14 +0300, Alex Rønne Petersen  
<xtzgzorex gmail.com> wrote:


 I really don't think this is a valid cause of concern in this day and  
 age.


Perhaps not for you.


 Look at Java and .NET. They're doing just fine even with all metadata  
 being available.


And that's why obfuscators exist.


 Plenty of commercial C++ products ship with RTTI these days, too (AAA  
 games, even).


Some people care about this, some don't. Don't forget that many of these  
AAA games you speak of come bundled with invasive security software, with  
rootkit/spyware traits. Wouldn't you say that "security through obscurity"  
is the lesser of two evils?


 What I mean to say is, I don't think we should let this "concern" hamper  
 runtime reflection in D, because it evidently is not as big of an issue  
 as people have made it out to be in the past. Security through obscurity  
 is bad, as the saying goes.


Do you speak for everyone, then?

I certainly don't want to give any extra help to crackers, cheaters and  
griefers. It's stupid to ignore such things, and doing so does a  
disservice to your users.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Aug 21 2011
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-21 15:13, Vladimir Panteleev wrote:

 On Sun, 21 Aug 2011 15:16:11 +0300, Jacob Carlborg <doob me.com> wrote:


 Yes, I know. But the point is to be able to do it without the use of
 mixins. It needs to work with third party types, otherwise there no
 use. It also needs to work when the static type is Object.

 You're already paying for the class info, which already contains
 getMembers. But this methods always returns an empty array and as far
 as I can see, it's not possible to set/get fields or call methods via
 these member instances.


 I think the best way to do this is to improve compile-time reflection to
 the point where it'd be possible to enumerate all declarations in a
 module and create run-time reflection information for them - then you'd
 just need to specify the modules for which you'd like run-time
 reflection. One thing to note is that there are use cases where more
 run-time information about a program's internals may not be desired,
 e.g. with commercial closed-source applications, so this would need to
 be optional.


This is for a serialization library where third party types need to be 
serializable. I don't like the idea of annotate everything that should 
be serializable.

-- 
/Jacob Carlborg
Aug 21 2011
next sibling parent reply Robert Clipsham <robert octarineparrot.com> writes:
On 21/08/2011 16:05, Jacob Carlborg wrote:

 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.


You don't need to annotate it:

void serialize(T)(T something);

You have access to all the CT reflection stuff, and can generate the 
necessary runtime reflection with that.

-- 
Robert
http://octarineparrot.com/
Aug 21 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-21 17:30, Robert Clipsham wrote:

 On 21/08/2011 16:05, Jacob Carlborg wrote:

 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.


 You don't need to annotate it:

 void serialize(T)(T something);

 You have access to all the CT reflection stuff, and can generate the
 necessary runtime reflection with that.


I already have a fully working serialization library. I'm hoping to make 
it better. The problem is when (de)serializing through a base class 
reference. Currently you need to register the subclass with the 
serializer. If T is Base and the actually runtime type is Sub it won't work.

-- 
/Jacob Carlborg
Aug 21 2011
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/21/11 1:19 PM, Jacob Carlborg wrote:

 On 2011-08-21 17:30, Robert Clipsham wrote:

 On 21/08/2011 16:05, Jacob Carlborg wrote:

 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.


 You don't need to annotate it:

 void serialize(T)(T something);

 You have access to all the CT reflection stuff, and can generate the
 necessary runtime reflection with that.


 I already have a fully working serialization library. I'm hoping to make
 it better. The problem is when (de)serializing through a base class
 reference. Currently you need to register the subclass with the
 serializer. If T is Base and the actually runtime type is Sub it won't
 work.


If we get to the point where adding one line of code inside or 
(preferably) outside a type to make it serializable, that would be great.

Andrei
Aug 21 2011
next sibling parent reply dsimcha <dsimcha yahoo.com> writes:
On 8/21/2011 6:35 PM, Andrei Alexandrescu wrote:

 On 8/21/11 1:19 PM, Jacob Carlborg wrote:

 On 2011-08-21 17:30, Robert Clipsham wrote:

 On 21/08/2011 16:05, Jacob Carlborg wrote:

 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.


 You don't need to annotate it:

 void serialize(T)(T something);

 You have access to all the CT reflection stuff, and can generate the
 necessary runtime reflection with that.


 I already have a fully working serialization library. I'm hoping to make
 it better. The problem is when (de)serializing through a base class
 reference. Currently you need to register the subclass with the
 serializer. If T is Base and the actually runtime type is Sub it won't
 work.


 If we get to the point where adding one line of code inside or
 (preferably) outside a type to make it serializable, that would be great.

 Andrei


I've been playing with Jacob's code and we're already at that point. 
Example (requires Jacob's Orange library to be installed):

import std.stdio, orange.serialization._, orange.serialization.archives._,
     std.exception;

class Base {
     string baseString;
}

class Derived : Base {
     string derivedString;
}

void main() {
     auto archiver = new XMLArchive!char();
     auto serializer = new Serializer(archiver);
     serializer.register!Derived();

     auto derived = new Derived;
     derived.baseString = "Base";
     derived.derivedString = "Derived";

     Base baseRef = derived;
     auto serialized = serializer.serialize(baseRef);
     auto baseRef2 = serializer.deserialize!(Base)(serialized);

     enforce(baseRef2.classinfo is Derived.classinfo);
     auto derived2 = cast(Derived) baseRef;
     writeln(derived2.baseString, ' ', derived2.derivedString);
}


The registration is not even necessary if you're serializing a class 
with a compile time type identical to its runtime type.  Of course the 
situation gets a little more complicated when classes have references to 
other classes (you need to register everything that could be 
transitively reachable) but it's still pretty good.
Aug 21 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/21/11 6:03 PM, dsimcha wrote:

 On 8/21/2011 6:35 PM, Andrei Alexandrescu wrote:

 On 8/21/11 1:19 PM, Jacob Carlborg wrote:

 On 2011-08-21 17:30, Robert Clipsham wrote:

 On 21/08/2011 16:05, Jacob Carlborg wrote:

 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.


 You don't need to annotate it:

 void serialize(T)(T something);

 You have access to all the CT reflection stuff, and can generate the
 necessary runtime reflection with that.


 I already have a fully working serialization library. I'm hoping to make
 it better. The problem is when (de)serializing through a base class
 reference. Currently you need to register the subclass with the
 serializer. If T is Base and the actually runtime type is Sub it won't
 work.


 If we get to the point where adding one line of code inside or
 (preferably) outside a type to make it serializable, that would be great.

 Andrei


 I've been playing with Jacob's code and we're already at that point.
 Example (requires Jacob's Orange library to be installed):

 import std.stdio, orange.serialization._, orange.serialization.archives._,
 std.exception;

 class Base {
 string baseString;
 }

 class Derived : Base {
 string derivedString;
 }

 void main() {
 auto archiver = new XMLArchive!char();
 auto serializer = new Serializer(archiver);
 serializer.register!Derived();

 auto derived = new Derived;
 derived.baseString = "Base";
 derived.derivedString = "Derived";

 Base baseRef = derived;
 auto serialized = serializer.serialize(baseRef);
 auto baseRef2 = serializer.deserialize!(Base)(serialized);

 enforce(baseRef2.classinfo is Derived.classinfo);
 auto derived2 = cast(Derived) baseRef;
 writeln(derived2.baseString, ' ', derived2.derivedString);
 }


 The registration is not even necessary if you're serializing a class
 with a compile time type identical to its runtime type. Of course the
 situation gets a little more complicated when classes have references to
 other classes (you need to register everything that could be
 transitively reachable) but it's still pretty good.


That's not how it should work. The line of code needs to be declarative, 
not imperative. "I want this class to be serializable" as opposed to "I 
want to serialize this class with this particular archiver".

Andrei
Aug 22 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-22 21:08, Andrei Alexandrescu wrote:

 On 8/21/11 6:03 PM, dsimcha wrote:

 On 8/21/2011 6:35 PM, Andrei Alexandrescu wrote:

 On 8/21/11 1:19 PM, Jacob Carlborg wrote:

 On 2011-08-21 17:30, Robert Clipsham wrote:

 On 21/08/2011 16:05, Jacob Carlborg wrote:

 This is for a serialization library where third party types need
 to be
 serializable. I don't like the idea of annotate everything that
 should
 be serializable.


 You don't need to annotate it:

 void serialize(T)(T something);

 You have access to all the CT reflection stuff, and can generate the
 necessary runtime reflection with that.


 I already have a fully working serialization library. I'm hoping to
 make
 it better. The problem is when (de)serializing through a base class
 reference. Currently you need to register the subclass with the
 serializer. If T is Base and the actually runtime type is Sub it won't
 work.


 If we get to the point where adding one line of code inside or
 (preferably) outside a type to make it serializable, that would be
 great.

 Andrei


 I've been playing with Jacob's code and we're already at that point.
 Example (requires Jacob's Orange library to be installed):

 import std.stdio, orange.serialization._,
 orange.serialization.archives._,
 std.exception;

 class Base {
 string baseString;
 }

 class Derived : Base {
 string derivedString;
 }

 void main() {
 auto archiver = new XMLArchive!char();
 auto serializer = new Serializer(archiver);
 serializer.register!Derived();

 auto derived = new Derived;
 derived.baseString = "Base";
 derived.derivedString = "Derived";

 Base baseRef = derived;
 auto serialized = serializer.serialize(baseRef);
 auto baseRef2 = serializer.deserialize!(Base)(serialized);

 enforce(baseRef2.classinfo is Derived.classinfo);
 auto derived2 = cast(Derived) baseRef;
 writeln(derived2.baseString, ' ', derived2.derivedString);
 }


 The registration is not even necessary if you're serializing a class
 with a compile time type identical to its runtime type. Of course the
 situation gets a little more complicated when classes have references to
 other classes (you need to register everything that could be
 transitively reachable) but it's still pretty good.


 That's not how it should work. The line of code needs to be declarative,
 not imperative. "I want this class to be serializable" as opposed to "I
 want to serialize this class with this particular archiver".

 Andrei


You mean you want something like this:

class Derived : Base
{
     mixin serializable;
}

Instead of:

serializer.register!Derived();

The only way I can think of how this could work is to store a global 
list of the serializable classes. Any other suggestion how this would 
work? Note that all this wouldn't be necessary if my original post was 
implemented.

-- 
/Jacob Carlborg
Aug 22 2011
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 08/22/2011 09:19 PM, Jacob Carlborg wrote:

 On 2011-08-22 21:08, Andrei Alexandrescu wrote:

 On 8/21/11 6:03 PM, dsimcha wrote:

 On 8/21/2011 6:35 PM, Andrei Alexandrescu wrote:

 On 8/21/11 1:19 PM, Jacob Carlborg wrote:

 On 2011-08-21 17:30, Robert Clipsham wrote:

 On 21/08/2011 16:05, Jacob Carlborg wrote:

 This is for a serialization library where third party types need
 to be
 serializable. I don't like the idea of annotate everything that
 should
 be serializable.


 You don't need to annotate it:

 void serialize(T)(T something);

 You have access to all the CT reflection stuff, and can generate the
 necessary runtime reflection with that.


 I already have a fully working serialization library. I'm hoping to
 make
 it better. The problem is when (de)serializing through a base class
 reference. Currently you need to register the subclass with the
 serializer. If T is Base and the actually runtime type is Sub it won't
 work.


 If we get to the point where adding one line of code inside or
 (preferably) outside a type to make it serializable, that would be
 great.

 Andrei


 I've been playing with Jacob's code and we're already at that point.
 Example (requires Jacob's Orange library to be installed):

 import std.stdio, orange.serialization._,
 orange.serialization.archives._,
 std.exception;

 class Base {
 string baseString;
 }

 class Derived : Base {
 string derivedString;
 }

 void main() {
 auto archiver = new XMLArchive!char();
 auto serializer = new Serializer(archiver);
 serializer.register!Derived();

 auto derived = new Derived;
 derived.baseString = "Base";
 derived.derivedString = "Derived";

 Base baseRef = derived;
 auto serialized = serializer.serialize(baseRef);
 auto baseRef2 = serializer.deserialize!(Base)(serialized);

 enforce(baseRef2.classinfo is Derived.classinfo);
 auto derived2 = cast(Derived) baseRef;
 writeln(derived2.baseString, ' ', derived2.derivedString);
 }


 The registration is not even necessary if you're serializing a class
 with a compile time type identical to its runtime type. Of course the
 situation gets a little more complicated when classes have references to
 other classes (you need to register everything that could be
 transitively reachable) but it's still pretty good.


 That's not how it should work. The line of code needs to be declarative,
 not imperative. "I want this class to be serializable" as opposed to "I
 want to serialize this class with this particular archiver".

 Andrei


 You mean you want something like this:

 class Derived : Base
 {
 mixin serializable;
 }

 Instead of:

 serializer.register!Derived();

 The only way I can think of how this could work is to store a global
 list of the serializable classes. Any other suggestion how this would
 work? Note that all this wouldn't be necessary if my original post was
 implemented.


I think that you maybe register too late in the process?


 auto archiver = new XMLArchive!char();
 auto serializer = new Serializer(archiver);
 serializer.register!Derived();


If it works that way, you have to register every class anew if you want 
to serialize it with another archiver, right?
Aug 22 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-22 22:00, Timon Gehr wrote:

 On 08/22/2011 09:19 PM, Jacob Carlborg wrote:

 On 2011-08-22 21:08, Andrei Alexandrescu wrote:

 On 8/21/11 6:03 PM, dsimcha wrote:

 On 8/21/2011 6:35 PM, Andrei Alexandrescu wrote:

 On 8/21/11 1:19 PM, Jacob Carlborg wrote:

 On 2011-08-21 17:30, Robert Clipsham wrote:

 On 21/08/2011 16:05, Jacob Carlborg wrote:

 This is for a serialization library where third party types need
 to be
 serializable. I don't like the idea of annotate everything that
 should
 be serializable.


 You don't need to annotate it:

 void serialize(T)(T something);

 You have access to all the CT reflection stuff, and can generate the
 necessary runtime reflection with that.


 I already have a fully working serialization library. I'm hoping to
 make
 it better. The problem is when (de)serializing through a base class
 reference. Currently you need to register the subclass with the
 serializer. If T is Base and the actually runtime type is Sub it
 won't
 work.


 If we get to the point where adding one line of code inside or
 (preferably) outside a type to make it serializable, that would be
 great.

 Andrei


 I've been playing with Jacob's code and we're already at that point.
 Example (requires Jacob's Orange library to be installed):

 import std.stdio, orange.serialization._,
 orange.serialization.archives._,
 std.exception;

 class Base {
 string baseString;
 }

 class Derived : Base {
 string derivedString;
 }

 void main() {
 auto archiver = new XMLArchive!char();
 auto serializer = new Serializer(archiver);
 serializer.register!Derived();

 auto derived = new Derived;
 derived.baseString = "Base";
 derived.derivedString = "Derived";

 Base baseRef = derived;
 auto serialized = serializer.serialize(baseRef);
 auto baseRef2 = serializer.deserialize!(Base)(serialized);

 enforce(baseRef2.classinfo is Derived.classinfo);
 auto derived2 = cast(Derived) baseRef;
 writeln(derived2.baseString, ' ', derived2.derivedString);
 }


 The registration is not even necessary if you're serializing a class
 with a compile time type identical to its runtime type. Of course the
 situation gets a little more complicated when classes have
 references to
 other classes (you need to register everything that could be
 transitively reachable) but it's still pretty good.


 That's not how it should work. The line of code needs to be declarative,
 not imperative. "I want this class to be serializable" as opposed to "I
 want to serialize this class with this particular archiver".

 Andrei


 You mean you want something like this:

 class Derived : Base
 {
 mixin serializable;
 }

 Instead of:

 serializer.register!Derived();

 The only way I can think of how this could work is to store a global
 list of the serializable classes. Any other suggestion how this would
 work? Note that all this wouldn't be necessary if my original post was
 implemented.


 I think that you maybe register too late in the process?

  > auto archiver = new XMLArchive!char();
  > auto serializer = new Serializer(archiver);
  > serializer.register!Derived();

 If it works that way, you have to register every class anew if you want
 to serialize it with another archiver, right?


Yes, that's how it currently works. I can make the "register" method 
static instead.

-- 
/Jacob Carlborg
Aug 22 2011
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/22/11 2:19 PM, Jacob Carlborg wrote:

 On 2011-08-22 21:08, Andrei Alexandrescu wrote:

 On 8/21/11 6:03 PM, dsimcha wrote:

 On 8/21/2011 6:35 PM, Andrei Alexandrescu wrote:

 On 8/21/11 1:19 PM, Jacob Carlborg wrote:

 On 2011-08-21 17:30, Robert Clipsham wrote:

 On 21/08/2011 16:05, Jacob Carlborg wrote:

 This is for a serialization library where third party types need
 to be
 serializable. I don't like the idea of annotate everything that
 should
 be serializable.


 You don't need to annotate it:

 void serialize(T)(T something);

 You have access to all the CT reflection stuff, and can generate the
 necessary runtime reflection with that.


 I already have a fully working serialization library. I'm hoping to
 make
 it better. The problem is when (de)serializing through a base class
 reference. Currently you need to register the subclass with the
 serializer. If T is Base and the actually runtime type is Sub it won't
 work.


 If we get to the point where adding one line of code inside or
 (preferably) outside a type to make it serializable, that would be
 great.

 Andrei


 I've been playing with Jacob's code and we're already at that point.
 Example (requires Jacob's Orange library to be installed):

 import std.stdio, orange.serialization._,
 orange.serialization.archives._,
 std.exception;

 class Base {
 string baseString;
 }

 class Derived : Base {
 string derivedString;
 }

 void main() {
 auto archiver = new XMLArchive!char();
 auto serializer = new Serializer(archiver);
 serializer.register!Derived();

 auto derived = new Derived;
 derived.baseString = "Base";
 derived.derivedString = "Derived";

 Base baseRef = derived;
 auto serialized = serializer.serialize(baseRef);
 auto baseRef2 = serializer.deserialize!(Base)(serialized);

 enforce(baseRef2.classinfo is Derived.classinfo);
 auto derived2 = cast(Derived) baseRef;
 writeln(derived2.baseString, ' ', derived2.derivedString);
 }


 The registration is not even necessary if you're serializing a class
 with a compile time type identical to its runtime type. Of course the
 situation gets a little more complicated when classes have references to
 other classes (you need to register everything that could be
 transitively reachable) but it's still pretty good.


 That's not how it should work. The line of code needs to be declarative,
 not imperative. "I want this class to be serializable" as opposed to "I
 want to serialize this class with this particular archiver".

 Andrei


 You mean you want something like this:

 class Derived : Base
 {
 mixin serializable;
 }

 Instead of:

 serializer.register!Derived();


Yes. Better yet:

class Derived : Base
{
     ...
}

mixin(serializable!Derived());

i.e. so you can add serialization to classes you can't edit.


 The only way I can think of how this could work is to store a global
 list of the serializable classes. Any other suggestion how this would
 work? Note that all this wouldn't be necessary if my original post was
 implemented.


A global structure sounds the way to go.


Andrei
Aug 22 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-22 22:26, Andrei Alexandrescu wrote:

 Yes. Better yet:

 class Derived : Base
 {
 ...
 }

 mixin(serializable!Derived());

 i.e. so you can add serialization to classes you can't edit.


Yeah, of course. So "The line of code needs to be declarative,
not imperative" was just a fancy way of saying that you want to move the 
class registration to global scope :)


 The only way I can think of how this could work is to store a global
 list of the serializable classes. Any other suggestion how this would
 work? Note that all this wouldn't be necessary if my original post was
 implemented.


 A global structure sounds the way to go.


 Andrei


Ok, then I just change "register" to a static method.

-- 
/Jacob Carlborg
Aug 22 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/22/11 11:30 PM, Jacob Carlborg wrote:

 On 2011-08-22 22:26, Andrei Alexandrescu wrote:

 Yes. Better yet:

 class Derived : Base
 {
 ...
 }

 mixin(serializable!Derived());

 i.e. so you can add serialization to classes you can't edit.


 Yeah, of course. So "The line of code needs to be declarative,
 not imperative" was just a fancy way of saying that you want to move the
 class registration to global scope :)


 The only way I can think of how this could work is to store a global
 list of the serializable classes. Any other suggestion how this would
 work? Note that all this wouldn't be necessary if my original post was
 implemented.


 A global structure sounds the way to go.


 Andrei


 Ok, then I just change "register" to a static method.


A static method of whom?

Andrei
Aug 22 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-23 08:52, Andrei Alexandrescu wrote:

 On 8/22/11 11:30 PM, Jacob Carlborg wrote:

 On 2011-08-22 22:26, Andrei Alexandrescu wrote:

 Yes. Better yet:

 class Derived : Base
 {
 ...
 }

 mixin(serializable!Derived());

 i.e. so you can add serialization to classes you can't edit.


 Yeah, of course. So "The line of code needs to be declarative,
 not imperative" was just a fancy way of saying that you want to move the
 class registration to global scope :)


 The only way I can think of how this could work is to store a global
 list of the serializable classes. Any other suggestion how this would
 work? Note that all this wouldn't be necessary if my original post was
 implemented.


 A global structure sounds the way to go.


 Andrei


 Ok, then I just change "register" to a static method.


 A static method of whom?

 Andrei


Well, "register" currently an instance method of Serializer so I would 
change it to be a static method of Serializer.

-- 
/Jacob Carlborg
Aug 23 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/23/11 12:55 AM, Jacob Carlborg wrote:

 On 2011-08-23 08:52, Andrei Alexandrescu wrote:

 On 8/22/11 11:30 PM, Jacob Carlborg wrote:

 Ok, then I just change "register" to a static method.


 A static method of whom?

 Andrei


 Well, "register" currently an instance method of Serializer so I would
 change it to be a static method of Serializer.


I think the ability of a class to be serialized would be independent of 
the notion of a serializer. To me, "this class is serializable" really 
means "this class has metadata associated with it that allows interested 
parties to serialize it". But perhaps this is splitting hairs.

Andrei
Aug 23 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-23 16:38, Andrei Alexandrescu wrote:

 On 8/23/11 12:55 AM, Jacob Carlborg wrote:

 On 2011-08-23 08:52, Andrei Alexandrescu wrote:

 On 8/22/11 11:30 PM, Jacob Carlborg wrote:

 Ok, then I just change "register" to a static method.


 A static method of whom?

 Andrei


 Well, "register" currently an instance method of Serializer so I would
 change it to be a static method of Serializer.


 I think the ability of a class to be serialized would be independent of
 the notion of a serializer. To me, "this class is serializable" really
 means "this class has metadata associated with it that allows interested
 parties to serialize it". But perhaps this is splitting hairs.

 Andrei


You don't want to have it in the class and don't want it in the 
serializer. I mean, it needs to be stored somewhere and I thought that a 
static method in Serializer would better than a completely global function.

Are you thinking about having another serialization library that can use 
this information as well? I'm not sure if that's good idea, different 
serialization implementations might need to do very different things 
with the information.

-- 
/Jacob Carlborg
Aug 23 2011
next sibling parent reply jdrewsen <jdrewsen nospam.com> writes:
Den 23-08-2011 17:03, Jacob Carlborg skrev:

 On 2011-08-23 16:38, Andrei Alexandrescu wrote:

 On 8/23/11 12:55 AM, Jacob Carlborg wrote:

 On 2011-08-23 08:52, Andrei Alexandrescu wrote:

 On 8/22/11 11:30 PM, Jacob Carlborg wrote:

 Ok, then I just change "register" to a static method.


 A static method of whom?

 Andrei


 Well, "register" currently an instance method of Serializer so I would
 change it to be a static method of Serializer.


 I think the ability of a class to be serialized would be independent of
 the notion of a serializer. To me, "this class is serializable" really
 means "this class has metadata associated with it that allows interested
 parties to serialize it". But perhaps this is splitting hairs.

 Andrei


 You don't want to have it in the class and don't want it in the
 serializer. I mean, it needs to be stored somewhere and I thought that a
 static method in Serializer would better than a completely global function.

 Are you thinking about having another serialization library that can use
 this information as well? I'm not sure if that's good idea, different
 serialization implementations might need to do very different things
 with the information.


It could be used for network transmissions. Correct me if I'm wrong but 
Orange serializes the entire object. When sending things over the 
network or when saving something to disk for that matter you most likely 
are only interested in serializing some of the fields of an object. I 
really think it would be nice to declaratively mark fields as 
serializable for certain purposes e.g.:

class Foo {
	int a;         // Send over network and saved to file
	int b;         // Saved to file
	ubyte[] cache; // not to be serialized
}

mixin(serialize!(Network, Foo, a);

mixin(serialize!(File, Foo, a);
mixin(serialize!(File, Foo, b);

// but still support not specifying each field
class Bar { int a; }
mixin(serialize!(File, Bar));


Another way would be to just declare a class as serializable and then 
for each serialization type (ie. Network, File) declare that they should 
skip a field. Actually I think I better like this approach since it 
would allow decoupling of serialization type and the declaration of 
serializability.

/Jonas
Aug 23 2011
next sibling parent reply "Marco Leise" <Marco.Leise gmx.de> writes:
Am 23.08.2011, 19:42 Uhr, schrieb jdrewsen <jdrewsen nospam.com>:


 Den 23-08-2011 17:03, Jacob Carlborg skrev:

 On 2011-08-23 16:38, Andrei Alexandrescu wrote:

 On 8/23/11 12:55 AM, Jacob Carlborg wrote:

 On 2011-08-23 08:52, Andrei Alexandrescu wrote:

 On 8/22/11 11:30 PM, Jacob Carlborg wrote:

 Ok, then I just change "register" to a static method.


 A static method of whom?

 Andrei


 Well, "register" currently an instance method of Serializer so I would
 change it to be a static method of Serializer.


 I think the ability of a class to be serialized would be independent of
 the notion of a serializer. To me, "this class is serializable" really
 means "this class has metadata associated with it that allows  
 interested
 parties to serialize it". But perhaps this is splitting hairs.

 Andrei


 You don't want to have it in the class and don't want it in the
 serializer. I mean, it needs to be stored somewhere and I thought that a
 static method in Serializer would better than a completely global  
 function.

 Are you thinking about having another serialization library that can use
 this information as well? I'm not sure if that's good idea, different
 serialization implementations might need to do very different things
 with the information.


 It could be used for network transmissions. Correct me if I'm wrong but  
 Orange serializes the entire object. When sending things over the  
 network or when saving something to disk for that matter you most likely  
 are only interested in serializing some of the fields of an object. I  
 really think it would be nice to declaratively mark fields as  
 serializable for certain purposes e.g.:

 class Foo {
 	int a;         // Send over network and saved to file
 	int b;         // Saved to file
 	ubyte[] cache; // not to be serialized
 }

 mixin(serialize!(Network, Foo, a);

 mixin(serialize!(File, Foo, a);
 mixin(serialize!(File, Foo, b);

 // but still support not specifying each field
 class Bar { int a; }
 mixin(serialize!(File, Bar));


 Another way would be to just declare a class as serializable and then  
 for each serialization type (ie. Network, File) declare that they should  
 skip a field. Actually I think I better like this approach since it  
 would allow decoupling of serialization type and the declaration of  
 serializability.

 /Jonas


These are good points Jonas. It's a good idea to read up on  
http://en.wikipedia.org/wiki/Serialization
I used Java for a while and I they have their reasons to require a  
'tag'/'annotation'/'interface' on serializable classes.

Java's choice to not make serializable a default is explained in short  
like this:
- structures like threads or files cannot maintain their semantics when  
serialized
- classes and libraries change over time, this adds extra work for the  
maintanance of serializable classes
- serialization means externalization of private attributes, even passwords

.NET adds the option to mark newly added fields as OptionalField, so a  
previously serialized instance can be loaded without error. NonSerialized  
marks a field as not serializable which is what Jonas mentioned. Like Java  
they work with interfaces to declare methods called after deserialization,  
so the 'cache' in the above example can be reinitialized correctly.
Aug 23 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-24 08:41, Marco Leise wrote:

 Am 23.08.2011, 19:42 Uhr, schrieb jdrewsen <jdrewsen nospam.com>:


 Den 23-08-2011 17:03, Jacob Carlborg skrev:

 On 2011-08-23 16:38, Andrei Alexandrescu wrote:

 On 8/23/11 12:55 AM, Jacob Carlborg wrote:

 On 2011-08-23 08:52, Andrei Alexandrescu wrote:

 On 8/22/11 11:30 PM, Jacob Carlborg wrote:

 Ok, then I just change "register" to a static method.


 A static method of whom?

 Andrei


 Well, "register" currently an instance method of Serializer so I would
 change it to be a static method of Serializer.


 I think the ability of a class to be serialized would be independent of
 the notion of a serializer. To me, "this class is serializable" really
 means "this class has metadata associated with it that allows
 interested
 parties to serialize it". But perhaps this is splitting hairs.

 Andrei


 You don't want to have it in the class and don't want it in the
 serializer. I mean, it needs to be stored somewhere and I thought that a
 static method in Serializer would better than a completely global
 function.

 Are you thinking about having another serialization library that can use
 this information as well? I'm not sure if that's good idea, different
 serialization implementations might need to do very different things
 with the information.


 It could be used for network transmissions. Correct me if I'm wrong
 but Orange serializes the entire object. When sending things over the
 network or when saving something to disk for that matter you most
 likely are only interested in serializing some of the fields of an
 object. I really think it would be nice to declaratively mark fields
 as serializable for certain purposes e.g.:

 class Foo {
 int a; // Send over network and saved to file
 int b; // Saved to file
 ubyte[] cache; // not to be serialized
 }

 mixin(serialize!(Network, Foo, a);

 mixin(serialize!(File, Foo, a);
 mixin(serialize!(File, Foo, b);

 // but still support not specifying each field
 class Bar { int a; }
 mixin(serialize!(File, Bar));


 Another way would be to just declare a class as serializable and then
 for each serialization type (ie. Network, File) declare that they
 should skip a field. Actually I think I better like this approach
 since it would allow decoupling of serialization type and the
 declaration of serializability.

 /Jonas


 These are good points Jonas. It's a good idea to read up on
 http://en.wikipedia.org/wiki/Serialization
 I used Java for a while and I they have their reasons to require a
 'tag'/'annotation'/'interface' on serializable classes.

 Java's choice to not make serializable a default is explained in short
 like this:
 - structures like threads or files cannot maintain their semantics when
 serialized
 - classes and libraries change over time, this adds extra work for the
 maintanance of serializable classes
 - serialization means externalization of private attributes, even passwords

 .NET adds the option to mark newly added fields as OptionalField, so a
 previously serialized instance can be loaded without error.
 NonSerialized marks a field as not serializable which is what Jonas
 mentioned. Like Java they work with interfaces to declare methods called
 after deserialization, so the 'cache' in the above example can be
 reinitialized correctly.


Orange supports NonSerialized fields and classes, (de)serialization 
events and several other ways to customize the serialization process. 
It's inspired by how serialization in .NET works.

For more information see the Orange unit tests: 
https://github.com/jacob-carlborg/orange/tree/master/tests

-- 
/Jacob Carlborg
Aug 23 2011
parent reply "Marco Leise" <Marco.Leise gmx.de> writes:
Am 24.08.2011, 08:58 Uhr, schrieb Jacob Carlborg <doob me.com>:


 On 2011-08-24 08:41, Marco Leise wrote:

 Am 23.08.2011, 19:42 Uhr, schrieb jdrewsen <jdrewsen nospam.com>:


 Den 23-08-2011 17:03, Jacob Carlborg skrev:

 On 2011-08-23 16:38, Andrei Alexandrescu wrote:

 On 8/23/11 12:55 AM, Jacob Carlborg wrote:

 On 2011-08-23 08:52, Andrei Alexandrescu wrote:

 On 8/22/11 11:30 PM, Jacob Carlborg wrote:

 Ok, then I just change "register" to a static method.


 A static method of whom?

 Andrei


 Well, "register" currently an instance method of Serializer so I  
 would
 change it to be a static method of Serializer.


 I think the ability of a class to be serialized would be independent  
 of
 the notion of a serializer. To me, "this class is serializable"  
 really
 means "this class has metadata associated with it that allows
 interested
 parties to serialize it". But perhaps this is splitting hairs.

 Andrei


 You don't want to have it in the class and don't want it in the
 serializer. I mean, it needs to be stored somewhere and I thought  
 that a
 static method in Serializer would better than a completely global
 function.

 Are you thinking about having another serialization library that can  
 use
 this information as well? I'm not sure if that's good idea, different
 serialization implementations might need to do very different things
 with the information.


 It could be used for network transmissions. Correct me if I'm wrong
 but Orange serializes the entire object. When sending things over the
 network or when saving something to disk for that matter you most
 likely are only interested in serializing some of the fields of an
 object. I really think it would be nice to declaratively mark fields
 as serializable for certain purposes e.g.:

 class Foo {
 int a; // Send over network and saved to file
 int b; // Saved to file
 ubyte[] cache; // not to be serialized
 }

 mixin(serialize!(Network, Foo, a);

 mixin(serialize!(File, Foo, a);
 mixin(serialize!(File, Foo, b);

 // but still support not specifying each field
 class Bar { int a; }
 mixin(serialize!(File, Bar));


 Another way would be to just declare a class as serializable and then
 for each serialization type (ie. Network, File) declare that they
 should skip a field. Actually I think I better like this approach
 since it would allow decoupling of serialization type and the
 declaration of serializability.

 /Jonas


 These are good points Jonas. It's a good idea to read up on
 http://en.wikipedia.org/wiki/Serialization
 I used Java for a while and I they have their reasons to require a
 'tag'/'annotation'/'interface' on serializable classes.

 Java's choice to not make serializable a default is explained in short
 like this:
 - structures like threads or files cannot maintain their semantics when
 serialized
 - classes and libraries change over time, this adds extra work for the
 maintanance of serializable classes
 - serialization means externalization of private attributes, even  
 passwords

 .NET adds the option to mark newly added fields as OptionalField, so a
 previously serialized instance can be loaded without error.
 NonSerialized marks a field as not serializable which is what Jonas
 mentioned. Like Java they work with interfaces to declare methods called
 after deserialization, so the 'cache' in the above example can be
 reinitialized correctly.


 Orange supports NonSerialized fields and classes, (de)serialization  
 events and several other ways to customize the serialization process.  
 It's inspired by how serialization in .NET works.

 For more information see the Orange unit tests:  
 https://github.com/jacob-carlborg/orange/tree/master/tests


Then I'll clearly use Orange when I have a need for serialization. Nice  
work!
I've taken a closer look at  
http://dsource.org/projects/orange/wiki/Features and I see that versioning  
is still missing. What are your plans there? What happens at the moment  
when I deserialize and object with missing/added fields?
Aug 24 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-24 10:10, Marco Leise wrote:

 Then I'll clearly use Orange when I have a need for serialization. Nice
 work!


Thanks.


 I've taken a closer look at
 http://dsource.org/projects/orange/wiki/Features and I see that
 versioning is still missing. What are your plans there? What happens at
 the moment when I deserialize and object with missing/added fields?


I actually haven't tested that with the new implementation. The idea is 
to have a callback that is called when a field is missing, or something 
else goes wrong. Then you can handle it however you want, throw an 
exception, do nothing or something else. The default callback will throw 
an exception.

You can manually implement versioning by adding a new field to a class, 
an integer, string or similar, containing a version number. Then you can 
implement the two methods "toData" and "fromData". When deserializing 
you being with deserializing the field containing the version and then 
you can do whatever you want depending on the version number.

-- 
/Jacob Carlborg
Aug 24 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-08-23 19:42, jdrewsen wrote:

 Den 23-08-2011 17:03, Jacob Carlborg skrev:

 On 2011-08-23 16:38, Andrei Alexandrescu wrote:

 On 8/23/11 12:55 AM, Jacob Carlborg wrote:

 On 2011-08-23 08:52, Andrei Alexandrescu wrote:

 On 8/22/11 11:30 PM, Jacob Carlborg wrote:

 Ok, then I just change "register" to a static method.


 A static method of whom?

 Andrei


 Well, "register" currently an instance method of Serializer so I would
 change it to be a static method of Serializer.


 I think the ability of a class to be serialized would be independent of
 the notion of a serializer. To me, "this class is serializable" really
 means "this class has metadata associated with it that allows interested
 parties to serialize it". But perhaps this is splitting hairs.

 Andrei


 You don't want to have it in the class and don't want it in the
 serializer. I mean, it needs to be stored somewhere and I thought that a
 static method in Serializer would better than a completely global
 function.

 Are you thinking about having another serialization library that can use
 this information as well? I'm not sure if that's good idea, different
 serialization implementations might need to do very different things
 with the information.


 It could be used for network transmissions. Correct me if I'm wrong but
 Orange serializes the entire object. When sending things over the
 network or when saving something to disk for that matter you most likely
 are only interested in serializing some of the fields of an object. I
 really think it would be nice to declaratively mark fields as
 serializable for certain purposes e.g.:


Yes, that's how it works, but there are several ways to customize the 
serialization process, see below. If you are interested, please look at 
the unit tests: https://github.com/jacob-carlborg/orange/tree/master/tests


 class Foo {
 int a; // Send over network and saved to file
 int b; // Saved to file
 ubyte[] cache; // not to be serialized
 }

 mixin(serialize!(Network, Foo, a);

 mixin(serialize!(File, Foo, a);
 mixin(serialize!(File, Foo, b);

 // but still support not specifying each field
 class Bar { int a; }
 mixin(serialize!(File, Bar));


 Another way would be to just declare a class as serializable and then
 for each serialization type (ie. Network, File) declare that they should
 skip a field. Actually I think I better like this approach since it
 would allow decoupling of serialization type and the declaration of
 serializability.

 /Jonas


Orange have several ways to solve this.

1. Mark a field as non-serialized

class Foo {
     int a;
     mixin NonSeriailzed!(a);
}

2. Mark a whole class as non-serialized

class Foo {
     mixin NonSeriailzed;
}

This will be handy for socket, thread, file and so on.

3. Custom serialization

class Foo {
     int a;

     void toData (Serializer serializer, Serializer.Data key) {
         // serialize the class how ever you want
     }

     void fromData (Serializer serializer, Serializer.Data key) {
         // deserialize the class how ever you want
     }
}

4. Custom non-intrusive serialization

class Foo {
     int a;
}

void serializeFoo (Foo foo, Serializer serializer, Serializer.Data key) {
     // serialize the class how ever you want
}

void deserializeFoo (Foo foo, Serializer serializer, Serializer.Data key) {
     // deserialize the class how ever you want
}

serializer.registerSerializer(Foo.classinfo.name, &serializeFoo);
serializer.registerDeserializer(Foo.classinfo.name, &deserializeFoo);

5. Events

class Foo {
     int a;

     void onSerializing () {
         // invoked before serializing of Foo
     }

     void onSerialized () {
         // invoked after serializing of Foo
     }

     void onDeserializing () {
         // invoked before deserializing of Foo
     }

     void onDeserialized () {
         // invoked after deserializing of Foo
     }

     mixin OnSerializing!(onSerializing);
     mixin OnSerialized!(onSerialized);
     mixin OnDeserializing!(onDeserializing);
     mixin OnDeserialized!(onDeserialized);
}



-- 
/Jacob Carlborg
Aug 23 2011
prev sibling parent reply "Martin Nowak" <dawg dawgfoto.de> writes:
On Tue, 23 Aug 2011 17:03:12 +0200, Jacob Carlborg <doob me.com> wrote:


 On 2011-08-23 16:38, Andrei Alexandrescu wrote:

 On 8/23/11 12:55 AM, Jacob Carlborg wrote:

 On 2011-08-23 08:52, Andrei Alexandrescu wrote:

 On 8/22/11 11:30 PM, Jacob Carlborg wrote:

 Ok, then I just change "register" to a static method.


 A static method of whom?

 Andrei


 Well, "register" currently an instance method of Serializer so I would
 change it to be a static method of Serializer.


 I think the ability of a class to be serialized would be independent of
 the notion of a serializer. To me, "this class is serializable" really
 means "this class has metadata associated with it that allows interested
 parties to serialize it". But perhaps this is splitting hairs.

 Andrei


 You don't want to have it in the class and don't want it in the  
 serializer. I mean, it needs to be stored somewhere and I thought that a  
 static method in Serializer would better than a completely global  
 function.

 Are you thinking about having another serialization library that can use  
 this information as well? I'm not sure if that's good idea, different  
 serialization implementations might need to do very different things  
 with the information.


I think the idea is to register something that provides the needed  
information at runtime and separate that from the serializer because it's  
useful on it's own.

Could be as simple as:

interface Serializable {
   string[] allMembers();
   Member getMember(string name);
}

interface Member {
   final void set(T)(T t) {
     setRaw(typeid(t), cast(void*)&t);
   }
   void setRaw(TypeInfo ti, void* p);

   final T get(T)() {
     return *cast(T*)enforce(getRaw(typeid(T)));
   }
   void* getRaw();
}

It's clearly interesting to make this available if you implement something  
along that line.
But I would not overstress it for there are dozen of different  
requirements profiles (performance, versioning, pre/post-serialize hooks).
So even if I am interested to use something similar for undo/redo or delta  
creation, I'm not sure one can find a one-for-all solution here
(i.e. the above approach seems way to dynamic for my needs).
Maybe Variant could help in setting/getting values.

martin
Aug 23 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-24 01:34, Martin Nowak wrote:

 On Tue, 23 Aug 2011 17:03:12 +0200, Jacob Carlborg <doob me.com> wrote:


 On 2011-08-23 16:38, Andrei Alexandrescu wrote:

 On 8/23/11 12:55 AM, Jacob Carlborg wrote:

 On 2011-08-23 08:52, Andrei Alexandrescu wrote:

 On 8/22/11 11:30 PM, Jacob Carlborg wrote:

 Ok, then I just change "register" to a static method.


 A static method of whom?

 Andrei


 Well, "register" currently an instance method of Serializer so I would
 change it to be a static method of Serializer.


 I think the ability of a class to be serialized would be independent of
 the notion of a serializer. To me, "this class is serializable" really
 means "this class has metadata associated with it that allows interested
 parties to serialize it". But perhaps this is splitting hairs.

 Andrei


 You don't want to have it in the class and don't want it in the
 serializer. I mean, it needs to be stored somewhere and I thought that
 a static method in Serializer would better than a completely global
 function.

 Are you thinking about having another serialization library that can
 use this information as well? I'm not sure if that's good idea,
 different serialization implementations might need to do very
 different things with the information.


 I think the idea is to register something that provides the needed
 information at runtime and separate that from the serializer because
 it's useful on it's own.

 Could be as simple as:

 interface Serializable {
 string[] allMembers();
 Member getMember(string name);
 }

 interface Member {
 final void set(T)(T t) {
 setRaw(typeid(t), cast(void*)&t);
 }
 void setRaw(TypeInfo ti, void* p);

 final T get(T)() {
 return *cast(T*)enforce(getRaw(typeid(T)));
 }
 void* getRaw();
 }

 It's clearly interesting to make this available if you implement
 something along that line.
 But I would not overstress it for there are dozen of different
 requirements profiles (performance, versioning, pre/post-serialize hooks).
 So even if I am interested to use something similar for undo/redo or
 delta creation, I'm not sure one can find a one-for-all solution here
 (i.e. the above approach seems way to dynamic for my needs).
 Maybe Variant could help in setting/getting values.

 martin


As soon as "setRaw" is called all static information is gone. I need to 
be able to iterate over T.tupleof.

-- 
/Jacob Carlborg
Aug 23 2011
next sibling parent reply "Martin Nowak" <dawg dawgfoto.de> writes:
On Wed, 24 Aug 2011 08:49:19 +0200, Jacob Carlborg <doob me.com> wrote:


 On 2011-08-24 01:34, Martin Nowak wrote:

 On Tue, 23 Aug 2011 17:03:12 +0200, Jacob Carlborg <doob me.com> wrote:


 On 2011-08-23 16:38, Andrei Alexandrescu wrote:

 On 8/23/11 12:55 AM, Jacob Carlborg wrote:

 On 2011-08-23 08:52, Andrei Alexandrescu wrote:

 On 8/22/11 11:30 PM, Jacob Carlborg wrote:

 Ok, then I just change "register" to a static method.


 A static method of whom?

 Andrei


 Well, "register" currently an instance method of Serializer so I  
 would
 change it to be a static method of Serializer.


 I think the ability of a class to be serialized would be independent  
 of
 the notion of a serializer. To me, "this class is serializable" really
 means "this class has metadata associated with it that allows  
 interested
 parties to serialize it". But perhaps this is splitting hairs.

 Andrei


 You don't want to have it in the class and don't want it in the
 serializer. I mean, it needs to be stored somewhere and I thought that
 a static method in Serializer would better than a completely global
 function.

 Are you thinking about having another serialization library that can
 use this information as well? I'm not sure if that's good idea,
 different serialization implementations might need to do very
 different things with the information.


 I think the idea is to register something that provides the needed
 information at runtime and separate that from the serializer because
 it's useful on it's own.

 Could be as simple as:

 interface Serializable {
 string[] allMembers();
 Member getMember(string name);
 }

 interface Member {
 final void set(T)(T t) {
 setRaw(typeid(t), cast(void*)&t);
 }
 void setRaw(TypeInfo ti, void* p);

 final T get(T)() {
 return *cast(T*)enforce(getRaw(typeid(T)));
 }
 void* getRaw();
 }

 It's clearly interesting to make this available if you implement
 something along that line.
 But I would not overstress it for there are dozen of different
 requirements profiles (performance, versioning, pre/post-serialize  
 hooks).
 So even if I am interested to use something similar for undo/redo or
 delta creation, I'm not sure one can find a one-for-all solution here
 (i.e. the above approach seems way to dynamic for my needs).
 Maybe Variant could help in setting/getting values.

 martin


 As soon as "setRaw" is called all static information is gone. I need to  
 be able to iterate over T.tupleof.


Not sure, I get what you mean.
Let me try to clarify this:

static Serializeable[TypeInfo] infos;

void register(T)() {
	infos[typeid(T)] = new SerializableImpl!T();
}

// Serializable binds static reflection to dynamic interface
class SerializeableImpl!T : Serializable {
	string[] allMembers() { array(__traits(allMembers, T)); }
	Member getMember(string name) {
	  if (mi = findMemberInfo(name)) {
             auto res = mi.clone(); // found immutable info, now create  
wrapper
	    res.address = this + mi.offset; // tell the clone where to find
	    return res;
	  }
	  return null;
	}
}

// Member binds purely static info to dynamic interface and an actual  
address
class MemberInfo!T : Member {
   override void setRaw(TypeInfo ti, void* p) {
     if (ti is typeid(T))
       *address = *cast(T*)p;
   }

   override void* getRaw(TypeInfo ti) {
     if (ti is typeid(T))
       return address;
   }

   T* address;
}

In object.d there is a lot of this partly implemented. Have a closer look  
at OffsetTypeInfo and MemberInfo_field.
You can definitely get all the information the compiler could provide for  
RTTI at compile time. If you try hard
enough you may even be able to store them at compile time. The only  
difference remaining is that you could not
access it through instance.classinfo but have to use a table lookup.

It would be great if there was a way to append data to classinfo at  
compile time. Like an extra pointer or an array of ubyte that can be set.
martin
Aug 24 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-25 05:31, Martin Nowak wrote:

 Not sure, I get what you mean.
 Let me try to clarify this:

 static Serializeable[TypeInfo] infos;

 void register(T)() {
 infos[typeid(T)] = new SerializableImpl!T();
 }

 // Serializable binds static reflection to dynamic interface
 class SerializeableImpl!T : Serializable {
 string[] allMembers() { array(__traits(allMembers, T)); }
 Member getMember(string name) {
 if (mi = findMemberInfo(name)) {
 auto res = mi.clone(); // found immutable info, now create wrapper
 res.address = this + mi.offset; // tell the clone where to find
 return res;
 }
 return null;
 }
 }

 // Member binds purely static info to dynamic interface and an actual
 address
 class MemberInfo!T : Member {
 override void setRaw(TypeInfo ti, void* p) {
 if (ti is typeid(T))
 *address = *cast(T*)p;
 }

 override void* getRaw(TypeInfo ti) {
 if (ti is typeid(T))
 return address;
 }

 T* address;
 }

 In object.d there is a lot of this partly implemented. Have a closer
 look at OffsetTypeInfo and MemberInfo_field.
 You can definitely get all the information the compiler could provide
 for RTTI at compile time. If you try hard
 enough you may even be able to store them at compile time. The only
 difference remaining is that you could not
 access it through instance.classinfo but have to use a table lookup.

 It would be great if there was a way to append data to classinfo at
 compile time. Like an extra pointer or an array of ubyte that can be set.
 martin


I have to take a good look at this but I can say right now that 
currently how "register" is implemented it needs access to instance 
methods and variables.

-- 
/Jacob Carlborg
Aug 25 2011
prev sibling parent reply "Martin Nowak" <dawg dawgfoto.de> writes:
It is actually already possible to store meta data in classinfos.

---
import std.stdio;

class A {
   int a;
   float b;
}

mixin template register(T) if(is(T == class)) {
   static this() {
     T.classinfo.m_offTi = [OffsetTypeInfo(0, typeid(int)),  
OffsetTypeInfo(4, typeid(float))];
   }
}

mixin register!A;

void main() {
   foreach(offTi; A.classinfo.offTi())
     writeln(offTi.offset, offTi.ti);
}
---

The compiler doesn't initialize the m_offTi array. But classinfos are in  
the initialized data section, at least for me, so you can write to them.

martin
Aug 24 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-25 06:30, Martin Nowak wrote:

 It is actually already possible to store meta data in classinfos.

 ---
 import std.stdio;

 class A {
 int a;
 float b;
 }

 mixin template register(T) if(is(T == class)) {
 static this() {
 T.classinfo.m_offTi = [OffsetTypeInfo(0, typeid(int)), OffsetTypeInfo(4,
 typeid(float))];
 }
 }

 mixin register!A;

 void main() {
 foreach(offTi; A.classinfo.offTi())
 writeln(offTi.offset, offTi.ti);
 }
 ---

 The compiler doesn't initialize the m_offTi array. But classinfos are in
 the initialized data section, at least for me, so you can write to them.

 martin


Seems to be very unsafe to rely on m_offTi in ClassInfo.

-- 
/Jacob Carlborg
Aug 25 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-22 00:35, Andrei Alexandrescu wrote:

 On 8/21/11 1:19 PM, Jacob Carlborg wrote:

 On 2011-08-21 17:30, Robert Clipsham wrote:

 On 21/08/2011 16:05, Jacob Carlborg wrote:

 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.


 You don't need to annotate it:

 void serialize(T)(T something);

 You have access to all the CT reflection stuff, and can generate the
 necessary runtime reflection with that.


 I already have a fully working serialization library. I'm hoping to make
 it better. The problem is when (de)serializing through a base class
 reference. Currently you need to register the subclass with the
 serializer. If T is Base and the actually runtime type is Sub it won't
 work.


 If we get to the point where adding one line of code inside or
 (preferably) outside a type to make it serializable, that would be great.

 Andrei


That's already what I have and it's only necessary to register 
subclasses that are serialized through base class references. If we get 
runtime reflection as I suggested in my original post, that wouldn't be 
needed. In addition to that I was hoping it would make it easier to 
implement a binary archive.

Project page: http://www.dsource.org/projects/orange
Source code: https://github.com/jacob-carlborg/orange

-- 
/Jacob Carlborg
Aug 21 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/22/11 1:41 AM, Jacob Carlborg wrote:

 On 2011-08-22 00:35, Andrei Alexandrescu wrote:

 On 8/21/11 1:19 PM, Jacob Carlborg wrote:

 On 2011-08-21 17:30, Robert Clipsham wrote:

 On 21/08/2011 16:05, Jacob Carlborg wrote:

 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.


 You don't need to annotate it:

 void serialize(T)(T something);

 You have access to all the CT reflection stuff, and can generate the
 necessary runtime reflection with that.


 I already have a fully working serialization library. I'm hoping to make
 it better. The problem is when (de)serializing through a base class
 reference. Currently you need to register the subclass with the
 serializer. If T is Base and the actually runtime type is Sub it won't
 work.


 If we get to the point where adding one line of code inside or
 (preferably) outside a type to make it serializable, that would be great.

 Andrei


 That's already what I have and it's only necessary to register
 subclasses that are serialized through base class references. If we get
 runtime reflection as I suggested in my original post, that wouldn't be
 needed.


FWIW I don't think D should pay the cost of reflection up-front and 
compulsively for all types.

If David's code example is what you have, I don't think Orange got 
"making a class serializable" properly yet.


Andrei
Aug 22 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-22 21:16, Andrei Alexandrescu wrote:

 FWIW I don't think D should pay the cost of reflection up-front and
 compulsively for all types.


So ClassInfo.getMembers is there because?


 If David's code example is what you have, I don't think Orange got
 "making a class serializable" properly yet.


 Andrei


You do understand that this is ONLY necessary when serializing through a 
base class reference? Other suggestions are welcome.

-- 
/Jacob Carlborg
Aug 22 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 8/22/11 2:22 PM, Jacob Carlborg wrote:

 On 2011-08-22 21:16, Andrei Alexandrescu wrote:

 FWIW I don't think D should pay the cost of reflection up-front and
 compulsively for all types.


 So ClassInfo.getMembers is there because?


At this point the implementation is unclear on what should and what 
shouldn't be part of compiler-supplied metadata.


 If David's code example is what you have, I don't think Orange got
 "making a class serializable" properly yet.


 Andrei


 You do understand that this is ONLY necessary when serializing through a
 base class reference? Other suggestions are welcome.


I do understand that point, which is unrelated to mine.


Andrei
Aug 22 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-22 22:27, Andrei Alexandrescu wrote:

 On 8/22/11 2:22 PM, Jacob Carlborg wrote:

 On 2011-08-22 21:16, Andrei Alexandrescu wrote:

 FWIW I don't think D should pay the cost of reflection up-front and
 compulsively for all types.


 So ClassInfo.getMembers is there because?


 At this point the implementation is unclear on what should and what
 shouldn't be part of compiler-supplied metadata.


 If David's code example is what you have, I don't think Orange got
 "making a class serializable" properly yet.


 Andrei


 You do understand that this is ONLY necessary when serializing through a
 base class reference? Other suggestions are welcome.


 I do understand that point, which is unrelated to mine.


 Andrei


Ok, I just want to make sure because I think that the library is working 
pretty well and turned out quite good with the, somewhat, limit 
possibilities the language give you, i.e. no runtime reflection.

-- 
/Jacob Carlborg
Aug 22 2011
prev sibling parent reply Kagamin <spam here.lot> writes:
Jacob Carlborg Wrote:


 I already have a fully working serialization library. I'm hoping to make 
 it better. The problem is when (de)serializing through a base class 
 reference. Currently you need to register the subclass with the 
 serializer. If T is Base and the actually runtime type is Sub it won't work.


In .net it won't work either. You have to specify known classes explicitly.
Aug 22 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-22 13:27, Kagamin wrote:

 Jacob Carlborg Wrote:


 I already have a fully working serialization library. I'm hoping to make
 it better. The problem is when (de)serializing through a base class
 reference. Currently you need to register the subclass with the
 serializer. If T is Base and the actually runtime type is Sub it won't work.


 In .net it won't work either. You have to specify known classes explicitly.


As far as I know, this is just because they chose to do it this way, not 
because of limitations in the language/runtime. .Net has runtime 
reflection, there's no technical need to add an attribute to indicate 
that a class is serializable.

-- 
/Jacob Carlborg
Aug 22 2011
prev sibling next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 21 Aug 2011 18:05:20 +0300, Jacob Carlborg <doob me.com> wrote:


 This is for a serialization library where third party types need to be  
 serializable. I don't like the idea of annotate everything that should  
 be serializable.


I think you misunderstood what I meant. I did not imply that one would  
need to annotate any types - only specify the modules for which run-time  
reflection should be available.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Aug 21 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-21 17:30, Vladimir Panteleev wrote:

 On Sun, 21 Aug 2011 18:05:20 +0300, Jacob Carlborg <doob me.com> wrote:


 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.


 I think you misunderstood what I meant. I did not imply that one would
 need to annotate any types - only specify the modules for which run-time
 reflection should be available.


Would that be possible to do for third party types available in a 
pre-compiled library?

-- 
/Jacob Carlborg
Aug 21 2011
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 21 Aug 2011 21:21:05 +0300, Jacob Carlborg <doob me.com> wrote:


 Would that be possible to do for third party types available in a  
 pre-compiled library?


If said types are in the .di files, sure, why not?

Otherwise, the answer is obvious... but as someone else mentioned in this  
thread, if the authors of said libraries choose to hide the source code or  
internals of the library, they probably won't be very friendly towards  
reflection either.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Aug 21 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-21 21:43, Vladimir Panteleev wrote:

 On Sun, 21 Aug 2011 21:21:05 +0300, Jacob Carlborg <doob me.com> wrote:


 Would that be possible to do for third party types available in a
 pre-compiled library?


 If said types are in the .di files, sure, why not?

 Otherwise, the answer is obvious... but as someone else mentioned in
 this thread, if the authors of said libraries choose to hide the source
 code or internals of the library, they probably won't be very friendly
 towards reflection either.


To be able to use a library you need parts of the source code, like the 
methods and fields available. And when that's already available what's 
wrong with some compiler/runtime changes to make it possible to use 
these through base class references.

-- 
/Jacob Carlborg
Aug 21 2011
prev sibling parent reply Kagamin <spam here.lot> writes:
Jacob Carlborg Wrote:


 This is for a serialization library where third party types need to be 
 serializable. I don't like the idea of annotate everything that should 
 be serializable.


I believe Adam Ruppe wrote serialization that works automatically.
Aug 22 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-22 13:37, Kagamin wrote:

 Jacob Carlborg Wrote:


 This is for a serialization library where third party types need to be
 serializable. I don't like the idea of annotate everything that should
 be serializable.


 I believe Adam Ruppe wrote serialization that works automatically.


My library works automatically for all types, except when serializing 
through a base class reference. Then the subclass have to be registered 
with the serializer.

-- 
/Jacob Carlborg
Aug 22 2011
prev sibling next sibling parent Robert Clipsham <robert octarineparrot.com> writes:
On 21/08/2011 14:13, Vladimir Panteleev wrote:

 On Sun, 21 Aug 2011 15:16:11 +0300, Jacob Carlborg <doob me.com> wrote:


 Yes, I know. But the point is to be able to do it without the use of
 mixins. It needs to work with third party types, otherwise there no
 use. It also needs to work when the static type is Object.

 You're already paying for the class info, which already contains
 getMembers. But this methods always returns an empty array and as far
 as I can see, it's not possible to set/get fields or call methods via
 these member instances.


 I think the best way to do this is to improve compile-time reflection to
 the point where it'd be possible to enumerate all declarations in a
 module and create run-time reflection information for them - then you'd
 just need to specify the modules for which you'd like run-time
 reflection. One thing to note is that there are use cases where more
 run-time information about a program's internals may not be desired,
 e.g. with commercial closed-source applications, so this would need to
 be optional.


I completely agree. I'd love to get rid of classinfo while we're at it 
to be honest, that's a heck of lot of bloat that isn't needed/can be 
done in a far nicer way.

-- 
Robert
http://octarineparrot.com/
Aug 21 2011
prev sibling parent reply Jakob Ovrum <jakobovrum+ng gmail.com> writes:
On 2011/08/21 22:13, Vladimir Panteleev wrote:

 On Sun, 21 Aug 2011 15:16:11 +0300, Jacob Carlborg <doob me.com> wrote:


 Yes, I know. But the point is to be able to do it without the use of
 mixins. It needs to work with third party types, otherwise there no
 use. It also needs to work when the static type is Object.

 You're already paying for the class info, which already contains
 getMembers. But this methods always returns an empty array and as far
 as I can see, it's not possible to set/get fields or call methods via
 these member instances.


 I think the best way to do this is to improve compile-time reflection to
 the point where it'd be possible to enumerate all declarations in a
 module and create run-time reflection information for them - then you'd
 just need to specify the modules for which you'd like run-time
 reflection. One thing to note is that there are use cases where more
 run-time information about a program's internals may not be desired,
 e.g. with commercial closed-source applications, so this would need to
 be optional.


I definitively think this is the way to go forward.

The only situation this method doesn't handle is the case of only having 
D interface files and binaries for a library you want to serialize. 
However, if the library is closed-source, one of the reasons for that 
might be that they don't want this kind of information to be available, 
and if that's the case and D would generate this runtime information by 
default, we'd start seeing obfuscators for D as which break the 
reflection functionality.

As for the other way around, where a closed-source library wanted to 
retain runtime reflection capabilties, they could always include the 
information by explicitly instantiating the runtime information template 
or however it would be implemented.

Besides, compile-time reflection for entire modules has many uses beyond 
implementing runtime reflection.
Aug 21 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-21 18:18, Jakob Ovrum wrote:

 On 2011/08/21 22:13, Vladimir Panteleev wrote:

 On Sun, 21 Aug 2011 15:16:11 +0300, Jacob Carlborg <doob me.com> wrote:


 Yes, I know. But the point is to be able to do it without the use of
 mixins. It needs to work with third party types, otherwise there no
 use. It also needs to work when the static type is Object.

 You're already paying for the class info, which already contains
 getMembers. But this methods always returns an empty array and as far
 as I can see, it's not possible to set/get fields or call methods via
 these member instances.


 I think the best way to do this is to improve compile-time reflection to
 the point where it'd be possible to enumerate all declarations in a
 module and create run-time reflection information for them - then you'd
 just need to specify the modules for which you'd like run-time
 reflection. One thing to note is that there are use cases where more
 run-time information about a program's internals may not be desired,
 e.g. with commercial closed-source applications, so this would need to
 be optional.


 I definitively think this is the way to go forward.

 The only situation this method doesn't handle is the case of only having
 D interface files and binaries for a library you want to serialize.
 However, if the library is closed-source, one of the reasons for that
 might be that they don't want this kind of information to be available,
 and if that's the case and D would generate this runtime information by
 default, we'd start seeing obfuscators for D as which break the
 reflection functionality.


The information is already available via .tupleof. This would make it 
available through base class references and with runtime strings.


 As for the other way around, where a closed-source library wanted to
 retain runtime reflection capabilties, they could always include the
 information by explicitly instantiating the runtime information template
 or however it would be implemented.

 Besides, compile-time reflection for entire modules has many uses beyond
 implementing runtime reflection.



-- 
/Jacob Carlborg
Aug 21 2011
prev sibling next sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 21 Aug 2011 06:55:52 -0400, Jacob Carlborg <doob me.com> wrote:


 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
      private int a;

      private void bar (int i)
      {
          a = i;
      }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language, only
 the compiler and runtime would need to be modified. To me, it doesn't
 look that complicated to implement, the runtime just needs to store an
 associative arrays that maps names of instance variables and methods to
 their addresses. This could probably be stored in the ClassInfo.
 Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.


I've basically done this entirely in library code: see
https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht
Aug 21 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-08-21 15:20, Robert Jacques wrote:

 On Sun, 21 Aug 2011 06:55:52 -0400, Jacob Carlborg <doob me.com> wrote:


 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
 private int a;

 private void bar (int i)
 {
 a = i;
 }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language, only
 the compiler and runtime would need to be modified. To me, it doesn't
 look that complicated to implement, the runtime just needs to store an
 associative arrays that maps names of instance variables and methods to
 their addresses. This could probably be stored in the ClassInfo.
 Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.


 I've basically done this entirely in library code: see
 https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht


The whole point was to have direct support in the language/runtime so 
these library solution were not needed. Does your solution work when the 
static type is Object and the runtime type is a subclass?

-- 
/Jacob Carlborg
Aug 21 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 21 Aug 2011 11:20:14 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-21 15:20, Robert Jacques wrote:

 On Sun, 21 Aug 2011 06:55:52 -0400, Jacob Carlborg <doob me.com> wrote:


 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
 private int a;

 private void bar (int i)
 {
 a = i;
 }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language, only
 the compiler and runtime would need to be modified. To me, it doesn't
 look that complicated to implement, the runtime just needs to store an
 associative arrays that maps names of instance variables and methods to
 their addresses. This could probably be stored in the ClassInfo.
 Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.


 I've basically done this entirely in library code: see
 https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht


 The whole point was to have direct support in the language/runtime so
 these library solution were not needed. Does your solution work when the
 static type is Object and the runtime type is a subclass?


Yes, so long as at some point the subclass is registered by variant, either
manually or auto-magically via reflection on another type. The docs had an
example of this:

         module example;
         class Grade   { real  mark;  }
         class Student { Grade grade; }
         void main(string[] args) {
             Variant.__register!Student;
             Variant grade = Object.factory("example.Grade");
             grade.mark(96.6);
             assert(grade.mark == 96.6);
         }

notice how Student had to be manually registered, but Grade was registered
auto-magically.
Aug 21 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-22 02:43, Robert Jacques wrote:

 On Sun, 21 Aug 2011 11:20:14 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-08-21 15:20, Robert Jacques wrote:

 On Sun, 21 Aug 2011 06:55:52 -0400, Jacob Carlborg <doob me.com> wrote:


 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
 private int a;

 private void bar (int i)
 {
 a = i;
 }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language,
 only
 the compiler and runtime would need to be modified. To me, it doesn't
 look that complicated to implement, the runtime just needs to store an
 associative arrays that maps names of instance variables and methods to
 their addresses. This could probably be stored in the ClassInfo.
 Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.


 I've basically done this entirely in library code: see
 https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht


 The whole point was to have direct support in the language/runtime so
 these library solution were not needed. Does your solution work when the
 static type is Object and the runtime type is a subclass?


 Yes, so long as at some point the subclass is registered by variant,
 either manually or auto-magically via reflection on another type. The
 docs had an example of this:

 module example;
 class Grade { real mark; }
 class Student { Grade grade; }
 void main(string[] args) {
 Variant.__register!Student;
 Variant grade = Object.factory("example.Grade");
 grade.mark(96.6);
 assert(grade.mark == 96.6);
 }

 notice how Student had to be manually registered, but Grade was
 registered auto-magically.


I already have this in my serialization library, I want to avoid that. 
That was the whole point of my original post.

-- 
/Jacob Carlborg
Aug 21 2011
prev sibling parent reply Mehrdad <wfunction hotmail.com> writes:
On 8/21/2011 3:55 AM, Jacob Carlborg wrote:

 What are the chances of D getting proper runtime reflection? Something 
 like this:

 class Foo
 {
     private int a;

     private void bar (int i)
     {
         a = i;
     }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should 
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language, 
 only the compiler and runtime would need to be modified. To me, it 
 doesn't look that complicated to implement, the runtime just needs to 
 store an associative arrays that maps names of instance variables and 
 methods to their addresses. This could probably be stored in the 
 ClassInfo. Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.


It's not /quite/ the same thing as you're asking, but I'm facing a 
problem that seems to fundamentally prevent dynamic typing to work 
correctly (which is related, but of course not the same thing as, 
reflection):
http://stackoverflow.com/questions/7102648/using-dynamic-typing-in-d-a-statically-typed-language

As much as I'd love to see it happen, I can't see how templates and 
reflection would actually work at run-time.
Aug 21 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-08-21 21:07, Mehrdad wrote:

 On 8/21/2011 3:55 AM, Jacob Carlborg wrote:

 What are the chances of D getting proper runtime reflection? Something
 like this:

 class Foo
 {
 private int a;

 private void bar (int i)
 {
 a = i;
 }
 }

 auto foo = new Foo;
 Object o = foo;
 o.setInstanceVariable("a", 3);
 assert(foo.a == 3);

 This would be useful as well:

 o.send("bar", 5);
 assert(foo.a == 5);
 assert(foo.a == o.getInstanceVariable("a"));

 Currently I have most use of the first example. These example should
 work regardless if the fields/methods are private, protected or public.

 As far as I know this wouldn't require any changes to the language,
 only the compiler and runtime would need to be modified. To me, it
 doesn't look that complicated to implement, the runtime just needs to
 store an associative arrays that maps names of instance variables and
 methods to their addresses. This could probably be stored in the
 ClassInfo. Something like this:

 Foo.classinfo.setInstanceVariable(foo, "a", 3);

 And this could be shorted to:

 o.setInstanceVariable("a", 3);

 "setInstanceVariable" could just forward the call to the class info.


 It's not /quite/ the same thing as you're asking, but I'm facing a
 problem that seems to fundamentally prevent dynamic typing to work
 correctly (which is related, but of course not the same thing as,
 reflection):
 http://stackoverflow.com/questions/7102648/using-dynamic-typing-in-d-a-statically-typed-language


 As much as I'd love to see it happen, I can't see how templates and
 reflection would actually work at run-time.


Why don't you just use a variant ?

-- 
/Jacob Carlborg
Aug 21 2011