www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Compiler patch for runtime reflection

reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
Hi,

Igor Stepanov has created a patch for DMD and Druntime which adds RTTI  
information for class and struct members.

Example:

import std.stdio;

class Foo
{
	static void PrintHello()
	{
		writeln("Hello");
	}
}
void main()
{
	auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
	assert(info.name == "PrintHello");
	auto print = cast(void function())info.pointer;
	print(); //prints "Hello"
}

While the inclusion of such functionality into the language remains a  
disputed matter, would anyone be interested in an unofficial patch for  
this?

Walter: would it be okay if the compiler changes were published as a  
GitHub fork, or should we stick to patches?

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Oct 21 2011
next sibling parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 21-10-2011 21:07, Vladimir Panteleev wrote:

 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }


Absolutely awesome!


 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?


Yes, very much. I would recommend setting up a fork on GitHub, and then 
adding it to a branch, e.g. 'reflection', if it's not going to be 
included in mainline DMD.


 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?


- Alex
Oct 21 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Fri, 21 Oct 2011 17:15:02 -0400, Alex Rønne Petersen <xtzgzorex gmail.com>
wrote:


 On 21-10-2011 21:07, Vladimir Panteleev wrote:

 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }


 Absolutely awesome!


 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?


 Yes, very much. I would recommend setting up a fork on GitHub, and then
 adding it to a branch, e.g. 'reflection', if it's not going to be
 included in mainline DMD.


Really? A runtime reflection system has been on the review queue for over six
months and I've brought the subject up on the newsgroup. I've received zero
feedback. So while I'm sure everyone want to check RTTI off the D features
list, I've not seen much real interest in it.
Oct 21 2011
next sibling parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 22-10-2011 05:36, Robert Jacques wrote:

 On Fri, 21 Oct 2011 17:15:02 -0400, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:


 On 21-10-2011 21:07, Vladimir Panteleev wrote:

 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }


 Absolutely awesome!


 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?


 Yes, very much. I would recommend setting up a fork on GitHub, and then
 adding it to a branch, e.g. 'reflection', if it's not going to be
 included in mainline DMD.


 Really? A runtime reflection system has been on the review queue for
 over six months and I've brought the subject up on the newsgroup. I've
 received zero feedback. So while I'm sure everyone want to check RTTI
 off the D features list, I've not seen much real interest in it.


I must have missed that. What should I search for to find your thread?

- Alex
Oct 22 2011
parent "Robert Jacques" <sandford jhu.edu> writes:
On Sat, 22 Oct 2011 06:51:00 -0400, Alex Rønne Petersen <xtzgzorex gmail.com>
wrote:

 On 22-10-2011 05:36, Robert Jacques wrote:

 On Fri, 21 Oct 2011 17:15:02 -0400, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:


 On 21-10-2011 21:07, Vladimir Panteleev wrote:

 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }


 Absolutely awesome!


 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?


 Yes, very much. I would recommend setting up a fork on GitHub, and then
 adding it to a branch, e.g. 'reflection', if it's not going to be
 included in mainline DMD.


 Really? A runtime reflection system has been on the review queue for
 over six months and I've brought the subject up on the newsgroup. I've
 received zero feedback. So while I'm sure everyone want to check RTTI
 off the D features list, I've not seen much real interest in it.


 I must have missed that. What should I search for to find your thread?


Don't bother; As I said, the thread went nowhere. However, if you'd like to
look at the code / docs for my proposed improved variant module:

https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht
https://jshare.johnshopkins.edu/rjacque2/public_html/variant.d
Oct 22 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-10-22 05:36, Robert Jacques wrote:

 On Fri, 21 Oct 2011 17:15:02 -0400, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:


 On 21-10-2011 21:07, Vladimir Panteleev wrote:

 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }


 Absolutely awesome!


 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?


 Yes, very much. I would recommend setting up a fork on GitHub, and then
 adding it to a branch, e.g. 'reflection', if it's not going to be
 included in mainline DMD.


 Really? A runtime reflection system has been on the review queue for
 over six months and I've brought the subject up on the newsgroup. I've
 received zero feedback. So while I'm sure everyone want to check RTTI
 off the D features list, I've not seen much real interest in it.


I must have completely missed that.

-- 
/Jacob Carlborg
Oct 22 2011
prev sibling next sibling parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 21.10.2011 21:07, schrieb Vladimir Panteleev:

 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?


I'd love to see proper runtime reflection support in D, including 
functionality to get information about available methods (their name and 
parameters) and a way to call them.
Something that is close to what Java offers would be great.

BTW: I don't really see the problem with providing this information 
(overhead-wise) - the information needs to be available once per 
class/struct, but objects of classes just need one pointer to it (other 
types don't even need that because they're not polymorphic and - like 
methods of structs - the address of the information is known at 
compile-time).

Cheers,
- Daniel
Oct 21 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Fri, 21 Oct 2011 17:23:17 -0400, Daniel Gibson <metalcaedes gmail.com> wrote:

 Am 21.10.2011 21:07, schrieb Vladimir Panteleev:

 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?


 I'd love to see proper runtime reflection support in D, including
 functionality to get information about available methods (their name and
 parameters) and a way to call them.


What do you mean by their 'parameters'? What about overloads? Attributes?
Arguments? Argument attributes?


 Something that is close to what Java offers would be great.


And what, exactly does JAVA offer? What works? What doesn't work? What's
missing?


 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).


1) Unused information is simply bloat: it increases exe size, slows the exe
down and increases the runtime memory footprint.
2) On a lot of systems (i.e. consoles, embedded, smart phones, tablets) memory
and disk space are both highly constrained resources that you don't want to
waste.
3) RTTI provides a back-door into a code-base; one that for many reasons you
may want to keep closed.
Oct 21 2011
next sibling parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 22-10-2011 05:48, Robert Jacques wrote:

 On Fri, 21 Oct 2011 17:23:17 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:

 Am 21.10.2011 21:07, schrieb Vladimir Panteleev:

 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?


 I'd love to see proper runtime reflection support in D, including
 functionality to get information about available methods (their name and
 parameters) and a way to call them.


 What do you mean by their 'parameters'? What about overloads?
 Attributes? Arguments? Argument attributes?


 Something that is close to what Java offers would be great.


 And what, exactly does JAVA offer? What works? What doesn't work? What's
 missing?


 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).


 1) Unused information is simply bloat: it increases exe size, slows the
 exe down and increases the runtime memory footprint.
 2) On a lot of systems (i.e. consoles, embedded, smart phones, tablets)
 memory and disk space are both highly constrained resources that you
 don't want to waste.
 3) RTTI provides a back-door into a code-base; one that for many reasons
 you may want to keep closed.


You could just introduce a -reflection switch to include reflection 
information.

- Alex
Oct 22 2011
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen  
<xtzgzorex gmail.com> wrote:


 You could just introduce a -reflection switch to include reflection  
 information.


As I mentioned in the previous thread on the subject, I think that ideally  
we should improve compile-time reflection so it's possible to generate  
runtime reflection information at compilation time. Something like this:

import reflection;

enum reflectionForStdStdio = generateReflectionForModule("std.stdio");

void main()
{
     reflectionForStdStdio.callFunction("writeln", "Hello, world!");
}

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Oct 22 2011
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 10/22/11, Vladimir Panteleev <vladimir thecybershadow.net> wrote:

 On Sat, 22 Oct 2011 13:50:32 +0300, Alex R=F8nne Petersen
 <xtzgzorex gmail.com> wrote:


 You could just introduce a -reflection switch to include reflection
 information.


 As I mentioned in the previous thread on the subject, I think that ideall=


y

 we should improve compile-time reflection so it's possible to generate
 runtime reflection information at compilation time. Something like this:

 import reflection;

 enum reflectionForStdStdio =3D generateReflectionForModule("std.stdio");

 void main()
 {
      reflectionForStdStdio.callFunction("writeln", "Hello, world!");
 }

 --
 Best regards,
   Vladimir                            mailto:vladimir thecybershadow.net


I think user properties would be a perfect fit for this. :)

 reflected
struct Foo {}
Oct 22 2011
parent reply =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 22-10-2011 15:20, Andrej Mitrovic wrote:

 On 10/22/11, Vladimir Panteleev<vladimir thecybershadow.net>  wrote:

 On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com>  wrote:


 You could just introduce a -reflection switch to include reflection
 information.


 As I mentioned in the previous thread on the subject, I think that ideally
 we should improve compile-time reflection so it's possible to generate
 runtime reflection information at compilation time. Something like this:

 import reflection;

 enum reflectionForStdStdio = generateReflectionForModule("std.stdio");

 void main()
 {
       reflectionForStdStdio.callFunction("writeln", "Hello, world!");
 }

 --
 Best regards,
    Vladimir                            mailto:vladimir thecybershadow.net


 I think user properties would be a perfect fit for this. :)

  reflected
 struct Foo {}


The only downside with either approach is that people need to explicitly 
mark their types as "reflectable", which can be a slight hindrance in 
the way of discoverability (and probably also complicates actual 
reflection - what if you try to get the return type of a reflected 
method and that type is not available? This would make for an *awful* 
lot of null checks).

In any case, if we need to mark stuff as reflectable in-source, I'd 
vouch for  reflected, though I would really prefer application-global 
reflection info.

- Alex
Oct 22 2011
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sat, 22 Oct 2011 16:35:57 +0300, Alex Rønne Petersen  
<xtzgzorex gmail.com> wrote:


 On 22-10-2011 15:20, Andrej Mitrovic wrote:

 On 10/22/11, Vladimir Panteleev<vladimir thecybershadow.net>  wrote:

 On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com>  wrote:


 You could just introduce a -reflection switch to include reflection
 information.


 As I mentioned in the previous thread on the subject, I think that  
 ideally
 we should improve compile-time reflection so it's possible to generate
 runtime reflection information at compilation time. Something like  
 this:

 import reflection;

 enum reflectionForStdStdio = generateReflectionForModule("std.stdio");

 void main()
 {
       reflectionForStdStdio.callFunction("writeln", "Hello, world!");
 }

 --
 Best regards,
    Vladimir                             
 mailto:vladimir thecybershadow.net


 I think user properties would be a perfect fit for this. :)

  reflected
 struct Foo {}


 The only downside with either approach


What do you mean by "either approach"? Which is the other one?

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Oct 22 2011
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 22-10-2011 17:18, Vladimir Panteleev wrote:

 On Sat, 22 Oct 2011 16:35:57 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:


 On 22-10-2011 15:20, Andrej Mitrovic wrote:

 On 10/22/11, Vladimir Panteleev<vladimir thecybershadow.net> wrote:

 On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:


 You could just introduce a -reflection switch to include reflection
 information.


 As I mentioned in the previous thread on the subject, I think that
 ideally
 we should improve compile-time reflection so it's possible to generate
 runtime reflection information at compilation time. Something like
 this:

 import reflection;

 enum reflectionForStdStdio = generateReflectionForModule("std.stdio");

 void main()
 {
 reflectionForStdStdio.callFunction("writeln", "Hello, world!");
 }

 --
 Best regards,
 Vladimir mailto:vladimir thecybershadow.net


 I think user properties would be a perfect fit for this. :)

  reflected
 struct Foo {}


 The only downside with either approach


 What do you mean by "either approach"? Which is the other one?


 reflected and generateReflectionForModule.

- Alex
Oct 22 2011
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 23 Oct 2011 01:54:52 +0300, Alex Rønne Petersen  
<xtzgzorex gmail.com> wrote:


 On 22-10-2011 17:18, Vladimir Panteleev wrote:

 On Sat, 22 Oct 2011 16:35:57 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:


 On 22-10-2011 15:20, Andrej Mitrovic wrote:

 On 10/22/11, Vladimir Panteleev<vladimir thecybershadow.net> wrote:

 On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:


 You could just introduce a -reflection switch to include reflection
 information.


 As I mentioned in the previous thread on the subject, I think that
 ideally
 we should improve compile-time reflection so it's possible to  
 generate
 runtime reflection information at compilation time. Something like
 this:

 import reflection;

 enum reflectionForStdStdio =  
 generateReflectionForModule("std.stdio");

 void main()
 {
 reflectionForStdStdio.callFunction("writeln", "Hello, world!");
 }

 --
 Best regards,
 Vladimir mailto:vladimir thecybershadow.net


 I think user properties would be a perfect fit for this. :)

  reflected
 struct Foo {}


 The only downside with either approach


 What do you mean by "either approach"? Which is the other one?


  reflected and generateReflectionForModule.

 - Alex


I don't see how my generateReflectionForModule idea requires any  
modification of the reflected code. The module can generate RTTI for all  
types used recursively from the starting point. If more precision is  
needed, it can be specified as generateReflectionForModule/etc. parameters.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Oct 23 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/23/2011 12:19 PM, Vladimir Panteleev wrote:

 On Sun, 23 Oct 2011 01:54:52 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:


 On 22-10-2011 17:18, Vladimir Panteleev wrote:

 On Sat, 22 Oct 2011 16:35:57 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:


 On 22-10-2011 15:20, Andrej Mitrovic wrote:

 On 10/22/11, Vladimir Panteleev<vladimir thecybershadow.net> wrote:

 On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:


 You could just introduce a -reflection switch to include reflection
 information.


 As I mentioned in the previous thread on the subject, I think that
 ideally
 we should improve compile-time reflection so it's possible to
 generate
 runtime reflection information at compilation time. Something like
 this:

 import reflection;

 enum reflectionForStdStdio =
 generateReflectionForModule("std.stdio");

 void main()
 {
 reflectionForStdStdio.callFunction("writeln", "Hello, world!");
 }

 --
 Best regards,
 Vladimir mailto:vladimir thecybershadow.net


 I think user properties would be a perfect fit for this. :)

  reflected
 struct Foo {}


 The only downside with either approach


 What do you mean by "either approach"? Which is the other one?


  reflected and generateReflectionForModule.

 - Alex


 I don't see how my generateReflectionForModule idea requires any
 modification of the reflected code. The module can generate RTTI for all
 types used recursively from the starting point. If more precision is
 needed, it can be specified as generateReflectionForModule/etc. parameters.


The module can generate RTTI for all types recursively from the starting 
point iff that information is statically available. It does not have to 
be. A module that comes as .di + static library binary could return a 
reference to a private class that has a publicly exported base class. 
How would you generate RTTI for a statically invisible class?

Eg:

// file A.d
module A;
class C{int x; ...}
private class D: C{int y; ...}

C factory(){ return new D; }


// file A.di
module A;
class C{int x; ...}

// file B.d
import A; // imports A.di file

enum reflectionForA = generateReflectionForModule("A");

void main(){
     C c = factory();
     auto s = reflectionForA.serialize(c); // disaster strikes.
}
Oct 23 2011
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 10/23/2011 01:06 PM, Timon Gehr wrote:

 On 10/23/2011 12:19 PM, Vladimir Panteleev wrote:

 On Sun, 23 Oct 2011 01:54:52 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:


 On 22-10-2011 17:18, Vladimir Panteleev wrote:

 On Sat, 22 Oct 2011 16:35:57 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:


 On 22-10-2011 15:20, Andrej Mitrovic wrote:

 On 10/22/11, Vladimir Panteleev<vladimir thecybershadow.net> wrote:

 On Sat, 22 Oct 2011 13:50:32 +0300, Alex Rønne Petersen
 <xtzgzorex gmail.com> wrote:


 You could just introduce a -reflection switch to include reflection
 information.


 As I mentioned in the previous thread on the subject, I think that
 ideally
 we should improve compile-time reflection so it's possible to
 generate
 runtime reflection information at compilation time. Something like
 this:

 import reflection;

 enum reflectionForStdStdio =
 generateReflectionForModule("std.stdio");

 void main()
 {
 reflectionForStdStdio.callFunction("writeln", "Hello, world!");
 }

 --
 Best regards,
 Vladimir mailto:vladimir thecybershadow.net


 I think user properties would be a perfect fit for this. :)

  reflected
 struct Foo {}


 The only downside with either approach


 What do you mean by "either approach"? Which is the other one?


  reflected and generateReflectionForModule.

 - Alex


 I don't see how my generateReflectionForModule idea requires any
 modification of the reflected code. The module can generate RTTI for all
 types used recursively from the starting point. If more precision is
 needed, it can be specified as generateReflectionForModule/etc.
 parameters.


 The module can generate RTTI for all types recursively from the starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?

 Eg:

 // file A.d
 module A;
 class C{int x; ...}
 private class D: C{int y; ...}

 C factory(){ return new D; }


 // file A.di
 module A;
 class C{int x; ...}


C factory();


 // file B.d
 import A; // imports A.di file

 enum reflectionForA = generateReflectionForModule("A");

 void main(){
 C c = factory();
 auto s = reflectionForA.serialize(c); // disaster strikes.
 }
Oct 23 2011
prev sibling next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 23 Oct 2011 14:06:42 +0300, Timon Gehr <timon.gehr gmx.ch> wrote:


 I don't see how my generateReflectionForModule idea requires any
 modification of the reflected code. The module can generate RTTI for all
 types used recursively from the starting point. If more precision is
 needed, it can be specified as generateReflectionForModule/etc.  
 parameters.


 The module can generate RTTI for all types recursively from the starting  
 point iff that information is statically available. It does not have to  
 be. A module that comes as .di + static library binary could return a  
 reference to a private class that has a publicly exported base class.  
 How would you generate RTTI for a statically invisible class?


The only way to prevent such problems is to have RTTI generated by the  
compiler and enabled always or by default.

However, is such functionality even desirable? It is equivalent to having  
a back door for accessing private class members (implementation details in  
general).

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Oct 23 2011
parent reply =?UTF-8?B?QWxleCBSw7hubmUgUGV0ZXJzZW4=?= <xtzgzorex gmail.com> writes:
On 23-10-2011 14:51, Vladimir Panteleev wrote:

 On Sun, 23 Oct 2011 14:06:42 +0300, Timon Gehr <timon.gehr gmx.ch> wrote:


 I don't see how my generateReflectionForModule idea requires any
 modification of the reflected code. The module can generate RTTI for all
 types used recursively from the starting point. If more precision is
 needed, it can be specified as generateReflectionForModule/etc.
 parameters.


 The module can generate RTTI for all types recursively from the
 starting point iff that information is statically available. It does
 not have to be. A module that comes as .di + static library binary
 could return a reference to a private class that has a publicly
 exported base class. How would you generate RTTI for a statically
 invisible class?


 The only way to prevent such problems is to have RTTI generated by the
 compiler and enabled always or by default.

 However, is such functionality even desirable? It is equivalent to
 having a back door for accessing private class members (implementation
 details in general).


Isn't that technically already possible by doing enough cast and pointer 
arithmetic hacks? I don't think it really makes a difference. If someone 
really wants to backdoor a type, they will find a way to do it.

- Alex
Oct 23 2011
parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Sun, 23 Oct 2011 18:20:32 +0300, Alex Rønne Petersen  
<xtzgzorex gmail.com> wrote:


 On 23-10-2011 14:51, Vladimir Panteleev wrote:

 On Sun, 23 Oct 2011 14:06:42 +0300, Timon Gehr <timon.gehr gmx.ch>  
 wrote:


 I don't see how my generateReflectionForModule idea requires any
 modification of the reflected code. The module can generate RTTI for  
 all
 types used recursively from the starting point. If more precision is
 needed, it can be specified as generateReflectionForModule/etc.
 parameters.


 The module can generate RTTI for all types recursively from the
 starting point iff that information is statically available. It does
 not have to be. A module that comes as .di + static library binary
 could return a reference to a private class that has a publicly
 exported base class. How would you generate RTTI for a statically
 invisible class?


 The only way to prevent such problems is to have RTTI generated by the
 compiler and enabled always or by default.

 However, is such functionality even desirable? It is equivalent to
 having a back door for accessing private class members (implementation
 details in general).


 Isn't that technically already possible by doing enough cast and pointer  
 arithmetic hacks? I don't think it really makes a difference. If someone  
 really wants to backdoor a type, they will find a way to do it.


Yes, but that's neither "safe" nor an actual language feature. Allowing  
RTTI to do the same would be both: my point was whether there should be a  
language feature for this (exploring unexposed types).

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Oct 23 2011
prev sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:
[snip]

 The module can generate RTTI for all types recursively from the starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?


You're not supposed to be able to. Runtime reflection should only apply to
public data members.
Oct 23 2011
next sibling parent reply =?ISO-8859-15?Q?Alex_R=F8nne_Petersen?= <xtzgzorex gmail.com> writes:
On 23-10-2011 18:03, Robert Jacques wrote:

 On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:
 [snip]

 The module can generate RTTI for all types recursively from the starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?


 You're not supposed to be able to. Runtime reflection should only apply
 to public data members.


I don't think that's necessarily a good idea. If you're doing something 
related to e.g. serialization and a type uses properties to hide fields 
(say, because the properties have in/out contracts), you would be unable 
to take those into account.

- Alex
Oct 23 2011
parent "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 23 Oct 2011 12:17:45 -0400, Alex Rønne Petersen <xtzgzorex gmail.com>
wrote:

 On 23-10-2011 18:03, Robert Jacques wrote:

 On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:
 [snip]

 The module can generate RTTI for all types recursively from the starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?


 You're not supposed to be able to. Runtime reflection should only apply
 to public data members.


 I don't think that's necessarily a good idea. If you're doing something
 related to e.g. serialization and a type uses properties to hide fields
 (say, because the properties have in/out contracts), you would be unable
 to take those into account.


A serialized message is part of the public API of your program. It is exposed
not only internally, but also to all those who interact with you. This includes
malicious hackers; one only has to take a look at the history of JSON parses to
see how serialization can be exploited on the web. Exposing private members
publicly is an extremely unsafe operation and a violation of programmer intent;
it should never be done auto-magically. If a type with private members wants to
support serialization, it can add the appropriate custom routines and data
verification checks to itself. If fact, it would seriously behoove us, if our
serialization library defaulted to requiring that a custom validation check be
run by each variable or one of its parents.
Oct 23 2011
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-23 18:03, Robert Jacques wrote:

 On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:
 [snip]

 The module can generate RTTI for all types recursively from the starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?


 You're not supposed to be able to. Runtime reflection should only apply
 to public data members.


It's not enough. Take this for example:

class Foo < ActiveRecord::Base
     before_save :bar

     private

     def bar
     end
end

The above code is an example from Ruby on Rails. The "before_save" call 
sets up a callback, "bar", that will called before saving the model. The 
callback method will be called using reflection.

-- 
/Jacob Carlborg
Oct 23 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Sun, 23 Oct 2011 13:48:03 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-10-23 18:03, Robert Jacques wrote:

 On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch> wrote:
 [snip]

 The module can generate RTTI for all types recursively from the starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?


 You're not supposed to be able to. Runtime reflection should only apply
 to public data members.


 It's not enough. Take this for example:

 class Foo < ActiveRecord::Base
      before_save :bar

      private

      def bar
      end
 end

 The above code is an example from Ruby on Rails. The "before_save" call
 sets up a callback, "bar", that will called before saving the model. The
 callback method will be called using reflection.


Using a short, cryptic stub from another programming language is not an
effective communication tool. Case in point: all I understand is that something
is returning a delegate to something, and reflection never enters the picture.

But, for sake of argument, let me assume that ActiveRecord needs to reflect Foo
for rails to work (i.e. ActiveRecord::Base's ctor reflects this). At a very
fundamental level, this isn't an argument for or against RTTI in D; in D you'd
do this with a mixin of some sort. When looking a use cases/features from other
languages, always remember to first ask oneself a) how would I do it in D
today? b) Is language X's solution 'better, faster, stronger' in some way?

I mean, considering that Ruby is a dynamic language, would bar even be
considered private from inside the super-class's ctor? Is Ruby's private even
comparable to D's private? Given that Ruby, by design, requires fields to be
private and for all external accesses to happen via methods, should we consider
a way to violate that contract an example of mis-feature?
Oct 23 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-23 20:54, Robert Jacques wrote:

 On Sun, 23 Oct 2011 13:48:03 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-10-23 18:03, Robert Jacques wrote:

 On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch>
 wrote:
 [snip]

 The module can generate RTTI for all types recursively from the
 starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?


 You're not supposed to be able to. Runtime reflection should only apply
 to public data members.


 It's not enough. Take this for example:

 class Foo < ActiveRecord::Base
 before_save :bar

 private

 def bar
 end
 end

 The above code is an example from Ruby on Rails. The "before_save" call
 sets up a callback, "bar", that will called before saving the model. The
 callback method will be called using reflection.


 Using a short, cryptic stub from another programming language is not an
 effective communication tool. Case in point: all I understand is that
 something is returning a delegate to something, and reflection never
 enters the picture.


What's happening is that the class method "before_save" is called with 
the symbol (think string) "bar". Then later ActiveRecord::Base will call 
the method corresponding to the string "bar" using reflection.

In D you would proabably do something like:

class Foo : ActiveRecord.Base
{
     private void bar () {}

     mixin beforeSave!(bar);
}

Now you still won't be able to call "bar" outside "Foo" if "bar" is private.


 But, for sake of argument, let me assume that ActiveRecord needs to
 reflect Foo for rails to work (i.e. ActiveRecord::Base's ctor reflects
 this). At a very fundamental level, this isn't an argument for or
 against RTTI in D; in D you'd do this with a mixin of some sort. When
 looking a use cases/features from other languages, always remember to
 first ask oneself a) how would I do it in D today? b) Is language X's
 solution 'better, faster, stronger' in some way?


You can think of it like this, just because you haven't come up with a 
reason for calling privte methods using reflection doesn't mean anyone 
else won't come up with a good reason. I tried here to give a reason.


 I mean, considering that Ruby is a dynamic language, would bar even be
 considered private from inside the super-class's ctor? Is Ruby's private
 even comparable to D's private? Given that Ruby, by design, requires
 fields to be private and for all external accesses to happen via
 methods, should we consider a way to violate that contract an example of
 mis-feature?


There is no constructor involved here. "before_save" is class method 
called when the class "Foo" is loaded.

You can do basically whatever you want with the code in Ruby. You can 
replace an existing method/class in the standard library with your own. 
You can easily add methods to existing classes. It's the programmer that 
is responsible to not mess up things, just as in D where you can cast 
away immutable, shared and do unsafe things with pointers.

-- 
/Jacob Carlborg
Oct 24 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 24 Oct 2011 03:30:02 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-10-23 20:54, Robert Jacques wrote:

 On Sun, 23 Oct 2011 13:48:03 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-10-23 18:03, Robert Jacques wrote:

 On Sun, 23 Oct 2011 07:06:42 -0400, Timon Gehr <timon.gehr gmx.ch>
 wrote:
 [snip]

 The module can generate RTTI for all types recursively from the
 starting
 point iff that information is statically available. It does not have to
 be. A module that comes as .di + static library binary could return a
 reference to a private class that has a publicly exported base class.
 How would you generate RTTI for a statically invisible class?


 You're not supposed to be able to. Runtime reflection should only apply
 to public data members.


 It's not enough. Take this for example:

 class Foo < ActiveRecord::Base
 before_save :bar

 private

 def bar
 end
 end

 The above code is an example from Ruby on Rails. The "before_save" call
 sets up a callback, "bar", that will called before saving the model. The
 callback method will be called using reflection.


 Using a short, cryptic stub from another programming language is not an
 effective communication tool. Case in point: all I understand is that
 something is returning a delegate to something, and reflection never
 enters the picture.


 What's happening is that the class method "before_save" is called with
 the symbol (think string) "bar". Then later ActiveRecord::Base will call
 the method corresponding to the string "bar" using reflection.

 In D you would proabably do something like:

 class Foo : ActiveRecord.Base
 {
      private void bar () {}

      mixin beforeSave!(bar);
 }

 Now you still won't be able to call "bar" outside "Foo" if "bar" is private.


Thanks. But how does super know to call "bar"? And if you can pass information
to super, couldn't you just pass a delegate? And isn't passing information to
super simply passing information to yourself?


 But, for sake of argument, let me assume that ActiveRecord needs to
 reflect Foo for rails to work (i.e. ActiveRecord::Base's ctor reflects
 this). At a very fundamental level, this isn't an argument for or
 against RTTI in D; in D you'd do this with a mixin of some sort. When
 looking a use cases/features from other languages, always remember to
 first ask oneself a) how would I do it in D today? b) Is language X's
 solution 'better, faster, stronger' in some way?


 You can think of it like this, just because you haven't come up with a
 reason for calling privte methods using reflection doesn't mean anyone
 else won't come up with a good reason. I tried here to give a reason.


I'm sorry if I've come across that way. I'm well aware with the reasons for
wanting access to private methods/fields via reflection and have mentioned them
in previous posts. What I've tried to point out is very eloquently stated by
fig. 6.3 on page 204 of TDPL: allowing reflection to bypass protection
guarantees essentially makes all declarations public in nature, if not extern.
That's a very heavy price to pay, just from a program maintenance perspective.
And if you consider someone writing medical or financial software, the privacy
concerns of exposing private variable to all become very real.

Back to my actual argument, what I was pointing out was that the toolboxes of
various languages differ heavily. Just because Ruby has a very cool library
which uses runtime reflection, one shouldn't simply state 'D must have runtime
reflection.' One needs to first decompose what the library is trying to achieve
from the language mechanisms used to implement it. Then you can ask yourself 1)
Can I do that in D? 2) How does D's solution compare to Ruby's?

My impression of what ActiveRecord, (which my be wrong), is that its using
runtime-reflection to do various meta-programming activities and that
under-the-hood, it stretching Ruby to the breaking point in order to do so. D,
on the other hand, has a ton of meta-programming specific features, and doesn't
need to rely on reflection to do so. This is partly to do with Ruby supporting
'eval' and thus making no distinction between compile-time and run-time
behavior; that's correct for Ruby, but D is a different kettle of fish.


 I mean, considering that Ruby is a dynamic language, would bar even be
 considered private from inside the super-class's ctor? Is Ruby's private
 even comparable to D's private? Given that Ruby, by design, requires
 fields to be private and for all external accesses to happen via
 methods, should we consider a way to violate that contract an example of
 mis-feature?


 There is no constructor involved here. "before_save" is class method
 called when the class "Foo" is loaded.


So it's the static ctor, then.


 You can do basically whatever you want with the code in Ruby. You can
 replace an existing method/class in the standard library with your own.
 You can easily add methods to existing classes. It's the programmer that
 is responsible to not mess up things, just as in D where you can cast
 away immutable, shared and do unsafe things with pointers.


Well one one hand, none of those things are allowed in SafeD (And you know,
that's not a resounding argument for reflection of private members). On the
other, you didn't answered my other two questions. In fact, the impression I'm
getting is that Ruby's private isn't comparable to D's private.
Oct 24 2011
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-24 17:37, Robert Jacques wrote:

 I'm sorry if I've come across that way. I'm well aware with the reasons
 for wanting access to private methods/fields via reflection and have
 mentioned them in previous posts. What I've tried to point out is very
 eloquently stated by fig. 6.3 on page 204 of TDPL: allowing reflection
 to bypass protection guarantees essentially makes all declarations
 public in nature, if not extern. That's a very heavy price to pay, just
 from a program maintenance perspective. And if you consider someone
 writing medical or financial software, the privacy concerns of exposing
 private variable to all become very real.


If someone is using a library and starts to mess with calling private 
methods they shouldn't they'll most likely mess things up for them self.


 Back to my actual argument, what I was pointing out was that the
 toolboxes of various languages differ heavily. Just because Ruby has a
 very cool library which uses runtime reflection, one shouldn't simply
 state 'D must have runtime reflection.' One needs to first decompose
 what the library is trying to achieve from the language mechanisms used
 to implement it. Then you can ask yourself 1) Can I do that in D? 2) How
 does D's solution compare to Ruby's?


Of course. I'm just showing that another language/library uses it and 
therefore there could be a chance that should be doable in D as well.


 My impression of what ActiveRecord, (which my be wrong), is that its
 using runtime-reflection to do various meta-programming activities and
 that under-the-hood, it stretching Ruby to the breaking point in order
 to do so. D, on the other hand, has a ton of meta-programming specific
 features, and doesn't need to rely on reflection to do so. This is
 partly to do with Ruby supporting 'eval' and thus making no distinction
 between compile-time and run-time behavior; that's correct for Ruby, but
 D is a different kettle of fish.


I doubt that ActiveRecord is stretching Ruby to the breaking point. On 
the other hand it feels like I'm stretching D's metaprogramming 
capabilities in my serialization library. That's just because D doesn't 
have proper reflection.

In Ruby there is very seldom reason to use "eval". That's because of the 
excellent support for runtime reflection. There's no clumsy syntax, no 
hacks, no need to instantiate reflectors and similar. You just call a 
method like you would call any other method:

class Foo
     def bar
     end
end

foo = Foo.new
foo.bar()
foo.send("bar")

In the above code, "bar" is called first using regular method call 
syntax and then using reflection.


 I mean, considering that Ruby is a dynamic language, would bar even be
 considered private from inside the super-class's ctor? Is Ruby's private
 even comparable to D's private? Given that Ruby, by design, requires
 fields to be private and for all external accesses to happen via
 methods, should we consider a way to violate that contract an example of
 mis-feature?


 There is no constructor involved here. "before_save" is class method
 called when the class "Foo" is loaded.


 So it's the static ctor, then.


I guess you could call it that.


 You can do basically whatever you want with the code in Ruby. You can
 replace an existing method/class in the standard library with your own.
 You can easily add methods to existing classes. It's the programmer that
 is responsible to not mess up things, just as in D where you can cast
 away immutable, shared and do unsafe things with pointers.


 Well one one hand, none of those things are allowed in SafeD (And you
 know, that's not a resounding argument for reflection of private
 members). On the other, you didn't answered my other two questions. In
 fact, the impression I'm getting is that Ruby's private isn't comparable
 to D's private.


If we talk about methods it's easiest. If you call a private method in 
Ruby using the regular method call syntax you get an error. If you do 
the same in D you get an error as well. The only difference is in Ruby 
it will be a runtime error and in D it will be a compile time error.

In Ruby you can bypass private using reflection with the "send" method. 
In D you can bypass private using a delegate.

In Ruby an instance variable is always private. I don't think you can 
declare a public instance variable, one uses setters/getters for that.
In D you can declare both private and public instance variables.

In Ruby you can get/set a value of an instance variable using 
reflection, instance_variable_set/get.
In D you can get/set a value of a private instance variable using 
compile time reflection, i.e. .tupleof.

Now, since we can call private methods and get/set private instance 
variables using the features we have now, why shouldn't we be able to do 
the same with runtime reflection?

BTW, I don't think it's a mis-feature.

-- 
/Jacob Carlborg
Oct 24 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 24 Oct 2011 12:38:12 -0400, Jacob Carlborg <doob me.com> wrote:


 On 2011-10-24 17:37, Robert Jacques wrote:

 I'm sorry if I've come across that way. I'm well aware with the reasons
 for wanting access to private methods/fields via reflection and have
 mentioned them in previous posts. What I've tried to point out is very
 eloquently stated by fig. 6.3 on page 204 of TDPL: allowing reflection
 to bypass protection guarantees essentially makes all declarations
 public in nature, if not extern. That's a very heavy price to pay, just
 from a program maintenance perspective. And if you consider someone
 writing medical or financial software, the privacy concerns of exposing
 private variable to all become very real.


 If someone is using a library and starts to mess with calling private
 methods they shouldn't they'll most likely mess things up for them self.


Aren't private methods by definition something people shouldn't be using?


 Back to my actual argument, what I was pointing out was that the
 toolboxes of various languages differ heavily. Just because Ruby has a
 very cool library which uses runtime reflection, one shouldn't simply
 state 'D must have runtime reflection.' One needs to first decompose
 what the library is trying to achieve from the language mechanisms used
 to implement it. Then you can ask yourself 1) Can I do that in D? 2) How
 does D's solution compare to Ruby's?


 Of course. I'm just showing that another language/library uses it and
 therefore there could be a chance that should be doable in D as well.


I understand and appreciate your intent. However, do so should be accompanied
by a critical dissection of the language/library, both to get those unfamiliar
up to speed and to frame the conversation in terms of how best can we improve D.


 My impression of what ActiveRecord, (which my be wrong), is that its
 using runtime-reflection to do various meta-programming activities and
 that under-the-hood, it stretching Ruby to the breaking point in order
 to do so. D, on the other hand, has a ton of meta-programming specific
 features, and doesn't need to rely on reflection to do so. This is
 partly to do with Ruby supporting 'eval' and thus making no distinction
 between compile-time and run-time behavior; that's correct for Ruby, but
 D is a different kettle of fish.


 I doubt that ActiveRecord is stretching Ruby to the breaking point.


:) I don't know myself. I've just heard that the internals of Rails are deep,
dark and full of non-idiomatic Rudy code.


 On
 the other hand it feels like I'm stretching D's metaprogramming
 capabilities in my serialization library. That's just because D doesn't
 have proper reflection.


I've written three serialization libraries and two versions of a runtime
reflection library; the first one was prior to the string bugs in CTFE being
fixed and ran DMD out of memory. I found serialization to be simple and
straight forward. And I found compiler bugs to be a greater issue with
reflection than any limits in D's capabilities. Indeed, given I wrote the same
library using two completely different mechanisms, gives me great hope for D.
So, I don't agree with your assessment, but I do understand where you're coming
from. A lot of D's metaprogramming capabilities are undocumented, and knowing
the right tool for the job is essential to getting nice, clean code.




 In Ruby there is very seldom reason to use "eval". That's because of the
 excellent support for runtime reflection. There's no clumsy syntax, no
 hacks, no need to instantiate reflectors and similar. You just call a
 method like you would call any other method:

 class Foo
      def bar
      end
 end

 foo = Foo.new
 foo.bar()
 foo.send("bar")

 In the above code, "bar" is called first using regular method call
 syntax and then using reflection.


 I mean, considering that Ruby is a dynamic language, would bar even be
 considered private from inside the super-class's ctor? Is Ruby's private
 even comparable to D's private? Given that Ruby, by design, requires
 fields to be private and for all external accesses to happen via
 methods, should we consider a way to violate that contract an example of
 mis-feature?


 There is no constructor involved here. "before_save" is class method
 called when the class "Foo" is loaded.


 So it's the static ctor, then.


 I guess you could call it that.


(Just checking, but you do know D has static ctors.)


 You can do basically whatever you want with the code in Ruby. You can
 replace an existing method/class in the standard library with your own.
 You can easily add methods to existing classes. It's the programmer that
 is responsible to not mess up things, just as in D where you can cast
 away immutable, shared and do unsafe things with pointers.


 Well one one hand, none of those things are allowed in SafeD (And you
 know, that's not a resounding argument for reflection of private
 members). On the other, you didn't answered my other two questions. In
 fact, the impression I'm getting is that Ruby's private isn't comparable
 to D's private.


 If we talk about methods it's easiest. If you call a private method in
 Ruby using the regular method call syntax you get an error. If you do
 the same in D you get an error as well. The only difference is in Ruby
 it will be a runtime error and in D it will be a compile time error.

 In Ruby you can bypass private using reflection with the "send" method.
 In D you can bypass private using a delegate.


Actually, you're not supposed to be able to; this is bug 1983.
http://d.puremagic.com/issues/show_bug.cgi?id=1983  and 
https://github.com/D-Programming-Language/dmd/pull/71


 In Ruby an instance variable is always private. I don't think you can
 declare a public instance variable, one uses setters/getters for that.
 In D you can declare both private and public instance variables.

 In Ruby you can get/set a value of an instance variable using
 reflection, instance_variable_set/get.
 In D you can get/set a value of a private instance variable using
 compile time reflection, i.e. .tupleof.


Yes, but only for source code you control.


 Now, since we can call private methods and get/set private instance
 variables using the features we have now, why shouldn't we be able to do
 the same with runtime reflection?

 BTW, I don't think it's a mis-feature.


Actually, baring bugs in DMD, .tupleof is the only way to access private fields
and there is _no way_ to access private members; And .tupleof only works if you
have access to the source code. Furthermore, .tupleof, god-send that it is,
isn't part of the D spec; unlike .stringof, it didn't make it into TDPL nor
into the D documentation. D's official method of compile-time reflection is
__traits. And __traits doesn't bypass protection attributes. So precedence
would be against runtime-reflection having access to private members.
Oct 24 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-25 07:24, Robert Jacques wrote:

 Aren't private methods by definition something people shouldn't be using?


Yes, that's what I mean. But sometimes it CAN be useful.


 :) I don't know myself. I've just heard that the internals of Rails are
 deep, dark and full of non-idiomatic Rudy code.


The parts that I've looked at from Rails are not that special.


 On
 the other hand it feels like I'm stretching D's metaprogramming
 capabilities in my serialization library. That's just because D doesn't
 have proper reflection.


 I've written three serialization libraries and two versions of a runtime
 reflection library; the first one was prior to the string bugs in CTFE
 being fixed and ran DMD out of memory. I found serialization to be
 simple and straight forward. And I found compiler bugs to be a greater
 issue with reflection than any limits in D's capabilities. Indeed, given
 I wrote the same library using two completely different mechanisms,
 gives me great hope for D. So, I don't agree with your assessment, but I
 do understand where you're coming from. A lot of D's metaprogramming
 capabilities are undocumented, and knowing the right tool for the job is
 essential to getting nice, clean code.


For example, getting the name of a instance variable:

T.tupleof[i].stringof[1 + T.stringof.length + 2 .. $];

I wouldn't consider that simple and straight forward. This works for 
both D1 and D2, don't know if D2 has a better way.

Your serialization library only works on public fields? Doesn't sound 
very useful.


 (Just checking, but you do know D has static ctors.)


Yes.


 Actually, you're not supposed to be able to; this is bug 1983.
 http://d.puremagic.com/issues/show_bug.cgi?id=1983 and
 https://github.com/D-Programming-Language/dmd/pull/71


I don't see what that has to do with private, it's about const, as far 
as I can see. I don't see how calling a private method through a 
delegate can be easily solved. I see three solutions:

1. Make it a compile time error to create a delegate from a private 
method. This would not allow to call that method via a delegate in the 
same module, which is possible if you call the method directly.

2. Implement some, most likely, very advanced data flow analysis in the 
compiler to detect when a delegate pointing to a private method escapes 
the module. Results in a compile time error. Will most likely not happen.

3. Make it a runtime error when the delegate is actually called. This 
would probably need some pieces of 2 as well. I don't think the 
community will be happy about adding runtime checks like these.

Oh, BTW the same needs to be done for function pointers as well since 
you can build a delegate out of a function pointer and an object.


 In Ruby an instance variable is always private. I don't think you can
 declare a public instance variable, one uses setters/getters for that.
 In D you can declare both private and public instance variables.

 In Ruby you can get/set a value of an instance variable using
 reflection, instance_variable_set/get.
 In D you can get/set a value of a private instance variable using
 compile time reflection, i.e. .tupleof.


 Yes, but only for source code you control.


You don't need control of the source, you only need the static type. Of 
course an opaque struct won't work.


 Actually, baring bugs in DMD, .tupleof is the only way to access private
 fields and there is _no way_ to access private members; And .tupleof
 only works if you have access to the source code. Furthermore, .tupleof,
 god-send that it is, isn't part of the D spec; unlike .stringof, it
 didn't make it into TDPL nor into the D documentation. D's official
 method of compile-time reflection is __traits. And __traits doesn't
 bypass protection attributes. So precedence would be against
 runtime-reflection having access to private members.


The compiler has .tupleof and it's mentioned in the spec at 
d-programming-language.org: http://d-programming-language.org/struct.html

-- 
/Jacob Carlborg
Oct 25 2011
next sibling parent reply Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
I said it once, but nobody responded.

Why not generate full reflection for everything by default and add an
attribute line  noreflect as an _optimization_ tool in cases, where
this thing is definitely not gonna be used in reflection?
Oct 25 2011
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 10/25/2011 11:24 AM, Gor Gyolchanyan wrote:

 I said it once, but nobody responded.

 Why not generate full reflection for everything by default and add an
 attribute line  noreflect as an _optimization_ tool in cases, where
 this thing is definitely not gonna be used in reflection?


As I see it, because you almost never need 'full reflection for 
everything', and idiomatic D code is not supposed to contain clutter 
like that.
Oct 25 2011
parent reply Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
You can always write  noreflect: at the start of your module and it
will work as expected. what's the problem with that?
Also, compiler would be able to automatically add that attribute to
any symbol, which is private and does not escape the module.
Again, not including reflection is purely _optimization_ issue and
should not be interfered with it's functionality.
Reflection is something you rarely have in mind when designing, for
example, classes.
Requiring explicit request for reflection defeats the purpose.

You could just as well say "writing public on every member is a
clutter". It is, but you don't have to do it to achieve the same
effect.

On Tue, Oct 25, 2011 at 2:58 PM, Timon Gehr <timon.gehr gmx.ch> wrote:

 On 10/25/2011 11:24 AM, Gor Gyolchanyan wrote:

 I said it once, but nobody responded.

 Why not generate full reflection for everything by default and add an
 attribute line  noreflect as an _optimization_ tool in cases, where
 this thing is definitely not gonna be used in reflection?


 As I see it, because you almost never need 'full reflection for everything',
 and idiomatic D code is not supposed to contain clutter like that.
Oct 25 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 07:30:04 -0400, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:

 You can always write  noreflect: at the start of your module and it
 will work as expected. what's the problem with that?
 Also, compiler would be able to automatically add that attribute to
 any symbol, which is private and does not escape the module.
 Again, not including reflection is purely _optimization_ issue and
 should not be interfered with it's functionality.
 Reflection is something you rarely have in mind when designing, for
 example, classes.
 Requiring explicit request for reflection defeats the purpose.

 You could just as well say "writing public on every member is a
 clutter". It is, but you don't have to do it to achieve the same
 effect.


You can always write  reflect: at the start of your module and will work as
expected. what's the problem with that?

Rhetoric aside, Timon wasn't referring to code clutter on behalf of the
programmer. He was referring to code bloat in the compiled program. The two are
very different things.

Also, just because reflection is something you rarely have in mind when
designing, doesn't mean it shouldn't be part of the design API. Reflection is
somewhat equivalent to the extern protection modifiers and so should warren
careful consideration. Furthermore, bugs with reflection, say due to a change
in the API, don't break during compilation; they break at runtime. And
practically, there's a decent change your customer, not your test suite, is
going to find it.
Oct 25 2011
next sibling parent reply Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
As i said before, enabling reflection manually is an act of enabling a
certain range of stuff you can do with the code. Forgetting to enable
reflection can lead to serious problems, when the reflection is
required.
disabling reflection manually is an act of optimization purely.
Forgetting to disable the reflection can only lead to unnecessary data
being stored.

The second scenario is much less error-prone.
yes, it _may_ lead to code bloat if you forget to disable it where it
is _absolutely positively unneeded_, but forgetting to enable it will
just make your code useless in many situations.

On Tue, Oct 25, 2011 at 7:20 PM, Robert Jacques <sandford jhu.edu> wrote:

 On Tue, 25 Oct 2011 07:30:04 -0400, Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com> wrote:

 You can always write  noreflect: at the start of your module and it
 will work as expected. what's the problem with that?
 Also, compiler would be able to automatically add that attribute to
 any symbol, which is private and does not escape the module.
 Again, not including reflection is purely _optimization_ issue and
 should not be interfered with it's functionality.
 Reflection is something you rarely have in mind when designing, for
 example, classes.
 Requiring explicit request for reflection defeats the purpose.

 You could just as well say "writing public on every member is a
 clutter". It is, but you don't have to do it to achieve the same
 effect.


 You can always write  reflect: at the start of your module and will work as
 expected. what's the problem with that?

 Rhetoric aside, Timon wasn't referring to code clutter on behalf of the
 programmer. He was referring to code bloat in the compiled program. The two
 are very different things.

 Also, just because reflection is something you rarely have in mind when
 designing, doesn't mean it shouldn't be part of the design API. Reflection
 is somewhat equivalent to the extern protection modifiers and so should
 warren careful consideration. Furthermore, bugs with reflection, say due to
 a change in the API, don't break during compilation; they break at runtime.
 And practically, there's a decent change your customer, not your test suite,
 is going to find it.
Oct 25 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 11:31:18 -0400, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:

 As i said before, enabling reflection manually is an act of enabling a
 certain range of stuff you can do with the code. Forgetting to enable
 reflection can lead to serious problems, when the reflection is
 required.


And when, precisely, is reflection _required_? Granted, in other languages it
is the only tool for the job. But we have a very big toolkit in D, and so far,
I have yet to see any example (including Rails) which _requires_ the holistic
runtime reflection proposed in these threads.


 disabling reflection manually is an act of optimization purely.


No, it isn't. Go read the rest of these threads for counter-arguments. Or the
Java Best Practices guide. Or search the Open Web Application Security Project
for reflection vulnerabilities.


 Forgetting to disable the reflection can only lead to unnecessary data
 being stored.


Alternatively, sensitive information being exposed.


 The second scenario is much less error-prone.


Incorrect, the second scenario is a security hole waiting to be exploited.


 yes, it _may_ lead to code bloat if you forget to disable it where it
 is _absolutely positively unneeded_,


Having written a (limited) runtime reflection library for D, yes it causes
bloat even if you only enable it where _absolutely positively needed_.


 but forgetting to enable it will
 just make your code useless in many situations.


Correction: it will make code designed to _require_ runtime-reflection
incompatible with code not designed to support reflection. Given the percentage
of code that requires reflection (very small) and the percentage of code which
wouldn't be compatible or trivially made to be compatible, (very small) I'm
fine with that.
Oct 25 2011
parent Ary Manzana <ary esperanto.org.ar> writes:
On 10/25/11 1:05 PM, Robert Jacques wrote:

 On Tue, 25 Oct 2011 11:31:18 -0400, Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com> wrote:

 As i said before, enabling reflection manually is an act of enabling a
 certain range of stuff you can do with the code. Forgetting to enable
 reflection can lead to serious problems, when the reflection is
 required.


 And when, precisely, is reflection _required_? Granted, in other
 languages it is the only tool for the job. But we have a very big
 toolkit in D, and so far, I have yet to see any example (including
 Rails) which _requires_ the holistic runtime reflection proposed in
 these threads.


This is completely true. Something like:

class Person < ActiveRecord::Base
   belongs_to :company
end

could be done at compile-time. Same as retrieving the schema from mysql: 
could be done at compile-time, like Nemerle does (or so I remember 
someone said that).

Me too, believes that greater compile-time capabilities will make almost 
all (but never all) runtime reflection unnecessary.
Oct 25 2011
prev sibling parent reply Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
Requiring explicit request for reflection contradicts D's "safety
first, performance second" policy.
lack of reflection is a safety issue too, because you won't be able to
do some checks.

On Tue, Oct 25, 2011 at 7:31 PM, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:

 As i said before, enabling reflection manually is an act of enabling a
 certain range of stuff you can do with the code. Forgetting to enable
 reflection can lead to serious problems, when the reflection is
 required.
 disabling reflection manually is an act of optimization purely.
 Forgetting to disable the reflection can only lead to unnecessary data
 being stored.

 The second scenario is much less error-prone.
 yes, it _may_ lead to code bloat if you forget to disable it where it
 is _absolutely positively unneeded_, but forgetting to enable it will
 just make your code useless in many situations.

 On Tue, Oct 25, 2011 at 7:20 PM, Robert Jacques <sandford jhu.edu> wrote:

 On Tue, 25 Oct 2011 07:30:04 -0400, Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com> wrote:

 You can always write  noreflect: at the start of your module and it
 will work as expected. what's the problem with that?
 Also, compiler would be able to automatically add that attribute to
 any symbol, which is private and does not escape the module.
 Again, not including reflection is purely _optimization_ issue and
 should not be interfered with it's functionality.
 Reflection is something you rarely have in mind when designing, for
 example, classes.
 Requiring explicit request for reflection defeats the purpose.

 You could just as well say "writing public on every member is a
 clutter". It is, but you don't have to do it to achieve the same
 effect.


 You can always write  reflect: at the start of your module and will work as
 expected. what's the problem with that?

 Rhetoric aside, Timon wasn't referring to code clutter on behalf of the
 programmer. He was referring to code bloat in the compiled program. The two
 are very different things.

 Also, just because reflection is something you rarely have in mind when
 designing, doesn't mean it shouldn't be part of the design API. Reflection
 is somewhat equivalent to the extern protection modifiers and so should
 warren careful consideration. Furthermore, bugs with reflection, say due to
 a change in the API, don't break during compilation; they break at runtime.
 And practically, there's a decent change your customer, not your test suite,
 is going to find it.
Oct 25 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 11:35:42 -0400, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:

 Requiring explicit request for reflection contradicts D's "safety
 first, performance second" policy.


Umm, it's "safe defaults, performance possible".


 lack of reflection is a safety issue too, because you won't be able to
 do some checks.


What would you use reflection to check?
Oct 25 2011
parent reply Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
I didn't mean reflection for violating access specifications. In that
case, of course, it's error-prone.
What i meant was a set of dynamic counterparts of static features of D:
* Dynamic interfaces
* Dynamic overloading
* Dynamic modules
* ...
That's what i had in mind.

On Tue, Oct 25, 2011 at 8:08 PM, Robert Jacques <sandford jhu.edu> wrote:

 On Tue, 25 Oct 2011 11:35:42 -0400, Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com> wrote:

 Requiring explicit request for reflection contradicts D's "safety
 first, performance second" policy.


 Umm, it's "safe defaults, performance possible".


 lack of reflection is a safety issue too, because you won't be able to
 do some checks.


 What would you use reflection to check?
Oct 25 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 12:14:58 -0400, Gor Gyolchanyan
<gor.f.gyolchanyan gmail.com> wrote:

 I didn't mean reflection for violating access specifications. In that
 case, of course, it's error-prone.


Thank you, I'm glad you agree.


 What i meant was a set of dynamic counterparts of static features of D:
 * Dynamic interfaces
 * Dynamic overloading
 * Dynamic modules
 * ...
 That's what i had in mind.


Care to explain what would each of those should look like?

Would dynamic interfaces be similar to isRange!T? i.e. a function which tests a
dynamic-object for certain fields/functions.

Would dynamic overload be a subset of prototype-style objects?

Would dynamic modules simply container for dynamic functions/objects? Something
more?
Oct 25 2011
parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
 Thank you, I'm glad you agree.


Of course. Reflection should _never_ expose private stuff.


 Would dynamic interfaces be similar to isRange!T? i.e. a function which tests
a dynamic-object for certain fields/functions.


Hmmm..... Yeap. Exactly. Anything else i would say would be a
derivative of it. I can't do it currently. Also, as discussed before,
i should never see private stuff this way (any any other way).


 Would dynamic overload be a subset of prototype-style objects?


Huh? Dynamic overloading is basically multi-methods. It chooses the
implementation based on run-time type of the parameters.
This could be further improved by allowing overloading, based on
arbitrary predicates (much like template constraints).


 Would dynamic modules simply container for dynamic functions/objects?


Basically, yes. Dynamic module would contain typemically typed
variables, functions, and types.


  Something more?


Cant thing of anything else ATM. Will tell you if i do.

On Tue, Oct 25, 2011 at 8:29 PM, Robert Jacques <sandford jhu.edu> wrote:

 On Tue, 25 Oct 2011 12:14:58 -0400, Gor Gyolchanyan
 <gor.f.gyolchanyan gmail.com> wrote:

 I didn't mean reflection for violating access specifications. In that
 case, of course, it's error-prone.


 Thank you, I'm glad you agree.


 What i meant was a set of dynamic counterparts of static features of D:
 * Dynamic interfaces
 * Dynamic overloading
 * Dynamic modules
 * ...
 That's what i had in mind.


 Care to explain what would each of those should look like?

 Would dynamic interfaces be similar to isRange!T? i.e. a function which
 tests a dynamic-object for certain fields/functions.

 Would dynamic overload be a subset of prototype-style objects?

 Would dynamic modules simply container for dynamic functions/objects?
 Something more?
Oct 25 2011
prev sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 03:23:40 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-10-25 07:24, Robert Jacques wrote:


[snip]


 On
 the other hand it feels like I'm stretching D's metaprogramming
 capabilities in my serialization library. That's just because D doesn't
 have proper reflection.


 I've written three serialization libraries and two versions of a runtime
 reflection library; the first one was prior to the string bugs in CTFE
 being fixed and ran DMD out of memory. I found serialization to be
 simple and straight forward. And I found compiler bugs to be a greater
 issue with reflection than any limits in D's capabilities. Indeed, given
 I wrote the same library using two completely different mechanisms,
 gives me great hope for D. So, I don't agree with your assessment, but I
 do understand where you're coming from. A lot of D's metaprogramming
 capabilities are undocumented, and knowing the right tool for the job is
 essential to getting nice, clean code.


 For example, getting the name of a instance variable:

 T.tupleof[i].stringof[1 + T.stringof.length + 2 .. $];

 I wouldn't consider that simple and straight forward. This works for
 both D1 and D2, don't know if D2 has a better way.


Not really, for the advanced user. I do find using "some string".length to be
cleaner in these cases where you have to slice .stringof. i.e.

     static Value from(T)(T value) {
         //...
         static if(is(T == struct) || is(T == class)) {
             Value[string] result;
             foreach(i,v;value.tupleof) {
                 result[value.tupleof[i].stringof["value.".length..$]] =
from(v);
             }
             return Value( result );
         }
         //..
     }

That said, I think anyone can read the above and understand what's happening.
And even writing that code would be straightforward is .stringof and .tupleof
were better documented.

Also, the first serialization library I wrote used __traits instead. It was
more awkward, mainly because __traits doesn't an 'allFields' member, just an
'allMembers'. If it did then the above becomes:

foreach(name;__traits(allFields,value) {
     result[name] = from(__traits(getMember, value, name));
}

which really is straightforward. Now I originally used 'allMembers' and then
filtered out everything I didn't want, which was fairly straightforward but
verbose.


 Your serialization library only works on public fields? Doesn't sound
 very useful.


In the interest of full disclosure, the library I wrote was quick and dirty and
does not handle ploy-morphism, cycle detection, aliasing, etc (mainly because
JSON doesn't); I used traits for my first one (so protection was observed) but
.tupleof for my second and third and was lazy about it. It's one line of code
to add protection to tuple of, but I've never gotten around to it. I figured my
code would be replace by orange or std.serialization or something.

That said, I have never serialized a private variable and have never wanted
too. The serialized objects are, for the most part, part of the external API to
my program. I have to do validation checks on the inputs and changing anything
takes a week to get the other teams to propagate the changes on their side.
There are also a couple of classes which I serialize in a binary format as a
poor mans database, but again, there's no need for anything in it to be private.

[snip]


 Actually, you're not supposed to be able to; this is bug 1983.
 http://d.puremagic.com/issues/show_bug.cgi?id=1983 and
 https://github.com/D-Programming-Language/dmd/pull/71


 I don't see what that has to do with private, it's about const, as far
 as I can see.


Both are protection attributes. The problem of taking the delegate of a mutable
member on a const object, is the exact same problem as taking the delegate of a
private member.

In short, you can't 'bypass private using a delegate.' That you can today, is
simple a bug in the compiler.


 I don't see how calling a private method through a
 delegate can be easily solved. I see three solutions:

 1. Make it a compile time error to create a delegate from a private
 method. This would not allow to call that method via a delegate in the
 same module, which is possible if you call the method directly.

 2. Implement some, most likely, very advanced data flow analysis in the
 compiler to detect when a delegate pointing to a private method escapes
 the module. Results in a compile time error. Will most likely not happen.

 3. Make it a runtime error when the delegate is actually called. This
 would probably need some pieces of 2 as well. I don't think the
 community will be happy about adding runtime checks like these.

 Oh, BTW the same needs to be done for function pointers as well since
 you can build a delegate out of a function pointer and an object.


Bypassing the protection type system and delegate escape are very different
things. The escape of a delegate is fully controlled by the coder of the
class/module; with bypass mechanisms, the library writer has no control.


 In Ruby an instance variable is always private. I don't think you can
 declare a public instance variable, one uses setters/getters for that.
 In D you can declare both private and public instance variables.

 In Ruby you can get/set a value of an instance variable using
 reflection, instance_variable_set/get.
 In D you can get/set a value of a private instance variable using
 compile time reflection, i.e. .tupleof.


 Yes, but only for source code you control.


 You don't need control of the source, you only need the static type. Of
 course an opaque struct won't work.


Sorry, that's what I meant by source you control. i.e. it is available to you
to read in some form. Runtime reflection would allow you access to knowledge of
opaque types.


 Actually, baring bugs in DMD, .tupleof is the only way to access private
 fields and there is _no way_ to access private members; And .tupleof
 only works if you have access to the source code. Furthermore, .tupleof,
 god-send that it is, isn't part of the D spec; unlike .stringof, it
 didn't make it into TDPL nor into the D documentation. D's official
 method of compile-time reflection is __traits. And __traits doesn't
 bypass protection attributes. So precedence would be against
 runtime-reflection having access to private members.


 The compiler has .tupleof and it's mentioned in the spec at
 d-programming-language.org: http://d-programming-language.org/struct.html


I stand corrected. Sorry, I just looked at TDPL's index, the page on Properties
and googled it. Now that I look for it, it's also on the Class page.

Still, __traits is still the full fledged compile-time reflection feature and
doesn't bypass protection attributes.
Oct 25 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-25 16:21, Robert Jacques wrote:

 On Tue, 25 Oct 2011 03:23:40 -0400, Jacob Carlborg <doob me.com> wrote:

 For example, getting the name of a instance variable:

 T.tupleof[i].stringof[1 + T.stringof.length + 2 .. $];

 I wouldn't consider that simple and straight forward. This works for
 both D1 and D2, don't know if D2 has a better way.


 Not really, for the advanced user. I do find using "some string".length
 to be cleaner in these cases where you have to slice .stringof. i.e.

 static Value from(T)(T value) {
 //...
 static if(is(T == struct) || is(T == class)) {
 Value[string] result;
 foreach(i,v;value.tupleof) {
 result[value.tupleof[i].stringof["value.".length..$]] = from(v);
 }
 return Value( result );
 }
 //..
 }

 That said, I think anyone can read the above and understand what's
 happening. And even writing that code would be straightforward is
 .stringof and .tupleof were better documented.


Yes, I found the snippet I use to get the name of a field by trail and 
error.


 Also, the first serialization library I wrote used __traits instead. It
 was more awkward, mainly because __traits doesn't an 'allFields' member,
 just an 'allMembers'. If it did then the above becomes:

 foreach(name;__traits(allFields,value) {
 result[name] = from(__traits(getMember, value, name));
 }

 which really is straightforward. Now I originally used 'allMembers' and
 then filtered out everything I didn't want, which was fairly
 straightforward but verbose.


__traits(getMember) is more clean when you the name of all the fields 
you want.


 Your serialization library only works on public fields? Doesn't sound
 very useful.


 In the interest of full disclosure, the library I wrote was quick and
 dirty and does not handle ploy-morphism, cycle detection, aliasing, etc
 (mainly because JSON doesn't); I used traits for my first one (so
 protection was observed) but .tupleof for my second and third and was
 lazy about it. It's one line of code to add protection to tuple of, but
 I've never gotten around to it. I figured my code would be replace by
 orange or std.serialization or something.


Well, things get a bit more complicated when the library needs to 
support ploy-morphism, cycle detection, aliasing and so on.


 That said, I have never serialized a private variable and have never
 wanted too. The serialized objects are, for the most part, part of the
 external API to my program. I have to do validation checks on the inputs
 and changing anything takes a week to get the other teams to propagate
 the changes on their side. There are also a couple of classes which I
 serialize in a binary format as a poor mans database, but again, there's
 no need for anything in it to be private.


Ok, I see.


 [snip]


 Actually, you're not supposed to be able to; this is bug 1983.
 http://d.puremagic.com/issues/show_bug.cgi?id=1983 and
 https://github.com/D-Programming-Language/dmd/pull/71


 I don't see what that has to do with private, it's about const, as far
 as I can see.


 Both are protection attributes. The problem of taking the delegate of a
 mutable member on a const object, is the exact same problem as taking
 the delegate of a private member.


I don't agree.


 In short, you can't 'bypass private using a delegate.' That you can
 today, is simple a bug in the compiler.


 I don't see how calling a private method through a
 delegate can be easily solved. I see three solutions:

 1. Make it a compile time error to create a delegate from a private
 method. This would not allow to call that method via a delegate in the
 same module, which is possible if you call the method directly.

 2. Implement some, most likely, very advanced data flow analysis in the
 compiler to detect when a delegate pointing to a private method escapes
 the module. Results in a compile time error. Will most likely not happen.

 3. Make it a runtime error when the delegate is actually called. This
 would probably need some pieces of 2 as well. I don't think the
 community will be happy about adding runtime checks like these.

 Oh, BTW the same needs to be done for function pointers as well since
 you can build a delegate out of a function pointer and an object.


 Bypassing the protection type system and delegate escape are very
 different things. The escape of a delegate is fully controlled by the
 coder of the class/module; with bypass mechanisms, the library writer
 has no control.


So how should this be fixed? Make it a compile error to create a 
delegate from a private method?


 In Ruby an instance variable is always private. I don't think you can
 declare a public instance variable, one uses setters/getters for that.
 In D you can declare both private and public instance variables.

 In Ruby you can get/set a value of an instance variable using
 reflection, instance_variable_set/get.
 In D you can get/set a value of a private instance variable using
 compile time reflection, i.e. .tupleof.


 Yes, but only for source code you control.


 You don't need control of the source, you only need the static type. Of
 course an opaque struct won't work.


 Sorry, that's what I meant by source you control. i.e. it is available
 to you to read in some form. Runtime reflection would allow you access
 to knowledge of opaque types.


Ok, actually I haven't thought of how runtime reflection would be have 
with an opaque type.


 Actually, baring bugs in DMD, .tupleof is the only way to access private
 fields and there is _no way_ to access private members; And .tupleof
 only works if you have access to the source code. Furthermore, .tupleof,
 god-send that it is, isn't part of the D spec; unlike .stringof, it
 didn't make it into TDPL nor into the D documentation. D's official
 method of compile-time reflection is __traits. And __traits doesn't
 bypass protection attributes. So precedence would be against
 runtime-reflection having access to private members.


 The compiler has .tupleof and it's mentioned in the spec at
 d-programming-language.org: http://d-programming-language.org/struct.html


 I stand corrected. Sorry, I just looked at TDPL's index, the page on
 Properties and googled it. Now that I look for it, it's also on the
 Class page.


No problem.


 Still, __traits is still the full fledged compile-time reflection
 feature and doesn't bypass protection attributes.


Yeah, but for a serialization library I want to be able to serialize 
private fields. This is known issue that this breaks encapsulation.

-- 
/Jacob Carlborg
Oct 25 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 12:38:18 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-10-25 16:21, Robert Jacques wrote:


[snip]

 Both are protection attributes. The problem of taking the delegate of a
 mutable member on a const object, is the exact same problem as taking
 the delegate of a private member.


 I don't agree.


[snip]


 Bypassing the protection type system and delegate escape are very
 different things. The escape of a delegate is fully controlled by the
 coder of the class/module; with bypass mechanisms, the library writer
 has no control.


 So how should this be fixed? Make it a compile error to create a
 delegate from a private method?


I think we are taking about different things. Today in D, due to a bug, for
example

foo.bar()

won't compile because it's private/not-const/etc. But

(&foo.bar)()

will compile. That 'feature' is what I thought you were referring to. If the
writer of a class wants to let a delegate of a private member function escape,
I don't have a fundamental problem with it. But I do have a problem with
someone who isn't the writer of the class doing so.

[snip]


 Still, __traits is still the full fledged compile-time reflection
 feature and doesn't bypass protection attributes.


 Yeah, but for a serialization library I want to be able to serialize
 private fields. This is known issue that this breaks encapsulation.


I understand that at times when serialization has to break encapsulation. My
opinion though, is that classes should have to opt in, before the serializer
goes ahead and breaks their carefully designed encapsulation. i.e. someone
should have to code review the class to make sure everything is kosher. For
code you are developing, this is as simple and straightforward as adding a
single mixin statement to the class.
Oct 25 2011
next sibling parent Kapps <Kapps NotValidEmail.com> writes:
The point of runtime reflection is not your own code, that's the thing.
By having opt-in reflection, you are at the mercy that every single 
component, every piece of code, every library, that you did not write, 
the author decided to enable reflection. It is an unreasonable assumption.


On 25/10/2011 11:44 PM, Robert Jacques wrote:

 I understand that at times when serialization has to break
 encapsulation. My opinion though, is that classes should have to opt in,
 before the serializer goes ahead and breaks their carefully designed
 encapsulation. i.e. someone should have to code review the class to make
 sure everything is kosher. For code you are developing, this is as
 simple and straightforward as adding a single mixin statement to the class.


It is the responsibility of the one using your code to determine whether 
they want take the risk and access private members.
Sometimes, it's needed. I have used private methods in production code 
when programming in .NET, just because the developers of it did not 
realize that something that's private would be useful to some people. It 
saved many, many, hours of pain and effort of rolling your own solution. 
If I upgrade the app to a newer version of .NET and the code breaks, 
that's the risk that was taken. Presumably the person accessing these 
things the author did not intend has taken steps to ensure it doesn't 
break if things change (or doesn't intend to link to a newer version of 
the library).
Oct 25 2011
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-10-26 07:44, Robert Jacques wrote:

 On Tue, 25 Oct 2011 12:38:18 -0400, Jacob Carlborg <doob me.com> wrote:

 On 2011-10-25 16:21, Robert Jacques wrote:


 [snip]

 Both are protection attributes. The problem of taking the delegate of a
 mutable member on a const object, is the exact same problem as taking
 the delegate of a private member.


 I don't agree.


 [snip]


 Bypassing the protection type system and delegate escape are very
 different things. The escape of a delegate is fully controlled by the
 coder of the class/module; with bypass mechanisms, the library writer
 has no control.


 So how should this be fixed? Make it a compile error to create a
 delegate from a private method?


 I think we are taking about different things. Today in D, due to a bug,
 for example

 foo.bar()

 won't compile because it's private/not-const/etc. But

 (&foo.bar)()

 will compile. That 'feature' is what I thought you were referring to. If
 the writer of a class wants to let a delegate of a private member
 function escape, I don't have a fundamental problem with it. But I do
 have a problem with someone who isn't the writer of the class doing so.


I'm referring to the above but, as I've already written and you haven't 
replied to is, I don't understand how it can be prevented.


 [snip]


 Still, __traits is still the full fledged compile-time reflection
 feature and doesn't bypass protection attributes.


 Yeah, but for a serialization library I want to be able to serialize
 private fields. This is known issue that this breaks encapsulation.


 I understand that at times when serialization has to break
 encapsulation. My opinion though, is that classes should have to opt in,
 before the serializer goes ahead and breaks their carefully designed
 encapsulation. i.e. someone should have to code review the class to make
 sure everything is kosher. For code you are developing, this is as
 simple and straightforward as adding a single mixin statement to the class.


It's not straightforward when serializing third party types. One of my 
goals when I started writing my serialization library was to, somewhere 
down the road, create a window/interface builder for DWT. The builder 
would (de)serialize the widgets available in DWT and save it as an XML 
file. Since I'm the current maintainer of DWT I could go and modify DWT 
to fit my needs but I don't think that's the right approach and will 
make merging future versions of SWT more difficult.

-- 
/Jacob Carlborg
Oct 26 2011
prev sibling parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 24.10.2011 17:37, schrieb Robert Jacques:

 That's a very heavy price to pay, just
 from a program maintenance perspective. And if you consider someone
 writing medical or financial software, the privacy concerns of exposing
 private variable to all become very real.
 


I don't think marking a method as private is a good way to hide critical
information. If you can actually call methods (like in real calls, not
some kind of webservice), you can as well use pointers to read the
memory directly.
Or he could run it in a debugger. Or (with root privileges) just look at
the whole memory of the process. Or ...

Cheers,
- Daniel
Oct 24 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 24 Oct 2011 16:34:11 -0400, Daniel Gibson <metalcaedes gmail.com> wrote:

 Am 24.10.2011 17:37, schrieb Robert Jacques:

 That's a very heavy price to pay, just
 from a program maintenance perspective. And if you consider someone
 writing medical or financial software, the privacy concerns of exposing
 private variable to all become very real.


 I don't think marking a method as private is a good way to hide critical
 information. If you can actually call methods (like in real calls, not
 some kind of webservice), you can as well use pointers to read the
 memory directly.
 Or he could run it in a debugger. Or (with root privileges) just look at
 the whole memory of the process. Or ...

 Cheers,
 - Daniel


By your argument, because can ever be truly secure there's no point to trying
to secure it. I don't buy that. There is a world of difference between
obj.reflect("SSN"), and trying to figure out 1) what objects hold SSN numbers
2) what's its field offset is and 3) what its memory representations is.
(because remember, you don't have the source code) And that's assuming you can
compile D code against the library and the library writer has directly exposed
the critical object. More likely, you'll have to repeat steps 1-3 multiple
times to find the data you're after. And runtime reflection means access will
get exposed across language barriers, e.g. in your application's macro language.

Furthermore, security focused languages seem to be centering around the concept
of capabilities, of which private, package, protected, public and extern are
the most rudimentary form of.

Getting back to your original point, I wasn't suggesting that private is a
panacea for security concerns, but it is one of the more powerful tools in our
tool box.
Oct 24 2011
parent reply Kapps <Kapps NotValidEmail.com> writes:
I really like Gor's idea of  noreflect for this. You have a network 
class you don't want as easily reversible, or a security class, or 
private fields such as password. You just mark the module, type, 
methods, or fields, as  noreflect. It could even disable compile-time 
reflection if desired.

It would be important, however, to be able to pass a flag to the 
compiler that marks everything as  noreflect. The idea of opt-in 
reflection is just silly. Standard example is a serialization library. 
Oh hey, you wanna serialize a field of type Vector3? Sorry, the person 
forgot to mark it as  reflected. Oh, you wanna serialize these 
properties of some class? Sorry, not reflected. Opt-out is definitely 
the way to go, as it gives the best of all worlds (and removes all 
overhead if you just pass in something like --version=noreflect to dmd).

On 24/10/2011 10:11 PM, Robert Jacques wrote:

 On Mon, 24 Oct 2011 16:34:11 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:

 Am 24.10.2011 17:37, schrieb Robert Jacques:

 That's a very heavy price to pay, just
 from a program maintenance perspective. And if you consider someone
 writing medical or financial software, the privacy concerns of exposing
 private variable to all become very real.


 I don't think marking a method as private is a good way to hide critical
 information. If you can actually call methods (like in real calls, not
 some kind of webservice), you can as well use pointers to read the
 memory directly.
 Or he could run it in a debugger. Or (with root privileges) just look at
 the whole memory of the process. Or ...

 Cheers,
 - Daniel


 By your argument, because can ever be truly secure there's no point to
 trying to secure it. I don't buy that. There is a world of difference
 between obj.reflect("SSN"), and trying to figure out 1) what objects
 hold SSN numbers 2) what's its field offset is and 3) what its memory
 representations is. (because remember, you don't have the source code)
 And that's assuming you can compile D code against the library and the
 library writer has directly exposed the critical object. More likely,
 you'll have to repeat steps 1-3 multiple times to find the data you're
 after. And runtime reflection means access will get exposed across
 language barriers, e.g. in your application's macro language.

 Furthermore, security focused languages seem to be centering around the
 concept of capabilities, of which private, package, protected, public
 and extern are the most rudimentary form of.

 Getting back to your original point, I wasn't suggesting that private is
 a panacea for security concerns, but it is one of the more powerful
 tools in our tool box.
Oct 25 2011
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-10-26 07:59, Kapps wrote:

 I really like Gor's idea of  noreflect for this. You have a network
 class you don't want as easily reversible, or a security class, or
 private fields such as password. You just mark the module, type,
 methods, or fields, as  noreflect. It could even disable compile-time
 reflection if desired.

 It would be important, however, to be able to pass a flag to the
 compiler that marks everything as  noreflect. The idea of opt-in
 reflection is just silly. Standard example is a serialization library.
 Oh hey, you wanna serialize a field of type Vector3? Sorry, the person
 forgot to mark it as  reflected. Oh, you wanna serialize these
 properties of some class? Sorry, not reflected. Opt-out is definitely
 the way to go, as it gives the best of all worlds (and removes all
 overhead if you just pass in something like --version=noreflect to dmd).


I agree, than can be SO annoying just because someone forgot to think 
about reflection/serialization.

-- 
/Jacob Carlborg
Oct 26 2011
prev sibling parent sclytrack <sclytrack fake.com> writes:
Runtime reflection on B, C, D not on A.
No reflection for private, protected.
[properties, methods, events]

class A
{
}

 reflection
class B: A
{
}


class C: B
{
}

class D: C
{
}




 I really like Gor's idea of  noreflect for this. You have a network
 class you don't want as easily reversible, or a security class, or
 private fields such as password. You just mark the module, type,
 methods, or fields, as  noreflect. It could even disable compile-time
 reflection if desired.
Jan 23 2012
prev sibling next sibling parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 22.10.2011 05:48, schrieb Robert Jacques:

 On Fri, 21 Oct 2011 17:23:17 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:

 Am 21.10.2011 21:07, schrieb Vladimir Panteleev:

 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?


 I'd love to see proper runtime reflection support in D, including
 functionality to get information about available methods (their name and
 parameters) and a way to call them.


 What do you mean by their 'parameters'? What about overloads?
 Attributes? Arguments? Argument attributes?


Primarily arguments. That should help identifying overloads.
But attributes and argument attributes are needed as well.


 Something that is close to what Java offers would be great.


 And what, exactly does JAVA offer? What works? What doesn't work? What's
 missing?


See
http://download.oracle.com/javase/6/docs/api/java/lang/Class.html
You can access constructors (look for one by it's parameters or get all 
of them), the same for methods (methods the class declares and all 
methods, i.e. also inherited ones), implemented interfaces, fields, ...

 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).


 1) Unused information is simply bloat: it increases exe size, slows the
 exe down and increases the runtime memory footprint.


Yeah, but (mostly) only per class, not per object (besides one pointer 
per object, which shouldn't be that bad and something like this seems is 
already there for the existing typeinfo)


 2) On a lot of systems (i.e. consoles, embedded, smart phones, tablets)
 memory and disk space are both highly constrained resources that you
 don't want to waste.
 3) RTTI provides a back-door into a code-base; one that for many reasons
 you may want to keep closed.


Yeah, it may make sense to be able to deactivate it.

Cheers,
- Daniel
Oct 22 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Sat, 22 Oct 2011 07:57:17 -0400, Daniel Gibson <metalcaedes gmail.com> wrote:

 Am 22.10.2011 05:48, schrieb Robert Jacques:


[snip]

 What do you mean by their 'parameters'? What about overloads?
 Attributes? Arguments? Argument attributes?


 Primarily arguments. That should help identifying overloads.
 But attributes and argument attributes are needed as well.


But handling overloads is the responsibility of the dispatch system. What would
be the use cases for this information?


 Something that is close to what Java offers would be great.


 And what, exactly does JAVA offer? What works? What doesn't work? What's
 missing?


 See
 http://download.oracle.com/javase/6/docs/api/java/lang/Class.html
 You can access constructors (look for one by it's parameters or get all
 of them), the same for methods (methods the class declares and all
 methods, i.e. also inherited ones), implemented interfaces, fields, ...


Thanks, I'll have a look see.


 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).


 1) Unused information is simply bloat: it increases exe size, slows the
 exe down and increases the runtime memory footprint.


 Yeah, but (mostly) only per class, not per object (besides one pointer
 per object, which shouldn't be that bad and something like this seems is
 already there for the existing typeinfo)


I was assuming the per object bloat was zero. Oh, and I did forget that RTTI
increases compile times.
Oct 22 2011
parent reply foobar <foo bar.com> writes:
Robert Jacques Wrote:


 On Sat, 22 Oct 2011 07:57:17 -0400, Daniel Gibson <metalcaedes gmail.com>
wrote:

 Am 22.10.2011 05:48, schrieb Robert Jacques:


 [snip]

 What do you mean by their 'parameters'? What about overloads?
 Attributes? Arguments? Argument attributes?


 Primarily arguments. That should help identifying overloads.
 But attributes and argument attributes are needed as well.


 
 But handling overloads is the responsibility of the dispatch system. What
would be the use cases for this information?
 

 Something that is close to what Java offers would be great.


 And what, exactly does JAVA offer? What works? What doesn't work? What's
 missing?


 See
 http://download.oracle.com/javase/6/docs/api/java/lang/Class.html
 You can access constructors (look for one by it's parameters or get all
 of them), the same for methods (methods the class declares and all
 methods, i.e. also inherited ones), implemented interfaces, fields, ...


 
 Thanks, I'll have a look see.
 

 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).


 1) Unused information is simply bloat: it increases exe size, slows the
 exe down and increases the runtime memory footprint.


 Yeah, but (mostly) only per class, not per object (besides one pointer
 per object, which shouldn't be that bad and something like this seems is
 already there for the existing typeinfo)


 
 I was assuming the per object bloat was zero. Oh, and I did forget that RTTI
increases compile times.


IMHO, RTTI should be a built-in feature of the language. 
1) You need one global system agreed upon by all clients. Otherwise, you could
end up in a situation where a single class has more than a single set of
metadata used in several different scenarios. 
2) This information is required anyway in the run-time itself. This is a
requirement for an accurate GC.
3) Other tools might use this info as well, debuggers, IDEs, etc.
4) It's with the spirit of D design - make the common/safe case the default. 

As others mentioned, there should be a compiler switch to turn this off. 
E.g. when developing for an embedded system where memory is scarce it makes
sense to not generate RTTI. It also makes sense to disable the GC and
preallocate the memory.
Oct 22 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Sat, 22 Oct 2011 14:38:22 -0400, foobar <foo bar.com> wrote:
[snip]

 IMHO, RTTI should be a built-in feature of the language.
 1) You need one global system agreed upon by all clients. Otherwise, you could
end up in a situation where a single class has more than a single set of
metadata used in several different scenarios.


Or a singleton class/struct. Yes, you should have a single global system, but
nothing says that the system has to be the language. Library solution are also
valid.


 2) This information is required anyway in the run-time itself. This is a
requirement for an accurate GC.


a) Not every user of D wants to use a GC. In fact, there's currently a push to
remove / improve all the GC mandatory features of D to support ref-counting /
manual memory usage.
b) RTTI is separate for GC BlockInfo. Yes, you could generate the  GC_BlockInfo
from the RTTI, but doing so would be so inefficient that its not worth
contemplating.


 3) Other tools might use this info as well, debuggers, IDEs, etc.


Debuggers and IDEs already have all this information, and a lot more.


 4) It's with the spirit of D design - make the common/safe case the default.


Since when has reflection ever been the common/safe case? .tupleof, despite
being a god-send, is one of the most unsafe features of D. My personal
impression of reflection is that it's one of those features that you only want
1% of the time, but if you don't have it, you're going to be pulling teeth to
work around it.


 As others mentioned, there should be a compiler switch to turn this off.
 E.g. when developing for an embedded system where memory is scarce it makes
sense to not generate RTTI. It also makes sense to disable the GC and
preallocate the memory.


Isn't this statement and statement 4) in conflict with each other? Unlike a GC,
turning off RTTI breaks everything that uses it.
Oct 22 2011
parent reply foobar <foo bar.com> writes:
Robert Jacques Wrote:


 On Sat, 22 Oct 2011 14:38:22 -0400, foobar <foo bar.com> wrote:
 [snip]

 IMHO, RTTI should be a built-in feature of the language.
 1) You need one global system agreed upon by all clients. Otherwise, you could
end up in a situation where a single class has more than a single set of
metadata used in several different scenarios.


 
 Or a singleton class/struct. Yes, you should have a single global system, but
nothing says that the system has to be the language. Library solution are also
valid.


But couldn't someone implement a different singleton class? How would you make
sure the system remains global? 


 

 2) This information is required anyway in the run-time itself. This is a
requirement for an accurate GC.


 
 a) Not every user of D wants to use a GC. In fact, there's currently a push to
remove / improve all the GC mandatory features of D to support ref-counting /
manual memory usage.


Agreed and it is or should be made possible to not use the GC. However the
default for D is GC. 


 b) RTTI is separate for GC BlockInfo. Yes, you could generate the 
GC_BlockInfo from the RTTI, but doing so would be so inefficient that its not
worth contemplating.


I'm not sure I follow. Are you discussing D's current GC implementation?
Generally speaking, an accurate GC AFAIK needs to have full type info. D's
current GC does not have full type info and can have false positives hence it's
a conservative GC. 


 

 3) Other tools might use this info as well, debuggers, IDEs, etc.


 
 Debuggers and IDEs already have all this information, and a lot more.
 

 4) It's with the spirit of D design - make the common/safe case the default.


 
 Since when has reflection ever been the common/safe case? .tupleof, despite
being a god-send, is one of the most unsafe features of D. My personal
impression of reflection is that it's one of those features that you only want
1% of the time, but if you don't have it, you're going to be pulling teeth to
work around it.


That's why it should be able to turn off RTTI generation if it is not needed
but it should be provided by default in order to prevent that pulling teeth
sensation. It's safer to provide an off switch rather than an on switch. 


 

 As others mentioned, there should be a compiler switch to turn this off.
 E.g. when developing for an embedded system where memory is scarce it makes
sense to not generate RTTI. It also makes sense to disable the GC and
preallocate the memory.


 
 Isn't this statement and statement 4) in conflict with each other? Unlike a
GC, turning off RTTI breaks everything that uses it.


where's the conflict? If my application does not use RTTI at all than the
'redundant' info can be removed. In fact, Thinking about it just know, It
should be a link-time optimization - only if the linker did not see any usage
of RTTI in the program, it'll remove the generated meta data.
Oct 22 2011
parent "Robert Jacques" <sandford jhu.edu> writes:
On Sat, 22 Oct 2011 17:37:40 -0400, foobar <foo bar.com> wrote:


 Robert Jacques Wrote:


 On Sat, 22 Oct 2011 14:38:22 -0400, foobar <foo bar.com> wrote:
 [snip]

 IMHO, RTTI should be a built-in feature of the language.
 1) You need one global system agreed upon by all clients. Otherwise, you could
end up in a situation where a single class has more than a single set of
metadata used in several different scenarios.


 Or a singleton class/struct. Yes, you should have a single global system, but
nothing says that the system has to be the language. Library solution are also
valid.


 But couldn't someone implement a different singleton class?


Sure.


 How would you make sure the system remains global?


You put it in Phobos.


 2) This information is required anyway in the run-time itself. This is a
requirement for an accurate GC.


 a) Not every user of D wants to use a GC. In fact, there's currently a push to
remove / improve all the GC mandatory features of D to support ref-counting /
manual memory usage.


 Agreed and it is or should be made possible to not use the GC. However the
default for D is GC.


But nothing depends on the GC. Things would depend on RTTI.


 b) RTTI is separate for GC BlockInfo. Yes, you could generate the 
GC_BlockInfo from the RTTI, but doing so would be so inefficient that its not
worth contemplating.


 I'm not sure I follow. Are you discussing D's current GC implementation?
 Generally speaking, an accurate GC AFAIK needs to have full type info. D's
current GC does not have full type info and can have false positives hence it's
a conservative GC.


The type info used for the GC and the type info for reflection are two very
different things. Saying that having one implies the other is fallacious.

[snip]


 4) It's with the spirit of D design - make the common/safe case the default.


 Since when has reflection ever been the common/safe case? .tupleof, despite
being a god-send, is one of the most unsafe features of D. My personal
impression of reflection is that it's one of those features that you only want
1% of the time, but if you don't have it, you're going to be pulling teeth to
work around it.


 That's why it should be able to turn off RTTI generation if it is not needed
but it should be provided by default in order to prevent that pulling teeth
sensation. It's safer to provide an off switch rather than an on switch.


So, let me get this right. If 99% of the time you don't want something, and 1%
of the time you do, it's your opinion is that the default should be the 1%
case? Because it's safer? By the way, I'm pretty sure we have very different
meaning of the word 'safe' here.


 As others mentioned, there should be a compiler switch to turn this off.
 E.g. when developing for an embedded system where memory is scarce it makes
sense to not generate RTTI. It also makes sense to disable the GC and
preallocate the memory.


 Isn't this statement and statement 4) in conflict with each other? Unlike a
GC, turning off RTTI breaks everything that uses it.


 where's the conflict?


Common features are by their very definition, pervasive and thus not
disable-able. Hence, your statement 4 and 5 are in conflict with each other.
Only one may be true.


 If my application does not use RTTI at all than the 'redundant' info can be
removed.


Generally speaking, no, actually it can't. I guess some form of LTO might be
able to do this, but we are getting into the realm of requiring `a sufficiently
smart compiler`.


 In fact, Thinking about it just know, It should be a link-time optimization -
only if the linker did not see any usage of RTTI in the program, it'll remove
the generated meta data.


Umm, no. That's not what LTO means. LTO means that instead of having a compiler
and a linker, you just have the compiler-front end and the compiler back-end.
And then, maybe, it can not generate the meta-data in the first place. Because
the compiler never is going to solve the 'was this ever accessed' problem.

Besides, D specifically is designed to allow 3rd party object files,
obfuscation, etc, which really isn't compatible with LTO.
Oct 22 2011
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Fri, 21 Oct 2011 23:48:51 -0400, Robert Jacques <sandford jhu.edu>  
wrote:


 On Fri, 21 Oct 2011 17:23:17 -0400, Daniel Gibson  
 <metalcaedes gmail.com> wrote:

 Am 21.10.2011 21:07, schrieb Vladimir Panteleev:

 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?


 I'd love to see proper runtime reflection support in D, including
 functionality to get information about available methods (their name and
 parameters) and a way to call them.


 What do you mean by their 'parameters'? What about overloads?  
 Attributes? Arguments? Argument attributes?


 Something that is close to what Java offers would be great.


 And what, exactly does JAVA offer? What works? What doesn't work? What's  
 missing?


 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).


 1) Unused information is simply bloat: it increases exe size, slows the  
 exe down and increases the runtime memory footprint.
 2) On a lot of systems (i.e. consoles, embedded, smart phones, tablets)  
 memory and disk space are both highly constrained resources that you  
 don't want to waste.
 3) RTTI provides a back-door into a code-base; one that for many reasons  
 you may want to keep closed.


I agree to all these points.  I think runtime reflection should be  
generated by the compiler as needed.  And the mechanism to determine "as  
needed" should be an attribute IMO, or an intrinsic function (i.e. if you  
have all the static information about a type, you should be able to  
generate the RTTI for it).  I do not want programs getting bigger than  
they already are for the sake of "just in case I want to use reflection"...

-Steve
Oct 24 2011
parent reply deadalnix <deadalnix gmail.com> writes:
Le 24/10/2011 16:20, Steven Schveighoffer a Ă©crit :

 On Fri, 21 Oct 2011 23:48:51 -0400, Robert Jacques <sandford jhu.edu>
 wrote:


 On Fri, 21 Oct 2011 17:23:17 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:

 Am 21.10.2011 21:07, schrieb Vladimir Panteleev:

 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?


 I'd love to see proper runtime reflection support in D, including
 functionality to get information about available methods (their name and
 parameters) and a way to call them.


 What do you mean by their 'parameters'? What about overloads?
 Attributes? Arguments? Argument attributes?


 Something that is close to what Java offers would be great.


 And what, exactly does JAVA offer? What works? What doesn't work?
 What's missing?


 BTW: I don't really see the problem with providing this information
 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it (other
 types don't even need that because they're not polymorphic and - like
 methods of structs - the address of the information is known at
 compile-time).


 1) Unused information is simply bloat: it increases exe size, slows
 the exe down and increases the runtime memory footprint.
 2) On a lot of systems (i.e. consoles, embedded, smart phones,
 tablets) memory and disk space are both highly constrained resources
 that you don't want to waste.
 3) RTTI provides a back-door into a code-base; one that for many
 reasons you may want to keep closed.


 I agree to all these points. I think runtime reflection should be
 generated by the compiler as needed. And the mechanism to determine "as
 needed" should be an attribute IMO, or an intrinsic function (i.e. if
 you have all the static information about a type, you should be able to
 generate the RTTI for it). I do not want programs getting bigger than
 they already are for the sake of "just in case I want to use reflection"...

 -Steve


Well, if the compile time reflection is good enough, you can do this 
without any runtime reflexion support in the compiler, just using a lib.

Why not support this option ?
Oct 24 2011
next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Mon, 24 Oct 2011 11:18:34 -0400, deadalnix <deadalnix gmail.com> wrote:
[snip]


 Well, if the compile time reflection is good enough, you can do this
 without any runtime reflexion support in the compiler, just using a lib.

 Why not support this option ?


Some of us are: https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht
Oct 24 2011
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 24 Oct 2011 11:18:34 -0400, deadalnix <deadalnix gmail.com> wrot=
e:


 Le 24/10/2011 16:20, Steven Schveighoffer a =C3=A9crit :

 On Fri, 21 Oct 2011 23:48:51 -0400, Robert Jacques <sandford jhu.edu>=





 wrote:


 On Fri, 21 Oct 2011 17:23:17 -0400, Daniel Gibson
 <metalcaedes gmail.com> wrote:

 Am 21.10.2011 21:07, schrieb Vladimir Panteleev:

 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds =










 =


 RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info =3D  =











 cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name =3D=3D "PrintHello");
 auto print =3D cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remain=










s a

 disputed matter, would anyone be interested in an unofficial patch=










  =


 for
 this?

 Walter: would it be okay if the compiler changes were published as=










 a

 GitHub fork, or should we stick to patches?


 I'd love to see proper runtime reflection support in D, including
 functionality to get information about available methods (their nam=








e  =


 and
 parameters) and a way to call them.


 What do you mean by their 'parameters'? What about overloads?
 Attributes? Arguments? Argument attributes?


 Something that is close to what Java offers would be great.


 And what, exactly does JAVA offer? What works? What doesn't work?
 What's missing?


 BTW: I don't really see the problem with providing this information=









 (overhead-wise) - the information needs to be available once per
 class/struct, but objects of classes just need one pointer to it  =









 (other
 types don't even need that because they're not polymorphic and - li=








ke

 methods of structs - the address of the information is known at
 compile-time).


 1) Unused information is simply bloat: it increases exe size, slows
 the exe down and increases the runtime memory footprint.
 2) On a lot of systems (i.e. consoles, embedded, smart phones,
 tablets) memory and disk space are both highly constrained resources=







 that you don't want to waste.
 3) RTTI provides a back-door into a code-base; one that for many
 reasons you may want to keep closed.


 I agree to all these points. I think runtime reflection should be
 generated by the compiler as needed. And the mechanism to determine "=




as

 needed" should be an attribute IMO, or an intrinsic function (i.e. if=





 you have all the static information about a type, you should be able =




to

 generate the RTTI for it). I do not want programs getting bigger than=





 they already are for the sake of "just in case I want to use  =





 reflection"...

 -Steve


 Well, if the compile time reflection is good enough, you can do this  =



 without any runtime reflexion support in the compiler, just using a li=


b.

 Why not support this option ?


As others have mentioned, there should be a standard way so we don't hav=
e  =

conflicting ways that don't mix.

Plus, it seems logical tome since the compiler is generating *some* RTTI=
,  =

it should generate *all* of it (when asked to do so).

-Steve
Oct 24 2011
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-10-21 21:07, Vladimir Panteleev wrote:

 Hi,

 Igor Stepanov has created a patch for DMD and Druntime which adds RTTI
 information for class and struct members.

 Example:

 import std.stdio;

 class Foo
 {
 static void PrintHello()
 {
 writeln("Hello");
 }
 }
 void main()
 {
 auto info = cast(OffsetTypeInfo_StaticMethod)Foo.classinfo.m_offTi[0];
 assert(info.name == "PrintHello");
 auto print = cast(void function())info.pointer;
 print(); //prints "Hello"
 }

 While the inclusion of such functionality into the language remains a
 disputed matter, would anyone be interested in an unofficial patch for
 this?

 Walter: would it be okay if the compiler changes were published as a
 GitHub fork, or should we stick to patches?


Very cool.

-- 
/Jacob Carlborg
Oct 22 2011
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/21/2011 12:07 PM, Vladimir Panteleev wrote:

 Walter: would it be okay if the compiler changes were published as a GitHub
 fork, or should we stick to patches?


I'd prefer as a patch, because forking the language can result in another
Phobos 
v Tango debacle, where everyone loses.
Oct 23 2011
parent Jacob Carlborg <doob me.com> writes:
On 2011-10-23 22:14, Walter Bright wrote:

 On 10/21/2011 12:07 PM, Vladimir Panteleev wrote:

 Walter: would it be okay if the compiler changes were published as a
 GitHub
 fork, or should we stick to patches?


 I'd prefer as a patch, because forking the language can result in
 another Phobos v Tango debacle, where everyone loses.


I though we used forks on GitHub with pull requests these days.

-- 
/Jacob Carlborg
Oct 24 2011
prev sibling parent reply Jonny Dee <jonnyd gmx.net> writes:
Hi,

I would love to see runtime reflection be available in D. I am currently
reading Walter Bright's "The D
Programming Language" and with every page I read I became more and more
fascinated and enthusiastic
about D. But then I started to wonder why I haven't read about runtime
reflection yet, so I did some Internet
research and was a bit disappointed this feature isn't available. D is really
cool, it's not only C done right,

language available yet. It's really a
pitty RTTI is not available. So I hope your approach will somehow influence D's
future :)

Keep going,
Jonny
Oct 25 2011
next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
Hear, hear!

On Tue, Oct 25, 2011 at 5:40 PM, Jonny Dee <jonnyd gmx.net> wrote:

 Hi,

 I would love to see runtime reflection be available in D. I am currently
reading Walter Bright's "The D
 Programming Language" and with every page I read I became more and more
fascinated and enthusiastic
 about D. But then I started to wonder why I haven't read about runtime
reflection yet, so I did some Internet
 research and was a bit disappointed this feature isn't available. D is really
cool, it's not only C done right,

language available yet. It's really a
 pitty RTTI is not available. So I hope your approach will somehow influence
D's future :)

 Keep going,
 Jonny
Oct 25 2011
prev sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:

 Hi,

 I would love to see runtime reflection be available in D. I am currently
reading Walter Bright's "The D
 Programming Language" and with every page I read I became more and more
fascinated and enthusiastic
 about D. But then I started to wonder why I haven't read about runtime
reflection yet, so I did some Internet
 research and was a bit disappointed this feature isn't available. D is really
cool, it's not only C done right,

language available yet. It's really a
 pitty RTTI is not available. So I hope your approach will somehow influence
D's future :)

 Keep going,
 Jonny


Hi Jonny,
Well, some RTTI is available. How much RTTI is needed, really depends on what
you're trying to do. Therefore whether D has or hasn't got RTTI depends heavily
on what any particular person means by RTTI. Do you have some use cases in
mind? Maybe prototype objects, duck-typing/casting or serialization? Or perhaps
something you wrote?

Sorry for all the questions, but I feel as if we don't have a resident
reflection expert in the dialog, or even the opinions of basic/experienced
users. We all know that 'RTTI' is good, but we're a little vague on the how and
why. (Well, beyond the use cases above)
Oct 25 2011
parent reply Jonny Dee <jonnyd gmx.net> writes:
Am 25.10.11 16:41, schrieb Robert Jacques:

 On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:

 Hi,

 I would love to see runtime reflection be available in D. I am
 currently reading Walter Bright's "The D
 Programming Language" and with every page I read I became more and
 more fascinated and enthusiastic
 about D. But then I started to wonder why I haven't read about runtime
 reflection yet, so I did some Internet
 research and was a bit disappointed this feature isn't available. D is
 really cool, it's not only C done right,

 programming language available yet. It's really a
 pitty RTTI is not available. So I hope your approach will somehow
 influence D's future :)

 Keep going,
 Jonny


 Hi Jonny,
 Well, some RTTI is available. How much RTTI is needed, really depends on
 what you're trying to do. Therefore whether D has or hasn't got RTTI
 depends heavily on what any particular person means by RTTI. Do you have
 some use cases in mind? Maybe prototype objects, duck-typing/casting or
 serialization? Or perhaps something you wrote?

 Sorry for all the questions, but I feel as if we don't have a resident
 reflection expert in the dialog, or even the opinions of
 basic/experienced users. We all know that 'RTTI' is good, but we're a
 little vague on the how and why. (Well, beyond the use cases above)


Hi Robert,

Well, before I tell you what I would like to see I'll cite Wikipedia [1]:
"
[...]
- Discover and modify source code constructions (such as code blocks, 
classes, methods, protocols, etc.) as a first-class object at runtime.
- Convert a string matching the symbolic name of a class or function 
into a reference to or invocation of that class or function.
[...]
"

Here is what I would dream of for arbitrary objects/classes (not 
necessarily known at compile-time):
- Query an object for its list of methods together with their 
signatures. Select a method, bind some values to its arguments, call it, 
and retrieve the return type (if any).
- Query an object for its public fields (at least), and provide a way to 
get/set their values.
- Query an object's class for all implemented interfaces and its base class.
- Query a module for all type definitions and provide a way to 
introspect these types in more detail. For instance, it would be really 
cool if I could find a class with name "Car" in module "cars", get a 
list of all defined constructors, select one, bind values to the 
constructor's parameters, and create a corresponding object.

Now you might ask where is the use case. I think, a very important one, 
besides creating GUI designers, is to be able to implement a Dependency 
Injection container [2,3,4] like, for example, the one provided by the 
Spring Framework for Java or .NET. In short, they allow you define how 
object trees should be build up solely by specifying this tree in an XML 
file. Within the XML file you can create new instances of classes of 
arbitrary types. You can specify which constructors to use for 
instantiation, and what values the constructor's arguments (if any) 
should have. You can even bind such an argument to an object previously 
created by the XML definition. You can call methods on existing objects, 
e.g. in order to call setter to further initialize an object with values 
you also define in that XML file. That XML file can be loaded when your 
program starts an make the DI container component instantiate all needed 
objects for you. There is no need for recompilation if the XML file is 
changed. Just restart your application.

Implementing such a DI container heavily depends on reflection, because 
the DI container component doesn't know anything about the objects to be 
created during runtime.

Qt also extends C++ with a reflection mechanism through the help of its 
meta object compiler (moc). It analyses the C++ source code, generates 
meta class definitions [6,7] and weaves them into your Qt class. Hence, 
in Qt, you can query an object for fields, methods, interfaces, etc. and 
you can call methods with arbitrary parameters, or you can instantiate a 
class using an arbitrary constructor. Consequently, somone implemented a 
DI container for C++ which is based on Qt and works more or less the 
same way the Spring DI container does. You can build up object trees 
simply by specifying such trees in an XML file.

I don't go into why dependency injection is a very powerful feature. 
This is Martin Fowler's [3] job ;) But when I program with C++ I miss 
such a flexible dependency injection mechanism a lot. And I hope this 
will eventually be available for D.

Cheers,
Jonny

[1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29
[2] http://en.wikipedia.org/wiki/Dependency_injection
[3] http://martinfowler.com/articles/injection.html
[4] 
http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29
[5] http://qtioccontainer.sourceforge.net/
[6] http://doc.qt.nokia.com/stable/qmetaobject.html
[7] 
http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx
Oct 25 2011
parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 25 Oct 2011 19:35:52 -0400, Jonny Dee <jonnyd gmx.net> wrote:

 Am 25.10.11 16:41, schrieb Robert Jacques:

 On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:

 Hi,

 I would love to see runtime reflection be available in D. I am
 currently reading Walter Bright's "The D
 Programming Language" and with every page I read I became more and
 more fascinated and enthusiastic
 about D. But then I started to wonder why I haven't read about runtime
 reflection yet, so I did some Internet
 research and was a bit disappointed this feature isn't available. D is
 really cool, it's not only C done right,

 programming language available yet. It's really a
 pitty RTTI is not available. So I hope your approach will somehow
 influence D's future :)

 Keep going,
 Jonny


 Hi Jonny,
 Well, some RTTI is available. How much RTTI is needed, really depends on
 what you're trying to do. Therefore whether D has or hasn't got RTTI
 depends heavily on what any particular person means by RTTI. Do you have
 some use cases in mind? Maybe prototype objects, duck-typing/casting or
 serialization? Or perhaps something you wrote?

 Sorry for all the questions, but I feel as if we don't have a resident
 reflection expert in the dialog, or even the opinions of
 basic/experienced users. We all know that 'RTTI' is good, but we're a
 little vague on the how and why. (Well, beyond the use cases above)


 Hi Robert,

 Well, before I tell you what I would like to see I'll cite Wikipedia [1]:
 "
 [...]
 - Discover and modify source code constructions (such as code blocks,
 classes, methods, protocols, etc.) as a first-class object at runtime.
 - Convert a string matching the symbolic name of a class or function
 into a reference to or invocation of that class or function.
 [...]
 "

 Here is what I would dream of for arbitrary objects/classes (not
 necessarily known at compile-time):
 - Query an object for its list of methods together with their
 signatures. Select a method, bind some values to its arguments, call it,
 and retrieve the return type (if any).
 - Query an object for its public fields (at least), and provide a way to
 get/set their values.
 - Query an object's class for all implemented interfaces and its base class.
 - Query a module for all type definitions and provide a way to
 introspect these types in more detail. For instance, it would be really
 cool if I could find a class with name "Car" in module "cars", get a
 list of all defined constructors, select one, bind values to the
 constructor's parameters, and create a corresponding object.

 Now you might ask where is the use case. I think, a very important one,
 besides creating GUI designers, is to be able to implement a Dependency
 Injection container [2,3,4] like, for example, the one provided by the
 Spring Framework for Java or .NET. In short, they allow you define how
 object trees should be build up solely by specifying this tree in an XML
 file. Within the XML file you can create new instances of classes of
 arbitrary types. You can specify which constructors to use for
 instantiation, and what values the constructor's arguments (if any)
 should have. You can even bind such an argument to an object previously
 created by the XML definition. You can call methods on existing objects,
 e.g. in order to call setter to further initialize an object with values
 you also define in that XML file. That XML file can be loaded when your
 program starts an make the DI container component instantiate all needed
 objects for you. There is no need for recompilation if the XML file is
 changed. Just restart your application.

 Implementing such a DI container heavily depends on reflection, because
 the DI container component doesn't know anything about the objects to be
 created during runtime.

 Qt also extends C++ with a reflection mechanism through the help of its
 meta object compiler (moc). It analyses the C++ source code, generates
 meta class definitions [6,7] and weaves them into your Qt class. Hence,
 in Qt, you can query an object for fields, methods, interfaces, etc. and
 you can call methods with arbitrary parameters, or you can instantiate a
 class using an arbitrary constructor. Consequently, somone implemented a
 DI container for C++ which is based on Qt and works more or less the
 same way the Spring DI container does. You can build up object trees
 simply by specifying such trees in an XML file.

 I don't go into why dependency injection is a very powerful feature.
 This is Martin Fowler's [3] job ;) But when I program with C++ I miss
 such a flexible dependency injection mechanism a lot. And I hope this
 will eventually be available for D.

 Cheers,
 Jonny

 [1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29
 [2] http://en.wikipedia.org/wiki/Dependency_injection
 [3] http://martinfowler.com/articles/injection.html
 [4]
 http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29
 [5] http://qtioccontainer.sourceforge.net/
 [6] http://doc.qt.nokia.com/stable/qmetaobject.html
 [7]
 http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx


Hi Jonny,
Thank you for your informative (and well cited) post. It has provided me with a
new take on an old design pattern and some enjoyable reading. In return, let me
outline my opinion of reflection in D today, and tomorrow, as it pertains to
your wish list.

Reflection in D today is very different from the host of VM languages that have
popularized the concept. Being a compiled systems language, actual runtime
self-modification is too virus like to become at a language level feature.
However, given the compilation speed of D, people have made proof of concept
libraries that essentially wrapped the compiler and dynamically loaded the
result. As LDC uses LLVM, which has a jit backend, I'd expect to see something
get into and D 'eval' library into etc eventually. (phobos uses the BOOST
license, which isn't compatible with LLVM).

Compile-time reflection and generation of code, on the other hand, is something
D does in spades. It fulfills your dream list, although I think module level
reflection might only be available in the github version. The API design is
still in flux and we are actively iterating / improving it as find new uses
cases and bugs. The current plan is to migrate all the traits functions over to
a special 'meta' namespace (i.e. __traits(allMembers,D) => meta.allMembers(T)
). Good solid libraries for each of the concepts I listed, (prototype objects,
duck-typing/casting or serialization), have been written using the compile-time
meta-programming features in D. So that's the good.

On the other hand, D's runtime capabilities are limited to object.factory, the
under implemented RTTI and library solutions which manually expose information
gathered by D's compile-time mechanisms. And so far, these tools have been more
than enough, from a functionality point of view. Most of our desire for better
runtime reflection stems from a desire for efficiency, composition, cleanliness
of user syntax and simplification of library code. These are all important
issues for the widespread use of reflection based libraries, but they're not 'I
can't implement X' issues.

As for the future, I believe that the division in D between compile-time and
run-time reflection warrants a serious look at the design of the run-time half
of the system. To that end, I believe that implementing reflection in a library
would be the best way to experiment and iterate an API. To that end, I have a
proposal in the review queue to improve std.variant which contains dynamic
dispatch (i.e. the ability to get/set public fields and call methods),
duck-typeing/casting and prototype-style objects. Notably, it's missing
introspection capabilities as thats what I'm most unsure about API wise, and
simplest to add. Designing reflection inside a library keeps Walter & Co's
bandwidth free for other bugs/features and provides a very good stress test of
D's meta-programming capabilities. (Both of which I think are good things)
(I am soliciting feedback, if you care to take a look:
https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht)

And there are many reasons we might want to experiment with D's runtime
reflection API instead of just copying someone. For example, take Dependency
Injection. Using Java-style reflection, DI suffers from the Reflection
injection and Unsafe Reflection security vulnerabilities. To do DI safely, you
have to validate your inputs and D's compile-time reflection provides a perfect
way to implement validated DI. Every time I hear about some form of injection,
be it SQL or JSON or Reflection, hit the news, makes me think that 5-10 years
from now well look back on the use of unvalidated dynamic code constructs the
same way we do about null terminated arrays today.
Oct 25 2011
parent reply Jonny Dee <jonnyd gmx.net> writes:
Hello Robert,

Am 26.10.11 07:16, schrieb Robert Jacques:

 On Tue, 25 Oct 2011 19:35:52 -0400, Jonny Dee <jonnyd gmx.net> wrote:

 Am 25.10.11 16:41, schrieb Robert Jacques:

 On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:

 [...]




 Hi Robert,

 Well, before I tell you what I would like to see I'll cite Wikipedia [1]:
 "
 [...]
 - Discover and modify source code constructions (such as code blocks,
 classes, methods, protocols, etc.) as a first-class object at runtime.
 - Convert a string matching the symbolic name of a class or function
 into a reference to or invocation of that class or function.
 [...]
 "

 Here is what I would dream of for arbitrary objects/classes (not
 necessarily known at compile-time):
 - Query an object for its list of methods together with their
 signatures. Select a method, bind some values to its arguments, call it,
 and retrieve the return type (if any).
 - Query an object for its public fields (at least), and provide a way to
 get/set their values.
 - Query an object's class for all implemented interfaces and its base
 class.
 - Query a module for all type definitions and provide a way to
 introspect these types in more detail. For instance, it would be really
 cool if I could find a class with name "Car" in module "cars", get a
 list of all defined constructors, select one, bind values to the
 constructor's parameters, and create a corresponding object.

 [...]

 Implementing such a DI container heavily depends on reflection, because
 the DI container component doesn't know anything about the objects to be
 created during runtime.

 Qt also extends C++ with a reflection mechanism through the help of its
 meta object compiler (moc). It analyses the C++ source code, generates
 meta class definitions [6,7] and weaves them into your Qt class. Hence,
 in Qt, you can query an object for fields, methods, interfaces, etc. and
 you can call methods with arbitrary parameters, or you can instantiate a
 class using an arbitrary constructor. Consequently, somone implemented a
 DI container for C++ which is based on Qt and works more or less the
 same way the Spring DI container does. You can build up object trees
 simply by specifying such trees in an XML file.

 I don't go into why dependency injection is a very powerful feature.
 This is Martin Fowler's [3] job ;) But when I program with C++ I miss
 such a flexible dependency injection mechanism a lot. And I hope this
 will eventually be available for D.

 Cheers,
 Jonny

 [1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29
 [2] http://en.wikipedia.org/wiki/Dependency_injection
 [3] http://martinfowler.com/articles/injection.html
 [4]
 http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29

 [5] http://qtioccontainer.sourceforge.net/
 [6] http://doc.qt.nokia.com/stable/qmetaobject.html
 [7]
 http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx


 Hi Jonny,
 Thank you for your informative (and well cited) post. It has provided me
 with a new take on an old design pattern and some enjoyable reading. In
 return, let me outline my opinion of reflection in D today, and
 tomorrow, as it pertains to your wish list.


Many thanks to you, too, for your very elaborate answer :)


 Reflection in D today is very different from the host of VM languages
 that have popularized the concept. Being a compiled systems language,
 actual runtime self-modification is too virus like to become at a
 language level feature. However, given the compilation speed of D,
 people have made proof of concept libraries that essentially wrapped the
 compiler and dynamically loaded the result. As LDC uses LLVM, which has
 a jit backend, I'd expect to see something get into and D 'eval' library
 into etc eventually. (phobos uses the BOOST license, which isn't
 compatible with LLVM).


I know, that "runtime self-modification" and runtime code generation is 
a "dangerous" feature. And there really are rare cases where using such 
an approach might justify the risc in using it. Although this feature is 
not on my wish list, it might be good for generating dynamic proxies to 
arbitrary object instances like they are used by some ORMs. See 
Hibernate/NHibernate, for example [1,2]. Another example is 
aspect-oriented programming. But while I can't see the exacty reason for 
it, such a feature might indeed be a feature which is more appropriate 
for VM languages.


 Compile-time reflection and generation of code, on the other hand, is
 something D does in spades. It fulfills your dream list, although I
 think module level reflection might only be available in the github
 version. The API design is still in flux and we are actively iterating /
 improving it as find new uses cases and bugs. The current plan is to
 migrate all the traits functions over to a special 'meta' namespace
 (i.e. __traits(allMembers,D) => meta.allMembers(T) ). Good solid
 libraries for each of the concepts I listed, (prototype objects,
 duck-typing/casting or serialization), have been written using the
 compile-time meta-programming features in D. So that's the good.


A absolutely agree! D's compile-time reflection is very good. And it's 
hard to top this.


 On the other hand, D's runtime capabilities are limited to
 object.factory, the under implemented RTTI and library solutions which
 manually expose information gathered by D's compile-time mechanisms. And
 so far, these tools have been more than enough, from a functionality
 point of view. Most of our desire for better runtime reflection stems
 from a desire for efficiency, composition, cleanliness of user syntax
 and simplification of library code. These are all important issues for
 the widespread use of reflection based libraries, but they're not 'I
 can't implement X' issues.


You are right, there IS certainly always some way to 'implement X'. But, 
as you know, there is always a consideration of the effort you need to 
implement X.


 As for the future, I believe that the division in D between compile-time
 and run-time reflection warrants a serious look at the design of the
 run-time half of the system. To that end, I believe that implementing
 reflection in a library would be the best way to experiment and iterate
 an API. To that end, I have a proposal in the review queue to improve
 std.variant which contains dynamic dispatch (i.e. the ability to get/set
 public fields and call methods), duck-typeing/casting and
 prototype-style objects. Notably, it's missing introspection
 capabilities as thats what I'm most unsure about API wise, and simplest
 to add. Designing reflection inside a library keeps Walter & Co's
 bandwidth free for other bugs/features and provides a very good stress
 test of D's meta-programming capabilities. (Both of which I think are
 good things)


I've got no problem with an approach which puts runtime reflection 
capabiities into a separate library. No matter were you look at, Java, 

however, how this might be done without compiler support. As already 
mentioned, Qt has its own moc compiler, which parses your C++ source 
code and generates the necessary infra structure. I'm still a beginner 
with respect to D, so I don't know what is really already possible, and 
what not. But my current feeling is that a similar approach would also 
be needed for D. If the D compiler itself, or another post-compiler 
component should generate the meta information for runtime reflection is 
another question. I could live with both, although I'd prefer the former 
built-in one.


 (I am soliciting feedback, if you care to take a look:
 https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht)


Thanks for this link, I'll certainly have a look at it.


 And there are many reasons we might want to experiment with D's runtime
 reflection API instead of just copying someone. For example, take
 Dependency Injection. Using Java-style reflection, DI suffers from the
 Reflection injection and Unsafe Reflection security vulnerabilities. To
 do DI safely, you have to validate your inputs and D's compile-time
 reflection provides a perfect way to implement validated DI. Every time
 I hear about some form of injection, be it SQL or JSON or Reflection,
 hit the news, makes me think that 5-10 years from now well look back on
 the use of unvalidated dynamic code constructs the same way we do about
 null terminated arrays today.


Considering security vulnerabilities is of course a very important 
matter. However, I think security must be assured a by software's 
design. Enforcing it solely by programming language constructs will not 
work for all cases. Particularly not, if this programming language 
allows direct access to a computer's memory, like C, C++, and D does. 
There is no sandbox out-of-the-box, where the compiled program runs in. 
So if you have a private field in a class that carries a password, one 
has to make sure it's only in memory as long as it is required. And if 
it is not required anymore one should clear it out with zeros, for 
example. So making a field private is by no means a secure solution. A 
hacker will not give up just because the official API declares a field 
to be private.

I consider 'unsafe reflection', as you call it, as a tool. Let's compare 
it to a knife, for instance. It is really a useful tool, I think nobody 
doubts, but at the same time it can be very dangerous. But inspite of 
this fact, everybody has not only one at home, I guess. Pointers are a 
useful tool, too. But they are also dangerous and still available in D, 
which is a good thing, because you can use this tool where needed. And 
actually, I don't think runtime reflection must be unsafe. The 
reflection mechanism provides type information for an object's 
properties, functions, arguments, etc. So validated DI is even possible 
with runtime reflection. Let's consider the the XML configuration of 
object trees once again. If you want to store a string to an int-field, 
for instance, then the DI container can refuse to do this, because it 
has access to all information required to enforce correct value types.

One more use case for reflection is data binding with GUI components. 
This approach is heavily used in Windows Presentation Foundation library 
[3,4]. GUI components can update an object's properties by using 
reflection. You don't need to register listeners for this purpose anymore.

That said, I don't think dynamic code constructs will be old-fashioned 
in 5-10 years, because you don't have to go an "unvalidated dynamic" way 
with runtime reflection.

BTW, as this thread also discusses an opt-in or opt-out implementation 
for runtime reflection. I'd prefer an opt-out way, too. Code bloat (I 
guess the binaries are meant) is not as bad as it might sound with 
today's memory sizes. And if one wants to avoid it for optimization 
purpose, one can do it. The use of D as a systems programming language 
for embedded systems is, as I've read somewhere, not a first citizen 
anyway, because you'll get a lot of code for the garbage collection 
mechanism.

Cheers,
Jonny

[1] http://en.wikipedia.org/wiki/Hibernate_%28Java%29
[2] http://en.wikipedia.org/wiki/Nhibernate
[3] http://en.wikipedia.org/wiki/Windows_Presentation_Foundation
[4] 
http://blogs.msdn.com/b/wpfsdk/archive/2006/10/19/wpf-basic-data-binding-faq.aspx
Oct 26 2011
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2011-10-26 20:20, Jonny Dee wrote:

 One more use case for reflection is data binding with GUI components.
 This approach is heavily used in Windows Presentation Foundation library
 [3,4]. GUI components can update an object's properties by using
 reflection. You don't need to register listeners for this purpose anymore.


Runtime reflection is used quite a lot in GUI programming on Mac OS X 
(at least the libraries/tools under the hood).

-- 
/Jacob Carlborg
Oct 26 2011
prev sibling next sibling parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Wed, 26 Oct 2011 11:20:47 -0700, Jonny Dee <jonnyd gmx.net> wrote:


 Hello Robert,

 Am 26.10.11 07:16, schrieb Robert Jacques:

 On Tue, 25 Oct 2011 19:35:52 -0400, Jonny Dee <jonnyd gmx.net> wrote:

 Am 25.10.11 16:41, schrieb Robert Jacques:

 On Tue, 25 Oct 2011 09:40:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:

 [...]




 Hi Robert,

 Well, before I tell you what I would like to see I'll cite Wikipedia  
 [1]:
 "
 [...]
 - Discover and modify source code constructions (such as code blocks,
 classes, methods, protocols, etc.) as a first-class object at runtime.
 - Convert a string matching the symbolic name of a class or function
 into a reference to or invocation of that class or function.
 [...]
 "

 Here is what I would dream of for arbitrary objects/classes (not
 necessarily known at compile-time):
 - Query an object for its list of methods together with their
 signatures. Select a method, bind some values to its arguments, call  
 it,
 and retrieve the return type (if any).
 - Query an object for its public fields (at least), and provide a way  
 to
 get/set their values.
 - Query an object's class for all implemented interfaces and its base
 class.
 - Query a module for all type definitions and provide a way to
 introspect these types in more detail. For instance, it would be really
 cool if I could find a class with name "Car" in module "cars", get a
 list of all defined constructors, select one, bind values to the
 constructor's parameters, and create a corresponding object.

 [...]

 Implementing such a DI container heavily depends on reflection, because
 the DI container component doesn't know anything about the objects to  
 be
 created during runtime.

 Qt also extends C++ with a reflection mechanism through the help of its
 meta object compiler (moc). It analyses the C++ source code, generates
 meta class definitions [6,7] and weaves them into your Qt class. Hence,
 in Qt, you can query an object for fields, methods, interfaces, etc.  
 and
 you can call methods with arbitrary parameters, or you can instantiate  
 a
 class using an arbitrary constructor. Consequently, somone implemented  
 a
 DI container for C++ which is based on Qt and works more or less the
 same way the Spring DI container does. You can build up object trees
 simply by specifying such trees in an XML file.

 I don't go into why dependency injection is a very powerful feature.
 This is Martin Fowler's [3] job ;) But when I program with C++ I miss
 such a flexible dependency injection mechanism a lot. And I hope this
 will eventually be available for D.

 Cheers,
 Jonny

 [1] http://en.wikipedia.org/wiki/Reflection_%28computer_programming%29
 [2] http://en.wikipedia.org/wiki/Dependency_injection
 [3] http://martinfowler.com/articles/injection.html
 [4]
 http://en.wikipedia.org/wiki/Spring_Framework#Inversion_of_Control_container_.28Dependency_injection.29

 [5] http://qtioccontainer.sourceforge.net/
 [6] http://doc.qt.nokia.com/stable/qmetaobject.html
 [7]
 http://blogs.msdn.com/b/willy-peter_schaub/archive/2010/06/03/unisa-chatter-reflection-using-qt.aspx


 Hi Jonny,
 Thank you for your informative (and well cited) post. It has provided me
 with a new take on an old design pattern and some enjoyable reading. In
 return, let me outline my opinion of reflection in D today, and
 tomorrow, as it pertains to your wish list.


 Many thanks to you, too, for your very elaborate answer :)


 Reflection in D today is very different from the host of VM languages
 that have popularized the concept. Being a compiled systems language,
 actual runtime self-modification is too virus like to become at a
 language level feature. However, given the compilation speed of D,
 people have made proof of concept libraries that essentially wrapped the
 compiler and dynamically loaded the result. As LDC uses LLVM, which has
 a jit backend, I'd expect to see something get into and D 'eval' library
 into etc eventually. (phobos uses the BOOST license, which isn't
 compatible with LLVM).


 I know, that "runtime self-modification" and runtime code generation is  
 a "dangerous" feature. And there really are rare cases where using such  
 an approach might justify the risc in using it. Although this feature is  
 not on my wish list, it might be good for generating dynamic proxies to  
 arbitrary object instances like they are used by some ORMs. See  
 Hibernate/NHibernate, for example [1,2]. Another example is  
 aspect-oriented programming. But while I can't see the exacty reason for  
 it, such a feature might indeed be a feature which is more appropriate  
 for VM languages.


 Compile-time reflection and generation of code, on the other hand, is
 something D does in spades. It fulfills your dream list, although I
 think module level reflection might only be available in the github
 version. The API design is still in flux and we are actively iterating /
 improving it as find new uses cases and bugs. The current plan is to
 migrate all the traits functions over to a special 'meta' namespace
 (i.e. __traits(allMembers,D) => meta.allMembers(T) ). Good solid
 libraries for each of the concepts I listed, (prototype objects,
 duck-typing/casting or serialization), have been written using the
 compile-time meta-programming features in D. So that's the good.


 A absolutely agree! D's compile-time reflection is very good. And it's  
 hard to top this.


 On the other hand, D's runtime capabilities are limited to
 object.factory, the under implemented RTTI and library solutions which
 manually expose information gathered by D's compile-time mechanisms. And
 so far, these tools have been more than enough, from a functionality
 point of view. Most of our desire for better runtime reflection stems
 from a desire for efficiency, composition, cleanliness of user syntax
 and simplification of library code. These are all important issues for
 the widespread use of reflection based libraries, but they're not 'I
 can't implement X' issues.


 You are right, there IS certainly always some way to 'implement X'. But,  
 as you know, there is always a consideration of the effort you need to  
 implement X.


 As for the future, I believe that the division in D between compile-time
 and run-time reflection warrants a serious look at the design of the
 run-time half of the system. To that end, I believe that implementing
 reflection in a library would be the best way to experiment and iterate
 an API. To that end, I have a proposal in the review queue to improve
 std.variant which contains dynamic dispatch (i.e. the ability to get/set
 public fields and call methods), duck-typeing/casting and
 prototype-style objects. Notably, it's missing introspection
 capabilities as thats what I'm most unsure about API wise, and simplest
 to add. Designing reflection inside a library keeps Walter & Co's
 bandwidth free for other bugs/features and provides a very good stress
 test of D's meta-programming capabilities. (Both of which I think are
 good things)


 I've got no problem with an approach which puts runtime reflection  
 capabiities into a separate library. No matter were you look at, Java,  

 however, how this might be done without compiler support. As already  
 mentioned, Qt has its own moc compiler, which parses your C++ source  
 code and generates the necessary infra structure. I'm still a beginner  
 with respect to D, so I don't know what is really already possible, and  
 what not. But my current feeling is that a similar approach would also  
 be needed for D. If the D compiler itself, or another post-compiler  
 component should generate the meta information for runtime reflection is  
 another question. I could live with both, although I'd prefer the former  
 built-in one.


 (I am soliciting feedback, if you care to take a look:
 https://jshare.johnshopkins.edu/rjacque2/public_html/variant.mht)


 Thanks for this link, I'll certainly have a look at it.


 And there are many reasons we might want to experiment with D's runtime
 reflection API instead of just copying someone. For example, take
 Dependency Injection. Using Java-style reflection, DI suffers from the
 Reflection injection and Unsafe Reflection security vulnerabilities. To
 do DI safely, you have to validate your inputs and D's compile-time
 reflection provides a perfect way to implement validated DI. Every time
 I hear about some form of injection, be it SQL or JSON or Reflection,
 hit the news, makes me think that 5-10 years from now well look back on
 the use of unvalidated dynamic code constructs the same way we do about
 null terminated arrays today.


 Considering security vulnerabilities is of course a very important  
 matter. However, I think security must be assured a by software's  
 design. Enforcing it solely by programming language constructs will not  
 work for all cases. Particularly not, if this programming language  
 allows direct access to a computer's memory, like C, C++, and D does.  
 There is no sandbox out-of-the-box, where the compiled program runs in.  
 So if you have a private field in a class that carries a password, one  
 has to make sure it's only in memory as long as it is required. And if  
 it is not required anymore one should clear it out with zeros, for  
 example. So making a field private is by no means a secure solution. A  
 hacker will not give up just because the official API declares a field  
 to be private.

 I consider 'unsafe reflection', as you call it, as a tool. Let's compare  
 it to a knife, for instance. It is really a useful tool, I think nobody  
 doubts, but at the same time it can be very dangerous. But inspite of  
 this fact, everybody has not only one at home, I guess. Pointers are a  
 useful tool, too. But they are also dangerous and still available in D,  
 which is a good thing, because you can use this tool where needed. And  
 actually, I don't think runtime reflection must be unsafe. The  
 reflection mechanism provides type information for an object's  
 properties, functions, arguments, etc. So validated DI is even possible  
 with runtime reflection. Let's consider the the XML configuration of  
 object trees once again. If you want to store a string to an int-field,  
 for instance, then the DI container can refuse to do this, because it  
 has access to all information required to enforce correct value types.

 One more use case for reflection is data binding with GUI components.  
 This approach is heavily used in Windows Presentation Foundation library  
 [3,4]. GUI components can update an object's properties by using  
 reflection. You don't need to register listeners for this purpose  
 anymore.

 That said, I don't think dynamic code constructs will be old-fashioned  
 in 5-10 years, because you don't have to go an "unvalidated dynamic" way  
 with runtime reflection.

 BTW, as this thread also discusses an opt-in or opt-out implementation  
 for runtime reflection. I'd prefer an opt-out way, too. Code bloat (I  
 guess the binaries are meant) is not as bad as it might sound with  
 today's memory sizes. And if one wants to avoid it for optimization  
 purpose, one can do it. The use of D as a systems programming language  
 for embedded systems is, as I've read somewhere, not a first citizen  
 anyway, because you'll get a lot of code for the garbage collection  
 mechanism.

 Cheers,
 Jonny

 [1] http://en.wikipedia.org/wiki/Hibernate_%28Java%29
 [2] http://en.wikipedia.org/wiki/Nhibernate
 [3] http://en.wikipedia.org/wiki/Windows_Presentation_Foundation
 [4]  
 http://blogs.msdn.com/b/wpfsdk/archive/2006/10/19/wpf-basic-data-binding-faq.aspx


I am a first time poster but I've been lurking around here for a while now  
and this topic is something that is extremely near and dear to my heart. I  

on WPF while it was still called Avalon. I know you mention that WPF uses  
Reflection, and on that statement I can only say that you are correct, for  
better or worse, usually worse. However, thats not the complete story. WPF  
uses something called a DependencyObject to implement objects with  
data-binding. And this is important as literally EVERY WPF object inherits  
this class, you can't build a WPF object without it. And every property  
you set on a WPF object is really two properties, one of which is a  
"public static DependencyProperty PropertyName", a DependencyProperty is  
also being a class that requires a DependencyObject to initialize it. In  
WPF you absolutely MUST declare this object as public. And while not  
strictly required Microsoft Best Practices dictate that the actual  
property is public as well.

My point is that even in WPF you don't actually NEED reflection of private  
members. The reason for this is that back during the design of Silverlight  
Microsoft knew that SL's reflection would NOT be able to reflect to  
private members and they needed to make sure that SL XAML would still work  
with WPF XAML. Remember, SL was originally called WPF/Everywhere. WPF has  
taken a lot of flack for the amount of reflection that it does precisely  
because it slows it down, but I think that it has other bigger problems.  
Personally, I've found the Measure and Arrange calls to be even  
significantly more expensive, but those can be mitigated with experience,  
the reflection can't, so it gets blamed. WPF has many other perf issues  
too, how they draw rounded corners makes grown men cry and any game  
programmer who wrote code that bad would be fired on the spot, a one  
thousand polygon circle anyone?

It's also interesting that you bring up WPF because the project I lead,  
The Horizon Project, is an attempt to create a open-source multi-platform  
framework that is as easy to use as the .NET Framework and one of it's  
core pillars is a declarative XML based UI design language. I am  
evaluating D and so far it is blowing C++ out of the water, (we'll need to  
have long conversations about strings though, I think .NET has the right  
answer here) but I have to admit that I am seriously missing Reflection.  
The problem is that there is currently no way to do data-binding without  
member-level reflection because it is a purely run-time concept. That  
said, I am researching ways around this, and I do have ideas ... but  
that's not for this forum.

For D, I have two notes. First, data-binding is the way of the future, you  
can't ignore it, or say that because it doesn't fit with philosophy 'X'  
that it shouldn't be done. The biggest reason that open-source projects  
die is that the industry moves in a way that conflicts with the projects  
philosophy and leaves the project behind. And this is a good thing, we are  
collective discovering better ways of doing things and part of that  
process is rejecting old philosophies. Adapt or die. The second is that  
you can't protect the programmer from all possible mistakes, and in terms  
of security their are many worse potential exploits that are much easier  
to reach for than Reflection. And to be honest, I've never actually heard  
of a successful Reflection exploit via the .NET Framework, although I am  
sure someone here knows of one...

However, I see no particular reason to reflect on private members by  
default, if you need to get data out, you should be using a public  
property or method to filter the data as needed, that's kind of the point  
of scoping. Private member reflection is NOT a requirement for UX, WPF  
being the case in point. I personally like the solution that Reflection in  
D is opt-out with both command-line options and  noreflect. I would add  
that a  deepreflect option be made available for those incorrigible souls  
who absolutely must reflect private members. I would also add that  
reflection should be default disabled by the compiler for code the works  
directly with pointers and direct memory access (malloc/free, new/delete,  
etc.) and the developer can only enable it using a compiler option.

For most of us, code-bloat is not an issue even in the slightest, todays  
computers can open multi-megabyte executables running JIT's (think .NET)  
in less than half a second (we have a working example where I work). In  
this case, a little extra bloat to make my life significantly easier is a  
non-choice, I'll take the bloat every time. However, I completely agree  
that there are cases where bloat is bad, anything system level (kernels,  
drivers, etc.), mobile or power-limited devices (although more of a  
toss-up), and others like games. But in the end, I have products to  
deliver, I need to best tools for the job, if that means using reflection,  
so be it, ideology should never enter the equation. In the case of  
reflection, it is my opinion that the compiler should provide a full range  
of tools and let the developers decide how much they want to use; this  
would make D superior to even .NET, as .NET is extremely limited in it's  
reflection control.

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.com/
Oct 26 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-26 21:53, Adam Wilson wrote:

 It's also interesting that you bring up WPF because the project I lead,
 The Horizon Project, is an attempt to create a open-source
 multi-platform framework that is as easy to use as the .NET Framework
 and one of it's core pillars is a declarative XML based UI design
 language. I am evaluating D and so far it is blowing C++ out of the
 water, (we'll need to have long conversations about strings though, I
 think .NET has the right answer here) but I have to admit that I am
 seriously missing Reflection. The problem is that there is currently no
 way to do data-binding without member-level reflection because it is a
 purely run-time concept. That said, I am researching ways around this,
 and I do have ideas ... but that's not for this forum.


Are you saying that you consider using D for this Horizon project? I can 
recommend you take a look at DWT: www.dsource.org/projects/dwt

Somewhere down the road I've planed to create an interface/window 
builder for DWT using XML or something similar. I'm thinking something 
like how it works on Mac OS X using Interface Builder.


 However, I see no particular reason to reflect on private members by
 default, if you need to get data out, you should be using a public
 property or method to filter the data as needed, that's kind of the
 point of scoping. Private member reflection is NOT a requirement for UX,
 WPF being the case in point. I personally like the solution that
 Reflection in D is opt-out with both command-line options and
  noreflect. I would add that a  deepreflect option be made available for
 those incorrigible souls who absolutely must reflect private members. I
 would also add that reflection should be default disabled by the
 compiler for code the works directly with pointers and direct memory
 access (malloc/free, new/delete, etc.) and the developer can only enable
 it using a compiler option.


I don't understand this one. Should the compiler disable reflection as 
soon as it sees malloc/free? On what level should it be disabled? I 
mean, the runtime needs to be able to use these functions to implement 
the memory manager, i.e. the garbage collector and other things as well.

-- 
/Jacob Carlborg
Oct 27 2011
parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Thu, 27 Oct 2011 00:14:35 -0700, Jacob Carlborg <doob me.com> wrote:


 Are you saying that you consider using D for this Horizon project? I can  
 recommend you take a look at DWT: www.dsource.org/projects/dwt

 Somewhere down the road I've planed to create an interface/window  
 builder for DWT using XML or something similar. I'm thinking something  
 like how it works on Mac OS X using Interface Builder.


I am, and I have looked at DWT. My problem with it is a one that is  
endemic to open-source UI framework. Microsoft recognized a decade ago  
that UI widgets whose look-and-feel is defined and controlled by the  
Operating System are going the way of the dodo. Out of that realization  
WPF was born. UI designers today want the ability to control every pixel  
of the UI's presentation. And the reasons for this are two-fold. The first  
is that it turns out that most OS designers are fantastically bad at UI  
and design in general. It takes epic piles of cash to pull of a decent one  
and even then there is a still a "programmers were here" look to it (with  
the notable exception of iOS/OSX where designers rule the roost). The  
second is product differentiation. Nobody wants an app that looks like  
every other app because it actually becomes impossible for the user to  
distinguish which app works best for them. Users ONLY look at the UI, and  
if the app doesn't look good, they wont "buy" it, even if it's free. This  
is non-negotiable. Users, when given two apps that do the same thing, even  
for different prices, will pick the prettier one every time, because the  
prettier one is perceived as being "better". It's called the  
Attractiveness Bias and it is a well-known principle in the design world.  
Who would you rather look at all day, Alessandra Ambrosio or Rosie  
O'Donnell? I rest my case.

I maintain that this is prime reason that Linux on the desktop has failed  
miserably, and I think Android proves my point. Android's key win is that  
it put a usable UI on top of Linux. People never had a problem with the  
price of Linux, they just couldn't stand to look at it. The Linux Desktop  
LOOKS industrial and it's apps for the most part look the same (I know of  
a few outliers that did a good job, but it isn't the norm). My point is  
that the day of cookie cutter apps is over. Anyone designing for that  
paradigm is history. Microsoft's latest UI paradigm, "Metro", is just a  
different cut-down version of WPF similar to Silverlight. Microsoft has no  
plans to go back to OS controlled UI styles; Metro and WPF are the plan  
for the next 15 years. (I attended the MS BUILD conference, they made this  
plan very ... nay, EXTREMELY clear).

Open-source is chronically behind the big boys with money, precisely  
because FOSS doesn't have the money to sling around for Testing and  
Usability Studies, and most FOSS guys don't want to mess around with that  
stuff anyways. You see FOSS guys tend to be engineers; they can put up  
with, and even like, industrial looking interfaces. But programmers also  
have a giant blind-spot when it comes to users. Most programmers view  
users as a lower species and assume that they will be delighted by  
whatever the programmer deigns to bequeath to them. But if you look at the  
successful people in the tech industry (*ahem* Steve Jobs) you'll find an  
attitude that is the exact opposite. Jobs was so focused on delivering  
what the user wanted that he would publicly berate any programmer who  
thought they knew better than the designer. While I don't necessarily  
agree with Jobs' management style, there is a reason why Apple is the  
second largest company in the world right now, and it has nothing to do  
with how good Apple's engineering is (which I hear is average at best).  
Despite programmers' best efforts, the world of technology is no longer  
controlled by programmers. For better or worse, users determine our course  
now. The open-source community would do well to embrace the user.

But without a first-class UI framework, that will never happen. In terms  
of capability and usability, both Apple and Microsoft have beat the best  
FOSS has to offer by a decade at least. I looked, searched, and scoured,  
but the fact of the matter is, even the usable FOSS UI offerings are  
pitiful when compared to the commercial offerings. The Horizon Project got  
it's start because there has been a trend in recent commercial UI  
offerings from towards increasing reliance on the operating system itself.  
Metro XAML just flat won't work on anything other than Windows 8,  
Silverlight is a second class citizen at MS now, Cocoa only works on Mac  
etc. My goal with The Horizon Project is to create First-Class UI  
Framework for multiple platforms so that programmers don't have to rewrite  
the UI from scratch for each new platform they want to support and then  
open-source it so that the commercial OS vendors can't pervert it for  
their own purposes. I want to put [some of] the power back into the  
programmers hands.

Apologies for the length, but this is a topic that is of some interest to  
me. :-)


 I don't understand this one. Should the compiler disable reflection as  
 soon as it sees malloc/free? On what level should it be disabled? I  
 mean, the runtime needs to be able to use these functions to implement  
 the memory manager, i.e. the garbage collector and other things as well.


The idea is to automatically prevent reflection access to sections of the  
program where using direct memory manipulation could potentially result in  
security holes but provide a way out for the programmer if they really  
wanted it. C/C++ is famous for programmer bugs around memory handling. If  
the compiler automatically disabled reflection, by sticking  noreflect in  
front of a function that used malloc or new, it could potentially prevent  
those types of memory manipulation flaws and help keep the Reflection  
attacks to a minimum. It was an idea that I was throwing out there. But I  
don't know enough about D yet to know if it's the right way to handle it.  
And I have to admit I am little confused though as I would hope that  
Reflection would be disabled on the GC, because I have never personally  
had a reason to reflect into the GC...

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.com/
Oct 27 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-27 20:42, Adam Wilson wrote:

 On Thu, 27 Oct 2011 00:14:35 -0700, Jacob Carlborg <doob me.com> wrote:


 Are you saying that you consider using D for this Horizon project? I
 can recommend you take a look at DWT: www.dsource.org/projects/dwt

 Somewhere down the road I've planed to create an interface/window
 builder for DWT using XML or something similar. I'm thinking something
 like how it works on Mac OS X using Interface Builder.


 I am, and I have looked at DWT. My problem with it is a one that is
 endemic to open-source UI framework. Microsoft recognized a decade ago
 that UI widgets whose look-and-feel is defined and controlled by the
 Operating System are going the way of the dodo. Out of that realization
 WPF was born. UI designers today want the ability to control every pixel
 of the UI's presentation. And the reasons for this are two-fold. The
 first is that it turns out that most OS designers are fantastically bad
 at UI and design in general. It takes epic piles of cash to pull of a
 decent one and even then there is a still a "programmers were here" look
 to it (with the notable exception of iOS/OSX where designers rule the
 roost).


I can agree to that.


 The second is product differentiation. Nobody wants an app that
 looks like every other app because it actually becomes impossible for
 the user to distinguish which app works best for them.


I want that. Because I know how the GUI works and it will be easy for me 
to learn if it follows the guidelines of the platform. Also I don't need 
to figure out if I can use the scroll wheel on the mouse on this, what 
looks like a, scroll bar. If the application have a native look and feel 
I know how to use the widgets.


 Users ONLY look
 at the UI, and if the app doesn't look good, they wont "buy" it, even if
 it's free. This is non-negotiable. Users, when given two apps that do
 the same thing, even for different prices, will pick the prettier one
 every time, because the prettier one is perceived as being "better".
 It's called the Attractiveness Bias and it is a well-known principle in
 the design world. Who would you rather look at all day, Alessandra
 Ambrosio or Rosie O'Donnell? I rest my case.


That's why I use Mac OS X where the native applications look good :)


 I maintain that this is prime reason that Linux on the desktop has
 failed miserably, and I think Android proves my point. Android's key win
 is that it put a usable UI on top of Linux. People never had a problem
 with the price of Linux, they just couldn't stand to look at it. The
 Linux Desktop LOOKS industrial and it's apps for the most part look the
 same (I know of a few outliers that did a good job, but it isn't the
 norm).


I can agree to that.


 My point is that the day of cookie cutter apps is over. Anyone
 designing for that paradigm is history. Microsoft's latest UI paradigm,
 "Metro", is just a different cut-down version of WPF similar to
 Silverlight. Microsoft has no plans to go back to OS controlled UI
 styles; Metro and WPF are the plan for the next 15 years. (I attended
 the MS BUILD conference, they made this plan very ... nay, EXTREMELY
 clear).


I don't know how Metro or WPF is implemented but how says they can't be 
the native look and feel of the OS.


 Open-source is chronically behind the big boys with money, precisely
 because FOSS doesn't have the money to sling around for Testing and
 Usability Studies, and most FOSS guys don't want to mess around with
 that stuff anyways. You see FOSS guys tend to be engineers; they can put
 up with, and even like, industrial looking interfaces. But programmers
 also have a giant blind-spot when it comes to users. Most programmers
 view users as a lower species and assume that they will be delighted by
 whatever the programmer deigns to bequeath to them. But if you look at
 the successful people in the tech industry (*ahem* Steve Jobs) you'll
 find an attitude that is the exact opposite. Jobs was so focused on
 delivering what the user wanted that he would publicly berate any
 programmer who thought they knew better than the designer. While I don't
 necessarily agree with Jobs' management style, there is a reason why
 Apple is the second largest company in the world right now, and it has
 nothing to do with how good Apple's engineering is (which I hear is
 average at best). Despite programmers' best efforts, the world of
 technology is no longer controlled by programmers. For better or worse,
 users determine our course now. The open-source community would do well
 to embrace the user.


I have no trouble in letting a designer decide how GUI should look like, 
in fact, as you say it would be better if they did. But not everyone can 
have that luxury. Since I'm no designer you do the best I can.


 But without a first-class UI framework, that will never happen. In terms
 of capability and usability, both Apple and Microsoft have beat the best
 FOSS has to offer by a decade at least. I looked, searched, and scoured,
 but the fact of the matter is, even the usable FOSS UI offerings are
 pitiful when compared to the commercial offerings. The Horizon Project
 got it's start because there has been a trend in recent commercial UI
 offerings from towards increasing reliance on the operating system
 itself. Metro XAML just flat won't work on anything other than Windows
 8, Silverlight is a second class citizen at MS now, Cocoa only works on
 Mac etc. My goal with The Horizon Project is to create First-Class UI
 Framework for multiple platforms so that programmers don't have to
 rewrite the UI from scratch for each new platform they want to support
 and then open-source it so that the commercial OS vendors can't pervert
 it for their own purposes. I want to put [some of] the power back into
 the programmers hands.

 Apologies for the length, but this is a topic that is of some interest
 to me. :-)


I's a topic of interest to me as well. But I, on the other hand, prefer 
a native look and fell of applications. Instead of having yet another 
application with yet another GUI that doesn't work properly. If I see 
something that looks like a scroll bar I assume I can use it like a 
scroll bar. But that's not true for many applications, instead they 
implement the bare minimum for having the scroll bar "work", i.e. I may 
not be able to scroll using the wheel on the mouse. A great example of 
this are scroll bars in games.

In my experience of non-native GUI's they always perform worse the 
native GUI's.

Anyway, you can still use DWT to draw your own widgets. Even if you 
don't use any native widgets you still need to be able to draw something 
on a surface. DWT has a platform independent way of doing that.

IBM Lotus Software and IBM Rational Software are built on SWT (which DWT 
is a port of) uses a lot a non-native widgets. There's an application 
called "Azureus" that had its own GUI that didn't look native at all, 
apparently it's called Vuze these days and it looks native.


 I don't understand this one. Should the compiler disable reflection as
 soon as it sees malloc/free? On what level should it be disabled? I
 mean, the runtime needs to be able to use these functions to implement
 the memory manager, i.e. the garbage collector and other things as well.


 The idea is to automatically prevent reflection access to sections of
 the program where using direct memory manipulation could potentially
 result in security holes but provide a way out for the programmer if
 they really wanted it. C/C++ is famous for programmer bugs around memory
 handling. If the compiler automatically disabled reflection, by sticking
  noreflect in front of a function that used malloc or new, it could
 potentially prevent those types of memory manipulation flaws and help
 keep the Reflection attacks to a minimum. It was an idea that I was
 throwing out there. But I don't know enough about D yet to know if it's
 the right way to handle it. And I have to admit I am little confused
 though as I would hope that Reflection would be disabled on the GC,
 because I have never personally had a reason to reflect into the GC...


What happens when you use class A from class B and the compiler has 
added  noreflect to class A? Will it add it to B as well? If not, how 
does the compiler know that?

There has been a similar discussion about having the compiler insert 
"pure" automatic on functions. But what might happens is a function that 
is called by another function change it implementation making it no 
longer pure. Which means your function will no longer be pure and you 
have no idea about it.

-- 
/Jacob Carlborg
Oct 28 2011
next sibling parent Gor Gyolchanyan <gor.f.gyolchanyan gmail.com> writes:
I think the compile-time reflection should be improved first, because
run-time reflection can be built upon that.
Currently D's compile-time reflection is not fully developed.
Oct 28 2011
prev sibling parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Fri, 28 Oct 2011 00:07:57 -0700, Jacob Carlborg <doob me.com> wrote:


 On 2011-10-27 20:42, Adam Wilson wrote:

 On Thu, 27 Oct 2011 00:14:35 -0700, Jacob Carlborg <doob me.com> wrote:


 Are you saying that you consider using D for this Horizon project? I
 can recommend you take a look at DWT: www.dsource.org/projects/dwt

 Somewhere down the road I've planed to create an interface/window
 builder for DWT using XML or something similar. I'm thinking something
 like how it works on Mac OS X using Interface Builder.


 I am, and I have looked at DWT. My problem with it is a one that is
 endemic to open-source UI framework. Microsoft recognized a decade ago
 that UI widgets whose look-and-feel is defined and controlled by the
 Operating System are going the way of the dodo. Out of that realization
 WPF was born. UI designers today want the ability to control every pixel
 of the UI's presentation. And the reasons for this are two-fold. The
 first is that it turns out that most OS designers are fantastically bad
 at UI and design in general. It takes epic piles of cash to pull of a
 decent one and even then there is a still a "programmers were here" look
 to it (with the notable exception of iOS/OSX where designers rule the
 roost).


 I can agree to that.


 The second is product differentiation. Nobody wants an app that
 looks like every other app because it actually becomes impossible for
 the user to distinguish which app works best for them.


 I want that. Because I know how the GUI works and it will be easy for me  
 to learn if it follows the guidelines of the platform. Also I don't need  
 to figure out if I can use the scroll wheel on the mouse on this, what  
 looks like a, scroll bar. If the application have a native look and feel  
 I know how to use the widgets.


Ahh, thats the true power of WPF. Presentation and Implementation are  
cleanly separated. I recent re-skinned every scrollbar in one of our WPF  
apps by writing the 30-line style once, placing it in the global resource  
dictionary with implicit styling enabled, and bam, every scrollbar in the  
app looks different, but functions in the exact same way any windows user  
would expect. There are cases where you can write code that calls up into  
the presentation layer and thus introducing a dependency on the  
presentation layer, but this is actively discouraged, and when I found one  
recently in a 3rd party vendors control, it was removed by the next  
release.
I can go on and on, but the point is, there are only two frameworks in  
existence that can do that are the WPF family and JavaFX. JavaFX is  
pathetic compared to WPF and is of course tied to an Oracle *shudder*  
controlled langauge. And a few months ago we had thousands of developers  
on the Silverlight forum looking for WPF family alternatives, all that  
they could find was JavaFX.


 Users ONLY look
 at the UI, and if the app doesn't look good, they wont "buy" it, even if
 it's free. This is non-negotiable. Users, when given two apps that do
 the same thing, even for different prices, will pick the prettier one
 every time, because the prettier one is perceived as being "better".
 It's called the Attractiveness Bias and it is a well-known principle in
 the design world. Who would you rather look at all day, Alessandra
 Ambrosio or Rosie O'Donnell? I rest my case.


 That's why I use Mac OS X where the native applications look good :)


 I maintain that this is prime reason that Linux on the desktop has
 failed miserably, and I think Android proves my point. Android's key win
 is that it put a usable UI on top of Linux. People never had a problem
 with the price of Linux, they just couldn't stand to look at it. The
 Linux Desktop LOOKS industrial and it's apps for the most part look the
 same (I know of a few outliers that did a good job, but it isn't the
 norm).


 I can agree to that.


 My point is that the day of cookie cutter apps is over. Anyone
 designing for that paradigm is history. Microsoft's latest UI paradigm,
 "Metro", is just a different cut-down version of WPF similar to
 Silverlight. Microsoft has no plans to go back to OS controlled UI
 styles; Metro and WPF are the plan for the next 15 years. (I attended
 the MS BUILD conference, they made this plan very ... nay, EXTREMELY
 clear).


 I don't know how Metro or WPF is implemented but how says they can't be  
 the native look and feel of the OS.


Nobody, in fact WPF has three default looks out of the box, one for  
Windows 2000, another for WinXP, and another for Aero (Vista+). WPF  
defaults to the correct one for that system unless you override it. It's  
just that in my experience, the default look is a lowest common  
demoninator, and not a good one.


 Open-source is chronically behind the big boys with money, precisely
 because FOSS doesn't have the money to sling around for Testing and
 Usability Studies, and most FOSS guys don't want to mess around with
 that stuff anyways. You see FOSS guys tend to be engineers; they can put
 up with, and even like, industrial looking interfaces. But programmers
 also have a giant blind-spot when it comes to users. Most programmers
 view users as a lower species and assume that they will be delighted by
 whatever the programmer deigns to bequeath to them. But if you look at
 the successful people in the tech industry (*ahem* Steve Jobs) you'll
 find an attitude that is the exact opposite. Jobs was so focused on
 delivering what the user wanted that he would publicly berate any
 programmer who thought they knew better than the designer. While I don't
 necessarily agree with Jobs' management style, there is a reason why
 Apple is the second largest company in the world right now, and it has
 nothing to do with how good Apple's engineering is (which I hear is
 average at best). Despite programmers' best efforts, the world of
 technology is no longer controlled by programmers. For better or worse,
 users determine our course now. The open-source community would do well
 to embrace the user.


 I have no trouble in letting a designer decide how GUI should look like,  
 in fact, as you say it would be better if they did. But not everyone can  
 have that luxury. Since I'm no designer you do the best I can.


Neither am I really, I am a programmer by trade who cut his teeth back in  

design isn't that hard, if one is willing to let go of their preferences.  
More to the point though, my company has no on staff designer, so it was  
either pay large sums of money to a consultant, or I as team lead could do  
it. I lost. To some degree it's a totally different way of thinking and I  
find that after a good design session (a week or two), it takes me a  
couple of days to realign my brain to left-brain-progammer-mode. But hey,  
they pay me, I jump to. *shrug* There is no mystical powers that are given  
to designers, anyone can make a good design.


 But without a first-class UI framework, that will never happen. In terms
 of capability and usability, both Apple and Microsoft have beat the best
 FOSS has to offer by a decade at least. I looked, searched, and scoured,
 but the fact of the matter is, even the usable FOSS UI offerings are
 pitiful when compared to the commercial offerings. The Horizon Project
 got it's start because there has been a trend in recent commercial UI
 offerings from towards increasing reliance on the operating system
 itself. Metro XAML just flat won't work on anything other than Windows
 8, Silverlight is a second class citizen at MS now, Cocoa only works on
 Mac etc. My goal with The Horizon Project is to create First-Class UI
 Framework for multiple platforms so that programmers don't have to
 rewrite the UI from scratch for each new platform they want to support
 and then open-source it so that the commercial OS vendors can't pervert
 it for their own purposes. I want to put [some of] the power back into
 the programmers hands.

 Apologies for the length, but this is a topic that is of some interest
 to me. :-)


 I's a topic of interest to me as well. But I, on the other hand, prefer  
 a native look and fell of applications. Instead of having yet another  
 application with yet another GUI that doesn't work properly. If I see  
 something that looks like a scroll bar I assume I can use it like a  
 scroll bar. But that's not true for many applications, instead they  
 implement the bare minimum for having the scroll bar "work", i.e. I may  
 not be able to scroll using the wheel on the mouse. A great example of  
 this are scroll bars in games.

 In my experience of non-native GUI's they always perform worse the  
 native GUI's.


Actually we found that by re-skinning our app intelligently we were able  
to load a large list of complex data 4x faster than the default skin. The  
reason for this is that in WPF all drawing is done the same way, it  
doesn't matter if it's a system skin or not, it's just a skin, there are  
no special shortcuts for the system skin to take. At it's heart WPF is  
based on DirectX9. This means that screen complexity is the ONLY limiting  
factor.

The other thing that WPF allows is data / presentation separation. I can  
pass a list box a chunk of data, and the data does not have to concern  
itself with how the presentation looks or vice versa. The listbox will  
display the data according to however the style is setup it. And THEN if I  
really want to I can change the presentation style on the fly based on  
user input. Can DWT do that in 2 lines of code? WPF Can. That's the power  
of reflection based data-binding. In fact I spend most of my time writing  
(much more compact) XAML code to do presentation layer work than I do  

work done on a screen. It's usually things like load this item based on a  
selection changed event or some such that XAML has no way to do (and  
shouldn't, thats implementation).


 Anyway, you can still use DWT to draw your own widgets. Even if you  
 don't use any native widgets you still need to be able to draw something  
 on a surface. DWT has a platform independent way of doing that.


Yea, but how much work is it? Can I create a completely new look for a  
control in 30 lines or less without modifying the implementation code at  
all to make the new look work? That's the power I'm after.


 IBM Lotus Software and IBM Rational Software are built on SWT (which DWT  
 is a port of) uses a lot a non-native widgets. There's an application  
 called "Azureus" that had its own GUI that didn't look native at all,  
 apparently it's called Vuze these days and it looks native.


That's IBM, they have supertanker loads of money, the open-source world  
does not. I expect IBM to re-skin anything they want to their  
specifications, they can afford it. WPF type apps are game changers in  
that they allow very small teams to create powerful, beautiful, apps in  
something less than a decade, and without requiring multi-million dollar  
investments. As an example, our flagship app (which we are keeping under  
tight wraps for the following reason) has the potential to completely wipe  
out multiple billions of dollars worth of corporate market capitalization  
and probably sink some long-time industry names in the process, all with a  
team  of less than 10 guys. At least, thats our goal. But that goal is  
only achievable because we can move orders of magnitude faster than they  
can. It also doesn't hurt that their interface looks like it was designed  
in 1999 (seriously) and is definitely a traditional type of framework (the  
standard Java one IIRC). Thank you WPF, it simply wouldn't be possible  
without you. So when I say the world is leaving behind traditional UI  
frameworks, I am not kidding; it just wont happen that quickly as corp IT  
loathes change, but it's already starting. (See: iOS and Android apps and  
CxO's demanding their internal apps on those platforms.)

Now imagine a world where not only can open-source compete with the  
big-boys, but take serious chunks out of their consumer mind-share. I want  
that. But, the only way that is happening is with a framework that allows  
open-source to compete at the same speed as the big boys can afford while  
delivering what the customer wants. Any app is only as good as the number  
of people using it.

It's not the big who eat the small, it's the fast who eat the slow. :-)


 I don't understand this one. Should the compiler disable reflection as
 soon as it sees malloc/free? On what level should it be disabled? I
 mean, the runtime needs to be able to use these functions to implement
 the memory manager, i.e. the garbage collector and other things as  
 well.


 The idea is to automatically prevent reflection access to sections of
 the program where using direct memory manipulation could potentially
 result in security holes but provide a way out for the programmer if
 they really wanted it. C/C++ is famous for programmer bugs around memory
 handling. If the compiler automatically disabled reflection, by sticking
  noreflect in front of a function that used malloc or new, it could
 potentially prevent those types of memory manipulation flaws and help
 keep the Reflection attacks to a minimum. It was an idea that I was
 throwing out there. But I don't know enough about D yet to know if it's
 the right way to handle it. And I have to admit I am little confused
 though as I would hope that Reflection would be disabled on the GC,
 because I have never personally had a reason to reflect into the GC...


 What happens when you use class A from class B and the compiler has  
 added  noreflect to class A? Will it add it to B as well? If not, how  
 does the compiler know that?

 There has been a similar discussion about having the compiler insert  
 "pure" automatic on functions. But what might happens is a function that  
 is called by another function change it implementation making it no  
 longer pure. Which means your function will no longer be pure and you  
 have no idea about it.



-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.com/
Oct 28 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-28 21:36, Adam Wilson wrote:

 Ahh, thats the true power of WPF. Presentation and Implementation are
 cleanly separated. I recent re-skinned every scrollbar in one of our WPF
 apps by writing the 30-line style once, placing it in the global
 resource dictionary with implicit styling enabled, and bam, every
 scrollbar in the app looks different, but functions in the exact same
 way any windows user would expect. There are cases where you can write
 code that calls up into the presentation layer and thus introducing a
 dependency on the presentation layer, but this is actively discouraged,
 and when I found one recently in a 3rd party vendors control, it was
 removed by the next release.
 I can go on and on, but the point is, there are only two frameworks in
 existence that can do that are the WPF family and JavaFX. JavaFX is
 pathetic compared to WPF and is of course tied to an Oracle *shudder*
 controlled langauge. And a few months ago we had thousands of developers
 on the Silverlight forum looking for WPF family alternatives, all that
 they could find was JavaFX.


It's not the look of the, in this example, scroll bar, it's the 
behavior. It may look like a scroll bar but it may not behave in all the 
ways as a real, native, scroll bar would.


 Nobody, in fact WPF has three default looks out of the box, one for
 Windows 2000, another for WinXP, and another for Aero (Vista+). WPF
 defaults to the correct one for that system unless you override it. It's
 just that in my experience, the default look is a lowest common
 demoninator, and not a good one.


To me it sounds like WPF is the native GUI. Then why are you complaining 
about native GUI's?


 Neither am I really, I am a programmer by trade who cut his teeth back

 but design isn't that hard, if one is willing to let go of their
 preferences. More to the point though, my company has no on staff
 designer, so it was either pay large sums of money to a consultant, or I
 as team lead could do it. I lost. To some degree it's a totally
 different way of thinking and I find that after a good design session (a
 week or two), it takes me a couple of days to realign my brain to
 left-brain-progammer-mode. But hey, they pay me, I jump to. *shrug*
 There is no mystical powers that are given to designers, anyone can make
 a good design.


I'm just saying that I'm not a good designer and if a designer is 
available I would prefer that they did the design.


 Actually we found that by re-skinning our app intelligently we were able
 to load a large list of complex data 4x faster than the default skin.
 The reason for this is that in WPF all drawing is done the same way, it
 doesn't matter if it's a system skin or not, it's just a skin, there are
 no special shortcuts for the system skin to take. At it's heart WPF is
 based on DirectX9. This means that screen complexity is the ONLY
 limiting factor.


If the changing the skin of an application can increase the data load by 
four times it sounds like something is seriously  wrong.

On Mac OS X a list only loads the data that is displayed so I don't 
think the amount of data matters. The same can be done using virtual 
tables in DWT.


 The other thing that WPF allows is data / presentation separation. I can
 pass a list box a chunk of data, and the data does not have to concern
 itself with how the presentation looks or vice versa. The listbox will
 display the data according to however the style is setup it. And THEN if
 I really want to I can change the presentation style on the fly based on
 user input. Can DWT do that in 2 lines of code? WPF Can. That's the
 power of reflection based data-binding. In fact I spend most of my time
 writing (much more compact) XAML code to do presentation layer work than

 of the work done on a screen. It's usually things like load this item
 based on a selection changed event or some such that XAML has no way to
 do (and shouldn't, thats implementation).


No, probably not. It's not as easy to change the theme in SWT/DWT since 
it's supposed to use the native look and feel.

It appears that you can at least set colors and similar properties using 
CSS: http://jaxenter.com/css-styling-in-eclipse-helios-32465.html


 Anyway, you can still use DWT to draw your own widgets. Even if you
 don't use any native widgets you still need to be able to draw
 something on a surface. DWT has a platform independent way of doing that.


 Yea, but how much work is it? Can I create a completely new look for a
 control in 30 lines or less without modifying the implementation code at
 all to make the new look work? That's the power I'm after.


No you probably can't. I don't know what you want to achieve with your 
project but I got the impression that you were looking to build a new 
GUI library that behaves something like WPF.

Regardless of how this GUI library works you need somehow to draw the 
widgets on the screen. I'm just saying that DWT has a platform 
independent way of doing this. On the other hand, if you're creating a 
completely new GUI library you probably would not want to depend on DWT. 
But instead creating bindings directly to the underlying drawing 
operations for Cairo, Quartz and DirectX or whatever Windows uses these 
days.

-- 
/Jacob Carlborg
Oct 29 2011
parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Sat, 29 Oct 2011 03:59:28 -0700, Jacob Carlborg <doob me.com> wrote:


 On 2011-10-28 21:36, Adam Wilson wrote:

 Ahh, thats the true power of WPF. Presentation and Implementation are
 cleanly separated. I recent re-skinned every scrollbar in one of our WPF
 apps by writing the 30-line style once, placing it in the global
 resource dictionary with implicit styling enabled, and bam, every
 scrollbar in the app looks different, but functions in the exact same
 way any windows user would expect. There are cases where you can write
 code that calls up into the presentation layer and thus introducing a
 dependency on the presentation layer, but this is actively discouraged,
 and when I found one recently in a 3rd party vendors control, it was
 removed by the next release.
 I can go on and on, but the point is, there are only two frameworks in
 existence that can do that are the WPF family and JavaFX. JavaFX is
 pathetic compared to WPF and is of course tied to an Oracle *shudder*
 controlled langauge. And a few months ago we had thousands of developers
 on the Silverlight forum looking for WPF family alternatives, all that
 they could find was JavaFX.


 It's not the look of the, in this example, scroll bar, it's the  
 behavior. It may look like a scroll bar but it may not behave in all the  
 ways as a real, native, scroll bar would.


 Nobody, in fact WPF has three default looks out of the box, one for
 Windows 2000, another for WinXP, and another for Aero (Vista+). WPF
 defaults to the correct one for that system unless you override it. It's
 just that in my experience, the default look is a lowest common
 demoninator, and not a good one.


 To me it sounds like WPF is the native GUI. Then why are you complaining  
 about native GUI's?


WPF isn't the native GUI, it just comes with skins that emulate the look  
of the native GUI. The native GUI in Windows is called Win32 GDI  
(Graphices Device Interface). WPF is built on top of DirectX which is  
analagous to OpenGL and is primarily used for games. DirectX completely  
bypasses the GDI and communicates directly with the hardware. It's primary  
claim to fame is that it is fast, but it's not as compatible, whereas GDI  
is compatible with everything that implements VESA (which is everything  
since the early 90's), but it's slow.


 Neither am I really, I am a programmer by trade who cut his teeth back

 but design isn't that hard, if one is willing to let go of their
 preferences. More to the point though, my company has no on staff
 designer, so it was either pay large sums of money to a consultant, or I
 as team lead could do it. I lost. To some degree it's a totally
 different way of thinking and I find that after a good design session (a
 week or two), it takes me a couple of days to realign my brain to
 left-brain-progammer-mode. But hey, they pay me, I jump to. *shrug*
 There is no mystical powers that are given to designers, anyone can make
 a good design.


 I'm just saying that I'm not a good designer and if a designer is  
 available I would prefer that they did the design.


 Actually we found that by re-skinning our app intelligently we were able
 to load a large list of complex data 4x faster than the default skin.
 The reason for this is that in WPF all drawing is done the same way, it
 doesn't matter if it's a system skin or not, it's just a skin, there are
 no special shortcuts for the system skin to take. At it's heart WPF is
 based on DirectX9. This means that screen complexity is the ONLY
 limiting factor.


 If the changing the skin of an application can increase the data load by  
 four times it sounds like something is seriously  wrong.

 On Mac OS X a list only loads the data that is displayed so I don't  
 think the amount of data matters. The same can be done using virtual  
 tables in DWT.


Ok, so I apologize for generalizing there. It turns out that WPF is really  
bad at drawing round shapes, and the default skins (which the exception of  
the Windows 2000 skin) rely heavily on rounded corners. We replaced the  
default skin with one that used square corners and had fewer Measure and  
Arrange stages (where WPF figures out where to place the controls) which  
are equally expensive. WPF has virtual lists as well, and we do use it,  
which also helped contribute to our speed gains, but it is not enabled by  
default. My complaint is that the working with WPF requires a non-trivial  
amount of esoteric experience with it to make it work well.


 The other thing that WPF allows is data / presentation separation. I can
 pass a list box a chunk of data, and the data does not have to concern
 itself with how the presentation looks or vice versa. The listbox will
 display the data according to however the style is setup it. And THEN if
 I really want to I can change the presentation style on the fly based on
 user input. Can DWT do that in 2 lines of code? WPF Can. That's the
 power of reflection based data-binding. In fact I spend most of my time
 writing (much more compact) XAML code to do presentation layer work than

 of the work done on a screen. It's usually things like load this item
 based on a selection changed event or some such that XAML has no way to
 do (and shouldn't, thats implementation).


 No, probably not. It's not as easy to change the theme in SWT/DWT since  
 it's supposed to use the native look and feel.

 It appears that you can at least set colors and similar properties using  
 CSS: http://jaxenter.com/css-styling-in-eclipse-helios-32465.html


 Anyway, you can still use DWT to draw your own widgets. Even if you
 don't use any native widgets you still need to be able to draw
 something on a surface. DWT has a platform independent way of doing  
 that.


 Yea, but how much work is it? Can I create a completely new look for a
 control in 30 lines or less without modifying the implementation code at
 all to make the new look work? That's the power I'm after.


 No you probably can't. I don't know what you want to achieve with your  
 project but I got the impression that you were looking to build a new  
 GUI library that behaves something like WPF.

 Regardless of how this GUI library works you need somehow to draw the  
 widgets on the screen. I'm just saying that DWT has a platform  
 independent way of doing this. On the other hand, if you're creating a  
 completely new GUI library you probably would not want to depend on DWT.  
 But instead creating bindings directly to the underlying drawing  
 operations for Cairo, Quartz and DirectX or whatever Windows uses these  
 days.


I am looking to create an open-source implementation of a GUI framework  
that is 'lookless', i.e. it's underlying implementation has no  
dependencies on how it looks and the look can be changed to however the  
designer wants it to look without having to worry about code. WPF and  
JavaFX are the only two frameworks that are examples of how this might  
work.

I absolutely agree that relying on something else is probably not the best  
idea, because for better or worse you end up inheriting their design  
choices, which may or may not work for your project. Since I am windows  
guy, the reference implementation is going to be DirectX. First I just  
have to create Direct2D bindings for D...

And another goal of the framework beyond platform independence is language  
independence. I have to admit that one of the big draws for me of D was  
the C++ ABI and COM compatibility of D while having a MUCH more modern  
capability set. However, it's still lacking in a few key areas for my  
project, and runtime reflection is one of them...

-- 
Adam Wilson
Project Coordinator
The Horizon Project
http://www.thehorizonproject.com/
Oct 29 2011
parent reply Jacob Carlborg <doob me.com> writes:
On 2011-10-29 22:11, Adam Wilson wrote:

 WPF isn't the native GUI, it just comes with skins that emulate the look
 of the native GUI. The native GUI in Windows is called Win32 GDI
 (Graphices Device Interface). WPF is built on top of DirectX which is
 analagous to OpenGL and is primarily used for games. DirectX completely
 bypasses the GDI and communicates directly with the hardware. It's
 primary claim to fame is that it is fast, but it's not as compatible,
 whereas GDI is compatible with everything that implements VESA (which is
 everything since the early 90's), but it's slow.


We don't have to argue about this but it sounds like WPF is a new native 
GUI, especially if it bypasses GDI.


 Ok, so I apologize for generalizing there. It turns out that WPF is
 really bad at drawing round shapes, and the default skins (which the
 exception of the Windows 2000 skin) rely heavily on rounded corners. We
 replaced the default skin with one that used square corners and had
 fewer Measure and Arrange stages (where WPF figures out where to place
 the controls) which are equally expensive. WPF has virtual lists as
 well, and we do use it, which also helped contribute to our speed gains,
 but it is not enabled by default. My complaint is that the working with
 WPF requires a non-trivial amount of esoteric experience with it to make
 it work well.


With DWT you only pass an additional flag when creating the table, if I 
recall correctly.


 I am looking to create an open-source implementation of a GUI framework
 that is 'lookless', i.e. it's underlying implementation has no
 dependencies on how it looks and the look can be changed to however the
 designer wants it to look without having to worry about code. WPF and
 JavaFX are the only two frameworks that are examples of how this might
 work.

 I absolutely agree that relying on something else is probably not the
 best idea, because for better or worse you end up inheriting their
 design choices, which may or may not work for your project. Since I am
 windows guy, the reference implementation is going to be DirectX. First
 I just have to create Direct2D bindings for D...


What's wrong with using the lowest level of the drawing operations 
available on a given OS. On Mac OS X it would be Quartz which is has 
been hardware accelerated since a very long time. On Linux it would 
probably be Cairo and on Windows you could use the lowest level of WPF. 
Why reinvent the wheel again?

If you want to roll your own drawing operations anyway why not have the 
reference implementation in OpenGL, it's available on basically all 
platforms including Windows.

OpenGL bindings and other things like SDL and FreeType: 
http://dsource.org/projects/derelict

DirectX bindings and bindings in general: 
http://dsource.org/projects/bindings

This is not my strongest side, but wouldn't you need native windows to 
draw your widgets on?


 And another goal of the framework beyond platform independence is
 language independence. I have to admit that one of the big draws for me
 of D was the C++ ABI and COM compatibility of D while having a MUCH more
 modern capability set. However, it's still lacking in a few key areas
 for my project, and runtime reflection is one of them...


Just for the record, the interface to C++ is very limited.

-- 
/Jacob Carlborg
Oct 30 2011
parent Mehrdad <wfunction hotmail.com> writes:
On 10/30/2011 4:03 AM, Jacob Carlborg wrote:

 On 2011-10-29 22:11, Adam Wilson wrote:

 WPF isn't the native GUI, it just comes with skins that emulate the 
 look of the native GUI. The native GUI in Windows is called Win32 GDI 
 (Graphices Device Interface). WPF is built on top of DirectX which is 
 analagous to OpenGL and is primarily used for games. DirectX 
 completely bypasses the GDI and communicates directly with the 
 hardware. It's primary claim to fame is that it is fast, but it's not 
 as compatible, whereas GDI is compatible with everything that 
 implements VESA (which is everything since the early 90's), but it's 
 slow.


 We don't have to argue about this but it sounds like WPF is a new 
 native GUI, especially if it bypasses GDI.


That's not the issue. The "nativeness" isn't regarding GDI vs. DirectX, 
but it's regarding the _windowing_ framework itself (which is separate 
from, although related to, the graphics framework).

_Native_ windows use the _system_ window classes (e.g. SysListView32) to 
display themselves to the user. WPF (which uses DirectX) and GTK (which 
uses GDI) and most frameworks, on the other hand, don't -- rather, they 
try to /look/ like native windows by creating custom windows and 
intercepting all of the drawing operations.

And, of course, they fail. This is indeed hard to notice on XP, but it's 
blindingly obvious on Windows Vista and later, because, for example, the 
buttons no longer "feel" native, even though they might "look" native -- 
e.g. the way they fade or get depressed (the timeout, the fade colors, 
etc.), the way they get focused, etc. is completely out of sync with the 
rest of the system's looks.

Since native controls use GDI internally, they obviously can't be drawn 
with DirectX. But the key point is that the inverse is not true -- if 
something uses GDI (e.g. GTK), that doesn't mean it's native. So no, WPF 
is not "native" in any way, and neither is GTK -- the only "native" 
frameworks are those that use the built-in windowing classes, which 
include MFC, ATL, SWT, and (I /think/) Qt. Most other frameworks don't 
do this.
Oct 30 2011
prev sibling next sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Wed, 26 Oct 2011 14:20:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:
[snip]

 Am 26.10.11 07:16, schrieb Robert Jacques:


[snip]


 I know, that "runtime self-modification" and runtime code generation is
 a "dangerous" feature.


As a point of semantics, actual "runtime self-modification" is no longer
available outside a VM; the practice is heavily used in viruses, worms, etc and
practically nowhere else, so both OS, hardware and anti-virus vendors have
ganged up against this practice. :)
Oct 27 2011
prev sibling parent "Robert Jacques" <sandford jhu.edu> writes:
On Wed, 26 Oct 2011 14:20:47 -0400, Jonny Dee <jonnyd gmx.net> wrote:
[snip]

 Am 26.10.11 07:16, schrieb Robert Jacques:


[snip]

 I've got no problem with an approach which puts runtime reflection
 capabiities into a separate library. No matter were you look at, Java,

 however, how this might be done without compiler support.


Long ago, when reflection in was being discussed one of the big reasons we
turned to compile-time reflection first, besides its synergy with other
meta-programming features, is because compile-time reflection enables
runtime-reflection. A decent portion of the community has always assumed that
D's runtime reflection support would come from a library exploiting D's
compile-time reflection. All it would take is someone willing to put in the
effort and time to do so. So far, that someone has been me. :)

My new Variant (https://jshare.johnshopkins.edu/rjacque2/publ
c_html/variant.mht) makes heavy use of compile-time reflection and provides
runtime-reflection capabilities. Currently, I have focused on dispatch, i.e.
the calling of member functions, the calling of static members and the setting
of fields, as I don't know what the introspection API should look like. I have
found that while __traits does provide all the information I need, it's often
not in the format I wish to consume it in. So I have been holding back on
implementation in order to solicit the feedback of the community.

As for ease of use, Variant generates reflection information for every type it
is exposed to. That includes all return types, fields and member arguments
types for every type assigned to or retrieved from Variant. Most of the time,
exposure happens naturally and everything 'Just Works' and no effort is needed.
For example:

Variant var = student;
assert(var.grade.mark == student.grade.mark)

Or

Variant var = Object.factory("example.Student");
assert(var.grade == (var.get!(Student)).grade);

Now it is possible for types, particularly sub-classes, to never be retrieved
from or assigned to, in which case only they can only use their
interfaces/base-classes type information, without some form of registration.
For example,

Variant.__register!Student;

And with the new module reflection capabilities, the hassle / annoyance of
registration can be reduced to that of an import statement. Or even a single
statement that would reflect all the imports in a module. (Registration works
via static ctors, so their's no worry about using reflection before
registration, or duplicate registrations, etc)

Now, this probably gets to the heart of why I don't like the concept of
 noreflect,  reflect or any other fixed reflection option: there is no right
answer to whether or not a class should support reflection. Trying to find a
single choice (or all/none) that fits the embedded, iOS/Driod/Windows 8,
Desktop, Server, etc developer is fool hardy. Letting each developer choose how
much or how little they need feels right to me. And in D we have the tools to
provide the granularity of control to make this declaration of reflection
intent as succinct as possible.
Oct 27 2011