www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Reflection Idea

reply "Craig Black" <cblack ara.com> writes:
Since DDL provides run-time reflection, and is finally ready to use, I 
thought it would be a good time to share my idea to improve performance when 
accessing metadata.

If we have an instance of a class and we want to it's metadata via a 
reflection database, we should be able to query the reflection database 
using a hash table lookup on the name of the class.

However, we could get the metadata more quickly if it was available via the 
TypeInfo class.  This could be made possible very easily if the TypeInfo 
class contained a reference to an IReflection interface.  This reference 
could be set by the reflection database at program startup.  This would 
provide a direct link to meta data for each class, and would make reflection 
a first class citizen in D.

-Craig
Sep 14 2006
parent reply Pragma <ericanderton yahoo.removeme.com> writes:
Craig Black wrote:

 Since DDL provides run-time reflection, and is finally ready to use, I 
 thought it would be a good time to share my idea to improve performance when 
 accessing metadata.


Technically DDL does, although I'm hesitant to call it that since it 
doesn't have all the typical reflection bells-and-whistles yet.


 
 If we have an instance of a class and we want to it's metadata via a 
 reflection database, we should be able to query the reflection database 
 using a hash table lookup on the name of the class.
 
 However, we could get the metadata more quickly if it was available via the 
 TypeInfo class.  This could be made possible very easily if the TypeInfo 
 class contained a reference to an IReflection interface.  This reference 
 could be set by the reflection database at program startup.  This would 
 provide a direct link to meta data for each class, and would make reflection 
 a first class citizen in D.


The only reason why I'd consider full-on reflection for D a bad move is 
that it'll bloat program sizes considerably.  This is kind of contrary 
to D's goal as a systems programming language.  Taking into 
consideration the sheer size of your average .map file is a good 
indicator of just how bad things can get.

So I'd caution the addition of adding such a suite to D as something 
optional if at all possible.  Even something minimalisitc would work - 
like the inclusion of just a hashtable of all the program's symbols 
(metadata as Craig put it above) that is *optionally* compiled in.

Otherwise, we're stuck with .map/.situ support under DDL - i.e. using 
your .map file as a runtime dependency for the program.  While this 
works well, and is such a dead-simple way of addressing the problem, 
it's also easy to get out of sync IMO.

[PSA]

Craig, if you're at all interested in hammering out a reflection 
interface (for DDL?), the DDL Community wiki there is open for your, and 
everyone else's abuse.

http://www.dsource.org/projects/ddl/wiki/Community

I have some reflection concepts, and a few other ideas, outlined under 
the DevNotes section as well:

http://www.dsource.org/projects/ddl/wiki/DevNotes

Enjoy!

-- 
- EricAnderton at yahoo
Sep 14 2006
next sibling parent "Craig Black" <cblack ara.com> writes:
"Pragma" <ericanderton yahoo.removeme.com> wrote in message 
news:eeclc4$2206$1 digitaldaemon.com...

 Craig Black wrote:

 Since DDL provides run-time reflection, and is finally ready to use, I 
 thought it would be a good time to share my idea to improve performance 
 when accessing metadata.


 Technically DDL does, although I'm hesitant to call it that since it 
 doesn't have all the typical reflection bells-and-whistles yet.


All in due time I'm sure.  At this point, I'd be happy with just the basics.


 The only reason why I'd consider full-on reflection for D a bad move is 
 that it'll bloat program sizes considerably.  This is kind of contrary to 
 D's goal as a systems programming language.  Taking into consideration the 
 sheer size of your average .map file is a good indicator of just how bad 
 things can get.

 So I'd caution the addition of adding such a suite to D as something 
 optional if at all possible.  Even something minimalisitc would work - 
 like the inclusion of just a hashtable of all the program's symbols 
 (metadata as Craig put it above) that is *optionally* compiled in.


Yes, optional of coarse.  DDL is a library, not a part of the D compiler.
However, an extra reference per TypeInfo instance shouldn't hurt too bad.
If not using reflection, the reference would simply be null.


 Otherwise, we're stuck with .map/.situ support under DDL - i.e. using your 
 .map file as a runtime dependency for the program.  While this works well, 
 and is such a dead-simple way of addressing the problem, it's also easy to 
 get out of sync IMO.

 [PSA]

 Craig, if you're at all interested in hammering out a reflection interface 
 (for DDL?), the DDL Community wiki there is open for your, and everyone 
 else's abuse.


I appreciate the offer very much.  However, at the moment I am very busy 
building the
next version of our software, which is written in C++.  I'm waiting on D to 
become
mature enough so that I can pitch it to my boss as something we cannot live 
without.
The first time I mentioned D to him as a useful technology, he asked if it 
provided
reflection, a feature that would be very useful for our product.  I had to 
admit that
it does not provide reflection yet.

It's a hard sale for a number of technical reasons.  I could elaborate, but 
probably
nobody cares all that much.  Anyway, perhaps I will take you up on that 
offer when I
have more time.

-Craig
Sep 15 2006
prev sibling next sibling parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Pragma wrote:

 Craig Black wrote:

 Since DDL provides run-time reflection, and is finally ready to use, I 
 thought it would be a good time to share my idea to improve 
 performance when accessing metadata.


 
 Technically DDL does, although I'm hesitant to call it that since it 
 doesn't have all the typical reflection bells-and-whistles yet.
 

 If we have an instance of a class and we want to it's metadata via a 
 reflection database, we should be able to query the reflection 
 database using a hash table lookup on the name of the class.

 However, we could get the metadata more quickly if it was available 
 via the TypeInfo class.  This could be made possible very easily if 
 the TypeInfo class contained a reference to an IReflection interface.  
 This reference could be set by the reflection database at program 
 startup.  This would provide a direct link to meta data for each 
 class, and would make reflection a first class citizen in D.


 
 The only reason why I'd consider full-on reflection for D a bad move is 
 that it'll bloat program sizes considerably.  This is kind of contrary 
 to D's goal as a systems programming language.  Taking into 
 consideration the sheer size of your average .map file is a good 
 indicator of just how bad things can get.
 
 So I'd caution the addition of adding such a suite to D as something 
 optional if at all possible.  Even something minimalisitc would work - 
 like the inclusion of just a hashtable of all the program's symbols 
 (metadata as Craig put it above) that is *optionally* compiled in.
 
 Otherwise, we're stuck with .map/.situ support under DDL - i.e. using 
 your .map file as a runtime dependency for the program.  While this 
 works well, and is such a dead-simple way of addressing the problem, 
 it's also easy to get out of sync IMO.
 
 [PSA]
 
 Craig, if you're at all interested in hammering out a reflection 
 interface (for DDL?), the DDL Community wiki there is open for your, and 
 everyone else's abuse.
 
 http://www.dsource.org/projects/ddl/wiki/Community
 
 I have some reflection concepts, and a few other ideas, outlined under 
 the DevNotes section as well:
 
 http://www.dsource.org/projects/ddl/wiki/DevNotes
 
 Enjoy!
 


Hum, it may be the case for runtime reflection, but as for compile-time 
reflection, it would have quite a lot of usefulness, and no 
performance(or other kind of) drawbacks of merely being available.
I'm not saying this about compile time reflection in the sense that it 
would serve as substitute to runtime reflection, it wouldn't. At least 
not in a complete, for-all-cases sense. I think both kinds have some 
uses that cannot be covered by the other kind.


-- 
Bruno Medeiros - MSc in CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Sep 22 2006
prev sibling parent reply Josh Stern <josh_usenet phadd.net> writes:
On Thu, 14 Sep 2006 18:32:05 -0400, Pragma wrote:


 Craig Black wrote:

 Since DDL provides run-time reflection, and is finally ready to use, I 
 thought it would be a good time to share my idea to improve performance when 
 accessing metadata.





 However, we could get the metadata more quickly if it was available via the 
 TypeInfo class.  This could be made possible very easily if the TypeInfo 
 class contained a reference to an IReflection interface.  This reference 
 could be set by the reflection database at program startup.  This would 
 provide a direct link to meta data for each class, and would make reflection 
 a first class citizen in D.


 
 The only reason why I'd consider full-on reflection for D a bad move is 
 that it'll bloat program sizes considerably.  This is kind of contrary 
 to D's goal as a systems programming language.  Taking into 
 consideration the sheer size of your average .map file is a good 
 indicator of just how bad things can get.


Different people undoubtably have contrasting ideas about what they'd
like to use reflection for.   In my mind, the most important application
of Java's reflection interface is the automation of object serialization.
Since I/O programming tends to be one of the most time consuming and
error-prone aspects of development, any sort of powerful automation
in this area is valuable.  I'm a newbie to D, but as far as I can tell
from the docs, the info to do something comparable isn't available in 
the binary files or the compile time interface.


 So I'd caution the addition of adding such a suite to D as something 
 optional if at all possible.  Even something minimalisitc would work - 
 like the inclusion of just a hashtable of all the program's symbols 
 (metadata as Craig put it above) that is *optionally* compiled in.


It could be supported without bloat by making the info either something
which is either statically accessed or else dynamically loaded from
a supplemental database file.  Opening up useful interfaces to the
language parser and using them  in separate tools (i.e. like Open C++)
is another possible approach to compile time reflection.
Sep 26 2006
parent reply Pragma <ericanderton yahoo.removeme.com> writes:
Josh Stern wrote:

 On Thu, 14 Sep 2006 18:32:05 -0400, Pragma wrote:
 

 Craig Black wrote:

 Since DDL provides run-time reflection, and is finally ready to use, I 
 thought it would be a good time to share my idea to improve performance when 
 accessing metadata.




 

 However, we could get the metadata more quickly if it was available via the 
 TypeInfo class.  This could be made possible very easily if the TypeInfo 
 class contained a reference to an IReflection interface.  This reference 
 could be set by the reflection database at program startup.  This would 
 provide a direct link to meta data for each class, and would make reflection 
 a first class citizen in D.


 The only reason why I'd consider full-on reflection for D a bad move is 
 that it'll bloat program sizes considerably.  This is kind of contrary 
 to D's goal as a systems programming language.  Taking into 
 consideration the sheer size of your average .map file is a good 
 indicator of just how bad things can get.


 
 Different people undoubtably have contrasting ideas about what they'd
 like to use reflection for.   In my mind, the most important application
 of Java's reflection interface is the automation of object serialization.
 Since I/O programming tends to be one of the most time consuming and
 error-prone aspects of development, any sort of powerful automation
 in this area is valuable.  I'm a newbie to D, but as far as I can tell
 from the docs, the info to do something comparable isn't available in 
 the binary files or the compile time interface.


I agree with you completely, Josh.  D really needs both a runtime and a 
compile-time reflection facility to suit all modes of use.  On the 
compile-time side, there have been some rather amazing efforts to use 
templates to provide traits and other info - but there's obviously some 
stuff missing from TypeInfo.


 

 So I'd caution the addition of adding such a suite to D as something 
 optional if at all possible.  Even something minimalisitc would work - 
 like the inclusion of just a hashtable of all the program's symbols 
 (metadata as Craig put it above) that is *optionally* compiled in.


 
 It could be supported without bloat by making the info either something
 which is either statically accessed or else dynamically loaded from
 a supplemental database file.  Opening up useful interfaces to the
 language parser and using them  in separate tools (i.e. like Open C++)
 is another possible approach to compile time reflection.


FWIW, I have the "dynamically loaded from a supplemental database file" 
part done.  DDL just uses the map file:

http://www.dsource.org/projects/ddl/wiki/Tutorial/UsingDDL/Reflect

//reflect.d
module reflect;
import ddl.DefaultRegistry;
import mango.io.Stdout;

void test(){
     Stdout.put("Hello DDL World"c).put(CR);
}

void main(){
     auto registry = new DefaultRegistry();
     auto inSitu = registry.load("reflect.map");

     auto testFn = inSitu.getDExport!(void function(),"reflect.test")();
     testFn();
}

It's not as pretty as Java's reflection suite, but IMO, it's only a few 
wrapper classes away from that level of utility.

-- 
- EricAnderton at yahoo
Sep 27 2006
parent reply user domain.invalid writes:
 
 FWIW, I have the "dynamically loaded from a supplemental database file" 
 part done.  DDL just uses the map file:
 
 http://www.dsource.org/projects/ddl/wiki/Tutorial/UsingDDL/Reflect
 
 //reflect.d
 module reflect;
 import ddl.DefaultRegistry;
 import mango.io.Stdout;
 
 void test(){
     Stdout.put("Hello DDL World"c).put(CR);
 }
 
 void main(){
     auto registry = new DefaultRegistry();
     auto inSitu = registry.load("reflect.map");
 
     auto testFn = inSitu.getDExport!(void function(),"reflect.test")();
     testFn();
 }
 
 It's not as pretty as Java's reflection suite, but IMO, it's only a few 
 wrapper classes away from that level of utility.
 


Isn't this dynamic binding rather than reflection? I think of reflection 
as the ability to find what the fields and methods are for an object. 
For example, with Java reflection has the following

Class c = someObject.getClass()
c.getFields();
c.getMethods();


Does DDL have the ability to lookup methods and fields? For example, how 
would you determine "reflect.test" without knowing it beforehand?

   Bradley
Sep 28 2006
parent Pragma <ericanderton yahoo.removeme.com> writes:
user domain.invalid wrote:

 FWIW, I have the "dynamically loaded from a supplemental database 
 file" part done.  DDL just uses the map file:

 http://www.dsource.org/projects/ddl/wiki/Tutorial/UsingDDL/Reflect

 //reflect.d
 module reflect;
 import ddl.DefaultRegistry;
 import mango.io.Stdout;

 void test(){
     Stdout.put("Hello DDL World"c).put(CR);
 }

 void main(){
     auto registry = new DefaultRegistry();
     auto inSitu = registry.load("reflect.map");

     auto testFn = inSitu.getDExport!(void function(),"reflect.test")();
     testFn();
 }

 It's not as pretty as Java's reflection suite, but IMO, it's only a 
 few wrapper classes away from that level of utility.


 
 Isn't this dynamic binding rather than reflection? I think of reflection 
 as the ability to find what the fields and methods are for an object. 
 For example, with Java reflection has the following
 
 Class c = someObject.getClass()
 c.getFields();
 c.getMethods();


Ahh, I see what you mean.  The example I cited was really more along the 
lines of binding, but it does employ a crude form of reflection to get 
to the binding step (searches through metadata for a kind of symbol).


 
 
 Does DDL have the ability to lookup methods and fields? 
 For example, how 
 would you determine "reflect.test" without knowing it beforehand?


Yes and no.  It does have the ability to lookup methods and fields, but 
at present, telling the difference from what you do and don't want is 
still very much DIY:

auto inSitu = registry.load("reflect.map");
foreach(module; insitu.getModules){
	foreach(symbol; module.getSymbols){
             // each symbol.name is a mangled D symbol
             // that can be inspected for it's type and namespace
             // symbol.address provides where it's actually located
	    // (ddl.Demangle.d can handle some aspects of this)
         }
}

ExporSymbols in the above cover just about everything: ModuleInfo, Class 
Definitions, ctors, dtors, functions, fields, methods and even vtables, 
initalizers, asserts and the rest of the zoo of D symbols exposed in 
.obj files.

So you can look for all the metadata you need, without having to bind to 
a given symbol.  Granted, it still feels like grocery shopping out of 
the "scratched and dented" bin.  Having a proper facade over this is a 
much needed, next step.

-- 
- EricAnderton at yahoo
Sep 28 2006