www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Finding class template instantiations via runtime reflection (for

reply Jean-Louis Leroy <jl leroy.nyc> writes:
It did not take long! Someone tried to create templatized open 
methods and it didn't work right of the box. I expected that, but 
in fact there may be a bit of hope. You cannot have virtual 
function templates in C++ or in D because the layout of the 
vtables have to be known at compile time - but openmethods 
creates its method tables at runtime so maybe it can be made to 
work.

I stumbled upon a problem very quickly: it seems that classes 
that come from class template instantiations are not registered 
in ModuleInfo - see below,

import std.stdio;

class Foo {}

class Bar(T) : Foo
{
   int i;
}

alias BarInt = Bar!int;

const barInt = new BarInt;

void main()
{
   foreach (mod; ModuleInfo) {
     foreach (c; mod.localClasses) {
       writeln(c);
     }
   }
}

...output:
modtemp.Foo
core.exception.RangeError
core.exception.AssertError
etc...

Neither 'Bar!int' nor 'BarInt' appear in 'localClasses'.

Ideas?
Sep 21 2017
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 9/21/17 4:32 PM, Jean-Louis Leroy wrote:

 It did not take long! Someone tried to create templatized open methods 
 and it didn't work right of the box. I expected that, but in fact there 
 may be a bit of hope. You cannot have virtual function templates in C++ 
 or in D because the layout of the vtables have to be known at compile 
 time - but openmethods creates its method tables at runtime so maybe it 
 can be made to work.
 
 I stumbled upon a problem very quickly: it seems that classes that come 
 from class template instantiations are not registered in ModuleInfo - 
 see below,
 
 import std.stdio;
 
 class Foo {}
 
 class Bar(T) : Foo
 {
    int i;
 }
 
 alias BarInt = Bar!int;
 
 const barInt = new BarInt;
 
 void main()
 {
    foreach (mod; ModuleInfo) {
      foreach (c; mod.localClasses) {
        writeln(c);
      }
    }
 }
 
 ....output:
 modtemp.Foo
 core.exception.RangeError
 core.exception.AssertError
 etc...
 
 Neither 'Bar!int' nor 'BarInt' appear in 'localClasses'.
 
 Ideas?
 


It used to be that classes were always in ModuleInfo (and that 
ModuleInfo was always generated). That's not so much the case any more, 
as there has been a push to reduce the runtime footprint and complexity 
(especially toward -betterC). This makes traditional runtime reflection 
more difficult and sporadic.

However, I would expect that if ModuleInfo is generated for a file, and 
some classes are added, then ALL classes instantiated in the module 
should be added.

I would say it's a bug.

-Steve
Sep 22 2017
next sibling parent =?UTF-8?Q?Ali_=c3=87ehreli?= <acehreli yahoo.com> writes:
On 09/22/2017 05:16 AM, Steven Schveighoffer wrote:

 On 9/21/17 4:32 PM, Jean-Louis Leroy wrote:



 it seems that classes that
 come from class template instantiations are not registered in
 ModuleInfo





 I would say it's a bug.

 -Steve


I think so too. I wanted to hear it from Steve first. :)

Ali
Sep 22 2017
prev sibling parent reply user1234 <user1234 12.hu> writes:
On Friday, 22 September 2017 at 12:16:52 UTC, Steven 
Schveighoffer wrote:

 On 9/21/17 4:32 PM, Jean-Louis Leroy wrote:

 It did not take long! Someone tried to create templatized open 
 methods and it didn't work right of the box. I expected that, 
 but in fact there may be a bit of hope. You cannot have 
 virtual function templates in C++ or in D because the layout 
 of the vtables have to be known at compile time - but 
 openmethods creates its method tables at runtime so maybe it 
 can be made to work.
 
 I stumbled upon a problem very quickly: it seems that classes 
 that come from class template instantiations are not 
 registered in ModuleInfo - see below,
 
 import std.stdio;
 
 class Foo {}
 
 class Bar(T) : Foo
 {
    int i;
 }
 
 alias BarInt = Bar!int;
 
 const barInt = new BarInt;
 
 void main()
 {
    foreach (mod; ModuleInfo) {
      foreach (c; mod.localClasses) {
        writeln(c);
      }
    }
 }
 
 ....output:
 modtemp.Foo
 core.exception.RangeError
 core.exception.AssertError
 etc...
 
 Neither 'Bar!int' nor 'BarInt' appear in 'localClasses'.
 
 Ideas?
 


 It used to be that classes were always in ModuleInfo (and that 
 ModuleInfo was always generated). That's not so much the case 
 any more, as there has been a push to reduce the runtime 
 footprint and complexity (especially toward -betterC). This 
 makes traditional runtime reflection more difficult and 
 sporadic.

 However, I would expect that if ModuleInfo is generated for a 
 file, and some classes are added, then ALL classes instantiated 
 in the module should be added.

 I would say it's a bug.

 -Steve


aliases are not symbol so it's expected that 'BarInt' doesn't 
appear in the items.
Would "Bar!int" be usable in Object.factory ?
Sep 23 2017
next sibling parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 9/24/17 12:30 AM, user1234 wrote:

 On Friday, 22 September 2017 at 12:16:52 UTC, Steven Schveighoffer wrote:

 It used to be that classes were always in ModuleInfo (and that 
 ModuleInfo was always generated). That's not so much the case any 
 more, as there has been a push to reduce the runtime footprint and 
 complexity (especially toward -betterC). This makes traditional 
 runtime reflection more difficult and sporadic.

 However, I would expect that if ModuleInfo is generated for a file, 
 and some classes are added, then ALL classes instantiated in the 
 module should be added.

 I would say it's a bug.


 
 aliases are not symbol so it's expected that 'BarInt' doesn't appear in 
 the items.


Correct, even with a fixed compiler, BarInt would not appear.


 Would "Bar!int" be usable in Object.factory ?


I'm not sure what the name of the class might be, as this capability 
depends completely on the compiler.

As it stands, no entry for the class type is made in the ModuleInfo, so 
there's no way to know what the "right name" is -- someone needs to add 
the capability, and that may be when the decision is made.

Note to readers, Object.factory uses exactly the mechanism Jean-Louis 
has used, so if his code can't find it, neither can Object.factory.

-Steve
Sep 25 2017
prev sibling parent Jean-Louis Leroy <jl leroy.nyc> writes:
On Sunday, 24 September 2017 at 04:30:26 UTC, user1234 wrote:

 aliases are not symbol so it's expected that 'BarInt' doesn't 
 appear in the items.


I didn't really expect that either.
Sep 25 2017
prev sibling next sibling parent bitwise <bitwise.pvt gmail.com> writes:
On Thursday, 21 September 2017 at 20:32:38 UTC, Jean-Louis Leroy 
wrote:

 Neither 'Bar!int' nor 'BarInt' appear in 'localClasses'.

 Ideas?


The information you can retrieve through localClasses is limited.

AFAIK it only retrieves plain non-template classes declared 
directly in the module. Last I checked, non-template classes 
added via mixin were skipped as well.

I'm not sure how much work can be expected in this area. Things 
related to runtime reflection have slowly been getting removed 
from druntime and dmd. I don't think there's much left of it.

Your best bet is to scrape the info you need on your own using 
D's traits:

https://dlang.org/phobos/std_traits.html
https://dlang.org/spec/traits.html

I would recommend trying to work with std.traits first (as 
opposed to __traits).

If you need to scan entire modules at a time, then start with 
something like this:

foreach(m; __traits(allMembers, fully.qualified.modulename))
{
     // __traits(getMember, ... , ...)
}
Sep 22 2017
prev sibling next sibling parent bitwise <bitwise.pvt gmail.com> writes:
On Thursday, 21 September 2017 at 20:32:38 UTC, Jean-Louis Leroy 
wrote:

 Neither 'Bar!int' nor 'BarInt' appear in 'localClasses'.

 Ideas?


The information you can retrieve through localClasses is limited.

AFAIK it only retrieves plain non-template classes declared 
directly in the module. Last I checked, non-template classes 
added via mixin were skipped as well.

I'm not sure how much work can be expected in this area. Things 
related to runtime reflection have slowly been getting removed 
from druntime and dmd. I don't think there's much left of it.

Your best bet is to scrape the info you need on your own using 
D's traits:

https://dlang.org/phobos/std_traits.html
https://dlang.org/spec/traits.html

I would recommend trying to work with std.traits first (as 
opposed to __traits).

If you need to scan entire modules at a time, then start with 
something like this:

foreach(m; __traits(allMembers, fully.qualified.modulename))
{
     // __traits(getMember, ... , ...)
}
Sep 22 2017
prev sibling next sibling parent bitwise <bitwise.pvt gmail.com> writes:
On Thursday, 21 September 2017 at 20:32:38 UTC, Jean-Louis Leroy 
wrote:

 Neither 'Bar!int' nor 'BarInt' appear in 'localClasses'.

 Ideas?


The information you can retrieve through localClasses is limited.

AFAIK it only retrieves plain non-template classes declared 
directly in the module. Last I checked, non-template classes 
added via mixin were skipped as well.

I'm not sure how much work can be expected in this area. Things 
related to runtime reflection have slowly been getting removed 
from druntime and dmd. I don't think there's much left of it.

Your best bet is to scrape the info you need on your own using 
D's traits:

https://dlang.org/phobos/std_traits.html
https://dlang.org/spec/traits.html

I would recommend trying to work with std.traits first (as 
opposed to __traits).

If you need to scan entire modules at a time, then start with 
something like this:

foreach(m; __traits(allMembers, fully.qualified.modulename))
{
     // __traits(getMember, ... , ...)
}
Sep 22 2017
prev sibling next sibling parent reply Jean-Louis Leroy <jl leroy.nyc> writes:
Thank you all for your input. I will file a bug report.
Sep 25 2017
parent reply Biotronic <simen.kjaras gmail.com> writes:
On Monday, 25 September 2017 at 13:45:05 UTC, Jean-Louis Leroy 
wrote:

 Thank you all for your input. I will file a bug report.


Just reopen the old one:
https://issues.dlang.org/show_bug.cgi?id=2484

--
   Biotronic
Sep 25 2017
parent reply Jean-Louis Leroy <jl leroy.nyc> writes:
On Monday, 25 September 2017 at 13:57:55 UTC, Biotronic wrote:

 On Monday, 25 September 2017 at 13:45:05 UTC, Jean-Louis Leroy 
 wrote:

 Thank you all for your input. I will file a bug report.


 Just reopen the old one:
 https://issues.dlang.org/show_bug.cgi?id=2484

 --
   Biotronic


Done, thanks.
Sep 25 2017
parent jmh530 <john.michael.hall gmail.com> writes:
On Monday, 25 September 2017 at 14:06:55 UTC, Jean-Louis Leroy 
wrote:

 Done, thanks.


You might add a link to this thread.
Sep 25 2017
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2017-09-21 22:32, Jean-Louis Leroy wrote:

 It did not take long! Someone tried to create templatized open methods 
 and it didn't work right of the box. I expected that, but in fact there 
 may be a bit of hope. You cannot have virtual function templates in C++ 
 or in D because the layout of the vtables have to be known at compile 
 time - but openmethods creates its method tables at runtime so maybe it 
 can be made to work.
 
 I stumbled upon a problem very quickly: it seems that classes that come 
 from class template instantiations are not registered in ModuleInfo - 
 see below,
 
 Ideas?


Not sure exactly what you need get access to. But you could try to 
inspect the symbol table at runtime, if what you need is available there.

Of course, that's a bit of a PITA, but you would not be dependent on any 
bugs getting fixed or new features/improvements implemented.

-- 
/Jacob Carlborg
Sep 25 2017
parent reply Jean-Louis Leroy <jl leroy.nyc> writes:
On Monday, 25 September 2017 at 14:17:13 UTC, Jacob Carlborg 
wrote:

 Of course, that's a bit of a PITA, but you would not be 
 dependent on any bugs getting fixed or new 
 features/improvements implemented.


How do you do that?
Sep 25 2017
parent reply Jacob Carlborg <doob me.com> writes:
On 2017-09-25 16:24, Jean-Louis Leroy wrote:

 On Monday, 25 September 2017 at 14:17:13 UTC, Jacob Carlborg wrote:

 Of course, that's a bit of a PITA, but you would not be dependent on 
 any bugs getting fixed or new features/improvements implemented.


 
 How do you do that?


On Posix you use "dlopen", passing in null as the filename to get a 
handle to the current executable. Then you iterate the sections until 
you find the symbol table. For macOS there's this reference for the 
Mach-O binary format [1]. For Mach-O files there's this helper function 
as well [3].

[1] 
https://web.archive.org/web/20090901205800/http://developer.apple.com/mac/library/documentation/DeveloperTools/Conceptual/MachORuntime/Reference/reference.html

[2] getsectbynamefromheader_64

-- 
/Jacob Carlborg
Sep 26 2017
parent reply Jean-Louis Leroy <jl leroy.nyc> writes:
On Tuesday, 26 September 2017 at 15:59:01 UTC, Jacob Carlborg 
wrote:

 On 2017-09-25 16:24, Jean-Louis Leroy wrote:

 On Monday, 25 September 2017 at 14:17:13 UTC, Jacob Carlborg 
 wrote:

 Of course, that's a bit of a PITA, but you would not be 
 dependent on any bugs getting fixed or new 
 features/improvements implemented.


 
 How do you do that?


 On Posix you use "dlopen", passing in null as the filename to 
 get a handle to the current executable. Then you iterate the 
 sections until you find the symbol table. For macOS there's 
 this reference for the Mach-O binary format [1]. For Mach-O 
 files there's this helper function as well [3].

 [1] 
 https://web.archive.org/web/20090901205800/http://developer.apple.com/mac/library/documentation/DeveloperTools/Conceptual/MachORuntime/Reference/reference.html

 [2] getsectbynamefromheader_64


Ah, I suspected that. I don't want to go down that path, what of 
Windows? But thanks anyway...
Sep 26 2017
parent Jacob Carlborg <doob me.com> writes:
On 2017-09-26 18:08, Jean-Louis Leroy wrote:


 Ah, I suspected that. I don't want to go down that path, what of 
 Windows? But thanks anyway...


It's possible to do the same on Windows (using LoadLibrary instead of 
dlopen). You would need to have separate code for each binary format. On 
Windows there are two, OMF used by DigitalMars and COFF used by 
Microsoft. DMD can output both. Linux and FreeBSD use ELF.

-- 
/Jacob Carlborg
Sep 27 2017
prev sibling parent reply b4s1L3 <b2.temp gmx.com> writes:
On Thursday, 21 September 2017 at 20:32:38 UTC, Jean-Louis Leroy 
wrote:

 It did not take long! Someone tried to create templatized open 
 methods and it didn't work right of the box. I expected that, 
 but in fact there may be a bit of hope. You cannot have virtual 
 function templates in C++ or in D because the layout of the 
 vtables have to be known at compile time - but openmethods 
 creates its method tables at runtime so maybe it can be made to 
 work.

 I stumbled upon a problem very quickly: it seems that classes 
 that come from class template instantiations are not registered 
 in ModuleInfo - see below,

 [...]

 Neither 'Bar!int' nor 'BarInt' appear in 'localClasses'.

 Ideas?


Yeah. You can setup a custom registry that maps names to their 
TypeInfo_Class.
I do something similar in iz (though the real point in iz is a 
GC-free factory but the principle of the registry would be the 
the same for you)

Example:

---
/+ dub.sdl:
    name "dub_script"
    dependency "iz" version="0.6.0"
+/
module dub_script;

import iz.memory, std.stdio;

class Foo(T){this(){writeln("fooDeInt");}}
TypeInfo_Class[string] registry; // you need that...

static this()
{
     registerFactoryClass!(Foo!int)(registry); // ...and that, 
maybe  with another name
}

void main()
{
     auto fooDeInt = iz.memory.factory(registry, "Foo!int");
     destruct(cast(Object) fooDeInt);
}
---
Sep 25 2017
parent Jean-Louis Leroy <jl leroy.nyc> writes:
On Tuesday, 26 September 2017 at 04:55:46 UTC, b4s1L3 wrote:

 On Thursday, 21 September 2017 at 20:32:38 UTC, Jean-Louis 
 Leroy wrote:

 [...]


 Yeah. You can setup a custom registry that maps names to their 
 TypeInfo_Class.
 I do something similar in iz (though the real point in iz is a 
 GC-free factory but the principle of the registry would be the 
 the same for you)

 Example:

 ---
 /+ dub.sdl:
    name "dub_script"
    dependency "iz" version="0.6.0"
 +/
 module dub_script;

 import iz.memory, std.stdio;

 class Foo(T){this(){writeln("fooDeInt");}}
 TypeInfo_Class[string] registry; // you need that...

 static this()
 {
     registerFactoryClass!(Foo!int)(registry); // ...and that, 
 maybe  with another name
 }

 void main()
 {
     auto fooDeInt = iz.memory.factory(registry, "Foo!int");
     destruct(cast(Object) fooDeInt);
 }
 ---


Yeah that's what I implemented yesterday (not pushed yet).

I have a prototype of a solution for using open methods and 
templates together - although the syntax is not light: it 
requires guidance from the user (a la C++ explicit template 
instantiation).
Sep 26 2017