digitalmars.D - std.reflection prototype
- bitwise (40/40) Mar 29 2015 I came across this post a while back and decided to implement it:
- Rikki Cattermole (6/45) Mar 29 2015 I love the idea of it.
- bitwise (5/10) Mar 29 2015 In the "time for std.reflection" convo, it seemed like Andre was
- Rikki Cattermole (5/16) Mar 29 2015 This definitely should be a library solution. But it seems silly, to not...
- bitwise (14/16) Mar 29 2015 In terms of what Andre was suggesting, I think my implementation
- Rikki Cattermole (28/42) Mar 29 2015 I have so many views upon all this, thanks to my new (not released as
- Kapps (18/34) Mar 29 2015 When I tried mine
- bitwise (14/24) Mar 30 2015 This sounds like a bug.. if it CAN be done with an executable
- bitwise (33/33) Mar 30 2015 https://github.com/D-Programming-Language/druntime/pull/775
- Jacob Carlborg (39/43) Mar 31 2015 Currently in object.d in druntime there's a template, RTInfo, declared
- bitwise (53/74) Apr 01 2015 If I'm understanding correctly, doing it this way is to avoid
- Jacob Carlborg (21/28) Apr 01 2015 As I said, the reason for implementing RTInfo for modules was to _not_
- bitwise (14/49) Apr 04 2015 Ok, I think I understand what you're suggesting now, which is
- Jacob Carlborg (8/20) Apr 05 2015 Yes, exactly. I don't know if it works with separate complication or
- bitwise (12/12) Apr 04 2015 One more question:
- Jacob Carlborg (5/13) Apr 05 2015 I don't know why they were not removed.
- ketmar (3/10) Mar 30 2015 'cause DMD on posix doesn't strip binaries by default. it's funny to=20
- Jacob Carlborg (7/8) Mar 29 2015 I made a pull request for that but unfortunately it hasn't been accepted...
- Rikki Cattermole (3/9) Mar 30 2015 You, me and Walter should have a chat then. I could pretty easily come
- Jacob Carlborg (22/24) Mar 30 2015 I've already come up with a way, any template with the @rtInfo UDA is
- bitwise (2/8) Apr 04 2015 Ok, I'm starting to understand this idea a little better. Is
- Rikki Cattermole (4/14) Apr 04 2015 I don't think so. I looked into it, and it looks to be pretty low
- bitwise (33/36) Apr 05 2015 I'm trying to get it figured out right now.
- bitwise (2/5) Apr 05 2015 Scratch that, I think this is what I'm looking for:
- Rikki Cattermole (2/7) Apr 05 2015 It most certainly is.
- bitwise (8/9) Apr 06 2015 I've continued to dig and try to reason about the history behind
- rcorre (39/39) Apr 13 2015 For me, having a solid reflection library like this is one of
- bitwise (58/72) Apr 13 2015 At this point, I've kinda hit a wall.
- Jacob Carlborg (5/10) Apr 14 2015 Yeah, this will require runtime reflection to avoid the registering of
- bitwise (5/14) Apr 14 2015 This is one of the main reasons I like the OffsetTypeInfo/offTi() option...
- Jacob Carlborg (6/10) Apr 15 2015 It needs to be possible to set and get a value of an instance variable
- bitwise (28/31) Apr 15 2015 Right now, this is the def:
- bitwise (26/28) Apr 15 2015 To clarify, I'm talking about doing doing things by Object pointer or vo...
- Jacob Carlborg (4/31) Apr 15 2015 I'm not sure, I would assume so. Why is this not safe in C++?
- bitwise (32/66) Apr 15 2015 One reason is that casting with multiple inheritance offsets the pointer...
- Jacob Carlborg (10/13) Apr 15 2015 In D I would assume this would eventually be possible:
- bitwise (55/62) Apr 16 2015 Ok, that sounds right. D has no multiple or virtual inheritance, so I
- Jacob Carlborg (13/21) Apr 17 2015 We need a "getMembers" on ClassInfo. Because "typeid" will resolve the
- bitwise (22/42) Apr 17 2015 Sorry, that's what I meant(using this on classes, not modules).
- bitwise (18/18) Apr 15 2015 One more thing.
- Baz (9/50) Mar 31 2015 This is needed for sure. For example do you count how many
- Gheorghe Gabriel (4/45) Sep 27 2017 Hi, your link is not working any more and I really need your
- bitwise (7/12) Sep 27 2017 I took the code down because there were design flaws I had to
- Gheorghe Gabriel (19/32) Sep 27 2017 Thank you, I really need it for my project.
I came across this post a while back and decided to implement it: http://forum.dlang.org/thread/juf7sk$16rl$1 digitalmars.com My implementation: https://github.com/bitwise-github/D-Reflection The above conversation seemed to stop abruptly, so I went on to assume that no one chose to champion the task. At the time, I looked around for other conversations or attempts at runtime reflection for D, but couldn't really find anything. I did find the ModuleInfo/reflection stuff in object.d, but it seemed like an effort that may have been abandoned. Also, the above conversation seemed to suggest it should be opt-in, which also made me wonder if the stuff in object.d was abandoned or for a different purpose. Looking again today, someone seems to have been working on it a bit.. For example, MemberInfo_field and MemberInfo_function were empty last time I checked. So what's the current state of things? Is anybody working on it? Although MemberInfo_field exists, it doesn't seem to be available from TypeInfo_Class... Is that the eventual goal? Is there anything I can do to help get things moving? Any comments on my implementation would be welcome as well. (https://github.com/bitwise-github/D-Reflection) main.d shows some general use cases, but basically, if the reflect(T) template is used on a class, that class, and any child types will be reflected. Recursive reflection only propagates downward, or else it could leak sideways and unnecessarily reflect several modules. Most of the reflection information is available at compile time. For example: enum name = reflectModule!(test).findClass("Test").findField("variable").name; pragma(msg, name); // "variable" will be outputted. To make a module available for runtime reflection, the following can be used: mixin(runtimeReflection!test); At this point, the above example can be rewritten as: string name = getModuleReflection("tests.test").findClass("Test3").findField("static_variable").name; writeln(name);
Mar 29 2015
On 30/03/2015 2:11 p.m., bitwise wrote:I came across this post a while back and decided to implement it: http://forum.dlang.org/thread/juf7sk$16rl$1 digitalmars.com My implementation: https://github.com/bitwise-github/D-Reflection The above conversation seemed to stop abruptly, so I went on to assume that no one chose to champion the task. At the time, I looked around for other conversations or attempts at runtime reflection for D, but couldn't really find anything. I did find the ModuleInfo/reflection stuff in object.d, but it seemed like an effort that may have been abandoned. Also, the above conversation seemed to suggest it should be opt-in, which also made me wonder if the stuff in object.d was abandoned or for a different purpose. Looking again today, someone seems to have been working on it a bit.. For example, MemberInfo_field and MemberInfo_function were empty last time I checked. So what's the current state of things? Is anybody working on it? Although MemberInfo_field exists, it doesn't seem to be available from TypeInfo_Class... Is that the eventual goal? Is there anything I can do to help get things moving? Any comments on my implementation would be welcome as well. (https://github.com/bitwise-github/D-Reflection) main.d shows some general use cases, but basically, if the reflect(T) template is used on a class, that class, and any child types will be reflected. Recursive reflection only propagates downward, or else it could leak sideways and unnecessarily reflect several modules. Most of the reflection information is available at compile time. For example: enum name = reflectModule!(test).findClass("Test").findField("variable").name; pragma(msg, name); // "variable" will be outputted. To make a module available for runtime reflection, the following can be used: mixin(runtimeReflection!test); At this point, the above example can be rewritten as: string name = getModuleReflection("tests.test").findClass("Test3").findField("static_variable").name; writeln(name);I love the idea of it. But first we need to finish off what support we damn well have in druntime. m_offTi for TypeInfo_Class currently is not being generated. We also need some form of RTInfo for modules. Not just for symbols. This alone will open up quite a few doors!
Mar 29 2015
But first we need to finish off what support we damn well have in druntime. m_offTi for TypeInfo_Class currently is not being generated. We also need some form of RTInfo for modules. Not just for symbols. This alone will open up quite a few doors!In the "time for std.reflection" convo, it seemed like Andre was talking about a library solution, and not something compiler generated.. has that idea since been thrown out? Has there been other conversations on this that I missed? Thanks
Mar 29 2015
On 30/03/2015 5:39 p.m., bitwise wrote:This definitely should be a library solution. But it seems silly, to not use code in druntime. Either that, or if it is like m_offTi, remove it. After all, it was never implemented.But first we need to finish off what support we damn well have in druntime. m_offTi for TypeInfo_Class currently is not being generated. We also need some form of RTInfo for modules. Not just for symbols. This alone will open up quite a few doors!In the "time for std.reflection" convo, it seemed like Andre was talking about a library solution, and not something compiler generated.. has that idea since been thrown out?Has there been other conversations on this that I missed?No.
Mar 29 2015
This definitely should be a library solution.In terms of what Andre was suggesting, I think my implementation is 90% of the way there, so if the will was there to add something like std.reflection to phobos, it wouldn't be much of a leap.But it seems silly, to not use code in druntime.I kind of agree. If I had things my way, all reflection info for all classes would be compiler generated so that it was guaranteed to be there for tools, interop and such. If people were really worried about code bloat, the reflection info could simply be turned off with a -no-reflection flag or something. In terms of playing around with the compiler and the type system, I'm not sure how helpful I could be right now. I'm pretty sure comprehension will not be a problem, but I've got quite a bit of reading to do before I'm up to speed with whats going on with dmd and the type system.
Mar 29 2015
On 30/03/2015 6:06 p.m., bitwise wrote:I have so many views upon all this, thanks to my new (not released as not ready) web service framework. But druntime right now is not ready for proper reflection. There needs to be some modifications to it and dmd. But mostly druntime for RTInfo. For example what shouldn't needed to be done. Here is my install information for my autoconfig system. (Allows for publically importing any module I want AND handling any symbol that the system has used. #Install RTInfo requires a single modification. [Based upon a forum thread](http://forum.dlang.org/post/majnjuhxdefjuqjlpbmv forum.dlang.org). ```D template RTInfo(T, string moduleName = __MODULE__) { enum RTInfo = cast(void*)0x12345678; static if (__traits(compiles, {import core.config;})) { import core.config; alias checkResult = TypeCheck!(T, moduleName); } } ``` COPY/core/configimports.d needs to be copied to import/core/configimports.d Also needed is object.di under imports to be modified. Add ``D public import core.configimports;`` anywhere you want.This definitely should be a library solution.In terms of what Andre was suggesting, I think my implementation is 90% of the way there, so if the will was there to add something like std.reflection to phobos, it wouldn't be much of a leap.But it seems silly, to not use code in druntime.I kind of agree. If I had things my way, all reflection info for all classes would be compiler generated so that it was guaranteed to be there for tools, interop and such. If people were really worried about code bloat, the reflection info could simply be turned off with a -no-reflection flag or something. In terms of playing around with the compiler and the type system, I'm not sure how helpful I could be right now. I'm pretty sure comprehension will not be a problem, but I've got quite a bit of reading to do before I'm up to speed with whats going on with dmd and the type system.
Mar 29 2015
On Monday, 30 March 2015 at 05:06:14 UTC, bitwise wrote:When I tried mine (https://shardsoft.com/stash/projects/SHARD/repos/shardtools/browse/source/Shard ools/Reflection.d), which was admittedly a while back, the template bloat was actually a significant problem. On Linux and OSX, the build for just the unittest library would generate an over 40MB executable. On Windows with Optlink though, it was only 4MB, so perhaps there are optimizations that could be made to reduce the size on OSX / Linux. Either way though, that's way too much bloat for something in druntime, and I don't think it's necessary. While it might be nice to use RTInfo to generate reflection data for your whole program, most people would be perfectly okay with at most a few modules, possibly recursively. Also, from what I can tell you generate a class for every symbol. This could generate a significant amount of bloat as well because it would generate further TypeInfo for each of these classes. I could be wrong here, but if that's the case, perhaps changing to structs would be a better choice if possible?This definitely should be a library solution.In terms of what Andre was suggesting, I think my implementation is 90% of the way there, so if the will was there to add something like std.reflection to phobos, it wouldn't be much of a leap.But it seems silly, to not use code in druntime.I kind of agree. If I had things my way, all reflection info for all classes would be compiler generated so that it was guaranteed to be there for tools, interop and such. If people were really worried about code bloat, the reflection info could simply be turned off with a -no-reflection flag or something. In terms of playing around with the compiler and the type system, I'm not sure how helpful I could be right now. I'm pretty sure comprehension will not be a problem, but I've got quite a bit of reading to do before I'm up to speed with whats going on with dmd and the type system.
Mar 29 2015
On Linux and OSX, the build for just the unittest library would generate an over 40MB executable. On Windows with Optlink though, it was only 4MB, so perhaps there are optimizations that could be made to reduce the size on OSX / Linux.This sounds like a bug.. if it CAN be done with an executable size of 4MB, why would it not be so on OSX/Linux? Also, i was reading through this pull request: https://github.com/D-Programming-Language/druntime/pull/775 and yglukhov mentioned something about needing 5 minutes and 12GB or ram to build phobos with whatever they're talking about. To be honest though, after reading the above pull-comments and this one below, I'm still confused about what they're trying to do. https://github.com/D-Programming-Language/dmd/pull/2271Also, from what I can tell you generate a class for every symbol. This could generate a significant amount of bloat as well because it would generate further TypeInfo for each of these classes. I could be wrong here, but if that's the case, perhaps changing to structs would be a better choice if possible?You're right that I am creating a class per symbol, along with a bunch of other templates. It probably would be a good idea to try and eliminate the templates where possible. I'm not sure why you are suggesting structs instead of classes though.
Mar 30 2015
https://github.com/D-Programming-Language/druntime/pull/775 https://github.com/D-Programming-Language/dmd/pull/2271 I've read over the comments of the above two pull-requests, and I'm confused about what is trying to be achieved. A ModuleInfo for each module already exists and is available at runtime. Each ModuleInfo has the method localClasses() to get all it's classes. Why not just add an array of MethodInfo and FieldInfo to TypeInfo_Class? It seems like the above two pulls are trying to add some kind of custom per-module info. For my purposes, some very basic info for suffice. At the very minimum, if TypeInfo_Class could be completed to contain these, I would be happy: // one per field class FieldInfo { string name(); TypeInfo type(); void *getAddress(void *instance); } // one per overload class MethodInfo { string name(); TypeInfo returnType(); TypeInfo[] parameterTypes(); bool isProperty(); bool isStatic(); void *getAddress(void *instance); } The above two would be more than enough to do serialization and runtime modification of unknown objects. What am I missing here?
Mar 30 2015
On 2015-03-30 23:31, bitwise wrote:https://github.com/D-Programming-Language/druntime/pull/775 https://github.com/D-Programming-Language/dmd/pull/2271 I've read over the comments of the above two pull-requests, and I'm confused about what is trying to be achieved.Currently in object.d in druntime there's a template, RTInfo, declared something like this: template RTInfo (T) { enum RTInfo = null; } The compiler will automatically instantiate this template for all user defined types. What the template evaluates to will be placed in the TypeInfo for that given type in the "rtInfo" property. This allows two things: * Add custom type checking on user defined types: template RTInfo (T) { static if (T.stringof == "Foo") static assert(false, "A type named 'Foo' is not allowed"); enum RTInfo = null; } * Add custom type info for a given type which is accessible at runtime: template RTInfo (T) { enum RTInfo = T.stringof; } The original reason why this was added to the language was to extract type info necessary to build a precise garbage collector. The problem with this is to use this feature you need to modify object.d in druntime. The first pull request tries to come up with a solution that doesn't require modifying druntime. The second pull request implemented the same thing as RTInfo but for modules instead of types. So the compiler would automatically instantiate this template with for each module it encounters during compilation. The main reason for implementing RTInfo for modules is to implement reflection. With the template instantiated for each module you could scan the module for class and methods and build up reflection data completely transparently without the user needing to register any types or similar. -- /Jacob Carlborg
Mar 31 2015
Currently in object.d in druntime there's a template, RTInfo, declared something like this: [snip]Thanks for the breakdown! Those pull requests make a lot more sense when read in the proper context ;)The main reason for implementing RTInfo for modules is to implement reflection. With the template instantiated for each module you could scan the module for class and methods and build up reflection data completely transparently without the user needing to register any types or similar.If I'm understanding correctly, doing it this way is to avoid making changes to the compiler, right? I don't understand this decision because it seems that most of the needed infrastructure is already built into ModuleInfo, and that it just needs to be completed. It would eliminate the problem of template/code bloat from a library like mine, and at the same time, would not require the user to register any types. The downside would be susceptibility to things like .NET Reflector, making binaries very easy to crack/decompile, and of course, the extra space it took for the metadata. But, the generation of the metadata could be made optional via command line switch. Metadata could either be disabled completely, or partially with something like --no-verbose-rtti, which could omit all the human readable portions of the metadata, while leaving behind whatever was needed for precise GC or whatever else may need it. Of course, the inverse would also be an option, to omit all type info and enable it manually via --verbose-rtti. Failing all of the above, I suppose my reflection library does the job well enough for my needs. Some work could be done to optimize the binary size, but feature-wise, I find it more than complete enough. However, I am thinking about making it more versatile in terms of what is generated, because that seems to be a common request. Something like this might make sense: module test; class Test1{} Reflected class Test2{} NotReflected class Test3{} /////////////////// module reflection; enum ReflectionMode { Inclusive, // include all classes unless specified Exclusive // exclude all classes unless specified } template moduleReflection(alias mod, ReflectionMode m) { ... } /////////////////// module test; mixin(moduleReflection!(test, ReflectionMode.Inclusive)); /////////////////// ReflectionMode.Inclusive would generate reflection for Test1 and Test2, ReflectionMode.Exclusive would generate reflection for only Test2. I'm not sure it's possible to make recursiveness of reflection optional without losing compile-time availability of major features though.This is needed for sure. For example do you count how many subjects on .learn are about __traits(allMembers,... or __traits(getMember,...Yeah, I agree these are confusing... along with type-tuples, and CTFE. I think all three of these are hard to grasp without knowledge of how a compiler works.About the design i would rather see this as a user choice, eg the reflexion as a mixin template that you can mix in a class or a struct you want to be filled with info. This remark is related to the size problem pointed in the previous posts.I would like to make things as flexible/optional as possible without being intrusive to user types. UDAs are about as far as I'm willing to go with per-type boilerplate.
Apr 01 2015
On 2015-04-02 02:28, bitwise wrote:If I'm understanding correctly, doing it this way is to avoid making changes to the compiler, right? I don't understand this decision because it seems that most of the needed infrastructure is already built into ModuleInfo, and that it just needs to be completed. It would eliminate the problem of template/code bloat from a library like mine, and at the same time, would not require the user to register any types.As I said, the reason for implementing RTInfo for modules was to _not_ have to register anything. There are other good use cases for both RTInfo and RMInfo (runtime module info), they are a more generic solution. Two other threads about unit testing [1], [2] is a good use case. RMInfo can be used to collect all unit test functions and create a custom runner. Here's [3] one example where a unit test runner makes it possible to have CTFE unit tests. This proof of concept only scans the current module, here RMInfo would be really handy to scan all modules. Here's [4] one example where RTInfo is used to check virtual methods. All virtual methods are required to be marked with virtual. There's a lot of missing info in ModuleInfo and TypeInfo. For example MemberInfo_function contains no information about parameters, return types, attributes and so on. [1] http://forum.dlang.org/thread/mfcgj3$12a0$1 digitalmars.com [2] http://forum.dlang.org/thread/mfci6o$13oa$1 digitalmars.com [3] http://forum.dlang.org/thread/ks1brj$1l6c$1 digitalmars.com [4] http://forum.dlang.org/thread/kok86c$126l$1 digitalmars.com -- /Jacob Carlborg
Apr 01 2015
On Thursday, 2 April 2015 at 06:57:25 UTC, Jacob Carlborg wrote:On 2015-04-02 02:28, bitwise wrote:Ok, I think I understand what you're suggesting now, which is that you want a library to be able to override RTInfo in order to add it's own metadata to all types, which raises the question, what if more than one library wants to add metadata? And I think this question was addressed by the suggestion of the AA for RTInfo where the module's fully qualified name was the key, which won't work because of separate compilation... right? So in my case I could just update my RTInfo to generate a reflection for each type, and make it accessible using a UFC or something. If I understand correctly though, the idea has been dismissed as being impossible due to separate compilation, right? Is there any leads on this at this point?If I'm understanding correctly, doing it this way is to avoid making changes to the compiler, right? I don't understand this decision because it seems that most of the needed infrastructure is already built into ModuleInfo, and that it just needs to be completed. It would eliminate the problem of template/code bloat from a library like mine, and at the same time, would not require the user to register any types.As I said, the reason for implementing RTInfo for modules was to _not_ have to register anything. There are other good use cases for both RTInfo and RMInfo (runtime module info), they are a more generic solution. Two other threads about unit testing [1], [2] is a good use case. RMInfo can be used to collect all unit test functions and create a custom runner. Here's [3] one example where a unit test runner makes it possible to have CTFE unit tests. This proof of concept only scans the current module, here RMInfo would be really handy to scan all modules. Here's [4] one example where RTInfo is used to check virtual methods. All virtual methods are required to be marked with virtual. There's a lot of missing info in ModuleInfo and TypeInfo. For example MemberInfo_function contains no information about parameters, return types, attributes and so on. [1] http://forum.dlang.org/thread/mfcgj3$12a0$1 digitalmars.com [2] http://forum.dlang.org/thread/mfci6o$13oa$1 digitalmars.com [3] http://forum.dlang.org/thread/ks1brj$1l6c$1 digitalmars.com [4] http://forum.dlang.org/thread/kok86c$126l$1 digitalmars.com
Apr 04 2015
On 2015-04-04 18:16, bitwise wrote:Ok, I think I understand what you're suggesting now, which is that you want a library to be able to override RTInfo in order to add it's own metadata to all types, which raises the question, what if more than one library wants to add metadata? And I think this question was addressed by the suggestion of the AA for RTInfo where the module's fully qualified name was the key, which won't work because of separate compilation... right?Yes, exactly. I don't know if it works with separate complication or not. But it will make it more complicated.So in my case I could just update my RTInfo to generate a reflection for each type, and make it accessible using a UFC or something. If I understand correctly though, the idea has been dismissed as being impossible due to separate compilation, right? Is there any leads on this at this point?I don't think it's impossible, but more it's complicated. I don't think I got a really good elaborated answer form Martin why he didn't like the AA approach, except for being complicated. -- /Jacob Carlborg
Apr 05 2015
One more question: Does anyone know why TypeInfo_class.getMembers() was removed? [1] I found an old post saying that it never worked and returned an empty array, so that is most likely the answer, but although getMembers was removed, Walter seems to have left behind the classes MemberInfo, MemberInfo_field, and MemberInfo_function, so I'm guessing there is still hope of having them implemented one day. [2] [1] https://github.com/D-Programming-Language/druntime/commit/f957b4ef7222bae5da7f3b4104ab42061dd41319#diff-7eac7eb46e31907f148813e793155274 [2] https://github.com/D-Programming-Language/druntime/blob/bd3f4cb2122edd1a7c107f86936c20bba15191b6/src/object.di#L276
Apr 04 2015
On 2015-04-04 21:49, bitwise wrote:One more question: Does anyone know why TypeInfo_class.getMembers() was removed? [1] I found an old post saying that it never worked and returned an empty array, so that is most likely the answerYes.but although getMembers was removed, Walter seems to have left behind the classes MemberInfo, MemberInfo_field, and MemberInfo_function, so I'm guessing there is still hope of having them implemented one day. [2]I don't know why they were not removed. -- /Jacob Carlborg
Apr 05 2015
On Mon, 30 Mar 2015 21:13:48 +0000, bitwise wrote:'cause DMD on posix doesn't strip binaries by default. it's funny to=20 scare windows people.=On Linux and OSX, the build for just the unittest library would generate an over 40MB executable. On Windows with Optlink though, it was only 4MB, so perhaps there are optimizations that could be made to reduce the size on OSX / Linux.=20 This sounds like a bug.. if it CAN be done with an executable size of 4MB, why would it not be so on OSX/Linux?
Mar 30 2015
On 2015-03-30 04:09, Rikki Cattermole wrote:We also need some form of RTInfo for modules.I made a pull request for that but unfortunately it hasn't been accepted yet. The reason seems to be that we need to come up with a way to do custom RTInfo without modifying druntime, that can also be used for this template. -- /Jacob Carlborg
Mar 29 2015
On 30/03/2015 7:59 p.m., Jacob Carlborg wrote:On 2015-03-30 04:09, Rikki Cattermole wrote:You, me and Walter should have a chat then. I could pretty easily come up with a way to add data into RTInfo.We also need some form of RTInfo for modules.I made a pull request for that but unfortunately it hasn't been accepted yet. The reason seems to be that we need to come up with a way to do custom RTInfo without modifying druntime, that can also be used for this template.
Mar 30 2015
On 2015-03-30 09:06, Rikki Cattermole wrote:You, me and Walter should have a chat then. I could pretty easily come up with a way to add data into RTInfo.I've already come up with a way, any template with the rtInfo UDA is treated the same way as RTInfo is now. The problem is then how to store the data in TypeInfo. Since it would be possible to have multiple data generated for a given type I was thinking it could be stored in an associative array. The keys would be the name of the module which generated the data and the the values would be the data. Something like this: module foo.bar; rtInfo template Foo (T) { enum Foo = "bar"; } assert(typeid(T).rtInfo["foo.bar"] == "bar"); If I recall correctly Martin Nowak didn't like this approach. The associate array would need to be built at load time of the application due to separate compilation. BTW, here [1] is the pull request and the reason why it was closed. [1] https://github.com/D-Programming-Language/dmd/pull/2271#issuecomment-59621060 -- /Jacob Carlborg
Mar 30 2015
I love the idea of it. But first we need to finish off what support we damn well have in druntime. m_offTi for TypeInfo_Class currently is not being generated. We also need some form of RTInfo for modules. Not just for symbols. This alone will open up quite a few doors!Ok, I'm starting to understand this idea a little better. Is anyone currently working on generating m_offTi?
Apr 04 2015
On 5/04/2015 4:41 a.m., bitwise wrote:I don't think so. I looked into it, and it looks to be pretty low hanging and easy to do. I'm just not setup *grumbles*.I love the idea of it. But first we need to finish off what support we damn well have in druntime. m_offTi for TypeInfo_Class currently is not being generated. We also need some form of RTInfo for modules. Not just for symbols. This alone will open up quite a few doors!Ok, I'm starting to understand this idea a little better. Is anyone currently working on generating m_offTi?
Apr 04 2015
I don't think so. I looked into it, and it looks to be pretty low hanging and easy to do. I'm just not setup *grumbles*.I'm trying to get it figured out right now. It looks like this will load all the static data for me: https://github.com/D-Programming-Language/druntime/blob/master/src/rt/sections_win32.d#L53 I'm guessing the TypeInfo will be part of the regular data segment. I'm a little confused why the ModuleInfo has it's own special segment.. Would that have to be done for all the MemberInfo too? Anyways, as far as I can tell, druntime would not have to be altered to implement offTi. Looking at the compiler now, I'm trying to figure out where the TypeInfo would be outputted. It seems like this is it: https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L1013 I'm pretty sure this function here is converting the typeinfo to a linked list of data that gets outputted to the object file: [code] TypeInfo_toDt(&s->Sdt, tid); [/code] https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L1027 But, inside that call, I find this: [code] TypeInfoDtVisitor v(pdt); d->accept(&v); [/code] https://github.com/D-Programming-Language/dmd/blob/master/src/typinf.c#L602 which finally leads me to... void TypeInfoDtVisitor::visit(TypeInfoClassDeclaration *d) { //printf("TypeInfoClassDeclaration::toDt() %s\n", tinfo->toChars()); assert(0); } which is not helpful. Anyways, I'll keep looking, but help is welcome :)
Apr 05 2015
I'm pretty sure this function here is converting the typeinfo to a linked list of data that gets outputted to the object file:Scratch that, I think this is what I'm looking for: https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L308......
Apr 05 2015
On 6/04/2015 6:08 a.m., bitwise wrote:It most certainly is.I'm pretty sure this function here is converting the typeinfo to a linked list of data that gets outputted to the object file:Scratch that, I think this is what I'm looking for: https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L308......
Apr 05 2015
It most certainly is.I've continued to dig and try to reason about the history behind this problem. This code has actually been around since D1, but was never completed: https://github.com/D-Programming-Language/dmd/blob/master/src/dsymbol.c#L1237 I think finishing up getMembers() rather than trying to output OffsetTypeInfo would be a better idea. Thoughts?
Apr 06 2015
For me, having a solid reflection library like this is one of the most important improvements D can make right now. After releasing jsonizer (a json serializer) I started talking to the authors of PainlessJson (another json serializer that was released around the same time) about how there was a substantial amount of overlap between our implementations, a large portion of which was related to reflection rather than actually handling json. While I won't speak for painlessjson, I can say that the techniques I used in jsonizer got pretty hacky -- there's a whole lot of __traits and one monstrous mixin template. For the most part, this looks really nice. Thanks for putting it together, bitwise! That being said, there are a few limitiations I see to this implementation: How can you get the user-defined attributes of a given ScopeRefl? It would be awesome to say: foreach(field ; reflect!Foo.fields) { static if (field.hasAttribute(...)) { ... } } I guess the difficulty here is that a user-defined attribute in D can be pretty much anything from a primitive type to a struct instance to some arbitrary symbol. How can you get the static type of a reflected field? For example: alias T = reflect!Foo.fields[0].field_type doSomethingWithType!T This is available in FieldRefImpl, but disappears on FieldRefl so it can be stuck into an array with a bunch of other FieldRefl, and I'm not sure if there is a way of getting that information back (other than resorting back to __traits on the user-end, which kind of defeats the purpose). If I understand correctly, more work needs to be done in druntime to better support reflection, but I've read this thread about 3 times and I'm still having a hard time figuring out whats going on :) If these changes were made to druntime, would they obviate the need for a reflection library, or simply make it easier to implement a reflection library? Is more help needed? If so, where should I start looking? I'm not familiar with druntime at all, but wouldn't mind trying to make sense of it.
Apr 13 2015
On Mon, 13 Apr 2015 08:57:26 -0400, rcorre <ryan rcorre.net> wrote:For me, having a solid reflection library like this is one of the most important improvements D can make right now.At this point, I've kinda hit a wall. Generating a hierarchical/object-oriented representation of the type system at compile-time using templates seems to be an all-or-nothing procedure. Below is an example of the base() method of ClassRefl: const(ClassRefl) base() const { static if(!is(T == Object)) { alias BaseClassesTuple!T base_type; return reflect!(base_type[0]); } } The method returns "reflect!(base_type[0])", which, when instantiated, will create another ClassRefl for the base class. That ClassRefl will also have a base() method, which will instantiate more reflections/templates, and so on... I also had a parent() method at one point, but since a module can be the parent of a class, it was possible that reflecting a single class using reflect!T could build entire reflection hierarchies for several modules. As soon as you reflect anything right now, all dependent reflections must also be instantiated. The amount of bloat(compiled binary size), and the increase in compile time would most likely be a deal breaker for a lot of people. My approach could probably be optimized, but I'm not sure it would be enough to make it usable. I'm out of ideas right now, but open to suggestions if you have a solution to the above problem.How can you get the static type of a reflected field? For example: alias T = reflect!Foo.fields[0].field_type doSomethingWithType!TI don't believe an answer exists with my current design.How can you get the user-defined attributes of a given ScopeRefl?I don't think this is possible either with the current design. The current design must also work at runtime, and since UDAs can be any symbol, I'm not sure there is a good way to generalize this. I could wrap all attributes in an AttributeRefl class, but there wouldn't be much to tell about the wrapped attribute beside it's type, and maybe a string representation of it(more bloat). I'm sure that more could be done to complement __traits and std.traits, but I don't think that includes using my reflection library in it's current state.I can say that the techniques I used in jsonizer got pretty hacky -- there's a whole lot of __traits and one monstrous mixin template.You may want to look at "Orange". https://github.com/jacob-carlborg/orange The API is clean, and non-invasive. The only downside for me, is that in order to serialize a class by a base pointer, you have to register it at compile time.If I understand correctly, more work needs to be done in druntime to better support reflection, but I've read this thread about 3 times and I'm still having a hard time figuring out whats going on :)The RTInfo(T) template would, in theory, automatically generate some custom metadata for every type. The compiler automatically instantiates the RTInfo(T) template for each type right now. The idea was suggested that a programmer could create their own RTInfo(T) that the compiler would use that instead. This option was thrown out because it doesn't work with separate compilation. Right now, RTInfo(T) is defined in object.d, which is implicitly imported into every module that you compile, but if you had your own RTInfo defined in your own module, it wouldn't be available to any other modules at compile time, nor would it be available to any third party libraries you may be using. The OffsetTypeInfo thing would be compiler generated. However, the oldest version of DMD I can find on github (2009) says this: // offTi[] dtdword(&dt, 0); dtdword(&dt, 0); // null for now, fix later So I'm not sure why it was never implemented. I may eventually attempt a pull request for the above fix.For the most part, this looks really nice. Thanks for putting it together, bitwise!Thanks ;)
Apr 13 2015
On 2015-04-14 07:03, bitwise wrote:You may want to look at "Orange". https://github.com/jacob-carlborg/orange The API is clean, and non-invasive. The only downside for me, is that in order to serialize a class by a base pointer, you have to register it at compile time.Yeah, this will require runtime reflection to avoid the registering of the type. -- /Jacob Carlborg
Apr 14 2015
On Tue, 14 Apr 2015 04:16:16 -0400, Jacob Carlborg <doob me.com> wrote:On 2015-04-14 07:03, bitwise wrote:This is one of the main reasons I like the OffsetTypeInfo/offTi() option. I think a "name" field would have to be added to OffsetTypeInfo for it to be useful, and I have a feeling that would increase the binary size by quite a bit.. So I'll have to test it and see.You may want to look at "Orange". https://github.com/jacob-carlborg/orange The API is clean, and non-invasive. The only downside for me, is that in order to serialize a class by a base pointer, you have to register it at compile time.Yeah, this will require runtime reflection to avoid the registering of the type.
Apr 14 2015
On 2015-04-14 18:33, bitwise wrote:This is one of the main reasons I like the OffsetTypeInfo/offTi() option. I think a "name" field would have to be added to OffsetTypeInfo for it to be useful, and I have a feeling that would increase the binary size by quite a bit.. So I'll have to test it and see.It needs to be possible to set and get a value of an instance variable based on it's name, through runtime reflection. It also needs to bypass protection, i.e. "private". -- /Jacob Carlborg
Apr 15 2015
On Wed, 15 Apr 2015 05:31:24 -0400, Jacob Carlborg <doob me.com> wrote:It needs to be possible to set and get a value of an instance variable based on it's name, through runtime reflection. It also needs to bypass protection, i.e. "private".Right now, this is the def: /** * Array of pairs giving the offset and type information for each * member in an aggregate. */ struct OffsetTypeInfo { size_t offset; /// Offset of member from start of object TypeInfo ti; /// TypeInfo for this member } If "string name" esd added, and then offTi[] esd actually populated, then I suppose you could do this: class Test { int a = 4; private int b = 5; void print(){ writeln(b); } } void main() { Test test = new Test; // offsetof would instead come from the TypeInfo/OffsetTypeInfo int* b = cast(int*)(cast(void*)test + Test.b.offsetof); *b = 1234; test.print(); } But AFAIK, this is NOT ok in C++ because of the way inheritance works.. is this safe in D?
Apr 15 2015
On Wed, 15 Apr 2015 11:26:48 -0400, bitwise <bitwise.pvt gmail.com> wrote:But AFAIK, this is NOT ok in C++ because of the way inheritance works.. is this safe in D?To clarify, I'm talking about doing doing things by Object pointer or void pointer, which seems to work fine: class Test { int a = 4; private int b = 5; void print(){ writeln(b); } } struct Test2 { int a = 4; private int b = 5; void print(){ writeln(b); } } void main() { Test test = new Test; Object obj = test; int* b = cast(int*)(cast(void*)obj + Test.b.offsetof); *b = 1234; test.print(); Test2 test2 = Test2(); void *obj = &test2; int* b = cast(int*)(obj + Test2.b.offsetof); *b = 1234; test2.print(); }
Apr 15 2015
On 2015-04-15 17:26, bitwise wrote:Right now, this is the def: /** * Array of pairs giving the offset and type information for each * member in an aggregate. */ struct OffsetTypeInfo { size_t offset; /// Offset of member from start of object TypeInfo ti; /// TypeInfo for this member } If "string name" esd added, and then offTi[] esd actually populated, then I suppose you could do this: class Test { int a = 4; private int b = 5; void print(){ writeln(b); } } void main() { Test test = new Test; // offsetof would instead come from the TypeInfo/OffsetTypeInfo int* b = cast(int*)(cast(void*)test + Test.b.offsetof); *b = 1234; test.print(); } But AFAIK, this is NOT ok in C++ because of the way inheritance works.. is this safe in D?I'm not sure, I would assume so. Why is this not safe in C++? -- /Jacob Carlborg
Apr 15 2015
On Wed, 15 Apr 2015 15:48:28 -0400, Jacob Carlborg <doob me.com> wrote:On 2015-04-15 17:26, bitwise wrote:One reason is that casting with multiple inheritance offsets the pointer, and I forget exactly virtual inheritance works, but I'm sure it breaks things too.. #include <iostream> using namespace std; class A { public: int a; }; class B { public: int b; }; class C { public: int c; }; class D : public A, public B, public C { public: int d; }; int main(int argc, const char * argv[]) { D *d = new D; cout << (intptr_t)d << endl; cout << (intptr_t)(C*)d << endl; cout << (intptr_t)(B*)d << endl; cout << (intptr_t)(A*)d << endl; return 0; } possible output: 1098416 1098424 1098420 1098416Right now, this is the def: /** * Array of pairs giving the offset and type information for each * member in an aggregate. */ struct OffsetTypeInfo { size_t offset; /// Offset of member from start of object TypeInfo ti; /// TypeInfo for this member } If "string name" esd added, and then offTi[] esd actually populated, then I suppose you could do this: class Test { int a = 4; private int b = 5; void print(){ writeln(b); } } void main() { Test test = new Test; // offsetof would instead come from the TypeInfo/OffsetTypeInfo int* b = cast(int*)(cast(void*)test + Test.b.offsetof); *b = 1234; test.print(); } But AFAIK, this is NOT ok in C++ because of the way inheritance works.. is this safe in D?I'm not sure, I would assume so. Why is this not safe in C++?
Apr 15 2015
On 2015-04-16 01:32, bitwise wrote:One reason is that casting with multiple inheritance offsets the pointer, and I forget exactly virtual inheritance works, but I'm sure it breaks things too..In D I would assume this would eventually be possible: class Foo { int a; } class Bar : Foo { int b; } Foo f = new Bar; auto offset = typeid(f).getMemeber("b").offset; auto ptr = cast(int*)(cast(void*)f + offset); *ptr = 4; -- /Jacob Carlborg
Apr 15 2015
On Thu, 16 Apr 2015 02:22:25 -0400, Jacob Carlborg <doob me.com> wrote:In D I would assume this would eventually be possible: class Foo { int a; } class Bar : Foo { int b; } Foo f = new Bar; auto offset = typeid(f).getMemeber("b").offset; auto ptr = cast(int*)(cast(void*)f + offset); *ptr = 4;Ok, that sounds right. D has no multiple or virtual inheritance, so I interfaces) which is likely designed to avoid these kinds of issues. I would be modifying the offTi() property though, not getMembers(). getMembers() actually works right now, although it's kinda useless. Basically, if you put any function named "getMembers" at module scope, the address of that function can be retrieved using ModuleInfo.xgetMembers(). /////////////// module mainMod; template isField(alias T) { enum hasInit = is(typeof(typeof(T).init)); enum isManifestConst = __traits(compiles, { enum e = T; }); enum isField = hasInit && !isManifestConst; } MemberInfo[] getMembers() { mixin("alias " ~ .stringof[7..$] ~ " mod;"); MemberInfo[] members; foreach(memberName; __traits(allMembers, mod)) { alias Identity!(__traits(getMember, mod, memberName)) member; static if(isField!member) members ~= new MemberInfo_field(memberName, typeid(member), 0); } return members; } int a = 1; int b = 2; int c = 3; void main() { foreach(mi; ModuleInfo) { if(mi.name == "mainModule") { alias MemberInfo[] function() getMembers_type; getMembers_type getMembers = cast(getMembers_type)mi.xgetMembers(); foreach(m; getMembers()) writeln(m.name); } } } Output: a b c //////////////// offTi() however, even though it's documented, returns an empty array. I'm not sure what the motivation originally was for having an array of TypeInfo and offsets without the field names, but the dead code predates D2. I suppose you could do binary serialization... Anyways, I've put the reflection library to rest for now. Hopefully I'll have some time in the next little while to try and get offTi() working.
Apr 16 2015
On 2015-04-17 01:25, bitwise wrote:Ok, that sounds right. D has no multiple or virtual inheritance, so I interfaces) which is likely designed to avoid these kinds of issues. I would be modifying the offTi() property though, not getMembers(). getMembers() actually works right now, although it's kinda useless. Basically, if you put any function named "getMembers" at module scope, the address of that function can be retrieved using ModuleInfo.xgetMembers().We need a "getMembers" on ClassInfo. Because "typeid" will resolve the dynamic type. The problem is this code: class Bar {} class Foo : Bar {} void main () { Bar a = new Foo; writeln(typeid(a)); } Compile time reflection cannot be used to get the members in this case. -- /Jacob Carlborg
Apr 17 2015
On Fri, 17 Apr 2015 05:30:04 -0400, Jacob Carlborg <doob me.com> wrote:On 2015-04-17 01:25, bitwise wrote:Sorry, that's what I meant(using this on classes, not modules). xgetMembers/getMembers seems to be designed so that the caller can write their own getMembers() function, whereas offTi() is designed to return an array that was automatically generated. offTi() is actually defined in TypeInfo: https://github.com/D-Programming-Language/druntime/blob/cfcf7480b2faea0af9ab6ddba8e3b0d9f05c4415/src/object_.d#L301 and overridden for TypeInfo_Class: https://github.com/D-Programming-Language/druntime/blob/cfcf7480b2faea0af9ab6ddba8e3b0d9f05c4415/src/object_.d#L831 but TypeInfo_Class.m_offTi is never populated: https://github.com/D-Programming-Language/dmd/blob/master/src/toobj.c#L436 So while this code technically works, there will be no output: class Test { int a = 4; int b = 5; int c = 6; } void main() { foreach(const(OffsetTypeInfo) off; typeid(Test).offTi()) writeln(off.offset); }Ok, that sounds right. D has no multiple or virtual inheritance, so I interfaces) which is likely designed to avoid these kinds of issues. I would be modifying the offTi() property though, not getMembers(). getMembers() actually works right now, although it's kinda useless. Basically, if you put any function named "getMembers" at module scope, the address of that function can be retrieved using ModuleInfo.xgetMembers().We need a "getMembers" on ClassInfo. Because "typeid" will resolve the dynamic type. The problem is this code: class Bar {} class Foo : Bar {} void main () { Bar a = new Foo; writeln(typeid(a)); } Compile time reflection cannot be used to get the members in this case.
Apr 17 2015
One more thing. I had to add a virtual destructor to A to get this to work: class A { public: int a; virtual ~A(){} }; // .... A *d2 = (A*)d; cout << boolalpha << (typeid(*d2) == typeid(D)) << endl; C *d3 = (C*)d; cout << boolalpha << (typeid(*d3) == typeid(D)) << endl; Output is now: true false Without virtual destructor: false false So you can't rely on serialization working properly using only TypeInfo/offsets
Apr 15 2015
On Monday, 30 March 2015 at 01:11:55 UTC, bitwise wrote:I came across this post a while back and decided to implement it: http://forum.dlang.org/thread/juf7sk$16rl$1 digitalmars.com My implementation: https://github.com/bitwise-github/D-Reflection The above conversation seemed to stop abruptly, so I went on to assume that no one chose to champion the task. At the time, I looked around for other conversations or attempts at runtime reflection for D, but couldn't really find anything. I did find the ModuleInfo/reflection stuff in object.d, but it seemed like an effort that may have been abandoned. Also, the above conversation seemed to suggest it should be opt-in, which also made me wonder if the stuff in object.d was abandoned or for a different purpose. Looking again today, someone seems to have been working on it a bit.. For example, MemberInfo_field and MemberInfo_function were empty last time I checked. So what's the current state of things? Is anybody working on it? Although MemberInfo_field exists, it doesn't seem to be available from TypeInfo_Class... Is that the eventual goal? Is there anything I can do to help get things moving? Any comments on my implementation would be welcome as well. (https://github.com/bitwise-github/D-Reflection) main.d shows some general use cases, but basically, if the reflect(T) template is used on a class, that class, and any child types will be reflected. Recursive reflection only propagates downward, or else it could leak sideways and unnecessarily reflect several modules. Most of the reflection information is available at compile time. For example: enum name = reflectModule!(test).findClass("Test").findField("variable").name; pragma(msg, name); // "variable" will be outputted. To make a module available for runtime reflection, the following can be used: mixin(runtimeReflection!test); At this point, the above example can be rewritten as: string name = getModuleReflection("tests.test").findClass("Test3").findField("static_variable").name; writeln(name);This is needed for sure. For example do you count how many subjects on .learn are about __traits(allMembers,... or __traits(getMember,... Run-time serialization is an example. About the design i would rather see this as a user choice, eg the reflexion as a mixin template that you can mix in a class or a struct you want to be filled with info. This remark is related to the size problem pointed in the previous posts.
Mar 31 2015
On Monday, 30 March 2015 at 01:11:55 UTC, bitwise wrote:I came across this post a while back and decided to implement it: http://forum.dlang.org/thread/juf7sk$16rl$1 digitalmars.com My implementation: https://github.com/bitwise-github/D-Reflection The above conversation seemed to stop abruptly, so I went on to assume that no one chose to champion the task. At the time, I looked around for other conversations or attempts at runtime reflection for D, but couldn't really find anything. I did find the ModuleInfo/reflection stuff in object.d, but it seemed like an effort that may have been abandoned. Also, the above conversation seemed to suggest it should be opt-in, which also made me wonder if the stuff in object.d was abandoned or for a different purpose. Looking again today, someone seems to have been working on it a bit.. For example, MemberInfo_field and MemberInfo_function were empty last time I checked. So what's the current state of things? Is anybody working on it? Although MemberInfo_field exists, it doesn't seem to be available from TypeInfo_Class... Is that the eventual goal? Is there anything I can do to help get things moving? Any comments on my implementation would be welcome as well. (https://github.com/bitwise-github/D-Reflection) main.d shows some general use cases, but basically, if the reflect(T) template is used on a class, that class, and any child types will be reflected. Recursive reflection only propagates downward, or else it could leak sideways and unnecessarily reflect several modules. Most of the reflection information is available at compile time. For example: enum name = reflectModule!(test).findClass("Test").findField("variable").name; pragma(msg, name); // "variable" will be outputted. To make a module available for runtime reflection, the following can be used: mixin(runtimeReflection!test); At this point, the above example can be rewritten as: string name = getModuleReflection("tests.test").findClass("Test3").findField("static_variable").name; writeln(name);Hi, your link is not working any more and I really need your implementation. Gabriel
Sep 27 2017
On Wednesday, 27 September 2017 at 20:38:51 UTC, Gheorghe Gabriel wrote:On Monday, 30 March 2015 at 01:11:55 UTC, bitwise wrote:I took the code down because there were design flaws I had to work out. I've reworked the code significantly at this point. I can throw a copy up on github in a day or two if you need it. There are still some issues to overcome though, which may or may not be a problem depending on what you're doing.[...]Hi, your link is not working any more and I really need your implementation. Gabriel
Sep 27 2017
On Thursday, 28 September 2017 at 01:33:35 UTC, bitwise wrote:On Wednesday, 27 September 2017 at 20:38:51 UTC, Gheorghe Gabriel wrote:Thank you, I really need it for my project. I want to load scripts in a 3D scene without compiling an restarting the whole engine. example: module game1.player; class Player : Actor { this(string name) { super(name) } ~this() {} void jump() { ... } Editable("Slider")("min = 0; max = 5;") ushort lives = 5; } In this moment I want to load this script and create a scene object. I should load Player class, then all its public methods, then its public members and check theire attributes.On Monday, 30 March 2015 at 01:11:55 UTC, bitwise wrote:I took the code down because there were design flaws I had to work out. I've reworked the code significantly at this point. I can throw a copy up on github in a day or two if you need it. There are still some issues to overcome though, which may or may not be a problem depending on what you're doing.[...]Hi, your link is not working any more and I really need your implementation. Gabriel
Sep 27 2017