www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Time for std.reflection

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Walter and I discussed the idea below a long time (years) ago. Most 
likely it's also been discussed in this newsgroup a couple of times. 
Given the state of the compiler back then, back then it seemed like a 
super cool idea that's entirely realizable, it would just take time for 
the compiler to become as capable as needed. Nowadays we're in shape to 
tackle it.

Here "it" is.

Back when runtime reflection was being discussed, my response was "let's 
focus on compile-time reflection, and then we can do run-time reflection 
on demand as a library". Though this might sound sensible, I initially 
didn't have a design. Now here's what we can do.

Currently we have information about symbols as __traits(...) intrinsics 
wrapped in nice but scattered ways. Now that CTFE is good enough to 
manipulate structs and arrays thereof, we have the possibility to 
finally approach things in a nicely unified, structured way.

First, we need to prime std.reflection with a few abstractions that 
characterize entities in a D program.

class ModuleInfo {
 property:
     string name();
     ImportInfo[] imports();
     DataInfo[] data();
     FunctionInfo[] functions();
     ClassInfo[] classes();
     StructInfo[] structs(); // includes unions
     TemplateInfo[] templates();
     EnumInfo[] enums();
     bool hasStaticCtor(), hasStaticDtor(),
       hasSharedCtor(), hasSharedDtor();
}

Probably there are a few more pieces of data, but you get the idea. Then 
for each of the entities mentioned above we have a similar definition. 
For example:

enum Protection { isPublic, isPackage, isProtected, isPrivate }

class ClassInfo {
 property:
     string name();
     string baseName();
     string parentName(); // if applicable, null otherwise
     string[] interfaces();
     bool isShared();
     Protection protection();
     DataMemberInfo[] data();
     MethodInfo[] methods();
     Object defaultConstructor();
     ...
}

Then for an e.g. method declaration we'd have:

class MethodInfo {
 property:
     string name();
     bool isStatic(), isFinal(), isOverride();
     Protection protection();
     string[] parameterTypes();
     string[] parameterNames();
}

Some details may vary, e.g. some may be straight members instead of 
properties etc. (I used properties to allude to use of lazy gathering of 
information).

So so far we have a nice collection of structured data associated with 
the entities in a D program. Note how this structuring differs yet has 
similar power to the primitives in std.traits; std.traits offers 
unstructured bits of information on demand (e.g. ParameterTypeNames) 
etc. but the objects above group information together per entity 
declared. All of the above goes in std.reflection, of course.

===========

On to primitives that return such data.

Given that D can (since relatively recently) create and manipulate class 
objects during compilation too, it follows that the classes above can be 
accessed in two ways - through compile-time API and run-time API. When 
possible, the APIs may even use the same functions; some other times 
they will be necessarily different.

There are two possible approaches to discovering such information. One 
is by fetching the ModuleInfo for the whole module and navigating it. 
Another one is by using search primitives from strings.

So we should have e.g.

// inside std.reflection
ModuleInfo getModuleInfo(string moduleName);

so a CT call would go like:

// client code
static info = getModuleInfo("std.algorithm");

whereas a run-time call would be:

// client code
auto info = getModuleInfo("std.algorithm");

In the latter case, the module needs to save all needed information for 
ri, so it should plant this:

// inside std.algorithm
mixin(makeModuleInfoAvailableDynamically());

The mixin would generate all information needed and would store it for 
later dynamic use.

A search API would go like e.g.

ClassInfo getClassInfo(string className);

In this case the class name could be qualified with module information etc.

===========

With this design we unify compile-time and run-time type manipulation in 
simple ways, by defining structured information about declarations that 
can be queried during compilation or dynamically.

Please chime in with thoughts. Would someone want to pioneer this project?


Andrei
Jul 21 2012
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Saturday, July 21, 2012 17:44:51 Andrei Alexandrescu wrote:

 With this design we unify compile-time and run-time type manipulation in
 simple ways, by defining structured information about declarations that
 can be queried during compilation or dynamically.
 
 Please chime in with thoughts.


Are you proposing that we replace std.traits or that std.reflection be built on 
top of it?

Having std.reflection will be awesome for a lot of stuff - particularly the
more 
complicated stuff - and would be a great addition to Phobos, but it seems like 
overkill for a lot of basic template constraints, so I wouldn't want to see it 
replace std.traits.

- Jonathan M Davis
Jul 21 2012
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/21/12 5:51 PM, Jonathan M Davis wrote:

 On Saturday, July 21, 2012 17:44:51 Andrei Alexandrescu wrote:

 With this design we unify compile-time and run-time type manipulation in
 simple ways, by defining structured information about declarations that
 can be queried during compilation or dynamically.

 Please chime in with thoughts.


 Are you proposing that we replace std.traits or that std.reflection be built on
 top of it?


On top. std.reflection would be a complete reflection solution, whereas 
std.traits gives casual type traits as needed.


 Having std.reflection will be awesome for a lot of stuff - particularly the
more
 complicated stuff - and would be a great addition to Phobos, but it seems like
 overkill for a lot of basic template constraints, so I wouldn't want to see it
 replace std.traits.


My thoughts exactly.


Andrei
Jul 21 2012
prev sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Sat, 21 Jul 2012 14:51:30 -0700, Jonathan M Davis <jmdavisProg gmx.com>  
wrote:


 On Saturday, July 21, 2012 17:44:51 Andrei Alexandrescu wrote:

 With this design we unify compile-time and run-time type manipulation in
 simple ways, by defining structured information about declarations that
 can be queried during compilation or dynamically.

 Please chime in with thoughts.


 Are you proposing that we replace std.traits or that std.reflection be  
 built on
 top of it?

 Having std.reflection will be awesome for a lot of stuff - particularly  
 the more
 complicated stuff - and would be a great addition to Phobos, but it  
 seems like
 overkill for a lot of basic template constraints, so I wouldn't want to  
 see it
 replace std.traits.

 - Jonathan M Davis


I too would want it in a different module But my read of Andrei's proposal  
is that it would be.

-- 
Adam Wilson
IRC: LightBender
Project Coordinator
The Horizon Project
http://www.thehorizonproject.org/
Jul 21 2012
prev sibling next sibling parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2012-07-21 21:44:51 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:


 Please chime in with thoughts.


Unifying runtime and compile-time reflection is a good idea, in theory 
at least. At least it'd make compile-time reflection easier. But when I 
think about the use case, unification the way you're proposing doesn't 
seem that great.

If you need runtime reflection in one class, you probably need it on 
all the derived classes too. That's the real problem runtime reflection 
solves: it allows you to know something about the subclasses you might 
otherwise not know about (because your code only know the base class).

So to make runtime reflection not a hassle, the thing that generates 
the runtime reflection info should be inherited somehow.

The other thing is: you very rarely need runtime reflection of 
everything. Especially, why would you ever need runtime reflection for 
a struct, where all the reflected info is available at compile time? 
And why create info for all fields and methods in a class in a case 
where all you need to get at is a pair of serialize/unserialize 
functions?


-- 
Michel Fortin
michel.fortin michelf.ca
http://michelf.ca/
Jul 21 2012
prev sibling next sibling parent reply "Kapps" <opantm2+spam gmail.com> writes:
I agree with most things proposed, however I am not a fan of the 
idea of mixing in runtime reflection info. Many times, you want 
reflection info from a type that is not your own, and thus I 
believe reflection should be generated by specifying a type. More 
importantly, a method should exist for recursively generating 
reflection info.

Also, I'd like to see a hierarchal approach to reflection data. 
The main advantage to std.reflection would be being able to use 
it at run-time, at which point we can't simply rely on templates, 
and instead if we want to store something we must rely on a base 
type. I think the best approach would be a hierarchal system 
where all reflection data derives from MemberInfo. My ideal API 
would like something like: 
https://workflowy.com/shared/62d4f791-e397-a86d-c018-09eab98b9927/
Jul 21 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/21/12 8:16 PM, Kapps wrote:

 I agree with most things proposed, however I am not a fan of the idea of
 mixing in runtime reflection info. Many times, you want reflection info
 from a type that is not your own, and thus I believe reflection should
 be generated by specifying a type. More importantly, a method should
 exist for recursively generating reflection info.


I confess I have trouble understanding each of the sentences above.


 Also, I'd like to see a hierarchal approach to reflection data. The main
 advantage to std.reflection would be being able to use it at run-time,
 at which point we can't simply rely on templates, and instead if we want
 to store something we must rely on a base type.


At no place in the proposed approach is the use of templates required or 
needed.


 I think the best
 approach would be a hierarchal system where all reflection data derives
 from MemberInfo. My ideal API would like something like:
 https://workflowy.com/shared/62d4f791-e397-a86d-c018-09eab98b9927/


I cringed at

MemberType -> { Module, Type, Field, Method, Parameter }

I was hoping to get away from slapping tagging on types just for the 
sake of using inheritance.


Andrei
Jul 21 2012
parent reply "Kapps" <opantm2+spam gmail.com> writes:
On Sunday, 22 July 2012 at 00:21:31 UTC, Andrei Alexandrescu 
wrote:

 On 7/21/12 8:16 PM, Kapps wrote:

 I agree with most things proposed, however I am not a fan of 
 the idea of
 mixing in runtime reflection info. Many times, you want 
 reflection info
 from a type that is not your own, and thus I believe 
 reflection should
 be generated by specifying a type. More importantly, a method 
 should
 exist for recursively generating reflection info.


 I confess I have trouble understanding each of the sentences 
 above.


What I meant was that when something requires reflection, it 
generally requires that everything it contains has reflection 
information as well. If using a mixin directly in the module 
itself, when a module does not include this mixin it will never 
get reflection information. This may or may not be desired. 
However it completely prevents many common uses of reflection, 
including serialization. So long as there is any dependency on 
any type or method that does not have reflection info, the entire 
operation would fail. This means that people will either start 
adding reflection to modules that do not need it in fear that 
someone may try to use a member of their module, or too sparsely 
thus making reflection all but unuseable for many common 
situations.


 Also, I'd like to see a hierarchal approach to reflection 
 data. The main
 advantage to std.reflection would be being able to use it at 
 run-time,
 at which point we can't simply rely on templates, and instead 
 if we want
 to store something we must rely on a base type.


 At no place in the proposed approach is the use of templates 
 required or needed.


What I mean by this is that people rely on the ability to use 
templates as a replacement to inheritance or common interfaces. 
For example, ranges rely on methods existing rather than using an 
interface for a range. With reflection, this isn't ideal. For 
example, if you wanted to get all the fields and properties in a 
module, including nested ones, you'd currently have to get all 
fields, in a module, add those, then get all types in a module, 
then all types in that type, then get all fields from those, etc. 
With a hierarchal approach where MemberInfo has a Children 
property, you'd simply get all children that are a FieldInfo or 
PropertyInfo, and recurse for all returned values that have 
children.


 I think the best
 approach would be a hierarchal system where all reflection 
 data derives
 from MemberInfo. My ideal API would like something like:
 https://workflowy.com/shared/62d4f791-e397-a86d-c018-09eab98b9927/


 I cringed at

 MemberType -> { Module, Type, Field, Method, Parameter }

 I was hoping to get away from slapping tagging on types just 
 for the sake of using inheritance.


Admittedly, the MemberType wouldn't be necessary and I don't see 
as having a huge benefit. In almost all cases you know what you 
want to operate on, inheritance simply makes the API cleaner and 
gives access to common methods more easily.
Jul 21 2012
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Sunday, July 22, 2012 05:19:39 Kapps wrote:

 What I meant was that when something requires reflection, it
 generally requires that everything it contains has reflection
 information as well. If using a mixin directly in the module
 itself, when a module does not include this mixin it will never
 get reflection information. This may or may not be desired.
 However it completely prevents many common uses of reflection,
 including serialization. So long as there is any dependency on
 any type or method that does not have reflection info, the entire
 operation would fail. This means that people will either start
 adding reflection to modules that do not need it in fear that
 someone may try to use a member of their module, or too sparsely
 thus making reflection all but unuseable for many common
 situations.


Runtime reflection _must_ be opt-in. We do _not_ want to make all types pay the 
cost of having it. That means that however it's done is going to require that 
every type that has it be marked in one way or another to enable that 
functionality. That's part of the cost of being a systems language. Whatever 
solution we come up with must take that into account.

- Jonathan M Davis
Jul 21 2012
next sibling parent "Kapps" <opantm2+spam gmail.com> writes:
On Sunday, 22 July 2012 at 03:47:16 UTC, Jonathan M Davis wrote:

 Runtime reflection _must_ be opt-in. We do _not_ want to make 
 all types pay the
 cost of having it. That means that however it's done is going 
 to require that
 every type that has it be marked in one way or another to 
 enable that
 functionality. That's part of the cost of being a systems 
 language. Whatever
 solution we come up with must take that into account.

 - Jonathan M Davis


But the whole point is that not every type has it; only those 
that are registered. It just happens that you can register types 
besides yourself, which is an absolute necessity for many common 
purposes of reflection. For serialization for example, the 
serializer / deserializer should create reflection information 
for all types contained in a class.
Jul 21 2012
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On 22/07/2012 05:46, Jonathan M Davis wrote:

 Runtime reflection _must_ be opt-in. We do _not_ want to make all types pay the
 cost of having it. That means that however it's done is going to require that
 every type that has it be marked in one way or another to enable that
 functionality.


I don't think both are that related here.

Yes, we are a system language, so this have to be opt-in. But no, this 
doesn't means that every type have to be marked to be reflected.

I'd expect from std.reflection that it is able to reflect recursively 
from the marked starting point.
Jul 22 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-07-22 14:04, deadalnix wrote:


 I'd expect from std.reflection that it is able to reflect recursively
 from the marked starting point.


I really hope so.

-- 
/Jacob Carlborg
Jul 22 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-07-22 02:16, Kapps wrote:

 I agree with most things proposed, however I am not a fan of the idea of
 mixing in runtime reflection info. Many times, you want reflection info
 from a type that is not your own, and thus I believe reflection should
 be generated by specifying a type. More importantly, a method should
 exist for recursively generating reflection info.


I agree.

-- 
/Jacob Carlborg
Jul 22 2012
prev sibling next sibling parent reply =?ISO-8859-1?Q?Alex_R=F8nne_Petersen?= <alex lycus.org> writes:
On 21-07-2012 23:44, Andrei Alexandrescu wrote:

 Walter and I discussed the idea below a long time (years) ago. Most
 likely it's also been discussed in this newsgroup a couple of times.
 Given the state of the compiler back then, back then it seemed like a
 super cool idea that's entirely realizable, it would just take time for
 the compiler to become as capable as needed. Nowadays we're in shape to
 tackle it.

 Here "it" is.

 Back when runtime reflection was being discussed, my response was "let's
 focus on compile-time reflection, and then we can do run-time reflection
 on demand as a library". Though this might sound sensible, I initially
 didn't have a design. Now here's what we can do.

 Currently we have information about symbols as __traits(...) intrinsics
 wrapped in nice but scattered ways. Now that CTFE is good enough to
 manipulate structs and arrays thereof, we have the possibility to
 finally approach things in a nicely unified, structured way.

 First, we need to prime std.reflection with a few abstractions that
 characterize entities in a D program.

 class ModuleInfo {
  property:
      string name();
      ImportInfo[] imports();
      DataInfo[] data();
      FunctionInfo[] functions();
      ClassInfo[] classes();
      StructInfo[] structs(); // includes unions
      TemplateInfo[] templates();
      EnumInfo[] enums();
      bool hasStaticCtor(), hasStaticDtor(),
        hasSharedCtor(), hasSharedDtor();
 }

 Probably there are a few more pieces of data, but you get the idea. Then
 for each of the entities mentioned above we have a similar definition.
 For example:

 enum Protection { isPublic, isPackage, isProtected, isPrivate }

 class ClassInfo {
  property:
      string name();
      string baseName();
      string parentName(); // if applicable, null otherwise
      string[] interfaces();
      bool isShared();
      Protection protection();
      DataMemberInfo[] data();
      MethodInfo[] methods();
      Object defaultConstructor();
      ...
 }

 Then for an e.g. method declaration we'd have:

 class MethodInfo {
  property:
      string name();
      bool isStatic(), isFinal(), isOverride();
      Protection protection();
      string[] parameterTypes();
      string[] parameterNames();
 }

 Some details may vary, e.g. some may be straight members instead of
 properties etc. (I used properties to allude to use of lazy gathering of
 information).

 So so far we have a nice collection of structured data associated with
 the entities in a D program. Note how this structuring differs yet has
 similar power to the primitives in std.traits; std.traits offers
 unstructured bits of information on demand (e.g. ParameterTypeNames)
 etc. but the objects above group information together per entity
 declared. All of the above goes in std.reflection, of course.

 ===========

 On to primitives that return such data.

 Given that D can (since relatively recently) create and manipulate class
 objects during compilation too, it follows that the classes above can be
 accessed in two ways - through compile-time API and run-time API. When
 possible, the APIs may even use the same functions; some other times
 they will be necessarily different.

 There are two possible approaches to discovering such information. One
 is by fetching the ModuleInfo for the whole module and navigating it.
 Another one is by using search primitives from strings.

 So we should have e.g.

 // inside std.reflection
 ModuleInfo getModuleInfo(string moduleName);

 so a CT call would go like:

 // client code
 static info = getModuleInfo("std.algorithm");

 whereas a run-time call would be:

 // client code
 auto info = getModuleInfo("std.algorithm");

 In the latter case, the module needs to save all needed information for
 ri, so it should plant this:

 // inside std.algorithm
 mixin(makeModuleInfoAvailableDynamically());

 The mixin would generate all information needed and would store it for
 later dynamic use.

 A search API would go like e.g.

 ClassInfo getClassInfo(string className);

 In this case the class name could be qualified with module information etc.

 ===========

 With this design we unify compile-time and run-time type manipulation in
 simple ways, by defining structured information about declarations that
 can be queried during compilation or dynamically.

 Please chime in with thoughts. Would someone want to pioneer this project?


 Andrei


I'm just curious about one thing: How do you plan to reify templates 
(they are Turing complete after all)?

-- 
Alex Rønne Petersen
alex lycus.org
http://lycus.org
Jul 21 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/21/12 8:28 PM, Alex Rønne Petersen wrote:

 I'm just curious about one thing: How do you plan to reify templates
 (they are Turing complete after all)?


I don't know.

Andrei

P.S. Please don't overquote.
Jul 21 2012
prev sibling next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On 21/07/2012 23:44, Andrei Alexandrescu wrote:

 Walter and I discussed the idea below a long time (years) ago. Most
 likely it's also been discussed in this newsgroup a couple of times.
 Given the state of the compiler back then, back then it seemed like a
 super cool idea that's entirely realizable, it would just take time for
 the compiler to become as capable as needed. Nowadays we're in shape to
 tackle it.

 Here "it" is.

 Back when runtime reflection was being discussed, my response was "let's
 focus on compile-time reflection, and then we can do run-time reflection
 on demand as a library". Though this might sound sensible, I initially
 didn't have a design. Now here's what we can do.

 Currently we have information about symbols as __traits(...) intrinsics
 wrapped in nice but scattered ways. Now that CTFE is good enough to
 manipulate structs and arrays thereof, we have the possibility to
 finally approach things in a nicely unified, structured way.

 First, we need to prime std.reflection with a few abstractions that
 characterize entities in a D program.

 class ModuleInfo {
  property:
 string name();
 ImportInfo[] imports();
 DataInfo[] data();
 FunctionInfo[] functions();
 ClassInfo[] classes();
 StructInfo[] structs(); // includes unions
 TemplateInfo[] templates();
 EnumInfo[] enums();
 bool hasStaticCtor(), hasStaticDtor(),
 hasSharedCtor(), hasSharedDtor();
 }

 Probably there are a few more pieces of data, but you get the idea. Then
 for each of the entities mentioned above we have a similar definition.
 For example:

 enum Protection { isPublic, isPackage, isProtected, isPrivate }

 class ClassInfo {
  property:
 string name();
 string baseName();
 string parentName(); // if applicable, null otherwise
 string[] interfaces();
 bool isShared();
 Protection protection();
 DataMemberInfo[] data();
 MethodInfo[] methods();
 Object defaultConstructor();
 ...
 }

 Then for an e.g. method declaration we'd have:

 class MethodInfo {
  property:
 string name();
 bool isStatic(), isFinal(), isOverride();
 Protection protection();
 string[] parameterTypes();
 string[] parameterNames();
 }

 Some details may vary, e.g. some may be straight members instead of
 properties etc. (I used properties to allude to use of lazy gathering of
 information).

 So so far we have a nice collection of structured data associated with
 the entities in a D program. Note how this structuring differs yet has
 similar power to the primitives in std.traits; std.traits offers
 unstructured bits of information on demand (e.g. ParameterTypeNames)
 etc. but the objects above group information together per entity
 declared. All of the above goes in std.reflection, of course.

 ===========

 On to primitives that return such data.

 Given that D can (since relatively recently) create and manipulate class
 objects during compilation too, it follows that the classes above can be
 accessed in two ways - through compile-time API and run-time API. When
 possible, the APIs may even use the same functions; some other times
 they will be necessarily different.

 There are two possible approaches to discovering such information. One
 is by fetching the ModuleInfo for the whole module and navigating it.
 Another one is by using search primitives from strings.

 So we should have e.g.

 // inside std.reflection
 ModuleInfo getModuleInfo(string moduleName);

 so a CT call would go like:

 // client code
 static info = getModuleInfo("std.algorithm");

 whereas a run-time call would be:

 // client code
 auto info = getModuleInfo("std.algorithm");

 In the latter case, the module needs to save all needed information for
 ri, so it should plant this:

 // inside std.algorithm
 mixin(makeModuleInfoAvailableDynamically());

 The mixin would generate all information needed and would store it for
 later dynamic use.

 A search API would go like e.g.

 ClassInfo getClassInfo(string className);

 In this case the class name could be qualified with module information etc.

 ===========

 With this design we unify compile-time and run-time type manipulation in
 simple ways, by defining structured information about declarations that
 can be queried during compilation or dynamically.

 Please chime in with thoughts. Would someone want to pioneer this project?


 Andrei


I don't understand. Are theses structures generated by the compiler or 
some kind of libraries at compile time on a per needed basis ?
Jul 21 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/21/12 8:41 PM, deadalnix wrote:

 I don't understand. Are theses structures generated by the compiler or
 some kind of libraries at compile time on a per needed basis ?


These are all library structures. Internally they use std.traits, which 
in turns uses compiler magic with __traits.

Andrei

P.S. What this thing wit quoting a long message to make a 1-line point? 
Is that a thing?
Jul 21 2012
next sibling parent deadalnix <deadalnix gmail.com> writes:
On 22/07/2012 06:48, Andrei Alexandrescu wrote:

 On 7/21/12 8:41 PM, deadalnix wrote:

 I don't understand. Are theses structures generated by the compiler or
 some kind of libraries at compile time on a per needed basis ?


 These are all library structures. Internally they use std.traits, which
 in turns uses compiler magic with __traits.

 Andrei


Sounds good, but what about derived classes ?


 P.S. What this thing wit quoting a long message to make a 1-line point?
 Is that a thing?


It is what happen when you editor collapse the text :D
Jul 22 2012
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-07-22 06:48, Andrei Alexandrescu wrote:


 P.S. What this thing wit quoting a long message to make a 1-line point?
 Is that a thing?


It's the new hip thing :)

-- 
/Jacob Carlborg
Jul 22 2012
prev sibling parent reply "sclytrack" <sclytrack thailand.com> writes:
 In the latter case, the module needs to save all needed 
 information for
 ri, so it should plant this:

 // inside std.algorithm
 mixin(makeModuleInfoAvailableDynamically());




Is there a way to get the current module at compile time?

__traits(allMembers, pkg.modulename)
Nov 19 2012
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Monday, 19 November 2012 at 11:43:36 UTC, sclytrack wrote:

You could do something like this:
string currentModule(alias T = {})() { return __traits(parent, 
T).stringof; }
Nov 19 2012
prev sibling next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/21/12, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> wrote:

 class ModuleInfo {
  property:
      string name();
      ImportInfo[] imports();
      DataInfo[] data();
      FunctionInfo[] functions();
      ClassInfo[] classes();
      StructInfo[] structs(); // includes unions
      TemplateInfo[] templates();
      EnumInfo[] enums();
      bool hasStaticCtor(), hasStaticDtor(),
        hasSharedCtor(), hasSharedDtor();
 }


Are class/struct/function/etc templates going to be stored in the
templates field? Then you'd have to tag each template with a type,
e.g. "class template" vs "function template" to be able to filter them
out.

Otherwise classes/functions/etc could have an optional
"TemplateTypeInfo[] typeParams" field so you could filter out
templated from non-templated types by checking their typeParams field,
e.g.: auto tempClasses = filter!(a => !empty(a.typeParams)
)(modinfo.classes);

I use a similar structure to what you've defined for my code generator
and it worked out nicely for me.
Jul 21 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/21/12 8:46 PM, Andrej Mitrovic wrote:

 On 7/21/12, Andrei Alexandrescu<SeeWebsiteForEmail erdani.org>  wrote:

 class ModuleInfo {
  property:
       string name();
       ImportInfo[] imports();
       DataInfo[] data();
       FunctionInfo[] functions();
       ClassInfo[] classes();
       StructInfo[] structs(); // includes unions
       TemplateInfo[] templates();
       EnumInfo[] enums();
       bool hasStaticCtor(), hasStaticDtor(),
         hasSharedCtor(), hasSharedDtor();
 }


 Are class/struct/function/etc templates going to be stored in the
 templates field? Then you'd have to tag each template with a type,
 e.g. "class template" vs "function template" to be able to filter them
 out.


Perhaps we could accommodate parameterized types together with 
non-parameterized types by having e.g. additional properties that are 
null for non-parameterized types.


 Otherwise classes/functions/etc could have an optional
 "TemplateTypeInfo[] typeParams" field so you could filter out
 templated from non-templated types by checking their typeParams field,
 e.g.: auto tempClasses = filter!(a =>  !empty(a.typeParams)
 )(modinfo.classes);

 I use a similar structure to what you've defined for my code generator
 and it worked out nicely for me.


That sounds great!

Andrei
Jul 21 2012
prev sibling next sibling parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Sat, Jul 21, 2012 at 11:44 PM, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:

Nice!


 class ModuleInfo {
  property:
     string name();
     ImportInfo[] imports();
     DataInfo[] data();
     FunctionInfo[] functions();
     ClassInfo[] classes();
     StructInfo[] structs(); // includes unions
     TemplateInfo[] templates();
     EnumInfo[] enums();
     bool hasStaticCtor(), hasStaticDtor(),
       hasSharedCtor(), hasSharedDtor();
 }


A few questions:

1) About templates, would class/struct/function templates be in there?
As in:

class Foo(T,U) { ... }
=>
class TemplateInfo {
 property:
    string name();
    FunctionInfo[] functions();
    ClassInfo[] classes();
    TemplateInfo[] templates(); // Yes, templates in templates are
possible, and so on, recursively
...
}

2) Why classes, as opposed to structs? Would inheritance/hierarchies
play a role there? Or reference semantics?
Note that between structs, classes, templates and modules there is a
lot of redundancy. A possibility could be to have an AggregateInfo
base class.

3) For a class, a (to my eyes) standard request would be to get the
parents chain, up to Object. Could that be added in ClassInfo, or as
an external free function? (edit: scratch that, you're talking about
lazy gathering below).

4) How would that allows queries like "Here is class C, give me all
its available subclasses."? Hmm, wait, I get it: extract classes from
the module, and recursively from imported modules. From these classes,
extract the parent classes and so on, until the search ranged over the
whole inheritance tree. I guess inheritance info could be standard
enough for std.reflection to provide such a search.

5) The compiler can emit JSON output giving a partial view of the same
information. As a long-term goal, I suggest these infos should be
compatible somehow. The JSON output should be enriched, and we should
ascertain that using std.json to read this kind of
automatically-generated information should give std.reflection infos
back.

6) Is really all the necessary info available through std.traits and
__traits? Imported modules and their subtilities (renamed functions,
etc) seem out of reach, no?

7) I know your examples are not complete, but don't forget aliases and
symbols, and module-level values. Since these can be of any type, I'm
not sure how they are managed in your scheme. I mean, you cannot have
IntInfo[], DoubleInfo[], ...

8) (more a remark than a question) To me, it's another example of a
tree-like structure. I humbly suggest we get a std.tree somewhere, to
manipulate generic n-ary trees: mapping on trees, searching
(extracting elements or whole subtrees), folding trees, etc. That is,
a module that deals with trees/graphs not so much as containers, but
as a way to represent the relationships between elements. I'm not sure
I'm so clear here, but ideally std.json, std.xml, std.reflection and
probably a few others should use it as a low-level basis.
Btw, I have tree / trees algo (including depth-first / breadth-first
ranges) / graph / graph algo (strongly connected components, search in
graphs, ...) on github, but it's code that predate std.container and
is in limbo, pending std.allocator.


Philippe
Jul 22 2012
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/22/12 9:39 AM, Philippe Sigaud wrote:

 1) About templates, would class/struct/function templates be in there?
 As in:

 class Foo(T,U) { ... }
 =>
 class TemplateInfo {
  property:
      string name();
      FunctionInfo[] functions();
      ClassInfo[] classes();
      TemplateInfo[] templates(); // Yes, templates in templates are
 possible, and so on, recursively
 ....
 }


Yah, ideally all entities definable in a D module should be available 
via reflection. But I focused on things that e.g. the user of a 
dynamically-loaded library would be interested in: functions, classes.

std.reflection could become the lynchpin for dynamic library use; once 
the library is loaded (with dlopen or such), the client needs to call 
getModuleInfo() (a C function that can be found with dlsym()) and then 
get access to pointers to functions necessary for doing all other work. 
(Note that my examples don't yet include pointers to executable code yet.)


 2) Why classes, as opposed to structs? Would inheritance/hierarchies
 play a role there? Or reference semantics?
 Note that between structs, classes, templates and modules there is a
 lot of redundancy. A possibility could be to have an AggregateInfo
 base class.


Initially I used struct, but then I figured reference semantics are more 
natural for storing cross-entity information, as you indeed took 
advantage of in your TemplateInfo.


 3) For a class, a (to my eyes) standard request would be to get the
 parents chain, up to Object. Could that be added in ClassInfo, or as
 an external free function? (edit: scratch that, you're talking about
 lazy gathering below).


I used strings because I assumed it's easy to just query the ClassInfo 
given a string. But yes, we could use references to ClassInfo and 
InterfaceInfo etc. in a transitive manner.


 4) How would that allows queries like "Here is class C, give me all
 its available subclasses."? Hmm, wait, I get it: extract classes from
 the module, and recursively from imported modules. From these classes,
 extract the parent classes and so on, until the search ranged over the
 whole inheritance tree. I guess inheritance info could be standard
 enough for std.reflection to provide such a search.


Something like that. Note that such a query is not particularly OO-ish, 
because getting a class' cone (totality of subclasses) works against the 
modularity that inheritance is meant for. I don't think we should make 
getting class cones particularly easy.


 5) The compiler can emit JSON output giving a partial view of the same
 information. As a long-term goal, I suggest these infos should be
 compatible somehow. The JSON output should be enriched, and we should
 ascertain that using std.json to read this kind of
 automatically-generated information should give std.reflection infos
 back.


Yes, that's a great connection that Walter and I discussed a bit.


 6) Is really all the necessary info available through std.traits and
 __traits? Imported modules and their subtilities (renamed functions,
 etc) seem out of reach, no?


We'll need indeed to enhance __traits with what's needed. Much of the 
point of std.reflection is to determine exactly what's there and what's 
needed. And that starts with the data structures design (the algorithmic 
aspects are minor).


 7) I know your examples are not complete, but don't forget aliases and
 symbols, and module-level values. Since these can be of any type, I'm
 not sure how they are managed in your scheme. I mean, you cannot have
 IntInfo[], DoubleInfo[], ...


I sort of eschewed part of that by using strings for types.


 8) (more a remark than a question) To me, it's another example of a
 tree-like structure. I humbly suggest we get a std.tree somewhere, to
 manipulate generic n-ary trees: mapping on trees, searching
 (extracting elements or whole subtrees), folding trees, etc. That is,
 a module that deals with trees/graphs not so much as containers, but
 as a way to represent the relationships between elements. I'm not sure
 I'm so clear here, but ideally std.json, std.xml, std.reflection and
 probably a few others should use it as a low-level basis.
 Btw, I have tree / trees algo (including depth-first / breadth-first
 ranges) / graph / graph algo (strongly connected components, search in
 graphs, ...) on github, but it's code that predate std.container and
 is in limbo, pending std.allocator.


Well you're the resident crazy-stuff-during-compilation guy. Did you try 
your trees during compilation?


Andrei
Jul 22 2012
next sibling parent reply "Kagamin" <spam here.lot> writes:
On Sunday, 22 July 2012 at 14:28:45 UTC, Andrei Alexandrescu 
wrote:

 Yah, ideally all entities definable in a D module should be 
 available via reflection. But I focused on things that e.g. the 
 user of a dynamically-loaded library would be interested in: 
 functions, classes.


Plugins are usually done by providing factory methods that create 
needed objects, the methods are usually obtained with dlsym.
Jul 22 2012
parent Wouter Verhelst <wouter grep.be> writes:
"Kagamin" <spam here.lot> writes:


 On Sunday, 22 July 2012 at 14:28:45 UTC, Andrei Alexandrescu wrote:

 Yah, ideally all entities definable in a D module should be
 available via reflection. But I focused on things that e.g. the user
 of a dynamically-loaded library would be interested in: functions,
 classes.


 Plugins are usually done by providing factory methods that create
 needed objects, the methods are usually obtained with dlsym.


That's what you'd do in a language that doesn't have something like D's
Object.factory(). In D's case, however, you'd have the "factory method"
be the ctor.

That's all hypothetical, however, since there's no D ABI for shared
objects yet...

-- 
The volume of a pizza of thickness a and radius z can be described by
the following formula:

pi zz a
Jul 22 2012
prev sibling next sibling parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Sun, Jul 22, 2012 at 4:28 PM, Andrei Alexandrescu
<SeeWebsiteForEmail erdani.org> wrote:


 std.reflection could become the lynchpin for dynamic library use; once the
 library is loaded (with dlopen or such), the client needs to call
 getModuleInfo() (a C function that can be found with dlsym()) and then get
 access to pointers to functions necessary for doing all other work. (Note
 that my examples don't yet include pointers to executable code yet.)


I wouldn't know. I have no experience with dlsym().


 2) Why classes, as opposed to structs? Would inheritance/hierarchies
 play a role there? Or reference semantics?
 Note that between structs, classes, templates and modules there is a
 lot of redundancy. A possibility could be to have an AggregateInfo
 base class.



 Initially I used struct, but then I figured reference semantics are more
 natural for storing cross-entity information, as you indeed took advantage
 of in your TemplateInfo.


I realized a few minutes after posting I answered my own question:
because this is a self-referencing structure (a tree, a graph), which
are easier to code with classes than structs.



 4) How would that allows queries like "Here is class C, give me all
 its available subclasses."? Hmm, wait, I get it: extract classes from
 the module, and recursively from imported modules. From these classes,
 extract the parent classes and so on, until the search ranged over the
 whole inheritance tree. I guess inheritance info could be standard
 enough for std.reflection to provide such a search.



 Something like that. Note that such a query is not particularly OO-ish,
 because getting a class' cone (totality of subclasses) works against the
 modularity that inheritance is meant for. I don't think we should make
 getting class cones particularly easy.


Right. Since people here asked this question, I thought that was a
common request in OO. I'm more a structs and mixins guy, myself.



 5) The compiler can emit JSON output giving a partial view of the same
 information. As a long-term goal, I suggest these infos should be
 compatible somehow. The JSON output should be enriched, and we should
 ascertain that using std.json to read this kind of
 automatically-generated information should give std.reflection infos
 back.



 Yes, that's a great connection that Walter and I discussed a bit.


Good to know.



 6) Is really all the necessary info available through std.traits and
 __traits? Imported modules and their subtilities (renamed functions,
 etc) seem out of reach, no?



 We'll need indeed to enhance __traits with what's needed. Much of the point
 of std.reflection is to determine exactly what's there and what's needed.
 And that starts with the data structures design (the algorithmic aspects are
 minor).


+1 for enhancing __traits locally.

- having __traits(allMembers, xxx) work on simple module names, and
not only on qualified package.module names
- having a way to get imports and a way to know whether they are
static / renaming import


As for the data structures, other people's designs allured to in this
thread seem similar to your proposal.

I have two other questions:

About functions: should they be subject to reflection also, or not?
They have no fields, inner functions are totally interned, etc. All a
user need is the 'interface', right? (name, and everything that's in
the type: return type, parameters, purity, etc)

About imports, what about inner imports, now that they are authorized
in almost any scope? My gut feeling right now is that a user does not
care if class C internally import std.algorithm in one of its methods,
but I could be wrong.



 7) I know your examples are not complete, but don't forget aliases and
 symbols, and module-level values. Since these can be of any type, I'm
 not sure how they are managed in your scheme. I mean, you cannot have
 IntInfo[], DoubleInfo[], ...



 I sort of eschewed part of that by using strings for types.


I see. They can be managed like fields in an aggregate (struct /
classes), as there are many similarities between D modules and classes
/ structs.

class ModuleInfo {
 property:
...
   FieldInfo[] data; // also used in StructInfo and ClassInfo
}

class FieldInfo {
 property:
    string name();
    bool isStatic();
    Protection protection();
    string type();
}

As long as this info is available at CT, FieldInfo.type can be
mixed-in and used in code.

what I'm not sure I get in your design is why some informations are
encoded in their own structure, like Protection above (the code is
copy-pasted from yours, I'd guess Protection is an enumeration), and
then some others are encoded as strings (types). Is that because the
values Protection can take are known in advance (and finite)?

I wondered whether a design like this could be interesting?:

abstract class FieldInfo {}

class Field(T) : FieldInfo {
 property:
    string name();
    bool isStatic();
    Protection protection();
    alias T Type;
}

But that doesn't cut it, AFAICT: different fields can be stored in a
FieldInfo[] array, but the type information is not easier to get,
anyway. So forget it.


This kind of manipulation is why I got interested in fully polymorphic
trees (tuples of tuples...), able to store any value, while keeping
the type information visible. The drastic consequence is to have a
tree type depend on its entire content. But I'm coming from Static
Typing Land here, whereas this introspection stuff is more dynamic.

Anyway, back to gobal values: aliases should be there also. A simple
AliasInfo class?



 Well you're the resident crazy-stuff-during-compilation guy.


Ah! I wish.

I had this wonderful idea of having code be parsed at CT, semantically
analyzed, transformed into some machine code at CT and... , oh wait.



 Did you try your trees during compilation?


Just did. They fail :) Either segmentation fault (core dumped) or an
error telling me class literals  cannot be returned from CTFE.
Hmm, this is old code. I'll have a look since in other projects, I can
obtain trees at CT.


Philippe
Jul 22 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 23-Jul-12 10:30, Philippe Sigaud wrote:

 Well you're the resident crazy-stuff-during-compilation guy.


 Ah! I wish.

 I had this wonderful idea of having code be parsed at CT, semantically
 analyzed, transformed into some machine code at CT and... , oh wait.


I kind of did it...
regex pattern  ---parse---> bytecode ---generate D code---> profit.

BTW you can avoid machine code, just use D code as glorified backend :)
(and get optimizations for free, yay!)


-- 
Dmitry Olshansky
Jul 22 2012
parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Mon, Jul 23, 2012 at 8:41 AM, Dmitry Olshansky <dmitry.olsh gmail.com> wrote:

 On 23-Jul-12 10:30, Philippe Sigaud wrote:

 Well you're the resident crazy-stuff-during-compilation guy.



 Ah! I wish.

 I had this wonderful idea of having code be parsed at CT, semantically
 analyzed, transformed into some machine code at CT and... , oh wait.



 I kind of did it...
 regex pattern  ---parse---> bytecode ---generate D code---> profit.

 BTW you can avoid machine code, just use D code as glorified backend :)
 (and get optimizations for free, yay!)


Hey, that was a joke, about implementing a D compiler in D, to be
executed at CT by the real D compiler to basically do its job a second
time, only worse :)
But yes, your regex work is a great example. I should have a look at
the code once more.
Jul 22 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 23-Jul-12 10:47, Philippe Sigaud wrote:

 On Mon, Jul 23, 2012 at 8:41 AM, Dmitry Olshansky <dmitry.olsh gmail.com>
wrote:

 On 23-Jul-12 10:30, Philippe Sigaud wrote:

 Well you're the resident crazy-stuff-during-compilation guy.



 Ah! I wish.

 I had this wonderful idea of having code be parsed at CT, semantically
 analyzed, transformed into some machine code at CT and... , oh wait.



 I kind of did it...
 regex pattern  ---parse---> bytecode ---generate D code---> profit.

 BTW you can avoid machine code, just use D code as glorified backend :)
 (and get optimizations for free, yay!)


 Hey, that was a joke, about implementing a D compiler in D, to be
 executed at CT by the real D compiler to basically do its job a second
 time, only worse :)


Yeah, now I think I got it. Could be a nice benchmark for CTFE though :)


 But yes, your regex work is a great example. I should have a look at
 the code once more.


I'd advice to wait a bit ... it's not really in a good shape (to read 
it) as there have been tons of improvements in CTFE since the last time 
I've touched it. I'll have to revisit it and remove some roundabout 
workarounds.

-- 
Dmitry Olshansky
Jul 22 2012
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2012-07-22 16:28, Andrei Alexandrescu wrote:


 Something like that. Note that such a query is not particularly OO-ish,
 because getting a class' cone (totality of subclasses) works against the
 modularity that inheritance is meant for. I don't think we should make
 getting class cones particularly easy.


I see no reason to make it difficult on purpose.

-- 
/Jacob Carlborg
Jul 22 2012
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 7/23/12 2:39 AM, Jacob Carlborg wrote:

 On 2012-07-22 16:28, Andrei Alexandrescu wrote:


 Something like that. Note that such a query is not particularly OO-ish,
 because getting a class' cone (totality of subclasses) works against the
 modularity that inheritance is meant for. I don't think we should make
 getting class cones particularly easy.


 I see no reason to make it difficult on purpose.


The reason is that often the cone would be incomplete and therefore 
confusing. It would also foster non-modular approaches to doing things.

Andrei
Jul 22 2012
prev sibling parent reply "David Nadlinger" <see klickverbot.at> writes:
On Sunday, 22 July 2012 at 13:39:31 UTC, Philippe Sigaud wrote:

 4) How would that allows queries like "Here is class C, give me 
 all
 its available subclasses."? Hmm, wait, I get it: extract 
 classes from
 the module, and recursively from imported modules. From these 
 classes,
 extract the parent classes and so on, until the search ranged 
 over the
 whole inheritance tree. I guess inheritance info could be 
 standard
 enough for std.reflection to provide such a search.


You can't do that without breaking the module system – as long 
as a class is not final (and then it doesn't make much sense to 
ask for its subclasses anyway), somebody can always extend it in 
a module completely separate from the code making the query.

To illustrate what I mean, let's assume you define a class Foo in 
a module A. The user imports A from a second module B, and 
potentially extends Foo there. Now, if it was possible to get all 
possible subclasses in A, this would lead to information leaking 
from B to A, while the import graph only allows the other 
direction. As a consequence, all kinds of issues related to order 
dependence, separate compilation, etc. would arise.

David
Jul 23 2012
parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Mon, Jul 23, 2012 at 2:43 PM, David Nadlinger <see klickverbot.at> wrote=
:
[getting subclasses]

 You can't do that without breaking the module system =E2=80=93 as long as=


 a class is

 not final (and then it doesn't make much sense to ask for its subclasses
 anyway), somebody can always extend it in a module completely separate fr=


om

 the code making the query.

 To illustrate what I mean, let's assume you define a class Foo in a modul=


e

 A. The user imports A from a second module B, and potentially extends Foo
 there. Now, if it was possible to get all possible subclasses in A, this
 would lead to information leaking from B to A, while the import graph onl=


y

 allows the other direction. As a consequence, all kinds of issues related=


 to

 order dependence, separate compilation, etc. would arise.


Yes, I know. That would be a purely local question: from module A,
what subclasses of class Foo do I see?
Anyway, maybe I'm warped by CLOS.
Jul 23 2012
prev sibling next sibling parent reply "Max Samukha" <maxsamukha gmail.com> writes:
On Saturday, 21 July 2012 at 21:44:52 UTC, Andrei Alexandrescu 
wrote:


 Please chime in with thoughts. Would someone want to pioneer 
 this project?


There are some questions.

1. static info = getModuleInfo("std.algorithm");

The language does not allow you to use CTFE parameter values as 
arguments to __traits/templates. Therefore, to be able to build 
meta-objects at compile-time, you would have to:

static info = getModuleInfo!"std.algorithm";

2. Then, what is the preferred way of referencing compiler 
objects - with strings or directly:

import std.algorithm;
static info = getModuleInfo!(std.algorithm);
?

I think using strings while we have direct access to those 
objects is not an incredibly good idea. Those strings will be 
mixed-in by getXxxInfo anyway:

ModuleInfo getModuleInfo(string name)()
{
     ...
     mixin("import " ~ name ~ ";");
     mixin("alias __traits(getMembers, " ~ name ~ ") members;");
     foreach (m; members)
     ...
}

3. auto info = getModuleInfo("std.algorithm");

There is no way to mutate global data structures at compile-time, 
therefore you would have to build, at run time, a data structure 
aggregating all meta-objects coming from various modules.

That could be achieved with static constructors:

module std.reflection;

// application-wide module info registry
private ModuleInfo[string] moduleInfos;

// run-time module info getter
string getModuleInfo(string s)
{
     return moduleInfos[s];
}

string makeModuleInfoAvailableDynamically()
{
     return q{
         shared static this() { shared static mi = 
getModuleInfo!(__traits(thisModule));
             moduleInfos[mi.name] = mi; }
     };
}

Problematic because mixing-in makeModuleInfoAvailableDynamically 
would mean that circular imports are no longer allowed for that 
module. Remember the whining babies complaining about this very 
use case a while ago?

What about changing the language so that static constructors 
marked with  system are exempted from circular dependency 
checking?

3. If you are against inheritance, why classes and not structs?

4. How the whole thing is intended to interact with dynamic link 
libraries?
Jul 22 2012
next sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Sun, Jul 22, 2012 at 5:10 PM, Max Samukha <maxsamukha gmail.com> wrote:


 The language does not allow you to use CTFE parameter values as arguments to
 __traits/templates. Therefore, to be able to build meta-objects at
 compile-time, you would have to:

 static info = getModuleInfo!"std.algorithm";


Maybe I don't get your comment, but AFAICT, the language does allow
you to use CTFE parameters values as arguments to templates:

template Twice(double d)
{
    enum Twice = d * 2;
}

double foo(double d)
{
    return d+1.0;
}

void main()
{
    enum t = Twice!(foo(1.0));
    pragma(msg, t);
}
Jul 22 2012
prev sibling next sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On Mon, 23 Jul 2012 08:32:37 +0200, Philippe Sigaud  
<philippe.sigaud gmail.com> wrote:


 Maybe I don't get your comment, but AFAICT, the language does allow
 you to use CTFE parameters values as arguments to templates:

 template Twice(double d)
 {
     enum Twice = d * 2;
 }

 double foo(double d)
 {
     return d+1.0;
 }

 void main()
 {
     enum t = Twice!(foo(1.0));
     pragma(msg, t);
 }


Wrong way around. Try this:

template Twice(double d)
{
     enum Twice = d * 2;
}

double foo(double d)
{
     return Twice!d;
}

void main()
{
     enum  t = foo(1.0);
     pragma(msg, t);
}

-- 
Simen
Jul 23 2012
prev sibling parent Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Mon, Jul 23, 2012 at 9:46 AM, Simen Kjaeraas <simen.kjaras gmail.com> wrote:


 Wrong way around. Try this:


 template Twice(double d)
 {
     enum Twice = d * 2;
 }

 double foo(double d)
 {
     return Twice!d;
 }

 void main()
 {
     enum  t = foo(1.0);
     pragma(msg, t);
 }


Ah, that, OK. And __ctfe does not help.
Would it really limit the proposed scheme for std.reflection?
Jul 23 2012
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 7/22/12, Philippe Sigaud <philippe.sigaud gmail.com> wrote:

 2) Why classes, as opposed to structs?


I think either way you'd need reference semantics. For example maybe
you're doing code-generation at compile-time but you need to rename a
class name in a typeinfo returned by std.reflection before doing any
processign. With reference semantics you only have to change one class
and all other types which refer to such a class will have access to
the new name.

You could use structs as well, and actually I use structs in my
codegenerator (with a similar layout to what Andrei posted). Each
struct (e.g. Class/Function) stores Symbols, which are structs with an
ID and a Type. I can look up each symbol in a SymTable which actually
holds the structures with data. So a Symbol is like a fake pointer,
and the SymTable would be the memory. I originally planned to use
classes but some serialization frameworks didn't work with those so I
settled using structs and a bit of template mixin magic instead.

All of this std.reflection talk is quite exciting actually. If you had
AST information about your entire D library you could do some really
cool things. You could make a better documentation generator than
ddoc, or export the AST into a file for a code-completion plugin, or
create a wrapper C library which enables other languages to use your D
library.
Jul 22 2012
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2012-07-21 23:44, Andrei Alexandrescu wrote:

 Walter and I discussed the idea below a long time (years) ago. Most
 likely it's also been discussed in this newsgroup a couple of times.
 Given the state of the compiler back then, back then it seemed like a
 super cool idea that's entirely realizable, it would just take time for
 the compiler to become as capable as needed. Nowadays we're in shape to
 tackle it.

 Here "it" is.

 Back when runtime reflection was being discussed, my response was "let's
 focus on compile-time reflection, and then we can do run-time reflection
 on demand as a library". Though this might sound sensible, I initially
 didn't have a design. Now here's what we can do.


I've been waiting for this :)

-- 
/Jacob Carlborg
Jul 22 2012
prev sibling next sibling parent reply Philippe Sigaud <philippe.sigaud gmail.com> writes:
On Sun, Jul 22, 2012 at 6:15 PM, Andrej Mitrovic
<andrej.mitrovich gmail.com> wrote:

 On 7/22/12, Philippe Sigaud <philippe.sigaud gmail.com> wrote:

 2) Why classes, as opposed to structs?


 I think either way you'd need reference semantics.


Yeah, I agree, reference semantics look better.
And since there is a lot of duplication (struct info is a subtype of
class info, itself looking very uch like a module info, I guess a
mini-hierarchy is possible).
So I'm convinced classes look like a good fit.



 You could use structs as well, and actually I use structs in my
 codegenerator (with a similar layout to what Andrei posted). Each
 struct (e.g. Class/Function) stores Symbols, which are structs with an
 ID and a Type. I can look up each symbol in a SymTable which actually
 holds the structures with data. So a Symbol is like a fake pointer,
 and the SymTable would be the memory. I originally planned to use
 classes but some serialization frameworks didn't work with those so I
 settled using structs and a bit of template mixin magic instead.


Could you describe your design in this thread, and possibly link to some code?



 All of this std.reflection talk is quite exciting actually.


It is!


 If you had
 AST information about your entire D library you could do some really
 cool things. You could make a better documentation generator than
 ddoc, or export the AST into a file for a code-completion plugin, or
 create a wrapper C library which enables other languages to use your D
 library.


You could do macros. 'nuff said.
Jul 22 2012
parent Jacob Carlborg <doob me.com> writes:
On 2012-07-23 08:35, Philippe Sigaud wrote:

 On Sun, Jul 22, 2012 at 6:15 PM, Andrej Mitrovic



 If you had
 AST information about your entire D library you could do some really
 cool things. You could make a better documentation generator than
 ddoc, or export the AST into a file for a code-completion plugin, or
 create a wrapper C library which enables other languages to use your D
 library.


 You could do macros. 'nuff said.


If I recall correctly, Scala uses its reflection API as a part off its 
macros.

-- 
/Jacob Carlborg
Jul 23 2012
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
Huge interest!
I wouldn't want to see static data structures bloat/clutter the exe though
in cases where it's not ever used/queried.
The object factory is already a serious problem.


On 22 July 2012 00:44, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>wrote:


 Walter and I discussed the idea below a long time (years) ago. Most likely
 it's also been discussed in this newsgroup a couple of times. Given the
 state of the compiler back then, back then it seemed like a super cool idea
 that's entirely realizable, it would just take time for the compiler to
 become as capable as needed. Nowadays we're in shape to tackle it.

 Here "it" is.

 Back when runtime reflection was being discussed, my response was "let's
 focus on compile-time reflection, and then we can do run-time reflection on
 demand as a library". Though this might sound sensible, I initially didn't
 have a design. Now here's what we can do.

 Currently we have information about symbols as __traits(...) intrinsics
 wrapped in nice but scattered ways. Now that CTFE is good enough to
 manipulate structs and arrays thereof, we have the possibility to finally
 approach things in a nicely unified, structured way.

 First, we need to prime std.reflection with a few abstractions that
 characterize entities in a D program.

 class ModuleInfo {
  property:
     string name();
     ImportInfo[] imports();
     DataInfo[] data();
     FunctionInfo[] functions();
     ClassInfo[] classes();
     StructInfo[] structs(); // includes unions
     TemplateInfo[] templates();
     EnumInfo[] enums();
     bool hasStaticCtor(), hasStaticDtor(),
       hasSharedCtor(), hasSharedDtor();
 }

 Probably there are a few more pieces of data, but you get the idea. Then
 for each of the entities mentioned above we have a similar definition. For
 example:

 enum Protection { isPublic, isPackage, isProtected, isPrivate }

 class ClassInfo {
  property:
     string name();
     string baseName();
     string parentName(); // if applicable, null otherwise
     string[] interfaces();
     bool isShared();
     Protection protection();
     DataMemberInfo[] data();
     MethodInfo[] methods();
     Object defaultConstructor();
     ...
 }

 Then for an e.g. method declaration we'd have:

 class MethodInfo {
  property:
     string name();
     bool isStatic(), isFinal(), isOverride();
     Protection protection();
     string[] parameterTypes();
     string[] parameterNames();
 }

 Some details may vary, e.g. some may be straight members instead of
 properties etc. (I used properties to allude to use of lazy gathering of
 information).

 So so far we have a nice collection of structured data associated with the
 entities in a D program. Note how this structuring differs yet has similar
 power to the primitives in std.traits; std.traits offers unstructured bits
 of information on demand (e.g. ParameterTypeNames) etc. but the objects
 above group information together per entity declared. All of the above goes
 in std.reflection, of course.

 ===========

 On to primitives that return such data.

 Given that D can (since relatively recently) create and manipulate class
 objects during compilation too, it follows that the classes above can be
 accessed in two ways - through compile-time API and run-time API. When
 possible, the APIs may even use the same functions; some other times they
 will be necessarily different.

 There are two possible approaches to discovering such information. One is
 by fetching the ModuleInfo for the whole module and navigating it. Another
 one is by using search primitives from strings.

 So we should have e.g.

 // inside std.reflection
 ModuleInfo getModuleInfo(string moduleName);

 so a CT call would go like:

 // client code
 static info = getModuleInfo("std.algorithm")**;

 whereas a run-time call would be:

 // client code
 auto info = getModuleInfo("std.algorithm")**;

 In the latter case, the module needs to save all needed information for
 ri, so it should plant this:

 // inside std.algorithm
 mixin(**makeModuleInfoAvailableDynamic**ally());

 The mixin would generate all information needed and would store it for
 later dynamic use.

 A search API would go like e.g.

 ClassInfo getClassInfo(string className);

 In this case the class name could be qualified with module information etc.

 ===========

 With this design we unify compile-time and run-time type manipulation in
 simple ways, by defining structured information about declarations that can
 be queried during compilation or dynamically.

 Please chime in with thoughts. Would someone want to pioneer this project?


 Andrei
Nov 19 2012
prev sibling parent "Malte Skarupke" <malteskarupke web.de> writes:
I like it. First a couple details:


 class ClassInfo {
  property:
     string name();
     string baseName();
     string parentName(); // if applicable, null otherwise
     string[] interfaces();
     bool isShared();
     Protection protection();
     DataMemberInfo[] data();
     MethodInfo[] methods();
     Object defaultConstructor();
     ...
 }


Why do you not use recursion here? Why not
ClassInfo base();
ClassInfo parent();
and then you get baseName by calling base.name;


 Then for an e.g. method declaration we'd have:

 class MethodInfo {
  property:
     string name();
     bool isStatic(), isFinal(), isOverride();
     Protection protection();
     string[] parameterTypes();
     string[] parameterNames();
 }


Do you think there'll be any way to get the actual types of the 
arguments and the return type back from the struct? I think that 
without compiler magic or an uncomfortable amount of templates it 
probably won't be possible, but I might be wrong.


Now to my big complaint:

I really dislike making the runtime api accessible through a 
mixin. I'd much prefer making it a non-member template. Reasons 
are
a) I don't want to store the runtime information in my type. A 
vtable pointer should be enough to get that information. Maybe 
use it as an index into a global immutable associative array. For 
modules and non-class types the name can act as a unique 
identifier.
b) It'd make it possible to get runtime information on types that 
I get from libraries.
c) It would lead to less coupling.
d) It would discourage writing types that rely on runtime 
reflection, but it would not make it impossible.
Nov 19 2012