digitalmars.D - AST macros
- Walter Bright (11/43) Mar 17 2007 this work?
- Walter Bright (2/30) Mar 17 2007 That would be something for much later.
- Walter Bright (2/22) Mar 17 2007 Right now, I see AST macros as manipulating expressions and statements,
- eao197 (38/40) Mar 18 2007 If I have understood you correctly the D macros will be used like
- Walter Bright (3/12) Mar 18 2007 Macros wouldn't be for adding declarations to classes, for that use
- eao197 (36/47) Mar 18 2007 I don't know how use template mixins for such case:
- Don Clugston (22/43) Mar 19 2007 This looks great already. However, I don't see any specific "abstract
- Walter Bright (13/61) Mar 19 2007 The beauty of it is that (a+b), before semantic analysis, is an AST! So
- Don Clugston (20/69) Mar 20 2007 Fantastic! Does this mean that you just use arg.stringof to get the
- Walter Bright (3/77) Mar 20 2007 Probably not.
- Bruno Medeiros (8/9) Mar 19 2007 Question: are AST macros going to use an AST hierarchy exactly like the
- Walter Bright (2/6) Mar 19 2007 You'll just write AST's as normal D code in normal D notation.
- Daniel Keep (24/47) Mar 19 2007 Walter,
- David B. Held (3/20) Mar 19 2007 I thought Tango had Fibers?
- Daniel Keep (15/37) Mar 19 2007 Cooperative multithreading does not a coroutine make. Fibers, IIRC, are
- Sean Kelly (6/36) Mar 19 2007 For what it's worth, Mikola Lysenko has coroutines as a part of his
- Sean Kelly (3/8) Mar 19 2007 My fault. It seems that's the one you wrote :-p
- David B. Held (7/13) Mar 20 2007 Wikipedia seems to think they are essentially the same thing:
- Daniel Keep (16/33) Mar 20 2007 Hmm... looks that way. I think mine are probably closer to generators
- Walter Bright (2/5) Mar 20 2007 I'm not understanding your example. Where is the macro?
- Daniel Keep (26/32) Mar 20 2007 I want "coro" to be the macro. Basically, I want it to look like a
Marcin Kuszczak wrote:Walter Bright wrote:invocation ofIt's pretty simple: macro foo(args) { ...syntax that gets inserted... } and the args and syntax is evaluated in the context of thethis work?the macro, not the definition of the macro. You'll be able to do things like: macro print(arg) { writefln(__FILE__, __LINE__, arg); } which get the file and line in the right context. There's no way to do that right now.And what about mixin templates? Couldn't they just be fixed to doFrom docs: " Unlike a template instantiation, a template mixin's body is evaluated within the scope where the mixin appears, not where the template declaration is defined. It is analogous to cutting and pasting thebody ofthe template into the location of the mixin. It is useful for injecting parameterized 'boilerplate' code, as well as for creating templatednestedfunctions, which is not possible with template instantiations. " For me it looks exactly like this what you want to do with macro. What would be a difference?Mixin templates have a fundamental problem with this - the mixin arguments are semantically evaluated before the mixin is instantiated. To manipulate AST's, it's necessary to get at them before the semantic analysis is done.For me it seems that the whole mixin thing is rather buggy andunusfull atthe moment - for reference see: http://www.digitalmars.com/d/archives/digitalmars/D/learn/3412.html#N3416They're not buggy, but it's possible they aren't the right design.
Mar 17 2007
I thought it would be too confusing to have 3 different mixin things.eao197 wrote:On naming why not use mixin, since they are so similar? mixin print(arg) { }What if you want the file in some other context? It would be nice to have a complete solution to that, which allows some sort of stack traversal. Although I'm not sure its possible, due to not wanting to keep this sort of information around at release time. ie: macro print(arg) { writefln(__FILE__[stack_level], __LINE__[stack_level], arg); } Or even better: Stack[0].Line; //Current line Stack[1].Line; //Line one level up Stack[0].File; Stack[0].Module; Stack[0].Function; Stack[0].NumbArgs; //Some form of reflection Stack[0].Arg[N]; //Access to the value of the argument (ie Turple of values) Stack[0].ArgIdentifier[N] //String name of the identifier Stack[0].FuncType //The type of function we are in (is it a macro, a compile time function a member, a regular function) ect...That would be something for much later.
Mar 17 2007
As I can see the content of macro body will be inserted into AST in the place of invocation. But there is not much differences with existing C/C++ macro handling (insertion of right __FILE__, __LINE__ is good thing anyway). Is there allowed any access to previous parsed entity? For example, can I define macro: macro some_class_extender(class_name) { ...modification of 'class_name' class structure... } class MyCoolClass { ... } some_class_extender( MyCoolClass ); and get the modified version of MyCoolClass after some_class_externder invocation? --Regards, Yauheni AkhotnikauRight now, I see AST macros as manipulating expressions and statements, not declarations.
Mar 17 2007
On Sun, 18 Mar 2007 05:00:34 +0300, Walter Bright <newshound digitalmars.com> wrote:Right now, I see AST macros as manipulating expressions and statements, not declarations.If I have understood you correctly the D macros will be used like class_eval/module_eval in Ruby -- for generation new content of the programm. For example, in Ruby I can write: class ProjectDescription def add_files( files ); ...; end def add_resources( resouces ); ...; end def self.define_singular_form_method( method ) class_eval %Q{ } end ... end The same trick will be available in D via macro: macro defineSingularFormMethod(methodName, paramType) { void methodName[0..$-1]( paramType a ) { methodName( [ a ] ); } } class ProjectDescription { // Methods which accept Array as argument. void addFiles( char[][] files ) { ... } void addResources( char[][] resources ) { ... } // Addition of singular form methods via macros. defineSingularFormMethod( addFiles, char[] ); defineSingularFormMethod( addResources, char[] ); } The main difference between Ruby's class_eval and D's macro is in the time of execution: in Ruby class_eval is executed in run-time, and in D macro is executed in compile-time (at syntax analyzing stage). Am I right? -- Regards, Yauheni Akhotnikau
Mar 18 2007
eao197 wrote:On Sun, 18 Mar 2007 05:00:34 +0300, Walter Bright <newshound digitalmars.com> wrote:Macros wouldn't be for adding declarations to classes, for that use template mixins.Right now, I see AST macros as manipulating expressions and statements, not declarations.If I have understood you correctly the D macros will be used like class_eval/module_eval in Ruby -- for generation new content of the programm. For example, in Ruby I can write:
Mar 18 2007
On Sun, 18 Mar 2007 23:30:54 +0300, Walter Bright = <newshound digitalmars.com> wrote:eao197 wrote:On Sun, 18 Mar 2007 05:00:34 +0300, Walter Bright =<newshound digitalmars.com> wrote:Right now, I see AST macros as manipulating expressions and =statements, not declarations.If I have understood you correctly the D macros will be used like =class_eval/module_eval in Ruby -- for generation new content of the =programm. For example, in Ruby I can write:Macros wouldn't be for adding declarations to classes, for that use =template mixins.I don't know how use template mixins for such case: // It isn't real D. template ValueAccessor(Type,Name) { Type Name; bool is_<Name>_defined; Type get<Name>() { if( !is_<Name>_defined ) throw new ValueNotDefined("<Name>"); return Name; } void set<Name>( Type value ) { Name =3D value; is_<Name>_defined =3D true; } } class SomeValueHolder { mixin ValueAccessor!( uint, DataCoding ); mixin ValueAccessor!( char[], SourceAddress ); mixin ValueAccessor!( char[], DestinationAddress ); ... // And another bunch of ValueAccessor calls. } auto holder =3D new SomeValueHolder; holder.setDataCoding( 0x01 ); holder.setSourceAddress( "..." ); This can be done via compile-time functions and mixin expression, but I = = don't think it is possible via template mixins. So I think it is better for me to wait while you implement your macro = system in D and see the result. I hope it will be very interesting. Good= = luck! Thank you for your patience. -- = Regards, Yauheni Akhotnikau
Mar 18 2007
Walter Bright wrote:Marcin Kuszczak wrote: > Walter Bright wrote: >> It's pretty simple: >> >> macro foo(args) >> { >> ...syntax that gets inserted... >> } >> >> and the args and syntax is evaluated in the context of the invocation of >> the macro, not the definition of the macro. You'll be able to do things >> like: >> >> macro print(arg) >> { >> writefln(__FILE__, __LINE__, arg); >> } >> >> which get the file and line in the right context. There's no way to do >> that right now.This looks great already. However, I don't see any specific "abstract syntax tree" syntax in this. Is that still a work-in-progress? An observation: My version of vector operation expression templates is mostly working now, using the existing mixins. It deals with ASTs by constructing a string representing the operations to be performed, and a tuple containing all of the parameters. The operations string is of the form "g+=((a+b)*c)+((d*e)+f)", where a, b, c... correspond to T[0], T[1], T[2]... of the tuple T. A compile-time function converts the string into postfix, and then another CFTE function converts that into optimised x87 asm, which is mixed in. Unexpectedly, I found that it's actually very nice to have flattened tuples. If an AST could recognise that two parameters are the same, it could avoid duplication in the tuple, something that a tree can't easily do: eg somevec+= othervec + somevec*3; could become "T[1]+=T[0]+(T[1]*3)" which allows better code generation in the final stage. It may be useful to separate the tree from the values in this way -- it's not clear to me that (for D) it's desirable to have function calls and parameters mixed together in the way that Lisp does.
Mar 19 2007
Don Clugston wrote:Walter Bright wrote:The beauty of it is that (a+b), before semantic analysis, is an AST! So there's no funky new grammar to learn.Marcin Kuszczak wrote: > Walter Bright wrote: >> It's pretty simple: >> >> macro foo(args) >> { >> ...syntax that gets inserted... >> } >> >> and the args and syntax is evaluated in the context of the invocation of >> the macro, not the definition of the macro. You'll be able to do things >> like: >> >> macro print(arg) >> { >> writefln(__FILE__, __LINE__, arg); >> } >> >> which get the file and line in the right context. There's no way to do >> that right now.This looks great already. However, I don't see any specific "abstract syntax tree" syntax in this. Is that still a work-in-progress?An observation: My version of vector operation expression templates is mostly working now, using the existing mixins. It deals with ASTs by constructing a string representing the operations to be performed, and a tuple containing all of the parameters. The operations string is of the form "g+=((a+b)*c)+((d*e)+f)", where a, b, c... correspond to T[0], T[1], T[2]... of the tuple T. A compile-time function converts the string into postfix, and then another CFTE function converts that into optimised x87 asm, which is mixed in.Wow! I think the macros can help by obviating the need for the user to do the mixin manually.Unexpectedly, I found that it's actually very nice to have flattened tuples. If an AST could recognise that two parameters are the same, it could avoid duplication in the tuple, something that a tree can't easily do: eg somevec+= othervec + somevec*3; could become "T[1]+=T[0]+(T[1]*3)" which allows better code generation in the final stage.There's a way to do this using specialization: macro foo(somevec : somevec+=othervec+somevec, othervec) which will only match for parameters of the form: foo( a += b + a); somevec => a othervec => b It works analogously to how you can match type patterns in templates using specializations.It may be useful to separate the tree from the values in this way -- it's not clear to me that (for D) it's desirable to have function calls and parameters mixed together in the way that Lisp does.
Mar 19 2007
Walter Bright wrote:Don Clugston wrote:Fantastic! Does this mean that you just use arg.stringof to get the expression? And 'arg' to evaluate it?Walter Bright wrote:The beauty of it is that (a+b), before semantic analysis, is an AST! So there's no funky new grammar to learn.You'll be able to do things like: >> >> macro print(arg) >> { >> writefln(__FILE__, __LINE__, arg); >> }This looks great already. However, I don't see any specific "abstract syntax tree" syntax in this. Is that still a work-in-progress?Actually usage is OK already. Here's an example: ------ auto p = Vec([1.0, 2, 18]); auto q = Vec([3.5L, 1.1, 3.8]); auto r = Vec([17.0f, 28.1, 1]); q -= ((p+r)*18.0L*314.1L - (p-r))* 35; real d = dot(r, p + 3.7*r); ------ And the only reason for Vec() is because otherwise you can't create array operations. In the first stage, the Vector structs returned by Vec() are removed, and only built-in real[], double[], and float[] vectors are put into the tuple. BTW the generated asm is really good quality; in each case, it generates the best code I can write by hand.An observation: My version of vector operation expression templates is mostly working now, using the existing mixins. It deals with ASTs by constructing a string representing the operations to be performed, and a tuple containing all of the parameters. The operations string is of the form "g+=((a+b)*c)+((d*e)+f)", where a, b, c... correspond to T[0], T[1], T[2]... of the tuple T. A compile-time function converts the string into postfix, and then another CFTE function converts that into optimised x87 asm, which is mixed in.Wow! I think the macros can help by obviating the need for the user to do the mixin manually.Now that could be useful... Are we going to able to write macros for member functions, and operator overloads?Unexpectedly, I found that it's actually very nice to have flattened tuples. If an AST could recognise that two parameters are the same, it could avoid duplication in the tuple, something that a tree can't easily do: eg somevec+= othervec + somevec*3; could become "T[1]+=T[0]+(T[1]*3)" which allows better code generation in the final stage.There's a way to do this using specialization: macro foo(somevec : somevec+=othervec+somevec, othervec) which will only match for parameters of the form: foo( a += b + a); somevec => a othervec => b It works analogously to how you can match type patterns in templates using specializations.
Mar 20 2007
Don Clugston wrote:Walter Bright wrote:I'm not totally sure yet.Don Clugston wrote:Fantastic! Does this mean that you just use arg.stringof to get the expression? And 'arg' to evaluate it?Walter Bright wrote:The beauty of it is that (a+b), before semantic analysis, is an AST! So there's no funky new grammar to learn.You'll be able to do things like: >> >> macro print(arg) >> { >> writefln(__FILE__, __LINE__, arg); >> }This looks great already. However, I don't see any specific "abstract syntax tree" syntax in this. Is that still a work-in-progress?Probably not.Actually usage is OK already. Here's an example: ------ auto p = Vec([1.0, 2, 18]); auto q = Vec([3.5L, 1.1, 3.8]); auto r = Vec([17.0f, 28.1, 1]); q -= ((p+r)*18.0L*314.1L - (p-r))* 35; real d = dot(r, p + 3.7*r); ------ And the only reason for Vec() is because otherwise you can't create array operations. In the first stage, the Vector structs returned by Vec() are removed, and only built-in real[], double[], and float[] vectors are put into the tuple. BTW the generated asm is really good quality; in each case, it generates the best code I can write by hand.An observation: My version of vector operation expression templates is mostly working now, using the existing mixins. It deals with ASTs by constructing a string representing the operations to be performed, and a tuple containing all of the parameters. The operations string is of the form "g+=((a+b)*c)+((d*e)+f)", where a, b, c... correspond to T[0], T[1], T[2]... of the tuple T. A compile-time function converts the string into postfix, and then another CFTE function converts that into optimised x87 asm, which is mixed in.Wow! I think the macros can help by obviating the need for the user to do the mixin manually.Now that could be useful... Are we going to able to write macros for member functions, and operator overloads?Unexpectedly, I found that it's actually very nice to have flattened tuples. If an AST could recognise that two parameters are the same, it could avoid duplication in the tuple, something that a tree can't easily do: eg somevec+= othervec + somevec*3; could become "T[1]+=T[0]+(T[1]*3)" which allows better code generation in the final stage.There's a way to do this using specialization: macro foo(somevec : somevec+=othervec+somevec, othervec) which will only match for parameters of the form: foo( a += b + a); somevec => a othervec => b It works analogously to how you can match type patterns in templates using specializations.
Mar 20 2007
Walter Bright wrote:...Question: are AST macros going to use an AST hierarchy exactly like the one in the DMD frontend, or will there be some changes? I say this because I think the DMD frontend AST hierarchy is in some cases quite strange, to say the least. -- Bruno Medeiros - MSc in CS/E student http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Mar 19 2007
Bruno Medeiros wrote:Question: are AST macros going to use an AST hierarchy exactly like the one in the DMD frontend, or will there be some changes? I say this because I think the DMD frontend AST hierarchy is in some cases quite strange, to say the least.You'll just write AST's as normal D code in normal D notation.
Mar 19 2007
Walter, I've just been going back over my old (and now fairly crusty) coroutine implementation, and realised that macros could make these a *major* feature of D. Ideally, I'd like to see a syntax like this:coro showfiles(char[] msg)(char[], char[]) // <-- input,output types { char[] path; while( (path = yield) != null ) // <-- yield is a protected method { writefln("%s: %s", msg, path); } }Which a macro could then unwind into something ugly that the user doesn't have to see:class showfiles : CoroutineT!(char[], char[], char[]) { mixin CoroutineMixin!(char[], char[], char[]); void run(char[] msg) { char[] path; while( (path = this.yield) != null ) { writefln("%s: %s", msg, path); } } }Which is basically how it looks now. Then, using it is nice and clean:scope sf = showfiles("Hi there"); sf("file1"); sf("file2");I've tried to come up with other syntaxes using current constructs, and I can't really get any better than having the user define a subclass, mixin a template, and then implement a run method. It just feels... unwieldy. Will the sort of syntax in the first example be possible? If not, do you have any thoughts on how to improve the above kind of construct (either now or with D 2.0)? -- Daniel -- int getRandomNumber() { return 4; // chosen by fair dice roll. // guaranteed to be random. } v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/
Mar 19 2007
Daniel Keep wrote:Walter, I've just been going back over my old (and now fairly crusty) coroutine implementation, and realised that macros could make these a *major* feature of D. Ideally, I'd like to see a syntax like this:I thought Tango had Fibers? Davecoro showfiles(char[] msg)(char[], char[]) // <-- input,output types { char[] path; while( (path = yield) != null ) // <-- yield is a protected method { writefln("%s: %s", msg, path); } }[...]
Mar 19 2007
David B. Held wrote:Daniel Keep wrote:Cooperative multithreading does not a coroutine make. Fibers, IIRC, are just stack-based threads. Coroutines are *entirely different*, although in this particular case they are implemented *using* stack threads. In any case, I have a working library right now... it's just that the syntax to use them is rather ugly and a bit messy. -- Daniel -- int getRandomNumber() { return 4; // chosen by fair dice roll. // guaranteed to be random. } v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/Walter, I've just been going back over my old (and now fairly crusty) coroutine implementation, and realised that macros could make these a *major* feature of D. Ideally, I'd like to see a syntax like this:I thought Tango had Fibers? Davecoro showfiles(char[] msg)(char[], char[]) // <-- input,output types { char[] path; while( (path = yield) != null ) // <-- yield is a protected method { writefln("%s: %s", msg, path); } }[...]
Mar 19 2007
Daniel Keep wrote:David B. Held wrote:For what it's worth, Mikola Lysenko has coroutines as a part of his StackThreads implementation, and it should be pretty easy to adapt them to use Tango's Fibers. The page is here: http://www.assertfalse.com/projects.shtml SeanDaniel Keep wrote:Cooperative multithreading does not a coroutine make. Fibers, IIRC, are just stack-based threads. Coroutines are *entirely different*, although in this particular case they are implemented *using* stack threads. In any case, I have a working library right now... it's just that the syntax to use them is rather ugly and a bit messy.Walter, I've just been going back over my old (and now fairly crusty) coroutine implementation, and realised that macros could make these a *major* feature of D. Ideally, I'd like to see a syntax like this:I thought Tango had Fibers? Davecoro showfiles(char[] msg)(char[], char[]) // <-- input,output types { char[] path; while( (path = yield) != null ) // <-- yield is a protected method { writefln("%s: %s", msg, path); } }[...]
Mar 19 2007
Sean Kelly wrote:For what it's worth, Mikola Lysenko has coroutines as a part of his StackThreads implementation, and it should be pretty easy to adapt them to use Tango's Fibers. The page is here: http://www.assertfalse.com/projects.shtmlMy fault. It seems that's the one you wrote :-p Sean
Mar 19 2007
Daniel Keep wrote:[...] Cooperative multithreading does not a coroutine make. Fibers, IIRC, are just stack-based threads. Coroutines are *entirely different*, although in this particular case they are implemented *using* stack threads. [...]Wikipedia seems to think they are essentially the same thing: http://en.wikipedia.org/wiki/Fiber_%28computer_science%29 Fibers are less than stack-based threads because they are cooperative. Whereas in a coroutine you write "return", in a fiber you write "yield". Otherwise, I believe they are closely isomorphic. Dave
Mar 20 2007
David B. Held wrote:Daniel Keep wrote:Hmm... looks that way. I think mine are probably closer to generators or channels than strictly coroutines, the difference being that Fibers just do a transfer of control to the given fiber, whereas mine work more like a function call. Actually, thinking about it, I'm not entirely sure if mine classify as coroutines or not... oh bother :P -- Daniel -- int getRandomNumber() { return 4; // chosen by fair dice roll. // guaranteed to be random. } v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/[...] Cooperative multithreading does not a coroutine make. Fibers, IIRC, are just stack-based threads. Coroutines are *entirely different*, although in this particular case they are implemented *using* stack threads. [...]Wikipedia seems to think they are essentially the same thing: http://en.wikipedia.org/wiki/Fiber_%28computer_science%29 Fibers are less than stack-based threads because they are cooperative. Whereas in a coroutine you write "return", in a fiber you write "yield". Otherwise, I believe they are closely isomorphic. Dave
Mar 20 2007
Daniel Keep wrote:Will the sort of syntax in the first example be possible? If not, do you have any thoughts on how to improve the above kind of construct (either now or with D 2.0)?I'm not understanding your example. Where is the macro?
Mar 20 2007
Walter Bright wrote:Daniel Keep wrote:I want "coro" to be the macro. Basically, I want it to look like a function, except it has a second argument list, where you can stick in the input/output types. You said before that you can do specialisations on syntax, so I suppose it might be something like... macro coro(name : name(typelist)(arglist){block}, typelist, arglist, block) { // blah } At this stage, I'm just trying to describe the kind of syntax I'd like to be able to make, and then you say "you can't do that, but you can do this", and then I say "well what if we changed this little bit here", and so forth until either one of us goes mad or we come to some sort of middle-ground :P Plus, I figure the more potential use-cases we show you, the better the end result will be :) -- Daniel -- int getRandomNumber() { return 4; // chosen by fair dice roll. // guaranteed to be random. } v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP http://hackerkey.com/Will the sort of syntax in the first example be possible? If not, do you have any thoughts on how to improve the above kind of construct (either now or with D 2.0)?I'm not understanding your example. Where is the macro?
Mar 20 2007