www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Thesis on metaprogramming in D

reply Leandro Lucarella <llucarella integratech.com.ar> writes:
Hello. My name is Leandro Lucarella, I'm finishing my grade on Computer 
Engineering in the UBA (University of Buenos Aires, Argentina) and I'm 
planning to do my thesis on meta-programming and I'd like to do it using 
D (because is the language that is closer to my ideal language, if it 
just had Python-like syntax... ;).

The center of my thesis would be adding meta-classes[1] to D, and maybe 
some other meta-programming features D could lack of (maybe some kind of 
access to the AST like XLR[2]).

I'd like to know if you think this is:
1) Doable.
2) Useful.
3) A feature that can be included in the official D specification.

In case it's not clear, I'd like to add support for meta-classes to the 
D language itself, not provide some nasty library to ease the job using 
the existing tools. If there is any D guru that is willing to help me 
out and give me some guidance, that would be great too.

TIA

[1] Like Python meta-classes but, of course, resolved at compile time.
     http://www.python.org/doc/essays/metaclasses/
[2] Extensible Language and Runtime
     http://xlr.sourceforge.net/

-- 
Leandro Lucarella
Integratech S.A.
4571-5252
Nov 27 2006
parent reply Steve Horne <stephenwantshornenospam100 aol.com> writes:
On Mon, 27 Nov 2006 15:23:33 -0300, Leandro Lucarella
<llucarella integratech.com.ar> wrote:


The center of my thesis would be adding meta-classes[1] to D, and maybe 
some other meta-programming features D could lack of (maybe some kind of 
access to the AST like XLR[2]).


I can't really offer any useful support, but if I understand what you
are saying, I can offer encouragement.

My view of the ideal handling of metaprogramming is that it should
have access to all the features that are available at runtime. The
only languages that I know that genuinely achieve that are the
Lisp-alikes such as Scheme.

My ideal is being able to define 'quoted' blocks that are parsed and
translated to ASTs, with those ASTs being processed by other code to
derive new ASTs which are then submitted for the final back-end
optimisation and code generation.

Which is not anti-templates as such. I like templates. It just means
that, in principle, you could define an alternative template mechanism
as a library.

The nearest you get to this at the moment (outside Lisp-alikes), at
least that I know of, are languages like Python where you can access
the AST, including the AST of generated code, at run-time.

Thanks for the XLR link, by the way - very interesting, and possibly
capable of exactly what I described above, though I've not read it
properly yet.

-- 
Remove 'wants' and 'nospam' from e-mail.
Nov 27 2006
next sibling parent reply "Andrey Khropov" <andkhropov_nosp m_mtu-net.ru> writes:
Steve Horne wrote:


 My view of the ideal handling of metaprogramming is that it should
 have access to all the features that are available at runtime. The
 only languages that I know that genuinely achieve that are the
 Lisp-alikes such as Scheme.
 
 My ideal is being able to define 'quoted' blocks that are parsed and
 translated to ASTs, with those ASTs being processed by other code to
 derive new ASTs which are then submitted for the final back-end
 optimisation and code generation.


Nemerle does exactly that way!

-- 
AKhropov
Nov 28 2006
next sibling parent reply Johan Granberg <lijat.meREM OVE.gmail.com> writes:
Andrey Khropov wrote:


 Steve Horne wrote:
 

 My view of the ideal handling of metaprogramming is that it should
 have access to all the features that are available at runtime. The
 only languages that I know that genuinely achieve that are the
 Lisp-alikes such as Scheme.
 
 My ideal is being able to define 'quoted' blocks that are parsed and
 translated to ASTs, with those ASTs being processed by other code to
 derive new ASTs which are then submitted for the final back-end
 optimisation and code generation.


 
 Nemerle does exactly that way!
 


Maybe this is a little of topic in a D news group but since nemerle came up,
What is the speed of nemerle related to other languages such as C D C++ ,
Java or python?
Nov 28 2006
parent reply Bill Baxter <wbaxter gmail.com> writes:
Johan Granberg wrote:

 Andrey Khropov wrote:
 



 Maybe this is a little of topic in a D news group but since nemerle came up,
 What is the speed of nemerle related to other languages such as C D C++ ,
 Java or python?




--bb
Nov 28 2006
parent "Andrey Khropov" <andkhropov_nosp m_mtu-net.ru> writes:
Bill Baxter wrote:





Yes, my experiments have shown that this is generally the case.
No surprise, because it's a statically typed language unlike Scheme/Lisp.

-- 
AKhropov
Nov 28 2006
prev sibling parent reply Don Clugston <dac nospam.com.au> writes:
Andrey Khropov wrote:

 Steve Horne wrote:
 

 My view of the ideal handling of metaprogramming is that it should
 have access to all the features that are available at runtime. The
 only languages that I know that genuinely achieve that are the
 Lisp-alikes such as Scheme.

 My ideal is being able to define 'quoted' blocks that are parsed and
 translated to ASTs, with those ASTs being processed by other code to
 derive new ASTs which are then submitted for the final back-end
 optimisation and code generation.


 
 Nemerle does exactly that way!


When I looked at the Nemerle website, I didn't see much that couldn't be 
done easily with D templates. It would be interesting to find something 
it can do, that D can't.
Nov 28 2006
next sibling parent reply Daniel Keep <daniel.keep+lists gmail.com> writes:
Don Clugston wrote:

 Andrey Khropov wrote:
 

 Steve Horne wrote:


 My view of the ideal handling of metaprogramming is that it should
 have access to all the features that are available at runtime. The
 only languages that I know that genuinely achieve that are the
 Lisp-alikes such as Scheme.

 My ideal is being able to define 'quoted' blocks that are parsed and
 translated to ASTs, with those ASTs being processed by other code to
 derive new ASTs which are then submitted for the final back-end
 optimisation and code generation.



 Nemerle does exactly that way!


 
 
 When I looked at the Nemerle website, I didn't see much that couldn't be 
 done easily with D templates. It would be interesting to find something 
 it can do, that D can't.
 



 def sr = System.IO.StreamReader ("SomeFile.txt");


Immutable run-time variables.  You can assign anything to them, and then 
it can't change.  AFAIK, D's const doesn't let you do that.


 someFunction(if (cond) expr_1 else expr_2);


No statements--everything is an expression.  And yes, it is VERY handy :)


 import some.namespace.SomeClass;


Import contents of static classes as well as modules.


 match (control) {
   | button is Button => ...
   | listv is ListView => ...
   | _ => ... // null case
 }


switch eat your heart out.  You can use literal, pattern matching... 
just about anything.


 def res2 = frobnicate (7, do_qux = true,
                           do_baz = false,
                           do_bar = true);


Named parameters.  D doesn't even have hashtable literals, so faking it 
would be a complete pain.


 ()


That's the void literal.  Basically, makes void act like any other type. 
  Try writing generic code where functions can have void return values, 
and you'll end up writing everything twice since you can't have void 
variables.


 variant RgbColor {
   | Red
   | Yellow
   | Green
   | Different {
       red : float;
       green : float;
       blue : float;
     }
 }


Different to a union since it remembers what kind of thing is being 
stored.  Fun trick with trees is to use a variant to store either a 
branch or a leaf.


 macro while_macro (cond, body)
 syntax ("while", "(", cond, ")", body) {
   <[
     def loop () {
       when ($cond) {
         $body;
         loop ()
       }
     }
     loop ()
   ]>
 }


Do THAT with templates :3


 def x = 3;
 System.Console.WriteLine ($"My value of x is $x and I'm happy");


 expands to


 def x = 3;
 System.Console.WriteLine ({
   def sb = System.Text.StringBuilder ("My value of x is ");
   sb.Append (x.ToString ());
   sb.Append (" and I'm happy");
   sb.ToString ()
 });




Even if you could fake the one just above that, there's no way you can 
do that in D.

It would be foolish to think that Nemerle isn't an amazingly powerful 
language.  D is good, and its' templates are very powerful, but they're 
not THAT powerful.

There is always more to learn :)

	-- Daniel
Nov 28 2006
next sibling parent Don Clugston <dac nospam.com.au> writes:
Daniel Keep wrote:

 Don Clugston wrote:

 Andrey Khropov wrote:


 Steve Horne wrote:


 My view of the ideal handling of metaprogramming is that it should
 have access to all the features that are available at runtime. The
 only languages that I know that genuinely achieve that are the
 Lisp-alikes such as Scheme.

 My ideal is being able to define 'quoted' blocks that are parsed and
 translated to ASTs, with those ASTs being processed by other code to
 derive new ASTs which are then submitted for the final back-end
 optimisation and code generation.



 Nemerle does exactly that way!



 When I looked at the Nemerle website, I didn't see much that couldn't 
 be done easily with D templates. It would be interesting to find 
 something it can do, that D can't.


 
  > def sr = System.IO.StreamReader ("SomeFile.txt");
 
 Immutable run-time variables.  You can assign anything to them, and then 
 it can't change.  AFAIK, D's const doesn't let you do that.
 
  > someFunction(if (cond) expr_1 else expr_2);
 
 No statements--everything is an expression.  And yes, it is VERY handy :)
 
  > import some.namespace.SomeClass;
 
 Import contents of static classes as well as modules.
 
  > match (control) {
  >   | button is Button => ...
  >   | listv is ListView => ...
  >   | _ => ... // null case
  > }
 
 switch eat your heart out.  You can use literal, pattern matching... 
 just about anything.
 
  > def res2 = frobnicate (7, do_qux = true,
  >                           do_baz = false,
  >                           do_bar = true);
 
 Named parameters.  D doesn't even have hashtable literals, so faking it 
 would be a complete pain.
 
  > ()
 
 That's the void literal.  Basically, makes void act like any other type. 
  Try writing generic code where functions can have void return values, 
 and you'll end up writing everything twice since you can't have void 
 variables.
 
  > variant RgbColor {
  >   | Red
  >   | Yellow
  >   | Green
  >   | Different {
  >       red : float;
  >       green : float;
  >       blue : float;
  >     }
  > }
 
 Different to a union since it remembers what kind of thing is being 
 stored.  Fun trick with trees is to use a variant to store either a 
 branch or a leaf.
 
  > macro while_macro (cond, body)
  > syntax ("while", "(", cond, ")", body) {
  >   <[
  >     def loop () {
  >       when ($cond) {
  >         $body;
  >         loop ()
  >       }
  >     }
  >     loop ()
  >   ]>
  > }
 
 Do THAT with templates :3


The 'lazy evaluation' page has a similar example. :-)


  >> def x = 3;
  >> System.Console.WriteLine ($"My value of x is $x and I'm happy");
  >
  > expands to
  >
  >> def x = 3;
  >> System.Console.WriteLine ({
  >>   def sb = System.Text.StringBuilder ("My value of x is ");
  >>   sb.Append (x.ToString ());
  >>   sb.Append (" and I'm happy");
  >>   sb.ToString ()
  >> });
 
 Even if you could fake the one just above that, there's no way you can 
 do that in D.


You could if we got the identifier() keyword which I've previously 
proposed. Admittedly, it still wouldn't be pretty. But there's really 
because there's no support for embedded variables.

But doing the same thing for
System.Console.WriteLine("My value of x is ", x, " and I'm happy");
is trivial with the new tuples.


 It would be foolish to think that Nemerle isn't an amazingly powerful 
 language.  D is good, and its' templates are very powerful, but they're 
 not THAT powerful.


I still have the impression that the functionality isn't much different.


 There is always more to learn :)
 
     -- Daniel
Nov 28 2006
prev sibling next sibling parent reply Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Daniel Keep wrote:

  > def sr = System.IO.StreamReader ("SomeFile.txt");
 
 Immutable run-time variables.  You can assign anything to them, and then 
 it can't change.  AFAIK, D's const doesn't let you do that.
 



transitively unchangeable, or just the immediate value?

-- 
Bruno Medeiros - MSc in CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Nov 30 2006
parent reply Daniel Keep <daniel.keep+lists gmail.com> writes:
Bruno Medeiros wrote:

 Daniel Keep wrote:
 

  > def sr = System.IO.StreamReader ("SomeFile.txt");

 Immutable run-time variables.  You can assign anything to them, and 
 then it can't change.  AFAIK, D's const doesn't let you do that.


 

 transitively unchangeable, or just the immediate value?
 


I'm not really sure what 'transitively unchangeable' means... >_<

Firstly, isn't 'final' just used for class members?  This is used for 
local variables (or members, if I remember correctly).  As for the 
second sentence, it's immutable storage.  So you can't change what's 
stored in the variable, but if you've stored a reference, then you can 
mutate what's being referenced.

	-- Daniel
Nov 30 2006
parent Bruno Medeiros <brunodomedeiros+spam com.gmail> writes:
Daniel Keep wrote:

 Bruno Medeiros wrote:

 Daniel Keep wrote:


  > def sr = System.IO.StreamReader ("SomeFile.txt");

 Immutable run-time variables.  You can assign anything to them, and 
 then it can't change.  AFAIK, D's const doesn't let you do that.



 transitively unchangeable, or just the immediate value?


 
 I'm not really sure what 'transitively unchangeable' means... >_<
 


'transitively unchangeable' or transitively const, means that in the 
case of references or pointers you can't change what is being referenced 
(and recursively so on). It's a transitive (recursive) read-only.


 Firstly, isn't 'final' just used for class members?  This is used for 
 local variables (or members, if I remember correctly).  As for the 
 second sentence, it's immutable storage.  So you can't change what's 
 stored in the variable, but if you've stored a reference, then you can 
 mutate what's being referenced.
 
     -- Daniel


Then that is just like 'final'. But I've realize now that this 'final' 

called 'readonly' 
(http://msdn2.microsoft.com/en-us/library/acdd6hb7.aspx), whereas I 

used on fields, but the Java 'final' can be used on any variable, such 
as locals. And like Don said D has that feature too, but hideously 
hidden behind 'const'. -_-'

-- 
Bruno Medeiros - MSc in CS/E student
http://www.prowiki.org/wiki4d/wiki.cgi?BrunoMedeiros#D
Nov 30 2006
prev sibling parent reply Don Clugston <dac nospam.com.au> writes:
Daniel Keep wrote:

 Don Clugston wrote:

 When I looked at the Nemerle website, I didn't see much that couldn't 
 be done easily with D templates. It would be interesting to find 
 something it can do, that D can't.


  > def sr = System.IO.StreamReader ("SomeFile.txt");
 
 Immutable run-time variables.  You can assign anything to them, and then 
 it can't change.  AFAIK, D's const doesn't let you do that.


It does, but only for class member variables. There are two 'const's in 
D, one for literal constants, and the other which is once-only assign, 
and should probably be called 'final' instead.

class SomeClass {
	const StreamReader sr;
	this(char [] filename) { sr = System.IO.StreamReader(filename);}
};

[snip]


  > macro while_macro (cond, body)
  > syntax ("while", "(", cond, ")", body) {
  >   <[
  >     def loop () {
  >       when ($cond) {
  >         $body;
  >         loop ()
  >       }
  >     }
  >     loop ()
  >   ]>
  > }
 
 Do THAT with templates :3


void While(lazy bool cond, void delegate() body) {
    for (; cond; ) body();
}

<g>
Or if I can't have for() as a primitive:

void While(lazy bool cond, void delegate() body) {
loopstart: if(cond) goto done;
     body();
     goto loopstart;
done:
}

 It would be foolish to think that Nemerle isn't an amazingly powerful 
 language.  D is good, and its' templates are very powerful, but they're 
 not THAT powerful.
 
 There is always more to learn :)


Very true.
It does seem, though, that Nemerle and D are exploring a quite similar 
'paradigm space' (so there's probably quite a bit each can learn from 
the other). Yet neither has really caught up with Lisp. Yet.
Nov 30 2006
next sibling parent reply Daniel Keep <daniel.keep+lists gmail.com> writes:
Don Clugston wrote:

 Daniel Keep wrote:

  > def sr = System.IO.StreamReader ("SomeFile.txt");

 Immutable run-time variables.  You can assign anything to them, and 
 then it can't change.  AFAIK, D's const doesn't let you do that.


 
 
 It does, but only for class member variables. There are two 'const's in 
 D, one for literal constants, and the other which is once-only assign, 
 and should probably be called 'final' instead.
 
 class SomeClass {
     const StreamReader sr;
     this(char [] filename) { sr = System.IO.StreamReader(filename);}
 };
 


The thing that *really* struck me was that when I wrote some stuff in 
Nemerle, it was amazing how often I didn't actually need mutable 
variables.  I wonder what affect having immutables in D would have 
optimisation-wise...


 [snip]
 

  > macro while_macro (cond, body)
  > syntax ("while", "(", cond, ")", body) {
  >   <[
  >     def loop () {
  >       when ($cond) {
  >         $body;
  >         loop ()
  >       }
  >     }
  >     loop ()
  >   ]>
  > }

 Do THAT with templates :3


 
 
 void While(lazy bool cond, void delegate() body) {
    for (; cond; ) body();
 }
 
 <g>
 Or if I can't have for() as a primitive:
 
 void While(lazy bool cond, void delegate() body) {
 loopstart: if(cond) goto done;
     body();
     goto loopstart;
 done:
 }


True, but it's not quite the same.  The D version has that outlying 
closing paren which is, frankly, really ugly.  Also, the D version is 
doing a function call, whereas the Nemerle version is actually expanding 
an AST.

Of course, these things could probably be solved with a little more work 
from the compiler: allowing trailing delegate literals, and inlining 
delegate literals.  *drool*


 It would be foolish to think that Nemerle isn't an amazingly powerful 
 language.  D is good, and its' templates are very powerful, but 
 they're not THAT powerful.

 There is always more to learn :)


 
 
 Very true.
 It does seem, though, that Nemerle and D are exploring a quite similar 
 'paradigm space' (so there's probably quite a bit each can learn from 
 the other). Yet neither has really caught up with Lisp. Yet.


No language will ever catch up to LISP: the only way to do that would be 
to become LISP, and then you have to deal with all those bloody 
parentheses :3

	-- Daniel
Nov 30 2006
next sibling parent Brad Anderson <brad dsource.org> writes:
Daniel Keep wrote:

 Don Clugston wrote:

 Daniel Keep wrote:

  > def sr = System.IO.StreamReader ("SomeFile.txt");

 Immutable run-time variables.  You can assign anything to them, and
 then it can't change.  AFAIK, D's const doesn't let you do that.



 It does, but only for class member variables. There are two 'const's
 in D, one for literal constants, and the other which is once-only
 assign, and should probably be called 'final' instead.

 class SomeClass {
     const StreamReader sr;
     this(char [] filename) { sr = System.IO.StreamReader(filename);}
 };


 
 The thing that *really* struck me was that when I wrote some stuff in
 Nemerle, it was amazing how often I didn't actually need mutable
 variables.  I wonder what affect having immutables in D would have
 optimisation-wise...


Not sure, but it would also help get closer to the Erlang model of concurrent
programming via message passing.  Erlang is all immutables, iirc.  CL-MUPROC,
a Lisp attempt at this Erlang model, treats immutability as a 'guideline' and
let's the developer be grown-up.  But it uses OS threads instead of Erlang's
lightweight processes. :(


 

 [snip]


  > macro while_macro (cond, body)
  > syntax ("while", "(", cond, ")", body) {
  >   <[
  >     def loop () {
  >       when ($cond) {
  >         $body;
  >         loop ()
  >       }
  >     }
  >     loop ()
  >   ]>
  > }

 Do THAT with templates :3



 void While(lazy bool cond, void delegate() body) {
    for (; cond; ) body();
 }

 <g>
 Or if I can't have for() as a primitive:

 void While(lazy bool cond, void delegate() body) {
 loopstart: if(cond) goto done;
     body();
     goto loopstart;
 done:
 }


 
 True, but it's not quite the same.  The D version has that outlying
 closing paren which is, frankly, really ugly.  Also, the D version is
 doing a function call, whereas the Nemerle version is actually expanding
 an AST.
 
 Of course, these things could probably be solved with a little more work
 from the compiler: allowing trailing delegate literals, and inlining
 delegate literals.  *drool*
 

 It would be foolish to think that Nemerle isn't an amazingly powerful
 language.  D is good, and its' templates are very powerful, but
 they're not THAT powerful.

 There is always more to learn :)



 Very true.
 It does seem, though, that Nemerle and D are exploring a quite similar
 'paradigm space' (so there's probably quite a bit each can learn from
 the other). Yet neither has really caught up with Lisp. Yet.


 
 No language will ever catch up to LISP: the only way to do that would be
 to become LISP, and then you have to deal with all those bloody
 parentheses :3


;)


 
     -- Daniel
Nov 30 2006
prev sibling parent Steve Horne <stephenwantshornenospam100 aol.com> writes:
On Fri, 01 Dec 2006 00:35:36 +0800, Daniel Keep
<daniel.keep+lists gmail.com> wrote:


Don Clugston wrote:



 It does seem, though, that Nemerle and D are exploring a quite similar 
 'paradigm space' (so there's probably quite a bit each can learn from 
 the other). Yet neither has really caught up with Lisp. Yet.


No language will ever catch up to LISP: the only way to do that would be 
to become LISP, and then you have to deal with all those bloody 
parentheses :3


Not sure I agree entirely.

In my view, the things that define the Lisp language are...

1.  The metaprogramming stuff, with a rather sparse core language
    being extended using library macros.

2.  The list based on pair (cons) objects.

3.  The explicit AST-structuring using parentheses.

Well, Nemerle has metaprogramming that is basically what Lisp does
with some extra tweaks. Nemerle has pair-object based lists. And
Nemerle has explicit AST-structuring using parentheses (it has a
pre-parse stage to handle this).

Nemerle is heavily Lisp influenced. It just takes ideas from other
places too. Personally, I think it may well have overtaken Lisp - just
as powerful (within the limits of the .NET platform), yet more
immediately usable.

-- 
Remove 'wants' and 'nospam' from e-mail.
Dec 01 2006
prev sibling parent Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Don Clugston wrote:

 Daniel Keep wrote:

 Do THAT with templates :3


 
 void While(lazy bool cond, void delegate() body) {
    for (; cond; ) body();
 }


Heh, incorrect answer. I don't see a template :P.
</nitpick>
Nov 30 2006
prev sibling parent Steve Horne <stephenwantshornenospam100 aol.com> writes:
On Tue, 28 Nov 2006 13:59:20 +0100, Don Clugston <dac nospam.com.au>
wrote:


Andrey Khropov wrote:

 Steve Horne wrote:
 

 My view of the ideal handling of metaprogramming is that it should
 have access to all the features that are available at runtime. The
 only languages that I know that genuinely achieve that are the
 Lisp-alikes such as Scheme.

 My ideal is being able to define 'quoted' blocks that are parsed and
 translated to ASTs, with those ASTs being processed by other code to
 derive new ASTs which are then submitted for the final back-end
 optimisation and code generation.


 
 Nemerle does exactly that way!




Cool!


When I looked at the Nemerle website, I didn't see much that couldn't be 
done easily with D templates. It would be interesting to find something 
it can do, that D can't.


That doesn't worry me much, to be honest. I don't want another Scheme
(or at least my perception of Scheme) where you start each journey not
by getting into your car, but by re-inventing the wheel.

The point in my view is to be able to opt of the standard toolkit when
in those special cases where it doesn't quite fit, without having to
opt out of the language completely.

And so I imagine most generic programming tasks in Nemerle would use
the .NET 2 generics. These aren't as flexible as C++ templates, let
alone D ones, but they have the advantage that they are closure based
and not prone to bloat. Nevertheless, there are useful things that you
can't easily do with them. It sounds like Nemerle lets you work around
that limitation when you need to.

Being a .NET thing, it's presumably more of an apps-level language
than systems-level, but maybe it has some transferrable ideas?

-- 
Remove 'wants' and 'nospam' from e-mail.
Nov 28 2006
prev sibling parent Leandro Lucarella <llucarella integratech.com.ar> writes:
Steve Horne escribió:

 On Mon, 27 Nov 2006 15:23:33 -0300, Leandro Lucarella
 <llucarella integratech.com.ar> wrote:
 

 The center of my thesis would be adding meta-classes[1] to D, and maybe 
 some other meta-programming features D could lack of (maybe some kind of 
 access to the AST like XLR[2]).


 
 I can't really offer any useful support, but if I understand what you
 are saying, I can offer encouragement.


Thanks for your support. Any other thing to say about the original topic?

I've noticed a tendency of this list to get rather silly[1]... I mean, 
off-topic =)

[1] I should posted this to a Python mailing list ;)

-- 
Leandro Lucarella
Integratech S.A.
4571-5252
Nov 30 2006