www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - D meta language ?

reply Georg Wrede <georg.wrede nospam.org> writes:
I've been thinkin hard on a meta language.

I've talked with folks, amongst them prof. Jaakko Järvi, who actually 
met Walter at the C++ meeting last fall.

Bad news: I seem to have spilled the beans, big time. After asking his 
opinion about some of the core concepts I'd been cooking, it occurred to 
me that -- this guy's office is down the hall from Bjarne Stroustrup!

If these guys stop dwelling _within_ the C++ template jungle, and start 
looking at it from above, then it'll be just 18 months before one of 
them holds a ground breaking lecture.   :-(   :-(


D meta language is an important thing. But now it is also urgent.

Is there a mailing list for it, or should I continue on my own track?
Mar 09 2005
next sibling parent reply =?ISO-8859-1?Q?Anders_F_Bj=F6rklund?= <afb algonet.se> writes:
Georg Wrede wrote:


 I've been thinkin hard on a meta language.


[...]

 D meta language is an important thing. But now it is also urgent.


What would the "D meta language" be ? Something like YACC, but for D ?
(maybe related to DGrammar http://www.dsource.org/projects/dgrammar/ ?)

Or something completely different ?

By definition it's "a language that can be used to describe languages"
Just wondering what your little project was, and why it was urgent...

--anders
Mar 09 2005
next sibling parent reply Georg Wrede <georg.wrede nospam.org> writes:
Anders F Björklund wrote:

 Georg Wrede wrote:
 

 I've been thinkin hard on a meta language.


 
 [...]
 

 D meta language is an important thing. But now it is also urgent.


 
 
 What would the "D meta language" be ? Something like YACC, but for D ?
 (maybe related to DGrammar http://www.dsource.org/projects/dgrammar/ ?)


BTW, where do I get the D grammar itself? (Other than copy/pasting from 
the docs, piece by piece. ;-)  )


 Or something completely different ?
 
 By definition it's "a language that can be used to describe languages"
 Just wondering what your little project was, and why it was urgent...


Well, the C++ preprocessor has grown into a language.

It's even Turing complete, so nobody can dispute it being a language on 
its own. The template system now is really a meta language. One could 
say that the STL is a language (of sorts), too, written with the syntax 
of the preprocessor/template grammar.

What I've been thinking of, is something like a preprocessor that is 
aware of the language. (At least, for the time being it would be a (more 
or less) preprocessor.) The goal is, however, to have something that is 
a genuine part of the compiler, and that is intimately "aware" of the 
"underlying" language. And could conduct two-way communication with "the 
rest of" the compiler.

"Preprocessor" here being merely a temporary technical solution.

The way I see the state of language development today, is that real 
productivity, real power of expression, real versatility, and adequate 
abstraction -- all should be possible using a single medium. Now, there 
have been numeros ambitious enterprises in this arena. Quite a few 
passing fads have been hyped as "the ultimate tool for _everything_".

Lisp, for example, is capable of expressing anything at all, with 
relative ease too (at least for the lisp gurus). But we don't see Lisp 
running everything. With C you can do "everything" too. But nobody uses 
it for abstract or high level stuff.

A single language cannot fit everything any more. The fields of 
application and levels of abstraction are way too disparate for a single 
syntax to manage, if we want it to still be usable for as much as 
possible. (And if we don't, the competition will!)

The time has come for languages that are genuine compounds of two 
separate levels of abstraction. Having two separate syntaxes "for one 
language" is no big deal today. C++ does it all the time. D has it (so 
far) in a smaller scale, if we think of D templates combined with 
mixins. The rules of this are different from "plain" D. Many machine 
code assemblers have had a two-level syntax for ages. The one I've 
played with, was Borland's TASM. Now, with TASM you could even do OO! 
The syntax and even support was there. And this was like 15 years ago.

Having two syntaxes lets each of these be tuned and optimised for its 
own half of the cake. Also, having two syntaxes helps keep a kind of 
separation within the applications written. This actually resembles what 
many programmers do right now: do the nitty gritty in C, make it a DLL, 
then do the UI or program logic in, say Python, or even Euphoria.

The D philosophy fits nicely here. D has always tried to be a language 
where you don't need a preprocessor. Somehow though, we seem to have a 
long way to go. For example, here's a snippet from the DMD distribution 
(written in C++, but anyway):

Expression *Parser::parseAssignExp()
{   Expression *e;
     Expression *e2;
     Loc loc;

     e = parseCondExp();
     while (1)
     {
         loc = this->loc;
         switch (token.value)
         {
#define X(tok,ector) case tok:  nextToken(); \
   e2 = parseAssignExp(); e = new ector(loc,e,e2); continue;

             X(TOKassign,    AssignExp);
             X(TOKaddass,    AddAssignExp);
             X(TOKminass,    MinAssignExp);
             X(TOKmulass,    MulAssignExp);
             X(TOKdivass,    DivAssignExp);
             X(TOKmodass,    ModAssignExp);
             X(TOKandass,    AndAssignExp);
             X(TOKorass,     OrAssignExp);
             X(TOKxorass,    XorAssignExp);
             X(TOKshlass,    ShlAssignExp);
             X(TOKshrass,    ShrAssignExp);
             X(TOKushrass,   UshrAssignExp);
             X(TOKcatass,    CatAssignExp);

#undef X
             default:                 break;
         }
         break;
     }     return e;
}

Doing this (what Walter's done here with the preprocessor) is just one 
example of what I want to be easy in D, too, without a separate 
preprocessor.

Another example would be a mathematician. He defines a notation to use 
in source code, and then is able to use it. Of course, one might 
criticize that for illegibility to the normal programmer. But more 
important is that it is legible to the professionals in their own field.

Say someone really needs to have (the infamous?) unquoted regular 
expression syntax, which was discussed here a couple of weeks ago. He 
could write a definition, and then just use any regular expressions 
within D code.

And I can't even begin to imagine what people like prof. Järvi could 
conjure up, given a tool like this.

So, whether easy or hairy stuff is needed, this would give more 
expressive power to D on so many new levels.

Our "metalanguage" should be aware of its surroundings. __LINE__ etc. 
are only the beginning. It should be able to get any information a D 
program can get -- about files, the machine, whatever. And all at 
compile time.

It should make the (revered, adminred, feared, sworn at) C++ template 
system look like an enormously hard to use toy.

Done right, it would be easy to grasp, easy to use, and therefore less 
error prone.
Mar 09 2005
next sibling parent reply xs0 <xs0 xs0.com> writes:
But what would it look like? I really like the idea of a meta-language, 
I just can't come up with something that fits in D and at the same time 
makes it extensible..

As I see it, two issues are involved:
- being able to specify new syntax, both locally (as the X example 
below) and globally, where the main problem is - what is the syntax for 
syntax, and how do you make it self-extensible; if it's not 
self-extensible, it's just another layer of templates, I guess..

- being able to modify existing code, because just defining syntax 
expansions (like X below) is not enough. For example, if you wanted to 
add aspects, you'd need to define the syntax, have the compiler parse 
the source, then do something like "for each aspect that is defined, go 
through all the method and class declarations, find those that match, 
and insert the code from the aspect there and there if that and that". 
Now that's basically a program in itself, so it's a good question 
whether it is not better to simply open the compiler to plugins, than to 
try to fit this in the language and have the compiler be practically a 
virtual machine..

Even after you're able to do that, there are issues of collisions (what 
if two meta-things want to modify the same, say, class?), infinite loops 
(code A produces B, code B produces Aa, which becomes Ba, which becomes 
Aaa, ...), optimization, etc.

Especially the issue of optimization seems a big problem to me... I 
mean, "A x B" where x is some nice unicode symbol is not that much 
better than mul(A,B) that people would swallow like 2x slower running 
time over it, in the case the compiler won't be able to optimize it. And 
AFAIK, the whole optimization thing is more or less quite specific cases 
of what gets changed to what (I may be wrong on this, I'm no expert).

OTOH, perhaps it's not that big a deal - there could still be major 
gains in productivity, which can often be far more important than speed..

For a final thought, I guess a real victory would be if the 
meta-language was so powerful, that the entire D we have now could be 
expressed in it (except perhaps some really really basic constructs -- 
you do need something to build with in the first place).


xs0


Georg Wrede wrote:

 Anders F Björklund wrote:
 

 Georg Wrede wrote:


 I've been thinkin hard on a meta language.



 [...]


 D meta language is an important thing. But now it is also urgent.




 What would the "D meta language" be ? Something like YACC, but for D ?
 (maybe related to DGrammar http://www.dsource.org/projects/dgrammar/ ?)


 
 
 BTW, where do I get the D grammar itself? (Other than copy/pasting from 
 the docs, piece by piece. ;-)  )
 

 Or something completely different ?

 By definition it's "a language that can be used to describe languages"
 Just wondering what your little project was, and why it was urgent...


 
 
 Well, the C++ preprocessor has grown into a language.
 
 It's even Turing complete, so nobody can dispute it being a language on 
 its own. The template system now is really a meta language. One could 
 say that the STL is a language (of sorts), too, written with the syntax 
 of the preprocessor/template grammar.
 
 What I've been thinking of, is something like a preprocessor that is 
 aware of the language. (At least, for the time being it would be a (more 
 or less) preprocessor.) The goal is, however, to have something that is 
 a genuine part of the compiler, and that is intimately "aware" of the 
 "underlying" language. And could conduct two-way communication with "the 
 rest of" the compiler.
 
 "Preprocessor" here being merely a temporary technical solution.
 
 The way I see the state of language development today, is that real 
 productivity, real power of expression, real versatility, and adequate 
 abstraction -- all should be possible using a single medium. Now, there 
 have been numeros ambitious enterprises in this arena. Quite a few 
 passing fads have been hyped as "the ultimate tool for _everything_".
 
 Lisp, for example, is capable of expressing anything at all, with 
 relative ease too (at least for the lisp gurus). But we don't see Lisp 
 running everything. With C you can do "everything" too. But nobody uses 
 it for abstract or high level stuff.
 
 A single language cannot fit everything any more. The fields of 
 application and levels of abstraction are way too disparate for a single 
 syntax to manage, if we want it to still be usable for as much as 
 possible. (And if we don't, the competition will!)
 
 The time has come for languages that are genuine compounds of two 
 separate levels of abstraction. Having two separate syntaxes "for one 
 language" is no big deal today. C++ does it all the time. D has it (so 
 far) in a smaller scale, if we think of D templates combined with 
 mixins. The rules of this are different from "plain" D. Many machine 
 code assemblers have had a two-level syntax for ages. The one I've 
 played with, was Borland's TASM. Now, with TASM you could even do OO! 
 The syntax and even support was there. And this was like 15 years ago.
 
 Having two syntaxes lets each of these be tuned and optimised for its 
 own half of the cake. Also, having two syntaxes helps keep a kind of 
 separation within the applications written. This actually resembles what 
 many programmers do right now: do the nitty gritty in C, make it a DLL, 
 then do the UI or program logic in, say Python, or even Euphoria.
 
 The D philosophy fits nicely here. D has always tried to be a language 
 where you don't need a preprocessor. Somehow though, we seem to have a 
 long way to go. For example, here's a snippet from the DMD distribution 
 (written in C++, but anyway):
 
 Expression *Parser::parseAssignExp()
 {   Expression *e;
     Expression *e2;
     Loc loc;
 
     e = parseCondExp();
     while (1)
     {
         loc = this->loc;
         switch (token.value)
         {
 #define X(tok,ector) case tok:  nextToken(); \
   e2 = parseAssignExp(); e = new ector(loc,e,e2); continue;
 
             X(TOKassign,    AssignExp);
             X(TOKaddass,    AddAssignExp);
             X(TOKminass,    MinAssignExp);
             X(TOKmulass,    MulAssignExp);
             X(TOKdivass,    DivAssignExp);
             X(TOKmodass,    ModAssignExp);
             X(TOKandass,    AndAssignExp);
             X(TOKorass,     OrAssignExp);
             X(TOKxorass,    XorAssignExp);
             X(TOKshlass,    ShlAssignExp);
             X(TOKshrass,    ShrAssignExp);
             X(TOKushrass,   UshrAssignExp);
             X(TOKcatass,    CatAssignExp);
 
 #undef X
             default:                 break;
         }
         break;
     }     return e;
 }
 
 Doing this (what Walter's done here with the preprocessor) is just one 
 example of what I want to be easy in D, too, without a separate 
 preprocessor.
 
 Another example would be a mathematician. He defines a notation to use 
 in source code, and then is able to use it. Of course, one might 
 criticize that for illegibility to the normal programmer. But more 
 important is that it is legible to the professionals in their own field.
 
 Say someone really needs to have (the infamous?) unquoted regular 
 expression syntax, which was discussed here a couple of weeks ago. He 
 could write a definition, and then just use any regular expressions 
 within D code.
 
 And I can't even begin to imagine what people like prof. Järvi could 
 conjure up, given a tool like this.
 
 So, whether easy or hairy stuff is needed, this would give more 
 expressive power to D on so many new levels.
 
 Our "metalanguage" should be aware of its surroundings. __LINE__ etc. 
 are only the beginning. It should be able to get any information a D 
 program can get -- about files, the machine, whatever. And all at 
 compile time.
 
 It should make the (revered, adminred, feared, sworn at) C++ template 
 system look like an enormously hard to use toy.
 
 Done right, it would be easy to grasp, easy to use, and therefore less 
 error prone.
Mar 10 2005
next sibling parent reply Georg Wrede <georg.wrede nospam.org> writes:
xs0 wrote:

 But what would it look like? I really like the idea of a meta-language, 
 I just can't come up with something that fits in D and at the same time 
 makes it extensible..


Well, at this point I think the best thing to do is to carefully think 
about what we _want_ it to be able to do. IMHO, if we try to come up 
with, say a syntax, or a way of "blending into D", then we unnecessarily 
create restraints. Both syntaxwise, but more important, we fence us into 
our newly made box, and that's the end of "out of the box" thinking. 
Right now is the time for the latter, and the longer we keep at it, the 
better we understand _what_ has to be done. Once that is clear, only 
then can we allow ourselves to think about _how_.


 As I see it, two issues are involved:
 - being able to specify new syntax, both locally (as the X example 
 below) and globally, where the main problem is - what is the syntax for 
 syntax, and how do you make it self-extensible; if it's not 
 self-extensible, it's just another layer of templates, I guess..


We shouldn't care! Just assume whatever we need to, we'll come up with 
in due time. First we need a good list of what we _want_. Suggestions 
and help, even (possibly hard) criticism is welcome too!

The C crowd _started_ with a syntax. Then kept appending to it, and we 
all see where that leads. They've never had a clue of where they are 
ultimately aiming. Well, we'gre gonna do it the other way around.


 - being able to modify existing code, because just defining syntax 
 expansions (like X below) is not enough. 


Right!


 For example, if you wanted to 
 add aspects, you'd need to define the syntax, have the compiler parse 
 the source, then do something like "for each aspect that is defined, go 
 through all the method and class declarations, find those that match, 
 and insert the code from the aspect there and there if that and that". 


The meta language idea should be able to accommodate that. But not like 
we specifically thought of aspects in particular, rather like if it is 
what we wanted it to be, and only after that somebody invents aspects, 
the meta language should handle that too. Or, more to the point, the 
programmer who wants to implement aspects. And what he has to write in 
meta to make it so should not be incomprehensible to others.


 Now that's basically a program in itself, so it's a good question 


Ecactly. And that is what I've meant each time I've mentioned Turing! I 
see no way around it. -- The C++ preprocessor language had reached 
Turing level without any of the C++ honchos ever even noticing. Several 
_years_ later somebody just happened to discover the fact.

My view is that cpp being TC is not a coincidence. Rather, it is a 
requisite. And therefore _we_ take that as a starting point.


 whether it is not better to simply open the compiler to plugins, than to 
 try to fit this in the language and have the compiler be practically a 
 virtual machine..


I would like to know more of this plugin idea.


 Even after you're able to do that, there are issues of collisions (what 
 if two meta-things want to modify the same, say, class?), infinite loops 
 (code A produces B, code B produces Aa, which becomes Ba, which becomes 
 Aaa, ...), optimization, etc.


We'll take care of them at the time. Had Walter got scared of circular 
imports too early -- but he didn't. Same with meta.


 Especially the issue of optimization seems a big problem to me... I 
 mean, "A x B" where x is some nice unicode symbol is not that much 
 better than mul(A,B) that people would swallow like 2x slower running 
 time over it, in the case the compiler won't be able to optimize it. And 
 AFAIK, the whole optimization thing is more or less quite specific cases 
 of what gets changed to what (I may be wrong on this, I'm no expert).
 
 OTOH, perhaps it's not that big a deal - there could still be major 
 gains in productivity, which can often be far more important than speed..


Optimization I hope happens after the meta phase. Of course, that does 
not give us the right to make meta inefficient. :-)


 For a final thought, I guess a real victory would be if the 
 meta-language was so powerful, that the entire D we have now could be 
 expressed in it (except perhaps some really really basic constructs -- 
 you do need something to build with in the first place).


Hmm. I wouldn't bet against it. OTOH, I'm not ready to have it as a goal 
either. Not at least right now.


 Georg Wrede wrote:
 

 Anders F Björklund wrote:


 Georg Wrede wrote:


 I've been thinkin hard on a meta language.




 [...]


 D meta language is an important thing. But now it is also urgent.





 What would the "D meta language" be ? Something like YACC, but for D ?
 (maybe related to DGrammar http://www.dsource.org/projects/dgrammar/ ?)




 BTW, where do I get the D grammar itself? (Other than copy/pasting 
 from the docs, piece by piece. ;-)  )


 Or something completely different ?

 By definition it's "a language that can be used to describe languages"
 Just wondering what your little project was, and why it was urgent...




 Well, the C++ preprocessor has grown into a language.

 It's even Turing complete, so nobody can dispute it being a language 
 on its own. The template system now is really a meta language. One 
 could say that the STL is a language (of sorts), too, written with the 
 syntax of the preprocessor/template grammar.

 What I've been thinking of, is something like a preprocessor that is 
 aware of the language. (At least, for the time being it would be a 
 (more or less) preprocessor.) The goal is, however, to have something 
 that is a genuine part of the compiler, and that is intimately "aware" 
 of the "underlying" language. And could conduct two-way communication 
 with "the rest of" the compiler.

 "Preprocessor" here being merely a temporary technical solution.

 The way I see the state of language development today, is that real 
 productivity, real power of expression, real versatility, and adequate 
 abstraction -- all should be possible using a single medium. Now, 
 there have been numeros ambitious enterprises in this arena. Quite a 
 few passing fads have been hyped as "the ultimate tool for _everything_".

 Lisp, for example, is capable of expressing anything at all, with 
 relative ease too (at least for the lisp gurus). But we don't see Lisp 
 running everything. With C you can do "everything" too. But nobody 
 uses it for abstract or high level stuff.

 A single language cannot fit everything any more. The fields of 
 application and levels of abstraction are way too disparate for a 
 single syntax to manage, if we want it to still be usable for as much 
 as possible. (And if we don't, the competition will!)

 The time has come for languages that are genuine compounds of two 
 separate levels of abstraction. Having two separate syntaxes "for one 
 language" is no big deal today. C++ does it all the time. D has it (so 
 far) in a smaller scale, if we think of D templates combined with 
 mixins. The rules of this are different from "plain" D. Many machine 
 code assemblers have had a two-level syntax for ages. The one I've 
 played with, was Borland's TASM. Now, with TASM you could even do OO! 
 The syntax and even support was there. And this was like 15 years ago.

 Having two syntaxes lets each of these be tuned and optimised for its 
 own half of the cake. Also, having two syntaxes helps keep a kind of 
 separation within the applications written. This actually resembles 
 what many programmers do right now: do the nitty gritty in C, make it 
 a DLL, then do the UI or program logic in, say Python, or even Euphoria.

 The D philosophy fits nicely here. D has always tried to be a language 
 where you don't need a preprocessor. Somehow though, we seem to have a 
 long way to go. For example, here's a snippet from the DMD 
 distribution (written in C++, but anyway):

 Expression *Parser::parseAssignExp()
 {   Expression *e;
     Expression *e2;
     Loc loc;

     e = parseCondExp();
     while (1)
     {
         loc = this->loc;
         switch (token.value)
         {
 #define X(tok,ector) case tok:  nextToken(); \
   e2 = parseAssignExp(); e = new ector(loc,e,e2); continue;

             X(TOKassign,    AssignExp);
             X(TOKaddass,    AddAssignExp);
             X(TOKminass,    MinAssignExp);
             X(TOKmulass,    MulAssignExp);
             X(TOKdivass,    DivAssignExp);
             X(TOKmodass,    ModAssignExp);
             X(TOKandass,    AndAssignExp);
             X(TOKorass,     OrAssignExp);
             X(TOKxorass,    XorAssignExp);
             X(TOKshlass,    ShlAssignExp);
             X(TOKshrass,    ShrAssignExp);
             X(TOKushrass,   UshrAssignExp);
             X(TOKcatass,    CatAssignExp);

 #undef X
             default:                 break;
         }
         break;
     }     return e;
 }

 Doing this (what Walter's done here with the preprocessor) is just one 
 example of what I want to be easy in D, too, without a separate 
 preprocessor.

 Another example would be a mathematician. He defines a notation to use 
 in source code, and then is able to use it. Of course, one might 
 criticize that for illegibility to the normal programmer. But more 
 important is that it is legible to the professionals in their own field.

 Say someone really needs to have (the infamous?) unquoted regular 
 expression syntax, which was discussed here a couple of weeks ago. He 
 could write a definition, and then just use any regular expressions 
 within D code.

 And I can't even begin to imagine what people like prof. Järvi could 
 conjure up, given a tool like this.

 So, whether easy or hairy stuff is needed, this would give more 
 expressive power to D on so many new levels.

 Our "metalanguage" should be aware of its surroundings. __LINE__ etc. 
 are only the beginning. It should be able to get any information a D 
 program can get -- about files, the machine, whatever. And all at 
 compile time.

 It should make the (revered, adminred, feared, sworn at) C++ template 
 system look like an enormously hard to use toy.

 Done right, it would be easy to grasp, easy to use, and therefore less 
 error prone.
Mar 10 2005
parent reply xs0 <xs0 xs0.com> writes:
Georg Wrede wrote:

 xs0 wrote:
 

 But what would it look like? I really like the idea of a 
 meta-language, I just can't come up with something that fits in D and 
 at the same time makes it extensible..


 
 
 Well, at this point I think the best thing to do is to carefully think 
 about what we _want_ it to be able to do. IMHO, if we try to come up 
 with, say a syntax, or a way of "blending into D", then we unnecessarily 
 create restraints. Both syntaxwise, but more important, we fence us into 
 our newly made box, and that's the end of "out of the box" thinking. 
 Right now is the time for the latter, and the longer we keep at it, the 
 better we understand _what_ has to be done. Once that is clear, only 
 then can we allow ourselves to think about _how_.


Well, the first thing I want is macros somewhat like C's, except not on 
textual level, so you can't define something like

#define BLAH(a,b) a + b +


I still want to be able to generate new identifiers:

#define

and SUM(int,double) becomes sum_int_double(int a, double b) {...}


Third, I wan't this:

func!(__myBasicTypes__).doit();

to get expanded to this:

func!(int).doit();
func!(double).doit();
func!(MyClass).doit();

for any "func" and "doit" i.e. whenever __myBasicTypes__ appears as a 
type, I want the expression to be expanded to three expressions with 
appropriate types. And not just in expressions, why not have template 
specializations for multiple types at the same time? Or function 
definitions:

func(__anyCharType[]) {
   // do something
}

becomes

func(char[])
func(wchar[])
func(dchar[])


Fourth, I want to be able to insert code from "outside" the 
method/func/class/whatever. Like:

#match C:class(*) {
     #prepend_or_create_method (C, static this()) {
         globals.all_classes~=#get(C.fullName);
     }
}

If we manage that, we rock (and Walter doesn't have to implement 
reflection :)



 As I see it, two issues are involved:
 - being able to specify new syntax, both locally (as the X example 
 below) and globally, where the main problem is - what is the syntax 
 for syntax, and how do you make it self-extensible; if it's not 
 self-extensible, it's just another layer of templates, I guess..


 
 We shouldn't care! Just assume whatever we need to, we'll come up with 
 in due time. First we need a good list of what we _want_. Suggestions 
 and help, even (possibly hard) criticism is welcome too!


Well, I think that just wildly brainstorming isn't going to be very 
productive. We should instead start by thinking of a few examples of 
what specifically we'd want to do, and conjure some way of encoding 
that. After the initial "spec" is done, we can see what else we'd like 
to do, and check whether it's already possible. If it is, great, if not, 
we amend and revise the "spec" to include both the old and new 
functionality. We can repeat this as long as we want, and in the end 
we'll have a solid syntax+semantics for the meta-D, and all that is left 
is implementing it, which will hopefully be relatively straightforward. 
OTOH, if we just dream about something, we'll only have dreams in the end.



 The C crowd _started_ with a syntax. Then kept appending to it, and we 
 all see where that leads. They've never had a clue of where they are 
 ultimately aiming. Well, we'gre gonna do it the other way around.


I'd say C started just as you're proposing - they thought of what they 
wanted to do and did it. The problem was that they later wanted to do 
additional stuff, and had to append to what was there already.

I know you're trying to say that we should have something that is 
infinitely self-extensible, so that we'll never have to "append" 
something to it, but even if we manage to do that, we'll eventually run 
into the same problems. I mean, there's only so many ways you can 
overload, say, '(' and ')', before it becomes ambiguous what exactly 
you're trying to do. After that happens, you need new symbols/syntax for 
your new feature, but eventually you still run out of them. In other 
words, even if you have an infinitely extensible language, you can't 
infinitely extend it (at least realistically - noone will want to use 
250 different operators, even if they're defined in Unicode).



 - being able to modify existing code, because just defining syntax 
 expansions (like X below) is not enough. 


 
 Right!


Hmm, after thinking about it, I don't think that's possible at all. Say 
you did manage to define aspects. You can't use them in the aspect 
defining code, so that's limitation number one (because it can't process 
itself - if an aspect is needed for aspects to work, they can't work, 
because they don't work before they work, or something :). Second, if 
some other extension during processing sees an aspect, it will skip it 
because it doesn't know what to do with it. Later, when the aspect gets 
applied, the first extension won't see the code again, so won't work 
there. If you do run it again, it is possible that it will break what 
the aspect extension did, which is possibly even worse.



 For example, if you wanted to add aspects, you'd need to define the 
 syntax, have the compiler parse the source, then do something like 
 "for each aspect that is defined, go through all the method and class 
 declarations, find those that match, and insert the code from the 
 aspect there and there if that and that". 


 
 
 The meta language idea should be able to accommodate that. But not like 
 we specifically thought of aspects in particular, rather like if it is 
 what we wanted it to be, and only after that somebody invents aspects, 
 the meta language should handle that too. Or, more to the point, the 
 programmer who wants to implement aspects. And what he has to write in 
 meta to make it so should not be incomprehensible to others.


Well, we can't make a language for things we can't foresee (I don't mean 
stuff like aspects specifically, but more like "something that modifies 
other code by pattern matching on identifiers")



 Now that's basically a program in itself, so it's a good question 


 
 Exactly. And that is what I've meant each time I've mentioned Turing! I 
 see no way around it. -- The C++ preprocessor language had reached 
 Turing level without any of the C++ honchos ever even noticing. Several 
 _years_ later somebody just happened to discover the fact.


Hmm, when you say C++ preprocessor, you do mean its template system, right?



 whether it is not better to simply open the compiler to plugins, than 
 to try to fit this in the language and have the compiler be 
 practically a virtual machine..


 
 I would like to know more of this plugin idea.


Well, if the compiler worked in some really well-defined way and had an 
open interface to extensions (as in .DLLs or similar), it could let the 
plugins work on its internal structures (like definitions for tokens and 
grammar before parsing and AST after parsing), which would basically 
mean "anything" is possible. But its not as simple as it sounds, 
especially the grammar part, grammars are very delicate...



 Even after you're able to do that, there are issues of collisions 
 (what if two meta-things want to modify the same, say, class?), 
 infinite loops (code A produces B, code B produces Aa, which becomes 
 Ba, which becomes Aaa, ...), optimization, etc.


 
 We'll take care of them at the time. Had Walter got scared of circular 
 imports too early -- but he didn't. Same with meta.


Well, it's not quite the same :) And those are issues that need to be 
considered straight from the start (esp. collisions), if you want 
something successful.


xs0
Mar 11 2005
parent Georg Wrede <georg.wrede nospam.org> writes:
xs0 wrote:

 Georg Wrede wrote:

 xs0 wrote:
 Well, at this point I think the best thing to do is to carefully think 
 about what we _want_ it to be able to do. IMHO, if we try to come up 
 with, say a syntax, or a way of "blending into D", then we 
 unnecessarily create restraints. Both syntaxwise, but more important, 
 we fence us into our newly made box, and that's the end of "out of the 
 box" thinking. Right now is the time for the latter, and the longer we 
 keep at it, the better we understand _what_ has to be done. Once that 
 is clear, only then can we allow ourselves to think about _how_.




Good list below!


 Well, the first thing I want is macros somewhat like C's, except not on 
 textual level, so you can't define something like
 
 #define BLAH(a,b) a + b +


That ought to be easy.

Definitely local scope is what should be used. Both textual level, and 
global scope in general "are for the other guys"!


 I still want to be able to generate new identifiers:
 
 #define
 
 and SUM(int,double) becomes sum_int_double(int a, double b) {...}


Ha! This revealed to me that the meta system could have a library of its 
own. Some of the usual things (like the one above) could be there all 
along. Then folks wouldn't have to actually write them, just look them 
up in the manual, and then activate.


 Third, I wan't this:
 
 func!(__myBasicTypes__).doit();
 
 to get expanded to this:
 
 func!(int).doit();
 func!(double).doit();
 func!(MyClass).doit();


So you are creating a method for a bunch of classes from a template?
Resembles multiple inheritance, and aspects?

This would definitely be doable in C++. But with D having method 
definitions inside the classes, I can't say off hand.

Probably it would still be doable, but you'd possibly need more work 
than with inheritance. (Here I'm deliberately overlooking the difference 
between int and double, vs classes.) You may want to be more specific on 
this one.


 for any "func" and "doit" i.e. whenever __myBasicTypes__ appears as a 
 type, I want the expression to be expanded to three expressions with 
 appropriate types. And not just in expressions, why not have template 
 specializations for multiple types at the same time? Or function 
 definitions:
 
 func(__anyCharType[]) {
   // do something
 }
 
 becomes
 
 func(char[])
 func(wchar[])
 func(dchar[])


That depends on the "// do something". That is, if it can handle all of 
the types. Then it should be doable.


 Fourth, I want to be able to insert code from "outside" the 
 method/func/class/whatever. Like:
 
 #match C:class(*) {
     #prepend_or_create_method (C, static this()) {
         globals.all_classes~=#get(C.fullName);
     }
 }


Gee, a question I have no answer for! You'd have to ask Walter about this.


 If we manage that, we rock (and Walter doesn't have to implement 
 reflection :)


I wouldn't be surprised if making this work would be the same amount of 
work for Walter.


 As I see it, two issues are involved:
 - being able to specify new syntax, both locally (as the X example 
 below) and globally, where the main problem is - what is the syntax 
 for syntax, and how do you make it self-extensible; if it's not 
 self-extensible, it's just another layer of templates, I guess..






My answer here missed your main point. :-(


 We shouldn't care! Just assume whatever we need to, we'll come up with 
 in due time. First we need a good list of what we _want_. Suggestions 
 and help, even (possibly hard) criticism is welcome too!


 
 Well, I think that just wildly brainstorming isn't going to be very 
 productive. We should instead start by thinking of a few examples of 
 what specifically we'd want to do, and conjure some way of encoding 
 that.


True. But just like with programming in general, the longer you abstain 
from the keyboard, the faster the overall app gets ready, and with 
better quality.


 After the initial "spec" is done, we can see what else we'd like 
 to do, and check whether it's already possible. If it is, great, if not, 
 we amend and revise the "spec" to include both the old and new 
 functionality. We can repeat this as long as we want, and in the end 
 we'll have a solid syntax+semantics for the meta-D, and all that is left 
 is implementing it, which will hopefully be relatively straightforward.


Yes. And this is a deep issue, so we ought to get folks tinking hard, 
before we know what we want in the spec.


 OTOH, if we just dream about something, we'll only have dreams in the end.


:-)


 The C crowd _started_ with a syntax. Then kept appending to it, and we 
 all see where that leads. They've never had a clue of where they are 
 ultimately aiming. Well, we'gre gonna do it the other way around.


 
 I'd say C started just as you're proposing - they thought of what they 
 wanted to do and did it. The problem was that they later wanted to do 
 additional stuff, and had to append to what was there already.
 
 I know you're trying to say that we should have something that is 
 infinitely self-extensible, so that we'll never have to "append" 
 something to it, 


Yes.


 but even if we manage to do that, we'll eventually run 
 into the same problems. I mean, there's only so many ways you can 
 overload, say, '(' and ')', before it becomes ambiguous what exactly 
 you're trying to do. After that happens, you need new symbols/syntax for 
 your new feature, but eventually you still run out of them. In other 
 words, even if you have an infinitely extensible language, you can't 
 infinitely extend it (at least realistically - noone will want to use 
 250 different operators, even if they're defined in Unicode).


That's where locality of scope comes to rescue. With this system 
statisticians, mathematicians, physicists, even astrologists, could 
reuse the same symbols -- just like in paper books today.


 - being able to modify existing code, because just defining syntax 
 expansions (like X below) is not enough. 


 Right!


 
 Hmm, after thinking about it, I don't think that's possible at all. Say 
 you did manage to define aspects. You can't use them in the aspect 
 defining code, so that's limitation number one (because it can't process 
 itself - if an aspect is needed for aspects to work, they can't work, 
 because they don't work before they work, or something :). Second, if 
 some other extension during processing sees an aspect, it will skip it 
 because it doesn't know what to do with it. Later, when the aspect gets 
 applied, the first extension won't see the code again, so won't work 
 there. If you do run it again, it is possible that it will break what 
 the aspect extension did, which is possibly even worse.
 

 For example, if you wanted to add aspects, you'd need to define the 
 syntax, have the compiler parse the source, then do something like 
 "for each aspect that is defined, go through all the method and class 
 declarations, find those that match, and insert the code from the 
 aspect there and there if that and that". 


 The meta language idea should be able to accommodate that. But not 
 like we specifically thought of aspects in particular, rather like if 
 it is what we wanted it to be, and only after that somebody invents 
 aspects, the meta language should handle that too. Or, more to the 
 point, the programmer who wants to implement aspects. And what he has 
 to write in meta to make it so should not be incomprehensible to others.


 
 Well, we can't make a language for things we can't foresee (I don't mean 
 stuff like aspects specifically, but more like "something that modifies 
 other code by pattern matching on identifiers")


About aspects, I really need more knowledge on them. So, on second 
thought I won't comment on them for the time being.

"Something that modifies other code by pattern matching on identifiers" 
is more or less what I'm thinking about in general with this meta thing.


 Now that's basically a program in itself, so it's a good question 


 Exactly. And that is what I've meant each time I've mentioned Turing! 
 I see no way around it. -- The C++ preprocessor language had reached 
 Turing level without any of the C++ honchos ever even noticing. 
 Several _years_ later somebody just happened to discover the fact.


 
 Hmm, when you say C++ preprocessor, you do mean its template system, right?


Yes. (Sorry for the ambiguous expression.)


 whether it is not better to simply open the compiler to plugins, than 
 to try to fit this in the language and have the compiler be 
 practically a virtual machine..


 I would like to know more of this plugin idea.


 
 Well, if the compiler worked in some really well-defined way and had an 
 open interface to extensions (as in .DLLs or similar), it could let the 
 plugins work on its internal structures (like definitions for tokens and 
 grammar before parsing and AST after parsing), which would basically 
 mean "anything" is possible. But its not as simple as it sounds, 
 especially the grammar part, grammars are very delicate...


Currently I imagine the meta thing and the compiler would work in 
cooperation. The meta would not change the compiler's structures, only 
ask it to add or change things. Hopefully not even that -- if I'm at all 
on the right track here.


 Even after you're able to do that, there are issues of collisions 
 (what if two meta-things want to modify the same, say, class?), 
 infinite loops (code A produces B, code B produces Aa, which becomes 
 Ba, which becomes Aaa, ...), optimization, etc.


 We'll take care of them at the time. Had Walter got scared of circular 
 imports too early -- but he didn't. Same with meta.


 
 Well, it's not quite the same :) And those are issues that need to be 
 considered straight from the start (esp. collisions), if you want 
 something successful.


Yes. Circularities I just cavalierly assume Walter could handle. Or at 
least detect, and error on them. So it's more like a "syntax issue". I 
mean, how to aid the programmer in not unnecessarily creating them and 
stumbling over compiler errors because of them.

Collisions, well, to begin with, we'd need to have a defined order of 
things. That's something even the application writer needs to be clear 
on, right? Meta or no meta.

If there is a clear order to what is done, then collisions should become 
a very much smaller issue.
Mar 11 2005
prev sibling parent Georg Wrede <georg.wrede nospam.org> writes:
xs0 wrote:

 As I see it, two issues are involved:
 - being able to specify new syntax, both locally (as the X example 
 below) and globally, where the main problem is - what is the syntax for 
 syntax, and how do you make it self-extensible; if it's not 
 self-extensible, it's just another layer of templates, I guess..


Forgot to answer this.

Of course it has to be sef-extensible. I see no other option. The entire 
point of a Decent Meta Language is, to be able to cope with the unforeseen.

It would have to be self-extensible, and self-modifiable. -- Without 
imposing either on the down-to-earth regular user of it.

It's like, D allows you to do OO or do without OO, or mix them. The Meta 
shoud be the same.

-------------

We could view (with some effort and will power) D itself as an asm 
metalanguage.

   "D is another asm compiler, but it wraps many of the most-used things 
in a convenient meta language, and has an extensive (in this context 
anyway) library (phobos), to relieve the programmer of writing the most 
usual things over and over. It handles high-level I/O, the usual data 
structures, and conversions for you."

But because it is not self-extensible and not self-modifiable, it can 
only (in a broad sense) be used with the kind of problems its creators 
had in mind. (( A derogatory sounding way of saying `it's for stuff 
you'd use C, C++, Java etc. for`.))
Mar 11 2005
prev sibling parent reply "Charlie Patterson" <charliep1 excite.com> writes:
I'm not sure where you want to go with this meta-language but I'm here to 
burst your bubble.  (-:  I always wanted a meta-langauge... until I saw one. 
There is one for C++ and it is indecipherable.  It seems the movement in 
langagues is towards cleaning up and pairing down, not making them more 
"tricky."

Lots of people still claim C++ is too hard precisely because you don't know 
what it will do under the covers for a simple assignment statement.  And 
keep in mind C++ and even Java and D have some basic "meta programming" with 
operator= (opAssign) type options.

If you really, really want to add whole new calculus to the language, well 
that is what I thought I wanted until I saw an example.  Most people just 
get disturbed by creating it and even just being forced to use the results.

Finally, I've been on a tear for the last few years that people over-use 
languages anyway.  I think I got the idea from a Myer's C++ book (and also 
that Andrescu template book).  I was following along with all the insanely 
detailed examples of using private inheritance with a customized new 
operator, etc, etc, when they started adjusting the C++ vtable in order to 
create some lisp-like capabilities (non-portable obviously).  That was the 
last straw for me.  If you have to *beat* the language, then you need to 
simplify your app... or suffer the ugly consequences of maintaing it.  Or 
writing a new language!  But at some level of power, people just drop off.

My two cents.

"Georg Wrede" <georg.wrede nospam.org> wrote in message 
news:422F7E8C.7040509 nospam.org...

 Anders F Björklund wrote:

 Georg Wrede wrote:


 I've been thinkin hard on a meta language.


 [...]


 D meta language is an important thing. But now it is also urgent.



 What would the "D meta language" be ? Something like YACC, but for D ?
 (maybe related to DGrammar http://www.dsource.org/projects/dgrammar/ ?)


 BTW, where do I get the D grammar itself? (Other than copy/pasting from 
 the docs, piece by piece. ;-)  )


 Or something completely different ?

 By definition it's "a language that can be used to describe languages"
 Just wondering what your little project was, and why it was urgent...


 Well, the C++ preprocessor has grown into a language.

 It's even Turing complete, so nobody can dispute it being a language on 
 its own. The template system now is really a meta language. One could say 
 that the STL is a language (of sorts), too, written with the syntax of the 
 preprocessor/template grammar.

 What I've been thinking of, is something like a preprocessor that is aware 
 of the language. (At least, for the time being it would be a (more or 
 less) preprocessor.) The goal is, however, to have something that is a 
 genuine part of the compiler, and that is intimately "aware" of the 
 "underlying" language. And could conduct two-way communication with "the 
 rest of" the compiler.

 "Preprocessor" here being merely a temporary technical solution.

 The way I see the state of language development today, is that real 
 productivity, real power of expression, real versatility, and adequate 
 abstraction -- all should be possible using a single medium. Now, there 
 have been numeros ambitious enterprises in this arena. Quite a few passing 
 fads have been hyped as "the ultimate tool for _everything_".

 Lisp, for example, is capable of expressing anything at all, with relative 
 ease too (at least for the lisp gurus). But we don't see Lisp running 
 everything. With C you can do "everything" too. But nobody uses it for 
 abstract or high level stuff.

 A single language cannot fit everything any more. The fields of 
 application and levels of abstraction are way too disparate for a single 
 syntax to manage, if we want it to still be usable for as much as 
 possible. (And if we don't, the competition will!)

 The time has come for languages that are genuine compounds of two separate 
 levels of abstraction. Having two separate syntaxes "for one language" is 
 no big deal today. C++ does it all the time. D has it (so far) in a 
 smaller scale, if we think of D templates combined with mixins. The rules 
 of this are different from "plain" D. Many machine code assemblers have 
 had a two-level syntax for ages. The one I've played with, was Borland's 
 TASM. Now, with TASM you could even do OO! The syntax and even support was 
 there. And this was like 15 years ago.

 Having two syntaxes lets each of these be tuned and optimised for its own 
 half of the cake. Also, having two syntaxes helps keep a kind of 
 separation within the applications written. This actually resembles what 
 many programmers do right now: do the nitty gritty in C, make it a DLL, 
 then do the UI or program logic in, say Python, or even Euphoria.

 The D philosophy fits nicely here. D has always tried to be a language 
 where you don't need a preprocessor. Somehow though, we seem to have a 
 long way to go. For example, here's a snippet from the DMD distribution 
 (written in C++, but anyway):

 Expression *Parser::parseAssignExp()
 {   Expression *e;
     Expression *e2;
     Loc loc;

     e = parseCondExp();
     while (1)
     {
         loc = this->loc;
         switch (token.value)
         {
 #define X(tok,ector) case tok:  nextToken(); \
   e2 = parseAssignExp(); e = new ector(loc,e,e2); continue;

             X(TOKassign,    AssignExp);
             X(TOKaddass,    AddAssignExp);
             X(TOKminass,    MinAssignExp);
             X(TOKmulass,    MulAssignExp);
             X(TOKdivass,    DivAssignExp);
             X(TOKmodass,    ModAssignExp);
             X(TOKandass,    AndAssignExp);
             X(TOKorass,     OrAssignExp);
             X(TOKxorass,    XorAssignExp);
             X(TOKshlass,    ShlAssignExp);
             X(TOKshrass,    ShrAssignExp);
             X(TOKushrass,   UshrAssignExp);
             X(TOKcatass,    CatAssignExp);

 #undef X
             default:                 break;
         }
         break;
     }     return e;
 }

 Doing this (what Walter's done here with the preprocessor) is just one 
 example of what I want to be easy in D, too, without a separate 
 preprocessor.

 Another example would be a mathematician. He defines a notation to use in 
 source code, and then is able to use it. Of course, one might criticize 
 that for illegibility to the normal programmer. But more important is that 
 it is legible to the professionals in their own field.

 Say someone really needs to have (the infamous?) unquoted regular 
 expression syntax, which was discussed here a couple of weeks ago. He 
 could write a definition, and then just use any regular expressions within 
 D code.

 And I can't even begin to imagine what people like prof. Järvi could 
 conjure up, given a tool like this.

 So, whether easy or hairy stuff is needed, this would give more expressive 
 power to D on so many new levels.

 Our "metalanguage" should be aware of its surroundings. __LINE__ etc. are 
 only the beginning. It should be able to get any information a D program 
 can get -- about files, the machine, whatever. And all at compile time.

 It should make the (revered, adminred, feared, sworn at) C++ template 
 system look like an enormously hard to use toy.

 Done right, it would be easy to grasp, easy to use, and therefore less 
 error prone.
Mar 10 2005
parent reply Georg Wrede <georg.wrede nospam.org> writes:
Charlie Patterson wrote:

 I'm not sure where you want to go with this meta-language but I'm here to 
 burst your bubble.  (-:  


Thanks! It NEEDS to be stabbed at, wrought, wrinkled, etc. It's the only 
way we'll prune the chaff off. Or maybe prune the whole deal if it 
doesn't deserve to live.


 I always wanted a meta-langauge... until I saw one. 
 There is one for C++ and it is indecipherable.  It seems the movement in 
 langagues is towards cleaning up and pairing down, not making them more 
 "tricky."


True. And the C++ one is so gross, it makes everybody give up.


 Lots of people still claim C++ is too hard precisely because you don't know 
 what it will do under the covers for a simple assignment statement.  And 
 keep in mind C++ and even Java and D have some basic "meta programming" with 
 operator= (opAssign) type options.


Right. Now, if Java and D already have some of this, then I think it 
just proves we can't live without one.


 If you really, really want to add whole new calculus to the language, well 
 that is what I thought I wanted until I saw an example.  Most people just 
 get disturbed by creating it and even just being forced to use the results.


Naaw, a new calculus is not the target. What I want, is a way to, at one 
end express repetitions and patterns in source with something smarter 
than just plain writing it all out (c.f. the DMD #define example), and 
at the other, a way to increase the expressivenes of the, er, "combined" 
language.

And it ABSOLUTELY has to be easy to write and understand. For example, 
simple #defines in cpp are pretty easy to understand, even for those 
who've never written one. But C++ templates get just unreadable pretty 
fast. And I blame their specific template language for it. I don't 
believe it _has_ to be so difficult.

The way to express a solution should never have to be more complicated 
than the solution itself.


 Finally, I've been on a tear for the last few years that people over-use 
 languages anyway.  I think I got the idea from a Myer's C++ book (and also 
 that Andrescu template book).  I was following along with all the insanely 
 detailed examples of using private inheritance with a customized new 
 operator, etc, etc, when they started adjusting the C++ vtable in order to 
 create some lisp-like capabilities (non-portable obviously).  That was the 
 last straw for me.  If you have to *beat* the language, then you need to 
 simplify your app... or suffer the ugly consequences of maintaing it.  Or 
 writing a new language!  But at some level of power, people just drop off.


True. D is a "practical language for practical programmers". We need a 
practical meta language, that is practical for practical purposes. And 
if we can give it expressive power that does not run out of stream right 
after the first block, without making either simple or hard things 
harder than they _inherently_ are, then we've succeeded.
Mar 10 2005
parent reply "Matthew" <admin stlsoft.dot.dot.dot.dot.org> writes:
"Georg Wrede" <georg.wrede nospam.org> wrote in message 
news:4230C679.40509 nospam.org...

 Charlie Patterson wrote:

 I'm not sure where you want to go with this meta-language but I'm 
 here to burst your bubble.  (-:


 Thanks! It NEEDS to be stabbed at, wrought, wrinkled, etc. It's the 
 only way we'll prune the chaff off. Or maybe prune the whole deal if 
 it doesn't deserve to live.


Being the dullard that I am, I have a real hard time imagining what 
you're talking about absent examples.

Can you cook up a couple - just small one's be ok - and that'd help me 
get into it?

Cheers

Matthew
Mar 10 2005
next sibling parent Georg Wrede <georg.wrede nospam.org> writes:
Matthew wrote:

 "Georg Wrede" <georg.wrede nospam.org> wrote:

Charlie Patterson wrote:


I'm not sure where you want to go with this meta-language but I'm 
here to burst your bubble.  (-:


Thanks! It NEEDS to be stabbed at, wrought, wrinkled, etc. It's the 
only way we'll prune the chaff off. Or maybe prune the whole deal if 
it doesn't deserve to live.


 
 Being the dullard that I am, I have a real hard time imagining what 
 you're talking about absent examples.
 
 Can you cook up a couple - just small one's be ok - and that'd help me 
 get into it?


OK. But I will not do examples on the language, only how it could be 
used. We need to have that covered, before we even start getting into 
the meta syntax.

Say you need to do a long case statement, like the C++ one:
---------------------
         switch (token.value)
         {
#define X(tok,ector) case tok:  nextToken(); \
   e2 = parseAssignExp(); e = new ector(loc,e,e2); continue;

             X(TOKassign,    AssignExp);
             X(TOKaddass,    AddAssignExp);
//etc......
             X(TOKushrass,   UshrAssignExp);
             X(TOKcatass,    CatAssignExp);

#undef X
             default:                 break;
         }
---------------------
One way to write it could be:
---------------------
switch (token.value)
{
     lkjsdflkjlklasldkjlasd    //  <-- the meta def
     TOKassign    AssignExp
     TOKaddass    AddAssignExp
//etc......
     TOKushrass   UshrAssignExp
     TOKcatass    CatAssignExp
     lksdjflksjdlf            //   <-- undo meta def
     default: break;
}
---------------------
Or, if we wanted to make it more readable to the C++ people, we'd do the 
meta def so that we could write the TOKassign lines just like they'd write:
---------------------
switch (token.value)
{
     lkjsdflkjlklasldkjlasd    //  <-- another meta def
             X(TOKassign,    AssignExp);
             X(TOKaddass,    AddAssignExp);
//etc......
             X(TOKushrass,   UshrAssignExp);
             X(TOKcatass,    CatAssignExp);
     lksdjflksjdlf            //   <-- undo meta def
     default: break;
}
---------------------


At this point everyting is open. For example: do we want the meta syntax 
to be "invisible"? (That means, an unconspicuous part of D.) Or do we 
want it to stand out like heck?

Unconspicuous:
---------------------
char[] foo = "lksdf";
int number = 8;
meta(create, kjdslfkjlskdjlsk);   // <-- a meta def here
tree appletree
    trunk
       mainBranch strong internetConnected
          hangar branch filesystemAware
       auxBranch temporary stdErrOutput
    roots
       mainRoot suction localNetConnected
          hardwareBranch tapesystem rw
             hwType HP9245
             hwType MaxstoreUnidriver
unmeta(last);                     // <-- undo meta def
int errlevel = 0;
---------------------

Conspicuous:
---------------------
char[] foo = "lksdf";
int number = 8;




tree appletree
    trunk
       mainBranch strong internetConnected
          hangar branch filesystemAware
       auxBranch temporary stdErrOutput
    roots
       mainRoot suction localNetConnected
          hardwareBranch tapesystem rw
             hwType HP9245
             hwType MaxstoreUnidriver




int errlevel = 0;

---------------------

The time for meta syntax is not yet. Issues like the above have to be 
thought out before that.


In the mean time:
We need more use cases!!

Tell me a few stumbling blocks you run into when writing STL for D!

Or other things that are currently hard in D and not in C++.


Another example:

---------------------
// excerpt from the Aaron Spelling far-out math library
// some formulas from the text book, compiled here
// these take and return floats
lksjdflksajdlfksadjlf // the metadef
    doubleSin(x) := sin(x) + sin(x);
    antiSin(x)   := sin(x + pi);
lksjdf // undo metadef
---------------------

Sorry, that example looks disgustingly trivial. Totally worthless. Let's 
write something worse, this time actually from a math book.  :-(

[[ Two hours later: Crap, I'll do this example later. I will, honest. In 
the meantime, let's see something else: ]]

Say, we need to do a lot of matrix literals. This might be part of some 
library code:

---------------------
LKAJSDLFKAJSLDKFJLSADKJF   // the metadef

eigen_schmeigen =
   2  3  2  0  1  0
  -3  0  0  0  1 -3
;

//... snip, a lot of them here

wamon_schmamon =
   0 -1  0 -1  1  0
   0  1  1  1  1  1
   1 -1 -1  0  0  1
;

LKAJSDF // end metadef
---------------------

So what's the use, you might ask. Point is, for a lot of things there 
are established ways of expressing them. Often, these are clearer than 
doing the same in regular programming language syntax. So, i bet some 
corporations, universities, oss, and others will contribute to our meta 
language library, and later the programmer only has to write:

---------------------
import uni_groeningen.matrix;
alias uni_groeningen.matrix.free_mat_notation.begin ug.begin;
alias uni_groeningen.matrix.free_mat_notation.end ug.end;
// ...
ug.begin;
    eigen_schmeigen =
      2  3  2  0  1  0
     -3  0  0  0  1 -3
    ;
    wamon_schmanon =
      0 -1  0 -1  1  0
      0  1  1  1  1  1
      1 -1 -1  0  0  1
    ;
ug.end;
---------------------

Sure, this opens the door for obfuscation C++ has never even seen. But 
hey, even if I have a kitchen knife in my kitchen, I don't stab everyone 
who comes there. (I'll leave NRA out of this. :-(  )



Somehow, though, I feel that examples from template programming (D, C++) 
might be needed here. What's something you currently wanted to do but can't?
Mar 11 2005
prev sibling parent Georg Wrede <georg.wrede nospam.org> writes:
Matthew wrote:

 Being the dullard that I am, I have a real hard time imagining what 
 you're talking about absent examples.


I'm not talking about absent examples. You are.
SORRY, couldn'resist!    :-)





                 Disclaimer:

The stuff below is idiotic. Pleas try to
look at the language part of it, not the
stupidity of the examples themselves...  :-/



This guy works in a company where most of the core competence is in 
using the Intel math coprocessor in creative ways. They do contracting 
for NASA, NSA, SHELL, and some even for CERN.

They have reams of code libraries to ease the use of the math 
coprocessor. However, the languages they've used so far (Forth, C, asm, 
C++, Lisp) have only been able to cater for some aspect of application 
development.

Currently they have separate staff groups for each language, and (the 
applications made being huge and complicated) most projects use the work 
from several of these groups.

They have tried to create a C++ template library, and a Lisp (actually 
Scheme) library, that'd let the programmer express his wishes both at 
low level as well as on the very high level this company needs. But 
nothing seems to work seamlessly, or efficiently.

Then they found D. An excerpt from the low level programming:

---- library code

module foo.bar;

lkajsdlkjaldskfjlsadk;

/* That was a meta definition that essentially creates the function 
wrapper for D function definitions that need low-level math coprosessor 
routines. Inside this kind of function definitions you can mix [Our 
Proprietary, Secret Notation] with regular D code. */

---- production code

import foo.bar;

alias foo.bar b;

float mLiff;

b.floatFunc  float aCoolLength(p1, p2, p3, p4)
{                     // all params float, unless other specified
    float m1, m2, m3;

    if ( p1 < p2 )
    {
       p1 p3 push
       sin
       p4 push
       log
       exp
       p2 inv
       mul
       m1 pop
    }

    if (m1 < p4)
       m2 = m1
    else
       m2 = m1 + p1;

    return m2;
}

void main()
{
    mLiff = aCoolLength(2.3, 4.5, 6.7, 8.9);
    writefln("The meaning of Life is: %f.2", mLiff);
}

Now, this was an example from the low level. I'd like to do one with 
high level too, but that'll take some more time.



Another example: This firm manufactures and sells sugar cube size mass 
produced brains. But model life cycle is short, and customers keep 
inventing unforeseen uses for them. Essentially, it is a cpu, some rom, 
some ram, and pins for 16 TTL connections programmable to be in or out, 
and an infrared interface for field programming. It also has 3 analog 
inputs and 3 outputs.

Their customers include toy manufacturers, mining robotics, fire 
departments, the military, and some unnamed government agencies. Often a 
project is to create the needed software for a new gadget -- and (with 
some unnamed customers) it has to be ready and bug free in a matter of 
hours. (They still remember the hulabaloo with a project during the Gulf 
War.)

They could make a huge library, have a language of their own, the works. 
But specs change, processors change, etc. So they need to do C like 
code, but it has to be convenient to do the behavioral primitives.

All the gadgets, once out in the field, are programmable with a small, 
interpreted language. End-users do field programming with this language. 
This firm designs and assembles these sugar-cube brains, and ships them 
with firmware customized for the retail gadget (or hand-held nuclear 
missile guidance system). They also sell the field programming software, 
and of course there are different UI versions for the toy makers and the 
marines.

They have found out that "We really want a C, but with faster 
development times, and vastly less bugs in the binaries."

Since most programs are just a single file, the low level gets defined 
in the same file as where it is used to build behavioral primitives. Few 
languages have been of help here.

Then they found D. Some excerpts, not necessarily from same project:

void step()
{
        if ( ! obstacle(ahead) )
        {
            unlock_brake
            forward.slow
            while (!obstacle(ahead) && distance < STEPDISTANCE )
            {}
            stop
        }
}

void deployPayload()
{
        unlockArm
        extendArm wristRight extendFinger1
        jawOpen smallExtendElbow jawClose
}

Well, this turned out to be intermediate level code. :-(
What they really wanted was to change having to write this (pure D code)

void function foo(){
      bar();
      barf();
      bar();
      barf();
      boo();
      doo(); dee(); dii();
      lee(); loo();
      luu();
      moo();
      soo();
}

to this:

void function foo(){
      bar
      barf
      bar
      barf
      boo
      doo dee dii
      lee loo
      luu
      moo
      soo
}

Most of their programming is just invoking tiny functions that neither 
take or return anything. Slapping lots of them in your code day in and 
day out gets tedious with all the "();" all over the place. Also, they 
wanted the code to look cleaner, since they discovered that it reduces 
bugs, and helps find errors, too.

So, their standard library merely defines that inside
   void fname()
you don't have to use either semicolons or parentheses.

They want, however to remain flexible. That's why they were so 
interested in D, with its superior meta code facilities. Now they are 
considering automatic creation of function sets.

The programmer would create all of the functions like:
int fname(int a = 1, int b = -1)
and the meta system would automatically create a set of these. This 
would be used where functions need to return, say, a success code, or 
length traversed. Most of the time that is not needed (well, in this 
example company, anyway), but sometimes it is needed.

For example the programmer creates:

// walk on water
int wow (int distance = 1, int careful = -1)
{
     // let's say careful == -1 means keep last level
     // The code here would use the params, and return
     // a value, in this case the distance traversed.
     // But most often it is not used.
}

Then the meta would create the following versions:

void  wow (int distance = 1, int careful = -1)
{
    // take none, return none
    // just walk one unit with default or
    // last used carefulness
}
int Wow (int distance = 1, int careful = -1)
{
    // return distance
}
int woW (int distance = 1, int careful = -1)
{
    // walk till obstacle
}
int WOW (int distance = 1, int careful = -1)
{
    // panic walk (careful at min, dist till obstacle)
}

And so on. (Since I haven't worked there, I don't know the standard 
permutations! But you get the drift, right?)

Those permutations would contain upper case letters in the name. The 
standard for which letter means what, has become plain obvious at the 
company, and it has been used for years, very successfully.

Problem was, that it was a constant source of bugs. People capped the 
wrong letter, etc. Now they see the end of those bugs. And since there 
is only one "copy" of actual function code, it stays in sync with itself.

Now they use this tac for most of the primitives. Walk, extendArm, 
drill, turn, sleep, all of them are now encoded with the 
three-first-letter-caps. Programmers are happy, and roll-out times 
smaller. This is also the default for any new kind of action primitives.



The way I see the meta language is, that it would not bee too hard to 
have it check and enforce the "self-made" syntaxes. Not automatically, 
but with writing meta code for the checking and enforcing -- and still 
be understandable to a normal person!
Mar 11 2005
prev sibling next sibling parent Georg Wrede <georg.wrede nospam.org> writes:
Anders F Björklund wrote:

 Georg Wrede wrote:
 

 I've been thinkin hard on a meta language.


 
 [...]
 

 D meta language is an important thing. But now it is also urgent.


 
 
 What would the "D meta language" be ? Something like YACC, but for D ?
 (maybe related to DGrammar http://www.dsource.org/projects/dgrammar/ ?)
 
 Or something completely different ?
 
 By definition it's "a language that can be used to describe languages"
 Just wondering what your little project was, and why it was urgent...
 
 --anders


Copy/pasting from another thread here. But read my other answer to your 
post first.



Why not push to get the enum-names exposed via reflection instead? I 
wouldn't
wish to have to re-type them all over again, in a different manner. 
Doesn't make
sense.

- Kris



In article <d0onfb$1afp$1 digitaldaemon.com>, Andrew Fedoniouk says...


Thanks a lot,

What about this:

enum E {
  ZERO,
  ONE,
  TWO
}

char[] toString(E e)
{
  static char[][] map =
  [
    E.ZERO  :"primordial",
    E.ONE   :"first",
    E.TWO   :"second"
  ];
  return map[e];
}

for simple continuous enums?






Things like the above would be handled with the meta language real neatly:

enum E {
some_meta_language_def_here;
ZERO   primordial
ONE    first
TWO    second
undefine_the_meta_def_here;
}

With a good meta language, even this would be _at_least_ as easy as with 
the C preprocessor.

The "some_meta_language_def_here" has to be easy to write, easy to 
understand, even if the language is capable of Awesome Deep Stuff too.

You want printable enums? "Do this."
You want reversed enums? "Do that."

You want a recursive compile time generated static tree structure to 
hold the Universal Pointers with a Riemann-Lembowski hyperbolic 
naturalised reverse transform in 9 dimensions? "Well, do this, and then 
type the definition in from your textbook."

You want enums where more than one enum-word has the same integer value? 
"Write it as below:"

enum E {
some_meta_language_def_here;
FOO BAR BARF
NOO NAR NARF
NIG
NAG
NUG
BOOBOO AUX_BOOBOO NOTHER_BOOBOO
undefine_the_meta_def_here;
}

And you'll get enums where FOO == BARF, NIG != NAG.

BTW, a lot of what Matthew, others (and even I) have suggested lately 
could be implemented with this meta language.

Carrying the thought a little further, once this language exists, then 
it might be worthwhile to refactor things between D and meta. For 
example, the current templates, mixins, and possibly other stuff, could 
be moved from D itself to "defined in terms of the metalanguage".

This may make the compiler core more simple and faster, for all I know.

New D features could _all_ be created in the metalanguage, at first. If 
it later turns out that they fit better in D itself, then so be it. The 
user should not have to need to know.
Mar 10 2005
prev sibling parent Georg Wrede <georg.wrede nospam.org> writes:
Anders F Björklund wrote:

 Georg Wrede wrote:

 D meta language is an important thing. But now it is also urgent.





 What would the "D meta language" be ? 


The way I see it now, having this meta language in D would not even slow 
down DMD at all, as long as you don't use meta constructs!

All that would be needed is a few more reserved words in the language. 
Unless they are encountered, then DMD speed should stay exactly the 
same. (I.e. industry shattering!) Of this I'm sure of.

If a local meta definition is encountered, then compiling would be 
slower for the part of source code where the definition is in scope.

With a lot of global scope definitions, compiling would be noticeably 
slower. But still a lot faster than C++.

Speaking of which, there is no reason at all to have definitions with a 
global effect. Locality, code hiding, etc. should be obeyed here too.
Mar 11 2005
prev sibling next sibling parent reply Norbert Nemec <Norbert Nemec-online.de> writes:
Georg Wrede schrieb:

 I've been thinkin hard on a meta language.
=20
 I've talked with folks, amongst them prof. Jaakko J=E4rvi, who actually=


=20

 met Walter at the C++ meeting last fall.
=20
 Bad news: I seem to have spilled the beans, big time. After asking his =



 opinion about some of the core concepts I'd been cooking, it occurred t=


o=20

 me that -- this guy's office is down the hall from Bjarne Stroustrup!
=20
 If these guys stop dwelling _within_ the C++ template jungle, and start=


=20

 looking at it from above, then it'll be just 18 months before one of=20
 them holds a ground breaking lecture.   :-(   :-(


Don't worry. The basic idea has been around for some time already.=20
Within C++, it will be hard to go far beyond the current state without=20
seriously breaking compatibility. It can only be new languages that can=20
take the next step.
Mar 09 2005
parent reply Georg Wrede <georg.wrede nospam.org> writes:
Norbert Nemec wrote:

 Georg Wrede schrieb:
 

 I've been thinkin hard on a meta language.

 I've talked with folks, amongst them prof. Jaakko Järvi, who actually 
 met Walter at the C++ meeting last fall.

 Bad news: I seem to have spilled the beans, big time. After asking his 
 opinion about some of the core concepts I'd been cooking, it occurred 
 to me that -- this guy's office is down the hall from Bjarne Stroustrup!

 If these guys stop dwelling _within_ the C++ template jungle, and 
 start looking at it from above, then it'll be just 18 months before 
 one of them holds a ground breaking lecture.   :-(   :-(


 
 
 Don't worry. The basic idea has been around for some time already. 
 Within C++, it will be hard to go far beyond the current state without 
 seriously breaking compatibility. It can only be new languages that can 
 take the next step.


Of course I worry. Guys of this caliber do things "right". And once 
they've figured out a replacement to their current pp/tl system, the new 
one will take care of the old pp/tl syntax just with a snap of fingers.

The goal, after all, once they start thinking "outside the box" is to 
create a universal meta language. For such a language it is a piece of 
cake to handle all of the current crap.

So, they'll get their Backward Compatibility, but this time as a _mere_ 
special case -- just one small thing among the tousands of grand things 
it'll handle.

One bad (for us, that is) thing that follows from this, is that once 
they've done this, then, given time, folks will rewrite the current STL, 
BOOST, etc. with the new syntax (possibly even skipping the 
bla<foo<bar>> notation), and this will inevitably result in a vast 
speed-up of compilation!!!!

So, I'm worried as hell. For D. But for Human Kind this is of course a 
Good Thing(TM).

-----------

If we have a smarter language to begin with (like D), then right now 
there is a window of opportunity for fame and reputation for D. But the 
window won't stay open for good. After that our meta language will be 
"just another thing somebody does `like what's done in C++`". Farewell, 
publicity, fame, hype, exponential sales. Back to our niche.

My real goof was in waking up this guy, from "dwelling withing the 
template jungle" to looking at it "from above", "outside the box". Men 
of this kind don't need more than that.
Mar 10 2005
next sibling parent reply Andy Friesen <andy ikagames.com> writes:
Georg Wrede wrote:

 My real goof was in waking up this guy, from "dwelling withing the 
 template jungle" to looking at it "from above", "outside the box". Men 
 of this kind don't need more than that.


Don't worry, you haven't pointed out anything he didn't know already. 
Lisp has been doing this for years now.  The majority of the syntax of 
the Nemerle language <http://nemerle.org> is implemented as macros. 
(even stuff like 'if' statements)

Also, Daveed Vandevoorde had a quasi-working implementation of such an 
extension for C++ as far back as 2003: 
<http://www.open-std.org/JTC1/SC22/WG21/docs/papers/2003/n1471.pdf>

  -- andy
Mar 10 2005
parent Georg Wrede <georg.wrede nospam.org> writes:
Andy Friesen wrote:

 Georg Wrede wrote:
 

 My real goof was in waking up this guy, from "dwelling withing the 
 template jungle" to looking at it "from above", "outside the box". Men 
 of this kind don't need more than that.


 
 
 Don't worry, you haven't pointed out anything he didn't know already. 
 Lisp has been doing this for years now.  The majority of the syntax of 
 the Nemerle language <http://nemerle.org> is implemented as macros. 
 (even stuff like 'if' statements)
 
 Also, Daveed Vandevoorde had a quasi-working implementation of such an 
 extension for C++ as far back as 2003: 
 <http://www.open-std.org/JTC1/SC22/WG21/docs/papers/2003/n1471.pdf>


Thanks!
Already peeked at 'em. Gonna take a closer look.
Mar 11 2005
prev sibling parent reply "Matthew" <admin stlsoft.dot.dot.dot.dot.org> writes:
"Georg Wrede" <georg.wrede nospam.org> wrote in message 
news:42300F90.40600 nospam.org...

 Norbert Nemec wrote:

 Georg Wrede schrieb:


 I've been thinkin hard on a meta language.

 I've talked with folks, amongst them prof. Jaakko Järvi, who 
 actually met Walter at the C++ meeting last fall.

 Bad news: I seem to have spilled the beans, big time. After asking 
 his opinion about some of the core concepts I'd been cooking, it 
 occurred to me that -- this guy's office is down the hall from 
 Bjarne Stroustrup!

 If these guys stop dwelling _within_ the C++ template jungle, and 
 start looking at it from above, then it'll be just 18 months before 
 one of them holds a ground breaking lecture.   :-(   :-(



 Don't worry. The basic idea has been around for some time already. 
 Within C++, it will be hard to go far beyond the current state 
 without seriously breaking compatibility. It can only be new 
 languages that can take the next step.


 Of course I worry. Guys of this caliber do things "right". And once 
 they've figured out a replacement to their current pp/tl system, the 
 new one will take care of the old pp/tl syntax just with a snap of 
 fingers.

 The goal, after all, once they start thinking "outside the box" is to 
 create a universal meta language. For such a language it is a piece of 
 cake to handle all of the current crap.

 So, they'll get their Backward Compatibility, but this time as a 
 _mere_ special case -- just one small thing among the tousands of 
 grand things it'll handle.

 One bad (for us, that is) thing that follows from this, is that once 
 they've done this, then, given time, folks will rewrite the current 
 STL, BOOST, etc. with the new syntax (possibly even skipping the 
 bla<foo<bar>> notation), and this will inevitably result in a vast 
 speed-up of compilation!!!!

 So, I'm worried as hell. For D. But for Human Kind this is of course a 
 Good Thing(TM).

 -----------


I'd chill out. I have not the slightest doubt that Walter's smart enough 
to do this. And I'm confident it'll be out before any C++ 
implementations.

But maybe I'm a starry-eyed optimist. :-)
Mar 10 2005
next sibling parent clayasaurus <clayasaurus gmail.com> writes:
Matthew wrote:

 "Georg Wrede" <georg.wrede nospam.org> wrote in message 
 news:42300F90.40600 nospam.org...
 

Norbert Nemec wrote:


Georg Wrede schrieb:



I've been thinkin hard on a meta language.

I've talked with folks, amongst them prof. Jaakko Järvi, who 
actually met Walter at the C++ meeting last fall.

Bad news: I seem to have spilled the beans, big time. After asking 
his opinion about some of the core concepts I'd been cooking, it 
occurred to me that -- this guy's office is down the hall from 
Bjarne Stroustrup!

If these guys stop dwelling _within_ the C++ template jungle, and 
start looking at it from above, then it'll be just 18 months before 
one of them holds a ground breaking lecture.   :-(   :-(



Don't worry. The basic idea has been around for some time already. 
Within C++, it will be hard to go far beyond the current state 
without seriously breaking compatibility. It can only be new 
languages that can take the next step.


Of course I worry. Guys of this caliber do things "right". And once 
they've figured out a replacement to their current pp/tl system, the 
new one will take care of the old pp/tl syntax just with a snap of 
fingers.

The goal, after all, once they start thinking "outside the box" is to 
create a universal meta language. For such a language it is a piece of 
cake to handle all of the current crap.

So, they'll get their Backward Compatibility, but this time as a 
_mere_ special case -- just one small thing among the tousands of 
grand things it'll handle.

One bad (for us, that is) thing that follows from this, is that once 
they've done this, then, given time, folks will rewrite the current 
STL, BOOST, etc. with the new syntax (possibly even skipping the 
bla<foo<bar>> notation), and this will inevitably result in a vast 
speed-up of compilation!!!!

So, I'm worried as hell. For D. But for Human Kind this is of course a 
Good Thing(TM).

-----------


 
 
 I'd chill out. I have not the slightest doubt that Walter's smart enough 
 to do this. And I'm confident it'll be out before any C++ 
 implementations.
 
 But maybe I'm a starry-eyed optimist. :-)
 
 


I remember Walter saying something about using '$' for something big, 
like meta language.

Besides, what would be so bad if D sat back, watched it happened, then 
found a better way to add it when it is ready?
Mar 10 2005
prev sibling parent reply Georg Wrede <georg.wrede nospam.org> writes:
Matthew wrote:

 "Georg Wrede" <georg.wrede nospam.org> wrote:


.......

The goal, after all, once they start thinking "outside the box" is to 
create a universal meta language. For such a language it is a piece of 
cake to handle all of the current crap.

So, they'll get their Backward Compatibility, but this time as a 
_mere_ special case -- just one small thing among the tousands of 
grand things it'll handle.




........

So, I'm worried as hell.


 
 I'd chill out. I have not the slightest doubt that Walter's smart enough 
 to do this. And I'm confident it'll be out before any C++ 
 implementations.


Course he is smart enough! But isn't this newsgroup here to let ideas 
cook, folks find the early worm, ask the Right Questions, etc.? (Not 
forgetting helping newbies, too.)


 But maybe I'm a starry-eyed optimist. :-)


Somehow I get the feeling there _is_ a meta language project, already 
well rolling forward -- behind the scenes. For some reason it's been 
awful quiet around here about its specifics.

Could be a lot of reasons. Maybe we'd crowd it with all these ideas, 
maybe he wants some peace with it, maybe Walter wants to save it for a 
Grand Opening, maybe it is "secret" to avoid embarrasment in case it folds?

Or, he's talking with superhumans in the academia and other 
Einstein/MahreshYogi look-alike hermits, the only ones who've Received 
the Knowledge from Above.

If this is the case, then I'd better shut up. I could always try to cook 
something up, and if it tastes good, then maybe try to earn some money 
with it? The it'd of course be a pure preprocessor.
Mar 11 2005
parent reply "Matthew" <admin stlsoft.dot.dot.dot.dot.org> writes:
 Matthew wrote:

 "Georg Wrede" <georg.wrede nospam.org> wrote:


 .......

The goal, after all, once they start thinking "outside the box" is to 
create a universal meta language. For such a language it is a piece 
of cake to handle all of the current crap.

So, they'll get their Backward Compatibility, but this time as a 
_mere_ special case -- just one small thing among the tousands of 
grand things it'll handle.




 ........

So, I'm worried as hell.


 I'd chill out. I have not the slightest doubt that Walter's smart 
 enough to do this. And I'm confident it'll be out before any C++ 
 implementations.


 Course he is smart enough! But isn't this newsgroup here to let ideas 
 cook, folks find the early worm, ask the Right Questions, etc.? (Not 
 forgetting helping newbies, too.)


It is. My point was that you seem to be over-reacting/stating.

I mean, let's be honest, D's not going to kick off C++ or Java/.NET in a 
single stroke, no matter how killer a feature it has. Any migration from 
them to it would take time, and be based on multipart user judgements. 
Even if D, right now, had the most amazing meta language ever seen, it 
wouldn't bring zillions of people over because (i) the language still 
has non-trivial flaws, (ii) the runtime library is still woefully 
inadequate, and (iii) MP is not all that important to most people (and 
I'd hazard a guess to say it never will be).

None of that is to say it's not desirable, and won't be a big component 
of the overall eventual picture of attractiveness. I just don't think 
it's anything to panic about.


 But maybe I'm a starry-eyed optimist. :-)


 Somehow I get the feeling there _is_ a meta language project, already 
 well rolling forward -- behind the scenes. For some reason it's been 
 awful quiet around here about its specifics.


I think there might. Walter's a thinker as well as a language porker. 
(Nomenclature: v. pork. To mess with, to hack. Can imply a degree of 
recklessness. Usage "pork away [on that library]". Derivation: my 
friends Leigh and Scott Perry, who were unfailingly bemused at my desire 
to get in and fix libraries which were flawed but for which they had 
adequate workarounds. <g>)


 Or, he's talking with superhumans in the academia and other 
 Einstein/MahreshYogi look-alike hermits, the only ones who've Received 
 the Knowledge from Above.


Well, I wasn't aware that there were any superhumans in academia. Maybe 
you're referring to people who use terminology with which you're 
unfamiliar? But that doesn't make them superhuman. (In fact, the more I 
learn about the rarefied heights of the programming world, the less 
impressed I am with anyone's omnipotence. Most acute, naturally, being 
my own. <g>)


 If this is the case, then I'd better shut up. I could always try to 
 cook something up, and if it tastes good, then maybe try to earn some 
 money with it? The it'd of course be a pure preprocessor.


Please don't shut up. You've intrigued me enormously about 
possibilities. It's just that I am so inexperienced in such matters - at 
least from a generic (not C++ TMP) perspective - that I am not able to 
grasp your intent without suitably simple examples. <g>
Mar 11 2005
parent reply Georg Wrede <georg.wrede nospam.org> writes:
Matthew wrote:

Matthew wrote:

"Georg Wrede" <georg.wrede nospam.org> wrote:

So, they'll get their Backward Compatibility, but this time as a 
_mere_ special case -- just one small thing among the tousands of 
grand things it'll handle.
So, I'm worried as hell.


I'd chill out. I have not the slightest doubt that Walter's smart 
enough to do this.


Course he is smart enough! But isn't this newsgroup here to let ideas 
cook, folks find the early worm, ask the Right Questions, etc.? (Not 
forgetting helping newbies, too.)


 It is. My point was that you seem to be over-reacting/stating.


A (delusional) sense of urgency, importance, and paranoia.

That has advanced the mankind, on several (actually most) occasions. Got 
us to the Moon, gave us (non-americans anyway) serious encryption, 
nuclear technology, good computers, submarines, GPS, teflon, solar 
panels. And Ada.

Complacency, that's what kills a language! (Quote of the day(TM))


 I mean, let's be honest, D's not going to kick off C++ or Java/.NET in a 
 single stroke, no matter how killer a feature it has.


I was thinkin more like the "extra" fame and hype D would get for doing 
something right, before the others. More press exposure won't hurt.

This "right" is, of course (so far) only my personal belief. Hopefully 
others will agree, in time. (Ha, and if I'm on a completely wrong track 
here, I wish to Bob somebody would kick down the stack of cartons I'm 
trying to lecture from. Before this becomes too public!! )

?Ha?s/on a completely wrong/not on precisely the right/    :-)

Right: Turing complete. Two-level. Infinitely extensible (honest!). Not 
only descriptive but procedural too.

In practice: No must-use. Comprehensible to the casual, or the newbie. 
Practical. Practical. Practical. Without self-promotional esotheric 
bells and whistles -- just get the job done. Any job.

I ADMIT THIS PLATE IS A BIT FULL. (No shit?!) But I have to admit, by 
now this is more than wishful thinking. I have (eh, like Walter "may 
be") some actual ideas on how to implement this. Why else would I sound 
so confident, even to a few specific questions by folks.


 Any migration from 
 them to it would take time, and be based on multipart user judgements. 
 Even if D, right now, had the most amazing meta language ever seen, it 
 wouldn't bring zillions of people over 


Actually, we live in the 21st century now. Times change, my friend. 
Obstacles are only as hard as you think. (Took me almost 50 years :-( to 
figure out.) Today we have something there's never been: we can turn 400 
million heads overnight. If 0.001% of these start actually thinking, 
we'll be well off.

Say there's a big article in PC-Magazine, that essentially says "we 
really think this is the language the industry will use in two years." 
Well, nothing will happen for a while. Six months from then, Walter's 
download server will crash.

Not enough, though. But have editorials saying essentially the same in 
DDJ and CUJ, hopefully one month after the PC-Magazine one, and we fly.


 because 
 (i) the language still has non-trivial flaws,
 (ii) the runtime library is still woefully inadequate, and 
 (iii) MP is not all that important to most people 
 (and I'd hazard a guess to say it never will be).


(i) We need to keep PC-Magazine ignorant of us till that's fixed. :-)

(ii) Now, that's something that will not proceed fast enough the way 
things currently are. We need to CHANGE THE SETUP around it. Do 
something DRASTIC. (Put it on dsource, or sourceforge, whatever! But 
something really has to be done to this, now! Projects on SourceForge 
have to have an "owner", so Walter wouldn't eve lose control!) Lots of 
talent just seeking a worthwhile, interesting project, with a long time 
future.

(iii) I fully and completely agree. MP is not for the average user. 
While easy things will be easy (like the X example in this thread), the 
regular (even professional) programmer will never use it.

MP (at least the way I see our concoction becoming used) is for 
corporate use, research, libraries, D development(!) -- and should be 
used invisibly to most things. It will find its use in a massive 
reduction of the size of app-field specific libraries, reducing 
complicity when doing "hairy stuff", and for programming language research.

(Issues like overloading on return type, unquoted regexps as anonymous 
functions, a serious string type or class -- or just regular D features 
investiagetd for inclusion or exlcusion.)

Most regular programmers will never know they've been using it! In time 
some things regularly done with MP will be incorporated directly in the 
D "core" (for speed of compilation, others just because the MP usage has 
shown them to be practical and useful (a new keyword, a new operator, 
new syntax for something)).


 None of that is to say it's not desirable, and won't be a big component 
 of the overall eventual picture of attractiveness. I just don't think 
 it's anything to panic about.


There's psychology here too. I used to sell SLR cameras at a major 
department store. We sold Canon, Nikon, Minolta, Olympus, and a half 
dozen now extinct brands. Customers dithering between two cameras ended 
up choosing the one with which "you could do this" that you couldn't do 
with the other. Most of them never noticed this themselves!!

Usually it was a feature (both they and I knew, but did't say) they'd 
never get around to using. But ultimately the choice was formulated as 
something "non-childish", like durability, long-term availability of 
accessories, quality, etc. And if the chosen one just happened to cost 
less than the other, then Economy was the Thing. (Wife'll be less 
angry!) (Oh, and if economy was the real reason, then the pretense was 
of course whatever else.)

I've participated in a few language shoot-outs. (University IT 
department curriculum planning meeting, customer board room, regular 
consulting with customers, mens' room, the local pub.) Any non-obvious 
choices end up like the SLR sales: ultimately the "cool thing" the other 
one doesn't have -- whether that is relevant to the choice at hand or not.

And believe it or not, the people who (by profession, or by position) 
definitely should not succumb to this, they're actually the worst.

We need to use this to our advantage. It's there for the taking!

Even the moron who'll _never_ be able to grasp even recursion, has to 
brag to his equal friend that he's Chosen the "infinitely extensible D" 
programming language.

Ditto the Pointy Haired boss over beer at the Country Club.


But maybe I'm a starry-eyed optimist. :-)


Somehow I get the feeling there _is_ a meta language project, already 
well rolling forward -- behind the scenes. For some reason it's been 
awful quiet around here about its specifics.


 
 I think there might. Walter's a thinker as well as a language porker. 


Then I might be happy. And not alone in the world.


Or, he's talking with superhumans in the academia and other 
Einstein/MahreshYogi look-alike hermits, the only ones who've Received 
the Knowledge from Above.


 
 Well, I wasn't aware that there were any superhumans in academia. 


Alfred Aho, Edsger Dijkstra, Donald Knuth, Linus Torvalds, Roger 
Penrose, Alan Turing.......

Not granting Superhuman Status to those who deserve it -- is arrogant. 
:-) :-P

OTOH, you might visit several hundred universities, and not find a 
single one. And that's a disgrace!


 Maybe 
 you're referring to people who use terminology with which you're 
 unfamiliar? 


LOL! I know what you mean! I had the good fortune to know my mother's 
cousin, who compulsively tried to pull that one off. At age 11 I 
noticed! The best thing was, mom had a lot of respect for him, and 
bought most of this crap -- so I couldn't shoot him down. I learned to 
"not notice".

Man, the lot of fun I've had through the years with these guys. And 
gals. Both in and out of academia. And none of them know!


 But that doesn't make them superhuman. (In fact, the more I 
 learn about the rarefied heights of the programming world, the less 
 impressed I am with anyone's omnipotence. Most acute, naturally, being 
 my own. <g>)


I was thinking of a few names I expressly did not list above. There's a 
particular example, a guy who is undeniably accomplished, who's known to 
everyone in the non-profit programming world, with a huge reputation. We 
sat a few hours drinking beer after his lecture at an academic 
convention. I was shattered to discover that he actually is a 
superhuman-wannabe, a man of lust and avarice, envy, and a petty soul.

(Maybe I should have spelled it like "superhuman wannabe"!)


<humor_intent quality="lame">

Omnipotence... not for us, but for D !!

</humor_intent>


If this is the case, then I'd better shut up. I could always try to 
cook something up, and if it tastes good, then maybe try to earn some 
money with it? The it'd of course be a pure preprocessor.


 
 Please don't shut up. You've intrigued me enormously about 
 possibilities. It's just that I am so inexperienced in such matters - at 
 least from a generic (not C++ TMP) perspective - that I am not able to 
 grasp your intent without suitably simple examples. <g>


Humble! :-)

Thanks! I'm relieved to hear this intrigues. I wish, if other people 
read this thread, that they'd at least be entertained, stimulated, and 
possibly motivated.

------------

Corollary:

I had a career revelation. The day FADPL (the Foundation for the 
Advancement of the D Programming Language, a US non-profit with 
tax-exempt status, later backed by Lawrence Livermore, GM, DoE, and 
others, hopefully to be started by somebody, some day), is looking for 
paid D evangelists, I'll submit my resume.

They could do a lot worse than hiring me.  :-)
Mar 12 2005
parent reply xs0 <xs0 xs0.com> writes:
 But I have to admit, by 
 now this is more than wishful thinking. I have (eh, like Walter "may 
 be") some actual ideas on how to implement this. Why else would I sound 
 so confident, even to a few specific questions by folks.


Well, for Bob's sake, what exactly do you want to implement and how do 
you propose it is done? :)

xs0
Mar 13 2005
parent reply Georg Wrede <georg.wrede nospam.org> writes:
xs0 wrote:

 But I have to admit, by now this is more than wishful thinking. I have 
 (eh, like Walter "may be") some actual ideas on how to implement this. 
 Why else would I sound so confident, even to a few specific questions 
 by folks.


 
 
 Well, for Bob's sake, what exactly do you want to implement and how do 
 you propose it is done? :)


Right now I'm trying to find a D grammar somewhere.

I'd like to add a couple of keywords to D. They'd be for defining meta 
things, and undoing the definitions. (At least in the beginning there'll 
be a define and an undefine. Later, with more experience and confidence, 
lexical scoping would work "automatically". Even more later global 
things could be defined. And after that, maybe local disabling of global 
things. Then I wonder about this-file-only defines, how this should work 
with importing files, and other, eh, more challenging obstacles.) (Oh, I 
forgot, there are no Obstacles, only Interesting Things Encountered on 
the way! :-O  )

I'm trying to find a language to use in this pre-prototyping (in reality 
I'll do 1-screenful short programs that get D source code, and output D 
source code (just like any preprocessor).

While I'm toying around (mostly to see what problems I've totally 
missed), with awk, sed, grep, possibly ed, maybe D, Euphoria, and 
whatever other seems to be nice for the particular mini thing, I am 
trying to figure out how to do it for real. (Heck, I may even end up 
searching around for my Perl book.)

Do what? Well, a pre-alpha version of a proof-of-concept kind of 
prototype. Or, if it looks like too much, then I'll try to make a couple 
different protorypes, each of which would demonstrate some part of the 
whole.

Something for #define X(a,b) foo(b,a);bar(a);blabla(a,b,a,b) kind of 
trivial C preprocessor things. Something else for having a snippet of 
self-made language inside a procedure instead of writing that too with D 
(see the sugar-cube robotics example in this thread).

And then maybe one or a couple of different try-outs for the hairy 
stuff. With that I dream you could do Lisp like stuff, that's really for 
"the superhuman". That's a bit hard, since I'm not.  :-(

-----

For these pathetic first attempts I'd either make small 
one-trick-preprocessors, or maybe later get a GDC version that I can 
compile, and then break the parser and lexer "trying to play Walter"! 
;-)  -- With neither his skills, experience, nor abilities.

If any of these one-trick things does anything useful at all, then I'll 
post them for others to laugh at. Then at least folks will get a glimpse 
of some parts of the whole. With any luck there'll be discussion and 
feedback, to keep us going on!

If we haven't given up by this time, then we'll try to do the same with 
some self-made-language inserters. Probably the really first ones will 
have some arbitrary but fixed language as the "inserted language", later 
the testers (you gyus) get versions actually capable of doing 
minilanguages defined in some input file by each of you -- for using and 
abusing them.

If we're still at it at this point, then it's time for the hairy stuff. 
I don't know how we'll test that, without asking some heavyweight 
hairy-stuffers. I mean, even if a particular preprocessor happened to 
actually work, where do we find the kind of guys who are actually 
capable of trying any hairy stuff on them? Or giving meaningful comments??

Phew! If we're /still/ here, then it'd be time to make a proposal for 
Walter.
Mar 13 2005
parent xs0 <xs0 xs0.com> writes:
After some thinking, I determined this has to be a pre/postprocessor, 
unless the compiler is completely redone to support dynamic grammar. 
Because, you obviously want to be able to define new tokens (like 
"aspect" or "-=>") and new syntax, but by the time the compiler is aware 
of that, parsing has already happened (and most probably failed). As for 
the scope, it would be nice to simply have two constructs, like #define 
for globals and #local
be renamed to make an obvious distinction from C's..

Hmm, after some more thought, it may make sense to move new syntax 
definitions completely out of D source, where they'll have no 
constraints/conflicts with the existing syntax, and it can be used with 
something like

#extension(aspect)

in the source. Local stuff will be somewhat limited then, but I guess it 
would be mostly used for stuff like the X macro, which shouldn't be too 
hard, and perhaps for mini operators, like -=>. OTOH, maybe these could 
be skipped altogether? You can write the extension normally and use 
something like

#with(graphOperator) {{




xs0

Georg Wrede wrote:

 xs0 wrote:
 

 But I have to admit, by now this is more than wishful thinking. I 
 have (eh, like Walter "may be") some actual ideas on how to implement 
 this. Why else would I sound so confident, even to a few specific 
 questions by folks.




 Well, for Bob's sake, what exactly do you want to implement and how do 
 you propose it is done? :)


 
 
 Right now I'm trying to find a D grammar somewhere.
 
 I'd like to add a couple of keywords to D. They'd be for defining meta 
 things, and undoing the definitions. (At least in the beginning there'll 
 be a define and an undefine. Later, with more experience and confidence, 
 lexical scoping would work "automatically". Even more later global 
 things could be defined. And after that, maybe local disabling of global 
 things. Then I wonder about this-file-only defines, how this should work 
 with importing files, and other, eh, more challenging obstacles.) (Oh, I 
 forgot, there are no Obstacles, only Interesting Things Encountered on 
 the way! :-O  )
 
 I'm trying to find a language to use in this pre-prototyping (in reality 
 I'll do 1-screenful short programs that get D source code, and output D 
 source code (just like any preprocessor).
 
 While I'm toying around (mostly to see what problems I've totally 
 missed), with awk, sed, grep, possibly ed, maybe D, Euphoria, and 
 whatever other seems to be nice for the particular mini thing, I am 
 trying to figure out how to do it for real. (Heck, I may even end up 
 searching around for my Perl book.)
 
 Do what? Well, a pre-alpha version of a proof-of-concept kind of 
 prototype. Or, if it looks like too much, then I'll try to make a couple 
 different protorypes, each of which would demonstrate some part of the 
 whole.
 
 Something for #define X(a,b) foo(b,a);bar(a);blabla(a,b,a,b) kind of 
 trivial C preprocessor things. Something else for having a snippet of 
 self-made language inside a procedure instead of writing that too with D 
 (see the sugar-cube robotics example in this thread).
 
 And then maybe one or a couple of different try-outs for the hairy 
 stuff. With that I dream you could do Lisp like stuff, that's really for 
 "the superhuman". That's a bit hard, since I'm not.  :-(
 
 -----
 
 For these pathetic first attempts I'd either make small 
 one-trick-preprocessors, or maybe later get a GDC version that I can 
 compile, and then break the parser and lexer "trying to play Walter"! 
 ;-)  -- With neither his skills, experience, nor abilities.
 
 If any of these one-trick things does anything useful at all, then I'll 
 post them for others to laugh at. Then at least folks will get a glimpse 
 of some parts of the whole. With any luck there'll be discussion and 
 feedback, to keep us going on!
 
 If we haven't given up by this time, then we'll try to do the same with 
 some self-made-language inserters. Probably the really first ones will 
 have some arbitrary but fixed language as the "inserted language", later 
 the testers (you gyus) get versions actually capable of doing 
 minilanguages defined in some input file by each of you -- for using and 
 abusing them.
 
 If we're still at it at this point, then it's time for the hairy stuff. 
 I don't know how we'll test that, without asking some heavyweight 
 hairy-stuffers. I mean, even if a particular preprocessor happened to 
 actually work, where do we find the kind of guys who are actually 
 capable of trying any hairy stuff on them? Or giving meaningful comments??
 
 Phew! If we're /still/ here, then it'd be time to make a proposal for 
 Walter.
Mar 13 2005
prev sibling next sibling parent reply "Lionello Lunesu" <lio lunesu.removethis.com> writes:
Just for my understanding: would a metalanguage allow you to define new 
operators on the fly?

This way, the current D language would in fact be an implementation using 
(its own) metalanguage for defining the existing operators and their 
precedence?

Lionello.
Mar 11 2005
parent reply Georg Wrede <georg.wrede nospam.org> writes:
Lionello Lunesu wrote:

 Just for my understanding: would a metalanguage allow you to define new 
 operators on the fly?


Yes.

Suppose you are into Graph Theory, and you need to create a lot of 
Directed Graphs. You could do the meta definition of the operator right 
where it is needed, or you could have it in a library for regular use.

Let's take the local case.

Class Vertex {
     Vertex[] toVertices, fromVertices;
     public addTo(Vertex v) { toVertices ~= v }
     public addFrom (Vertex v) { fromVertices ~= v }
}


void main () {
     Vertex a = new Vertex;
     ... and b,c,d,e....

Now, we need to create the connections. For this we need an operator, 
let's call it "-=>" and define it.

     metadefinition ( lkjdsallkjakjsf ); //Nobody knows the syntax

Then just go on with the graph:

     a -=> b;
     c -=> d;
     d -=> a;
     c -=> b;

} // end main.

Our connecting operator is in scope only in main itself, so we won't 
pollute all of the application source code.

The meta definitions should obey normal D scoping rules.

/////////////////////////////////

Now, let's take the module case.

module foo.bar;

meta myGraphOperator (lkjsdfljsadl);  // metadefinition, that
                         // defines the operator -=>
// still unknown syntax.  :-)


Production code would be:

import foo.bar;

Class Vertex {
     Vertex[] toVertices, fromVertices;
     public addTo(Vertex v) { toVertices ~= v }
     public addFrom (Vertex v) { fromVertices ~= v }
}

void main () {
     Vertex a = new Vertex;
     ... and b,c,d,e....

// Now, we need to create the connections.
// For this we need the operator.

     foo.bar.myGraphOperator; // get it into this scope

// Then just go on with the graph:

     a -=> b;
     c -=> d;
     d -=> a;
     c -=> b;

} // end main.


 This way, the current D language would in fact be an implementation using 
 (its own) metalanguage for defining the existing operators and their 
 precedence?


I think it could.

Of course, compiling would then be slower!
Mar 11 2005
parent "Lionello Lunesu" <lio lunesu.removethis.com> writes:
Thanks a lot for your example! I finally know what all the (totally 
justified) fuzz is about!

Thanks again..

Lionello.
Mar 13 2005
prev sibling parent reply Norbert Nemec <Norbert Nemec-online.de> writes:
Georg Wrede schrieb:

 I've been thinkin hard on a meta language.


Word from the wise: Don't try to reinvent the wheel but learn from those 
to tried before (even if they failed)

There are loads of programming languages or add-ons do existing 
languages out there that allow meta-programming in some way. I don't 
have all the references at hand that I have read over the past years. 
There have been thorough scientific investigations of the matter. I 
doubt it is possible to outstrip those projects by starting from scratch.

The point of D is not to give it the perfect features for 
meta-programming but designing features in such a way that they are 
practical, fit nicely into the general language and don't get in the way 
with regular programming.

A preprocessor is not such a great idea, since it will generally split 
the language into two levels causing all kinds of problems. Walter had 
good reason to drop the concept of a preprocessor.

In general, the big pieces of the meta-language already exist in D: 
templates. These already contain a complete functional programming 
language that can be used for meta-programming. Several pieces are 
missing for full comfort and usability, but these are details that have 
to be fixed.

The meta-language formed by templates certainly is not very handy when 
trying to do regular calculations at compile time, but then, it has to 
be clear that this is not the point of a meta-language. Meta-programming 
has the very specific purpose of doing code transformations and for 
this, templates are excellently designed. It is not a general functional 
language, but a highly specialized language. In principle, you can do 
anything, but going beyond the special purpose of the meta-language will 
be both awkward and inefficient.

Now: if you can identify single weak points of the existing 
meta-language and propose clear solutions, these may have a chance. For 
doing a complete redesign of templates, the language is a bit too 
mature, though.

Ciao,
Norbert
Mar 14 2005
parent reply Georg Wrede <georg.wrede nospam.org> writes:
Norbert Nemec wrote:

 Georg Wrede schrieb:
 

 I've been thinkin hard on a meta language.


 
 Word from the wise: Don't try to reinvent the wheel but learn from those 
 to tried before (even if they failed)
 
 There are loads of programming languages or add-ons do existing 
 languages out there that allow meta-programming in some way. I don't 
 have all the references at hand that I have read over the past years. 
 There have been thorough scientific investigations of the matter. I 
 doubt it is possible to outstrip those projects by starting from scratch.


Thanks. I welcome this and other criticisms. Really! The harder, the 
better. (If anybody can actually convince me that this is either 
impossible or useless, I'll be thankful. The sooner, the more thankful. 
Then I could stop wasting time and bandwidth, and maybe find something 
else where I feel I can contribute to D. :-)  )

If you happen to find some (especially in your own opinion) good 
references, I'd appreciate. Both of successful things, and those 
explaining what or why something should not be done, or is downright 
proved impossible.


 The point of D is not to give it the perfect features for 
 meta-programming but designing features in such a way that they are 
 practical, fit nicely into the general language and don't get in the way 
 with regular programming.


I fully agree with all of that.

If somebody wants to do ultra heavy duty meta programming, they probably 
would do it in Lisp anyway. (Or something.)

However, what I'm _specifically_ out to get, is a syntax that is not 
derived from currently existing template language solutions. (D, C++, 
and the like.)

I can't help feeling that path is a dead end.

While easier for trivial (or at least not too complicated) template 
transforms, I somehow feel (lack of Ph.D. in this area confines me to a 
feelings-only basis here, appologies to all), that such an approach 
turns counter productive already in moderately advanced contexts. 
(Already the  Recursive Templates example 
(http://www.digitalmars.com/d/template.html) seems unnecessarily 
complicated to me. You need to use two separate templates, and an enum. 
And all for something that I wish could be expressed as simply as the 
non-template recursive function.

Not to mention, that I (hope) I'm not the only one who has to look hard 
at what's being done there, even if recursion itself is very familiar to 
me.

Every language is like a person. It appears one way at the surface, and 
once you really get to know them, then you (most often) realise that 
some of the things you thought one could do or couldn't, actually are 
the other way around. (I wrote about that at length in my Master's Thesis.)

The old amongst us remember Mbasic. (That was before MS-DOS and the PC. 
Mbasic was by Microsoft, for the CP/M operating system.) My personal 
_feeling_ was that programming effort grew exponentially with program 
size. When I found Turbo Pascal, the effort seemed to grow 
logarithmically! (I know this is not so, but I'm talking about the 
feeling. Oh, the joy!)

With templates, I again have a feeling. And that is, effort grows too 
fast. (Unscientific, I admit!) Doing easy stuff is easy, even 
next-to-easy is next-to-easy. But already moderate is hard.

NOTE: I think meta and the existing template system, are not mutually 
exclusive! Even if you can do some things in both.


 A preprocessor is not such a great idea, since it will generally split 
 the language into two levels causing all kinds of problems. Walter had 
 good reason to drop the concept of a preprocessor.


True. For the time being, however, when we toy around with what may be 
done with meta language, it seems like less work to have it as a 
preprocessor. I fully agree that (if done at all) this should be done 
respecting Walter's ideas:

  - context free grammar
  - no preprocessor
  - straightforward and practical


 In general, the big pieces of the meta-language already exist in D: 
 templates. These already contain a complete functional programming 
 language that can be used for meta-programming. Several pieces are 
 missing for full comfort and usability, but these are details that have 
 to be fixed.


Yes. And my quest here should in no way slow down that work!


 The meta-language formed by templates certainly is not very handy when 
 trying to do regular calculations at compile time, but then, it has to 
 be clear that this is not the point of a meta-language.


Strictly speaking then, should we call this, er, "quest", something else 
than meta? I'd have no problem with that. (I assume this thread has 
given a pretty good idea to everybody of what I'm aiming at.)


 Meta-programming 
 has the very specific purpose of doing code transformations and for 
 this, templates are excellently designed. It is not a general functional 
 language, but a highly specialized language. 


Code transformations, yes. But I want one to be able to do arbitrary 
code transformations. (Yes, yes, I know, this is reaching for the stars, 
a wild goose chase. And even if we pull this off, the documentation has 
to explicitly state that this is not _supposed_ to be used for 
everything. This _should_ be used sparingly, if at all. And only when 
templates, modules, OO, etc. fail -- and even then "don't try this at 
home". But at this point, know that your job _will_ get done with this.)

Of course, already one could download (Windows people, Others have these 
already) awk, sed, perl, scheme, and do code transformations limited 
only by their own imagination.

But I want more. To reach this "more", this needs to be inside a 
language. (Just like Walter already did with the templates!) Only then 
can we get some serious value-add. (( Not to mention, if this was 
entirely a preprocessor thing, then the Other Guys could use it on their 
out-dated language! :-)  ))

It's a matter of principle. This thing has to be without arbitrary 
limits. Just like a word processor with "unlimited line length, 
unlimited file size". Everybody knows that this doesn't work:

$ wget -O - -r -l 9999 "ftp://archive.loc.gov/texts" | $EDIT -

no matter how "unlimited" the editor is in theory, or what a monster 
mainframe you're currently using. Still it matters for many practical 
purposes that you have "unlimited ll/fs". Same with what I'm pursuing. 
Nobody wants to have the compiler invent the entire source code of the 
Google engine. But we don't want arbitrary limits here either.


 In principle, you can do anything, 


I assume the example applications of meta in this thread would not be 
included in this "anything"?


 but going beyond the special purpose of the meta-language will 
 be both awkward and inefficient.


I wish I can prove different. (I'm not sure at all, but I'm making a 
hell of a try at it.)


 Now: if you can identify single weak points of the existing 
 meta-language and propose clear solutions, these may have a chance.


I will, if I find any.


 For doing a complete redesign of templates, the language is 
 a bit too mature, though.


Yes, let's not tear down what we already have.
Mar 15 2005
next sibling parent reply Matthias Becker <Matthias_member pathlink.com> writes:
(Already the  Recursive Templates example 
(http://www.digitalmars.com/d/template.html) seems unnecessarily 
complicated to me. You need to use two separate templates, and an enum. 
And all for something that I wish could be expressed as simply as the 
non-template recursive function.


I just looked at the example and it is wrong! The factorial of 0 is 1. So it
should be:












Well after doing some meta-programming in C++ this simple example isn't hard to
read.

But how to do this in a simpler way?









Might be cool. But that is a totaly different kind of meta-programming than I
thought would be discussed in this thread. I thought it would be something like
"compiler-plugins" that work on the AST (abstract synthax tree), so we can
define our own language constructs. I never thought about any kind of
compiletime-calculation.

-- Matthias Becker
Mar 15 2005
parent Georg Wrede <georg.wrede nospam.org> writes:
Matthias Becker wrote:

(Already the  Recursive Templates example 
(http://www.digitalmars.com/d/template.html) seems unnecessarily 
complicated to me. You need to use two separate templates, and an enum. 
And all for something that I wish could be expressed as simply as the 
non-template recursive function.




....

 Well after doing some meta-programming in C++ this simple example 
 isn't hard to read.
 
 But how to do this in a simpler way?
 









Yes.


 Might be cool. But that is a totaly different kind of meta-programming than I
 thought would be discussed in this thread. I thought it would be something like
 "compiler-plugins" that work on the AST (abstract synthax tree), so we can
 define our own language constructs. 


Ultimately, I hope, this will be the case.

But I have to admit, if I had created D (that is, if I were Walter :-) ) 
it would take an army to force me to open up the AST at compile time. 
I'd fear this leads to intractable error messages, false errors in 
plainly correct code, people believing DMD is broken, and all just 
because of bugs in people's own meta code.

If this meta thing advances well, then that might become reality. But I 
wouldn't bet it'd be before D 3.0. Even with luck, 2.0 seems remote.

What I do, however, think is possible, is that a kind of non-AST-messing 
version might be incorporated in D 2.0. This would of course be more 
limited than the ultimate version, but it's ok. We really do need some 
time to get used to this sort of thing, see what it might be applied to, 
see if folks use it to do something unexpectedly cool, etc. In doing 
that, we'll have good time to figure out a smooth and practical syntax 
for the 3.0 meta thing.

(Of course not forgetting that none of this might ever even become 
reality! Creativity, perseverance, hard work, but most of all, a good 
dose of plain good luck is needed.)


 I never thought about any kind of compiletime-calculation.


Compile-time calculation opens a new field for us. It gives 
possibilities that are hard to reach otherwise.
Mar 16 2005
prev sibling parent Norbert Nemec <Norbert Nemec-online.de> writes:
Sorry, that I don't go into the details of your mail. The points you 
bring up contains much that I've considered before as well. The topic 
meta-programming is extremely deep. The more you think about it, the 
less you know what way you should go. That's probably why I never really 
stuck with one concept long enough to really elaborate it.

In any case, what you write sounds somewhat like what is tried with the 
OpenC++ project:

	http://opencxx.sourceforge.net/

I never managed go beyond scratching the surface of this, and it seems 
like it is just far too powerful to make it into mainstream development.

There also is an OpenJava project, that takes the ideas and ports them 
to Java. Obviously, the concept does not depend too much on the details 
of the language. Maybe OpenD would be just the project that you were 
thinking of?

I think the best place to start reading about OpenC++ would be the 
programmers guide by Shigeru Chiba:

	http://www.csg.is.titech.ac.jp/~chiba/pub/SPL96-024.ps.gz

I never found the time to dig into it, but maybe you are interested?
Mar 16 2005