www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - custom AST library

reply "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
Hi,

I'm currently making some custom AST parser for D :

http://pastebin.com/1hmnTUbY -- AST Parser

http://pastebin.com/ZRc5be1p -- Sum example

Right now both lexer and syntax parser are just awfull and they don't work 
in CTFE because of class usage. But example looks rather interesting, with 
such approach (if it works) one can create full list comprehension like 


I tried to change class to struct, but failed :-( Any ideas how to make it 
work in CTFE ?
Nov 25 2010
parent reply "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
Hm... there is some limitations in example code, concerning local variables 
in nested functions... here is workaround   http://pastebin.com/mQaKXaYY
Nov 25 2010
next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
I'm just having a look at this now, looks rather interesting! I'd love
to be able to use some Python-like list comprehension tricks in my D
code.

P.S. not sure if it's your fault, but you're code has spaces between
"q" and "{", which gives a compiler error. q{} has to start without
spaces between q and {.   ;)

On 11/25/10, Alex_Dovhal <alex_dovhal yahoo.com> wrote:

 Hm... there is some limitations in example code, concerning local variables
 in nested functions... here is workaround   http://pastebin.com/mQaKXaYY
Dec 21 2010
parent reply "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"Andrej Mitrovic" wrote:

 I'm just having a look at this now, looks rather interesting! I'd love
 to be able to use some Python-like list comprehension tricks in my D
 code.


Yes list comprehension is also possible this way. But it is not very eye 
handy right now even if fully implemented, because one can't use variables 
external to the scope of sum or listcomp... templates, also those 
parenthesis and braces..., so must write:

float a = 1 ;
int n = 10 ;
float x = mixin(sumImpl(q{i=0:10; a*i})) ;
float y = mixin(sumImpl(q{i=1:n; i*mixin(prodImpl(q{j=0:i ; i+j}))}));

instead of:

float a = 1 ;
int n = 10 ;
float x = sum!(q{i=0:10; a*i}) ;
float y = sum!(q{i=1:n; i*prod!(q{j=0:i ; i+j})});

I made it as my first attempt to use D, that's why maybe it's not solid or 
well designed, so any ideas are welcomed.
Dec 21 2010
parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
That's why I think it might be a better idea to simply use "DSL
blocks" throughout your code. Instead of having to mixin each
template, you would only have one template where you have all the
rules for your DSL:

float a = 1 ;
int n = 10 ;
// more D code..

mixin(MyDSL!(q{
    // Here you use the syntax defined by the "MyDSL" language
    float x = sum!(i=0:10; a*i) ;
    float y = sum!(i=1:n; i*prod!(j=0:i ; i+j);
    // More MyDSL code..
}));

// Here you use D syntax again, and D will know about float x and float y.

Not to mention that you could potentially have some control over
syntax highlighting (if you have a capable editor :p). For example you
could write a little script in VIM which detects the use of "MyDSL!()"
template, and highlights the code inside the block differently than
the one outside the block.



On 12/21/10, Alex_Dovhal <alex_dovhal yahoo.com> wrote:

 "Andrej Mitrovic" wrote:

 I'm just having a look at this now, looks rather interesting! I'd love
 to be able to use some Python-like list comprehension tricks in my D
 code.


 Yes list comprehension is also possible this way. But it is not very eye
 handy right now even if fully implemented, because one can't use variables
 external to the scope of sum or listcomp... templates, also those
 parenthesis and braces..., so must write:

 float a = 1 ;
 int n = 10 ;
 float x = mixin(sumImpl(q{i=0:10; a*i})) ;
 float y = mixin(sumImpl(q{i=1:n; i*mixin(prodImpl(q{j=0:i ; i+j}))}));

 instead of:

 float a = 1 ;
 int n = 10 ;
 float x = sum!(q{i=0:10; a*i}) ;
 float y = sum!(q{i=1:n; i*prod!(q{j=0:i ; i+j})});

 I made it as my first attempt to use D, that's why maybe it's not solid or
 well designed, so any ideas are welcomed.
Dec 21 2010
parent reply "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"Andrej Mitrovic" <andrej.mitrovich gmail.com> wrote:

 That's why I think it might be a better idea to simply use "DSL
 blocks" throughout your code. Instead of having to mixin each
 template, you would only have one template where you have all the
 rules for your DSL:

 float a = 1 ;
 int n = 10 ;
 // more D code..

 mixin(MyDSL!(q{
    // Here you use the syntax defined by the "MyDSL" language
    float x = sum!(i=0:10; a*i) ;
    float y = sum!(i=1:n; i*prod!(j=0:i ; i+j);
    // More MyDSL code..
 }));

 // Here you use D syntax again, and D will know about float x and float y.

 Not to mention that you could potentially have some control over
 syntax highlighting (if you have a capable editor :p). For example you
 could write a little script in VIM which detects the use of "MyDSL!()"
 template, and highlights the code inside the block differently than
 the one outside the block.


Very interesting idea :), one note - sum!(...) in DSL is not template any 
more, so one can change a design as he wants, say:

mixin(MyDSL_Impl(q{
    float x = sum {i=0:10 ; a * i};
    float y = sum {i=1:n; i*prod{j=0:i; i+j}} ;
}));

this MyDSL_Impl only scans for "name" "{" text "}" and replaces it with 
mixin(nameImpl(q{text})) ;

User can put entire function body or even almost full module (if don't care 
about compile time) into that DSL in order not to change from DSL to normal 
code and back. One more thing to think about - how to show correct error 
line number in there?
Dec 21 2010
parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 12/21/10, Alex_Dovhal <alex_dovhal yahoo.com> wrote:

 User can put entire function body or even almost full module (if don't care
 about compile time) into that DSL in order not to change from DSL to normal
 code and back.


Yep. You could even take it a step further and use some form of escape
mechanism, which would leave D code alone but parse the rest of the
string as a DSL. E.g.:

mixin(MyDSL!(q{

    $(D)
    float a = 1 ;    // will be mixed in as is
    int n = 10 ;
    void test(int t)
    {
        writeln(t);
    }
    $(/D)  // end of D code

   x = sum(i=0:10; a*i) ;  // will be parsed and reconstructed as D code
   y = sum(i=1:n; i*prod!(j=0:i ; i+j);
}));

So now you wouldn't have to write a lot of MyDSL blocks in-between D code.


 One more thing to think about - how to show correct error
 line number in there?


See https://docs.google.com/leaf?id=0B-f1J0HiG2ovNmQzNzM1ODYtNDM3ZC00N2I4LWI2YmUtYTIyM2Y2MzcyZWIw&sort=name&layout=list&num=50&pli=1
, there's a unittests.d module in /src in the archive. It uses a
parameter that is default initialized to the line (size_t line =
__LINE__))

I guess from there you could chomp up your string into an array for
each new line, and if your DSL parser finds a syntax error it would
report the line number as e.g. currentLineNumber + line, where
currentLineNumber is the line number you're currently parsing, e.g.
DSLStringArray[currentLineNumber] << this line has the error.

I think that's about it (but I haven't had my nighttime coffee yet so
don't trust me fully on this one :p).
Dec 21 2010
prev sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
SORRY.

Disregard that, it was my fault. My reindenter f*ked it up. You're
code is fine. :)

On 12/21/10, Andrej Mitrovic <andrej.mitrovich gmail.com> wrote:

 I'm just having a look at this now, looks rather interesting! I'd love
 to be able to use some Python-like list comprehension tricks in my D
 code.

 P.S. not sure if it's your fault, but you're code has spaces between
 "q" and "{", which gives a compiler error. q{} has to start without
 spaces between q and {.   ;)

 On 11/25/10, Alex_Dovhal <alex_dovhal yahoo.com> wrote:

 Hm... there is some limitations in example code, concerning local
 variables
 in nested functions... here is workaround   http://pastebin.com/mQaKXaYY
Dec 21 2010