www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Scanner / Parser for D2

reply Arild Boes <aboesx gmail.com> writes:
Hello,

I'd like to have a c++ parser for D. I realized, that the full source 
code of DMD was avaliable and went to fetch it. (thanks for that btw!).

However it seems rather complex, and I sure could use some quick 
insights regarding how the lexer / scanner part could be used, favorably 
seperately, from the rest of the compiler.

The simple goal is to fiddle with color coding in vs 2008 with the 
language SDK, and then perhaps move on to more interesting stuff.
All it really needs for the simple stuff is a scanner. I previously did 
some succesfull tests with a sablecc generated parser and some toy grammar.

I havent really had the time to examine the dmd source in detail yet, so 
i am hedgeing my bets here, not at all expecting to get a full detailed 
walktrough. Hints in the general direction would be much appreciated though.

postscript: being a silent news reader and D fan for some time, I would 
like to utilize this post to express my gratitude for the kickass 
language D is, and for the work everyone has put into it.

Thanks in advance.
Apr 22 2009
next sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2009-04-22 16:33:05 -0400, Arild Boes <aboesx gmail.com> said:


 Hello,
 
 I'd like to have a c++ parser for D. I realized, that the full source 
 code of DMD was avaliable and went to fetch it. (thanks for that btw!).
 
 However it seems rather complex, and I sure could use some quick 
 insights regarding how the lexer / scanner part could be used, 
 favorably seperately, from the rest of the compiler.
 
 The simple goal is to fiddle with color coding in vs 2008 with the 
 language SDK, and then perhaps move on to more interesting stuff.


I've done something like it in my D plugin for Xcode. For Xcode 3 I've 
created a grammar for Xcode's built-in scanner, but for earlier 
versions I'm using the lexer from an old version of DMD. Perhaps you'd 
like to take a look.
<http://michelf.com/projects/d-for-xcode/>


-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
Apr 22 2009
parent Arild Boes <aboesx gmail.com> writes:
Michel Fortin skrev:

 I've done something like it in my D plugin for Xcode. For Xcode 3 I've 
 created a grammar for Xcode's built-in scanner, but for earlier versions 
 I'm using the lexer from an old version of DMD. Perhaps you'd like to 
 take a look.
 <http://michelf.com/projects/d-for-xcode/>
 
 


Thanks, that might be worth a look :)

/ab
Apr 23 2009
prev sibling parent reply "Unknown W. Brackets" <unknown simplemachines.org> writes:
I went and rewrote the bulk of the main parser/lexer in a way that was 
more VSIP-friendly, which worked out for me.  Unfortunately, it's just a 
separate thing, and I need to update it badly at this point.

One problem is, you really want to be able to scan/lex by line.  If you 
don't do some caching/optimization based on that in Visual Studio, 
you'll be eating dirt.  The design of the parser and lexer in D doesn't 
really lend itself to that, as-is.

But, I never went much farther than colored tokens and brace matching, 
and it's been a few months since I've had time to mess with it.

You may also be interested in Dave Sieber's DCoder, which was on dsource 
iirc, although it was built trying to use yacc/etc. for D, which didn't 
really work out well at the time.  It was for 2003 iirc.

-[Unknown]


Arild Boes wrote:

 Hello,
 
 I'd like to have a c++ parser for D. I realized, that the full source 
 code of DMD was avaliable and went to fetch it. (thanks for that btw!).
 
 However it seems rather complex, and I sure could use some quick 
 insights regarding how the lexer / scanner part could be used, favorably 
 seperately, from the rest of the compiler.
 
 The simple goal is to fiddle with color coding in vs 2008 with the 
 language SDK, and then perhaps move on to more interesting stuff.
 All it really needs for the simple stuff is a scanner. I previously did 
 some succesfull tests with a sablecc generated parser and some toy grammar.
 
 I havent really had the time to examine the dmd source in detail yet, so 
 i am hedgeing my bets here, not at all expecting to get a full detailed 
 walktrough. Hints in the general direction would be much appreciated 
 though.
 
 postscript: being a silent news reader and D fan for some time, I would 
 like to utilize this post to express my gratitude for the kickass 
 language D is, and for the work everyone has put into it.
 
 Thanks in advance.
Apr 23 2009
parent reply Arild Boes <abo dae.dk> writes:
Thanks.

I had not really thought about efficiency of the scanner with regards to 
realtime usage, and you are proberbly right about scanning/lexing on a 
line by line basis. Maybe the right thing to do, is to write a custom 
scanner that accomodates that. I was trying to avoid this however, 
because changes in the language, means changing the lexer obviously.
An viable alternative would be to use some scanner generator, that suits 
my needs. Now only the grammar needs to be updated.

The DCoder thing sounds interesting.

/ab

Unknown W. Brackets skrev:

 I went and rewrote the bulk of the main parser/lexer in a wa


y that was

 more VSIP-friendly, which worked out for me.  Unfortunately, it's just a 
 separate thing, and I need to update it badly at this point.
 
 One problem is, you really want to be able to scan/lex by line.  If you 
 don't do some caching/optimization based on that in Visual Studio, 
 you'll be eating dirt.  The design of the parser and lexer in D doesn't 
 really lend itself to that, as-is.
 
 But, I never went much farther than colored tokens and brace matching, 
 and it's been a few months since I've had time to mess with it.
 
 You may also be interested in Dave Sieber's DCoder, which was on dsource 
 iirc, although it was built trying to use yacc/etc. for D, which didn't 
 really work out well at the time.  It was for 2003 iirc.
 
 -[Unknown]
Apr 24 2009
parent "Unknown W. Brackets" <unknown simplemachines.org> writes:
Just be warned, I spent time trying to make the grammar correct and I 
just ran into shift/reduce conflicts.  I don't think D's grammar 
(especially with nested things) can be properly understood by flex and 
bison.

I may be wrong, though, I never liked .y and .l files, so it may just be 
that I was bad at them.

What I wanted to do, eventually, was try to reuse more of DMD's existing 
code.  I mean, it's only a few key routines that aren't newline 
friendly, and if a proper plan was laid out Walter might even be 
friendly to changing them, I'd guess.

-[Unknown]


Arild Boes wrote:

 Thanks.
 
 I had not really thought about efficiency of the scanner with regards to 
 realtime usage, and you are proberbly right about scanning/lexing on a 
 line by line basis. Maybe the right thing to do, is to write a custom 
 scanner that accomodates that. I was trying to avoid this however, 
 because changes in the language, means changing the lexer obviously.
 An viable alternative would be to use some scanner generator, that suits 
 my needs. Now only the grammar needs to be updated.
 
 The DCoder thing sounds interesting.
 
 /ab
 
 Unknown W. Brackets skrev:

 I went and rewrote the bulk of the main parser/lexer in a wa


 
 y that was

 more VSIP-friendly, which worked out for me.  Unfortunately, it's just 
 a separate thing, and I need to update it badly at this point.

 One problem is, you really want to be able to scan/lex by line.  If 
 you don't do some caching/optimization based on that in Visual Studio, 
 you'll be eating dirt.  The design of the parser and lexer in D 
 doesn't really lend itself to that, as-is.

 But, I never went much farther than colored tokens and brace matching, 
 and it's been a few months since I've had time to mess with it.

 You may also be interested in Dave Sieber's DCoder, which was on 
 dsource iirc, although it was built trying to use yacc/etc. for D, 
 which didn't really work out well at the time.  It was for 2003 iirc.

 -[Unknown]
Apr 25 2009