• Mike James (8/8) Aug 29 2014 Hi,
• ketmar via Digitalmars-d-learn (6/12) Aug 29 2014 On Fri, 29 Aug 2014 13:41:20 +0000
• Ary Borenszweig (7/14) Aug 29 2014 I believe this is just an abstract description of how it works. It

"Mike James" <foo bar.com> writes:

Hi,

Looking at the DMD Source Guide it says "The lexer transforms the 
file into an array of tokens."

Why is this step taken instead of, say, just calling a function 
that returns the next token (or however many required for the 
look-ahead)?

Regards,
   -=mike=-

ketmar via Digitalmars-d-learn <digitalmars-d-learn puremagic.com> writes:

On Fri, 29 Aug 2014 13:41:20 +0000
Mike James via Digitalmars-d-learn <digitalmars-d-learn puremagic.com>
wrote:

 Looking at the DMD Source Guide it says "The lexer transforms the=20
 file into an array of tokens."
=20
 Why is this step taken instead of, say, just calling a function=20
 that returns the next token (or however many required for the=20
 look-ahead)?

D parser is somewhat complicated and it can peek alot of tokens from
input stream without consuming 'em. so it's just easier to convert
source code to tokens and then work with token stream.

Ary Borenszweig <ary esperanto.org.ar> writes:

On 8/29/14, 10:41 AM, Mike James wrote:
 Hi,

 Looking at the DMD Source Guide it says "The lexer transforms the file
 into an array of tokens."

 Why is this step taken instead of, say, just calling a function that
 returns the next token (or however many required for the look-ahead)?

 Regards,
    -=mike=-

I believe this is just an abstract description of how it works. It 
actually uses something like next token with a freelist of tokens if it 
needs to do some lookahead.

https://github.com/D-Programming-Language/dmd/blob/master/src/lexer.h#L260
https://github.com/D-Programming-Language/dmd/blob/master/src/lexer.h#L261
https://github.com/D-Programming-Language/dmd/blob/master/src/lexer.h#L237