• Tove (16/16) Feb 27 2014 Projects such as Pegged and our CTFE regex engine often serve as

"Tove" <tove fransson.se> writes:

Projects such as Pegged and our CTFE regex engine often serve as 
poster-children of what is possible in D and many agree they are 
among the more important projects.

I was thinking, after std.lexer is accepted, we have a stable 
interface, but no matter how great the code is and even if it 
beats the already superlative DMD lexer, it will _NOT_ be fast 
during CTFE.

It is also often stressed in this very forum that it's paramount 
for a lexer to be beyond fast.

For the above reasons, I propose that the compiler would offer an 
interface to retrieve an already lexed buffer, similar in concept 
to the existing token string q{...}

"Token strings open with the characters q{ and close with the 
token }. In between must be valid D tokens."

By definition they contain only valid tokens... the compiler 
would only have to create a range compatible with std.lexer...