• snowCat (8/8) Mar 10 2018 Recently, the task of implementing the FALSE programming language
• =?UTF-8?B?UsOpbXkgTW91w6t6YQ==?= (25/34) Mar 10 2018 I implemented a small concatenative language in 2016 and used

snowCat <disconnectix gmail.com> writes:

Recently, the task of implementing the FALSE programming language 
on D has come up and I have faced the problem of splitting the 
FALSE expression into a list of strings from its typical 
elements: blocks of comments, numbers, strings and operators.
How to properly implement such a partition, tell me please.
Or at least tell me how to describe his grammar in Pegged.
Your name in the article about the implementation of my blog is 
guaranteed.

=?UTF-8?B?UsOpbXkgTW91w6t6YQ==?= <remy.moueza-dontspamme gmail.com> writes:

On Saturday, 10 March 2018 at 11:47:48 UTC, snowCat wrote:
 Recently, the task of implementing the FALSE programming 
 language on D has come up and I have faced the problem of 
 splitting the FALSE expression into a list of strings from its 
 typical elements: blocks of comments, numbers, strings and 
 operators.
 How to properly implement such a partition, tell me please.
 Or at least tell me how to describe his grammar in Pegged.
 Your name in the article about the implementation of my blog is 
 guaranteed.

I implemented a small concatenative language in 2016 and used 
Pegged for the lexing.
I uploaded the source on github: 
https://github.com/remy-j-a-moueza/stacky.
The Pegged grammar is at the beginning of `stacky.d`.
Pegged is also pretty well documented in its wiki and tutorial 
(https://github.com/PhilippeSigaud/Pegged/wiki/Pegged-Tutorial).

As for a hand made lexer, the last time I clearly remember 
working on one was in 2007, on D1, trying to lex C++.
I copied part of that code here: 
https://gist.github.com/remy-j-a-moueza/8909819cbf972430bfbb16dff768b97d

The algorithm is mainly:
- try to match a regular expression
-- on a match: create a token for what has been matched, move 
forward in your string input, loop to the next iteration,
-- on error: try again with another regular expression.
By the end you should have accumulated your tokens.

There is a lot of resources online about parsing, in various 
programming languages and libraries, way better than what I have 
provided. Whatever the tools, the principles stay the same. 
Whatever you can learn somewhere else will still be usable in D, 
just a bit differently.

I hope this help.
Have fun.