www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How to best implement a DSL?

reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:
Hi, I'm seeking for ideas/comments/experiences how to best implement a 
DSL in D.

What I would like to do is something like this:

	... my D code ...

	my-dsl {
		... my multi-line DSL code ...
		trade 100 shares(x) when (time < 20:00) and timingisright()
	}


	... my D code ...


Some things that circle in my head:
* Can the D parser somehow be missued for a DSL? So I can skip all the 
generic features for types etc.?

* I could use a PEG grammer for parsing the DSL, but this leads to 
quite some overhead for a tiny DSL.

* For static DSL code I would like to use CTFE to convert it into D code
	* Does this requires a CTFE compatible PEG parser tookit?
	* Could I somehow call an external program during compilation which 
gets the DSL block as input and returns D code?

* For dynamic DSL code I think I need to create something like an interpreter
	* How can I reference D variables from DSL code? Is there a lookup 
meachnisam or do I have to create a dictonary?
	* Is it possible to populate such a dictonary via CTFE?


-- 
Robert M. Münch
http://www.saphirion.com
smarter | better | faster
Jul 28 2018
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 29/07/2018 2:59 AM, Robert M. Münch wrote:

 Hi, I'm seeking for ideas/comments/experiences how to best implement a 
 DSL in D.
 
 What I would like to do is something like this:
 
      ... my D code ...
 
      my-dsl {
          ... my multi-line DSL code ...
          trade 100 shares(x) when (time < 20:00) and timingisright()
      }
 
 
      ... my D code ...
 
 
 Some things that circle in my head:
 * Can the D parser somehow be missued for a DSL? So I can skip all the 
 generic features for types etc.?


Let's go with no.


 * I could use a PEG grammer for parsing the DSL, but this leads to quite 
 some overhead for a tiny DSL.
 
 * For static DSL code I would like to use CTFE to convert it into D code
      * Does this requires a CTFE compatible PEG parser tookit?


Yes.


      * Could I somehow call an external program during compilation which 
 gets the DSL block as input and returns D code?


No. But you can pre-process.


 * For dynamic DSL code I think I need to create something like an 
 interpreter
      * How can I reference D variables from DSL code? Is there a lookup 
 meachnisam or do I have to create a dictonary?


Registration, but like you said, a giant dictionary.


      * Is it possible to populate such a dictonary via CTFE?


Sort of, it can be registered, but the actual execution of the 
registration occurs at runtime.



But you're slightly over thinking this. Write an interpreter and a 
parser. The fact that the parser can work at CTFE is irrelevant and 
almost a footnote on the page of details ;).

If you want to be clever you could generate D code and mix it in based 
upon what was parsed. But you'd still want that dictionary.
Jul 28 2018
parent reply =?iso-8859-1?Q?Robert_M._M=FCnch?= <robert.muench saphirion.com> writes:
On 2018-07-28 15:43:12 +0000, rikki cattermole said:


     * Could I somehow call an external program during compilation which 
 gets the DSL block as input and returns D code?


 
 No. But you can pre-process.


Yes, sure, but this complicates the build-system. I preferr to use as 
few parts as possible.



     * Is it possible to populate such a dictonary via CTFE?


 
 Sort of, it can be registered, but the actual execution of the 
 registration occurs at runtime.


Yes, no problem. How is this done?


 But you're slightly over thinking this. Write an interpreter and a 
 parser. The fact that the parser can work at CTFE is irrelevant and 
 almost a footnote on the page of details ;).


Not really if you think about deployment. I want to create a single 
executable, no external files. So I somehow have to get the scripts 
included or use the compiler as generator. The main use-case is, that I 
want to use a very declarative approach for some UI parts.


 If you want to be clever you could generate D code and mix it in based 
 upon what was parsed. But you'd still want that dictionary.


Yes, mixing in generate D code was the idea. However, I think a good 
included infrastructure that support creating DSLs might help a lot to 
simplify this.


-- 
Robert M. Münch
http://www.saphirion.com
smarter | better | faster
Jul 28 2018
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 29/07/2018 4:53 AM, Robert M. Münch wrote:

 On 2018-07-28 15:43:12 +0000, rikki cattermole said:
 
 
      * Could I somehow call an external program during compilation which 
 gets the DSL block as input and returns D code?
 
 
 No. But you can pre-process.
 
 
 Yes, sure, but this complicates the build-system. I preferr to use as 
 few parts as possible.
 
 
 
      * Is it possible to populate such a dictonary via CTFE?
 
 
 Sort of, it can be registered, but the actual execution of the 
 registration occurs at runtime.
 
 
 Yes, no problem. How is this done?


Usually either a mixin template or a string mixin that plops a module 
constructor down with it calling a register function with whatever data 
you happened to compute/create using CTFE.


 But you're slightly over thinking this. Write an interpreter and a 
 parser. The fact that the parser can work at CTFE is irrelevant and 
 almost a footnote on the page of details ;).
 
 
 Not really if you think about deployment. I want to create a single 
 executable, no external files. So I somehow have to get the scripts 
 included or use the compiler as generator. The main use-case is, that I 
 want to use a very declarative approach for some UI parts.


You missed my point here.
There is nothing special about parsing at CTFE, you're just restricted 
as to the language features you can use (e.g. no extern's), that's it.
Jul 28 2018
parent Matthew OConnor <thegreendragon gmail.com> writes:
On Saturday, 28 July 2018 at 17:01:22 UTC, rikki cattermole wrote:

 You missed my point here.
 There is nothing special about parsing at CTFE, you're just 
 restricted as to the language features you can use (e.g. no 
 extern's), that's it.


Is there an example of how this could be done?
Aug 24 2018
prev sibling next sibling parent Mr.Bingo <Bingo Namo.com> writes:
On Saturday, 28 July 2018 at 14:59:31 UTC, Robert M. Münch wrote:

 Hi, I'm seeking for ideas/comments/experiences how to best 
 implement a DSL in D.

 What I would like to do is something like this:

 	... my D code ...

 	my-dsl {
 		... my multi-line DSL code ...
 		trade 100 shares(x) when (time < 20:00) and timingisright()
 	}


 	... my D code ...


 Some things that circle in my head:
 * Can the D parser somehow be missued for a DSL? So I can skip 
 all the generic features for types etc.?

 * I could use a PEG grammer for parsing the DSL, but this leads 
 to quite some overhead for a tiny DSL.

 * For static DSL code I would like to use CTFE to convert it 
 into D code
 	* Does this requires a CTFE compatible PEG parser tookit?
 	* Could I somehow call an external program during compilation 
 which gets the DSL block as input and returns D code?

 * For dynamic DSL code I think I need to create something like 
 an interpreter
 	* How can I reference D variables from DSL code? Is there a 
 lookup meachnisam or do I have to create a dictonary?
 	* Is it possible to populate such a dictonary via CTFE?


Why not simply turn your dsl in to D code? If it is pre-existing 
you'll need a simple mapping.

trade 100 shares(x) when (time < 20:00) and timingisright()

could be written any number of ways:

trade(100.shares(x).when("time < 20.0", "&", "timingisright()")

Simply construct your grammar around D types and use UFSC, 
members, and ranges and then either use it directly or map to it.

If the grammar is relatively small but simply lots of variations 
then it should be rather easy(e.g., if you just have trade, 
shares, and when then it would be very easy and the work would in 
implementation.
Jul 29 2018
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2018-07-28 16:59, Robert M. Münch wrote:

 Hi, I'm seeking for ideas/comments/experiences how to best implement a
 DSL in D.

 What I would like to do is something like this:

      ... my D code ...

      my-dsl {
          ... my multi-line DSL code ...
          trade 100 shares(x) when (time < 20:00) and timingisright()
      }


      ... my D code ...


 Some things that circle in my head:
 * Can the D parser somehow be missued for a DSL? So I can skip all the
 generic features for types etc.?


Yes. But it would require your DSL to be syntactically valid D code (but 
not semantically). DMD can be used as a library but is lacking quite a 
bit i this regard. For example, you cannot get the original source code 
from an AST node because the AST node only stores where in the source 
code the node starts, not where it ends. Some AST nodes doesn't contain 
location information at all. Also, unless you want to write the DSL in a 
string literal you need to build some kind of pre-processor that 
converts the DSL to valid D code.

-- 
/Jacob Carlborg
Jul 31 2018