www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - AST like coding syntax. Easy upgrade!

reply "Prudence" <Pursuit Happyness.All> writes:
template X(Y)
{
	string X = Y.stringof;
}

auto s = X({int 3;})

Of course, doesn't work!!

But having the ability to pass code that isn't contained in a 
string is very useful!!


1. A new code keyword, similar to alias. Can only be used as 
template parameters. If you are worried about backwards 
compatibility, call it _code, or __code, or __code__... I don't 
care, only sticks and stones hurt me.

2. The argument is only checked for syntaxical errors as the 
argument. It obviously will be checked. If we allow for other 
types of code besides D, then this won't work. We could extend 
code to allow for other languages(which would require their 
syntax checking algorithms): code(python).

e.g.,

template PythonParser X(code(python) pythoncode)
{
     // Parses pythoncode and coverts it into D or passes it to 
the python interpreter or whatever...
}

In this case, D will know it is python code, check for a python 
code parser and deal with it(use an external lib to check it or 
have built in grammars for such things).

No code is ever actually executed by this extension in the 
compiler, so it's somewhat of a trivial addition. All the 
processing is added by the programmer.

3. stringof returns the code as a string. e.g. 
pythoncode.stringof is simply the string representation of the 
block of code passed.


This complements D's mixin mechanisms by getting out of having to 
put code in strings, which are nearly a clusterfuck for complex 
code.



4. This opens the door to allow for one to migrate code easier to 
D. Suppose you have to transfer a large code base in, say, 
javascript. Instead of having to convert it all by hand, you 
could have something like


template IncludeJS(code(javscript))
{
    magiccookie(javscript.stringof);
}

... converts and integrates the javascript code into D or simply 
interprets it and returns the error code, or whatever. It's not 
unfeasible to think that someone could write the magiccookie that 
brings in all the objects, functions, and such into D to be 
consumed in D code. I've seen this done for several scripting 
languages such as lua and js.
Sep 06 2015
next sibling parent reply "Zoadian" <no no.no> writes:
On Sunday, 6 September 2015 at 19:32:58 UTC, Prudence wrote:

 template X(Y)
 {
 	string X = Y.stringof;
 }

 [...]



as you'd have to write a parser for other languages why not just 
use strings? you can already do this:

template X(string Y)
{
	enum X = Y;
}

auto s = X!q{int 3;};

obviously X has to be a compiletime js->d compiler.
Sep 06 2015
next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 6 September 2015 at 20:22:23 UTC, Zoadian wrote:

 obviously X has to be a compiletime js->d compiler.


Just a fun fact: my script.d's interpreter is itself CTFEable in 
modern dmd!


import arsd.script;

void main() {
         // script.d is similar to but not identical to javascript
         // so run that code in the interpreter and fetch a D
         // type right out all at compile time...
         enum a = interpret("var a = 5; a += 2; a;").get!int;
         pragma(msg, a);
}


$ dmd d ~/arsd/jsvar ~/arsd/script
7


jsvar.d and script.d can be found here:
https://github.com/adamdruppe/arsd
Sep 06 2015
next sibling parent reply "Prudence" <Pursuit Happyness.All> writes:
On Sunday, 6 September 2015 at 20:38:44 UTC, Adam D. Ruppe wrote:

 On Sunday, 6 September 2015 at 20:22:23 UTC, Zoadian wrote:

 obviously X has to be a compiletime js->d compiler.


 Just a fun fact: my script.d's interpreter is itself CTFEable 
 in modern dmd!


 import arsd.script;

 void main() {
         // script.d is similar to but not identical to 
 javascript
         // so run that code in the interpreter and fetch a D
         // type right out all at compile time...
         enum a = interpret("var a = 5; a += 2; a;").get!int;
         pragma(msg, a);
 }


 $ dmd d ~/arsd/jsvar ~/arsd/script
 7


 jsvar.d and script.d can be found here:
 https://github.com/adamdruppe/arsd



Yeah, but wouldn't it be so much nicer? (and probably debuggable 
inline)

interpret({
     var a = 5;
     a += 2;
     a;
}

With full compiler error support? (if it's not D, the external 
parser could return the error, line and position info)

Or, maybe better yet, have the concept of "code strings". which 
are strings that are suppose to be interpreted as code. This then 
means the compiler just has to do a syntax check for errors 
before it does anything else with them(semantics will be checked 
lazy, which is already implemented).
Sep 06 2015
next sibling parent reply "cym13" <cpicard openmailbox.org> writes:
On Sunday, 6 September 2015 at 21:16:18 UTC, Prudence wrote:

 Or, maybe better yet, have the concept of "code strings". which 
 are strings that are suppose to be interpreted as code. This 
 then means the compiler just has to do a syntax check for 
 errors before it does anything else with them(semantics will be 
 checked lazy, which is already implemented).


There already is a kind of "code string":

     interpret(q{
          var a = 2;
          var b += a;
     });

It doesn't do any kind of syntax check, but there again how do 
you want to have syntax check for any language? The D compiler is 
a D compiler, it can't support js syntax or whatever.
Sep 06 2015
next sibling parent reply "bitwise" <bitwise.pvt gmail.com> writes:
On Sunday, 6 September 2015 at 22:37:16 UTC, cym13 wrote:

 On Sunday, 6 September 2015 at 21:16:18 UTC, Prudence wrote:

[...]


 There already is a kind of "code string":

     interpret(q{
          var a = 2;
          var b += a;
     });

 It doesn't do any kind of syntax check, but there again how do 
 you want to have syntax check for any language? The D compiler 
 is a D compiler, it can't support js syntax or whatever.


Many IDEs support multiple languages and can infer language 
automatically by syntax. It's probably much more difficult than 
it seems, but I suppose one of these IDEs could be made to parse 
and infer D token strings separately.
Sep 06 2015
parent reply "cym13" <cpicard openmailbox.org> writes:
On Sunday, 6 September 2015 at 23:00:21 UTC, bitwise wrote:

 On Sunday, 6 September 2015 at 22:37:16 UTC, cym13 wrote:

 On Sunday, 6 September 2015 at 21:16:18 UTC, Prudence wrote:

[...]


 There already is a kind of "code string":

     interpret(q{
          var a = 2;
          var b += a;
     });

 It doesn't do any kind of syntax check, but there again how do 
 you want to have syntax check for any language? The D compiler 
 is a D compiler, it can't support js syntax or whatever.


 Many IDEs support multiple languages and can infer language 
 automatically by syntax. It's probably much more difficult than 
 it seems, but I suppose one of these IDEs could be made to 
 parse and infer D token strings separately.


Sure, but the support for that will be an external tool, it 
doesn't have anything to do in the D compiler. q{} strings are 
meant to be seen specially by editors, they won't highlight them 
the same way for example, it is then the editor's job to detect 
other languages if it wants to. D has done his job in the matter.
Sep 06 2015
parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Sunday, 6 September 2015 at 23:38:51 UTC, cym13 wrote:

 On Sunday, 6 September 2015 at 23:00:21 UTC, bitwise wrote:

 On Sunday, 6 September 2015 at 22:37:16 UTC, cym13 wrote:

 On Sunday, 6 September 2015 at 21:16:18 UTC, Prudence wrote:

[...]


 There already is a kind of "code string":

     interpret(q{
          var a = 2;
          var b += a;
     });

 It doesn't do any kind of syntax check, but there again how 
 do you want to have syntax check for any language? The D 
 compiler is a D compiler, it can't support js syntax or 
 whatever.


 Many IDEs support multiple languages and can infer language 
 automatically by syntax. It's probably much more difficult 
 than it seems, but I suppose one of these IDEs could be made 
 to parse and infer D token strings separately.


 Sure, but the support for that will be an external tool, it 
 doesn't have anything to do in the D compiler. q{} strings are 
 meant to be seen specially by editors, they won't highlight 
 them the same way for example, it is then the editor's job to 
 detect other languages if it wants to. D has done his job in 
 the matter.


Editors will have a hard time highlighting q{} strings 
differently, since they'll need to understand the semantics in 
order to know how the string will be parsed. Compare it to Ruby's 
heredoc, where the chosen terminator string can be used as an 
hint(https://github.com/joker1007/vim-ruby-heredoc-syntax). Sure, 
it may be just a convention, but an easily kept one that can make 
programmers' life easier.

You can't do that with D's q{} strings, unless you hard-code into 
the editor's relevant syntax file the templates that use them, 
just like the regular syntax of the language.
Sep 06 2015
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Monday, 7 September 2015 at 00:34:20 UTC, Idan Arye wrote:

 Compare it to Ruby's heredoc, where the chosen terminator 
 string can be used as an 
 hint(https://github.com/joker1007/vim-ruby-heredoc-syntax).


Or D's heredoc strings, yes, we have them too: 
http://dlang.org/lex.html (search for "heredoc" or "delimited 
string" on that page)

For example:

---
void main() {
         pragma(msg, q"RUBY
   def foo
     if a.nil?
       puts :lol
     end
   end
RUBY");
}
---
Sep 06 2015
prev sibling parent reply anonymous <anonymous example.com> writes:
On Monday 07 September 2015 00:37, cym13 wrote:


 There already is a kind of "code string":
 
      interpret(q{
           var a = 2;
           var b += a;
      });
 
 It doesn't do any kind of syntax check, but there again how do 
 you want to have syntax check for any language? The D compiler is 
 a D compiler, it can't support js syntax or whatever.


There's a very basic syntax check: Token strings (q{...}) go through 
tokenization. Compilation fails when the contents aren't valid tokens. For 
example, q{'} fails with "Error: unterminated character constant".
Sep 06 2015
next sibling parent reply "cym13" <cpicard openmailbox.org> writes:
On Sunday, 6 September 2015 at 23:40:58 UTC, anonymous wrote:

 On Monday 07 September 2015 00:37, cym13 wrote:


 There already is a kind of "code string":
 
      interpret(q{
           var a = 2;
           var b += a;
      });
 
 It doesn't do any kind of syntax check, but there again how do 
 you want to have syntax check for any language? The D compiler 
 is a D compiler, it can't support js syntax or whatever.


 There's a very basic syntax check: Token strings (q{...}) go 
 through tokenization. Compilation fails when the contents 
 aren't valid tokens. For example, q{'} fails with "Error: 
 unterminated character constant".


Then you can't put anything but D-like code... A string is way 
better for that purpose, the q{} gives a handle to the editor and 
I can't see a thing in there that can't (and shoudln't) be done 
at the library level.
Sep 06 2015
parent "cym13" <cpicard openmailbox.org> writes:
On Sunday, 6 September 2015 at 23:48:30 UTC, cym13 wrote:

 On Sunday, 6 September 2015 at 23:40:58 UTC, anonymous wrote:

 On Monday 07 September 2015 00:37, cym13 wrote:


 There already is a kind of "code string":
 
      interpret(q{
           var a = 2;
           var b += a;
      });
 
 It doesn't do any kind of syntax check, but there again how 
 do you want to have syntax check for any language? The D 
 compiler is a D compiler, it can't support js syntax or 
 whatever.


 There's a very basic syntax check: Token strings (q{...}) go 
 through tokenization. Compilation fails when the contents 
 aren't valid tokens. For example, q{'} fails with "Error: 
 unterminated character constant".




Ah, just reread the post, I was OT sorry for my last post.
Sep 06 2015
prev sibling parent reply "Idan Arye" <GenericNPC gmail.com> writes:
On Sunday, 6 September 2015 at 23:40:58 UTC, anonymous wrote:

 On Monday 07 September 2015 00:37, cym13 wrote:


 There already is a kind of "code string":
 
      interpret(q{
           var a = 2;
           var b += a;
      });
 
 It doesn't do any kind of syntax check, but there again how do 
 you want to have syntax check for any language? The D compiler 
 is a D compiler, it can't support js syntax or whatever.


 There's a very basic syntax check: Token strings (q{...}) go 
 through tokenization. Compilation fails when the contents 
 aren't valid tokens. For example, q{'} fails with "Error: 
 unterminated character constant".


That's not considered as syntax check - that's an earlier stage 
of the compilation process called "lexical 
analysis"(https://en.wikipedia.org/wiki/Lexical_analysis)
Sep 06 2015
parent reply anonymous <anonymous example.com> writes:
On Monday 07 September 2015 02:24, Idan Arye wrote:


 That's not considered as syntax check - that's an earlier stage 
 of the compilation process called "lexical 
 analysis"(https://en.wikipedia.org/wiki/Lexical_analysis)


From the Wikipedia article: "a lexer is generally combined with a parser, 
which together analyze the syntax".
Sep 06 2015
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 07-Sep-2015 08:25, anonymous wrote:

 On Monday 07 September 2015 02:24, Idan Arye wrote:


 That's not considered as syntax check - that's an earlier stage
 of the compilation process called "lexical
 analysis"(https://en.wikipedia.org/wiki/Lexical_analysis)


  From the Wikipedia article: "a lexer is generally combined with a parser,
 which together analyze the syntax".


No in this case obviously.

-- 
Dmitry Olshansky
Sep 07 2015
prev sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 6 September 2015 at 21:16:18 UTC, Prudence wrote:

 Yeah, but wouldn't it be so much nicer? (and probably 
 debuggable inline)

 interpret({
     var a = 5;
     a += 2;
     a;
 }


Not really because that already more-or-less works today (add a q 
before that { and it will compile).

The interpreter throws exceptions when it gets a bad program; 
syntax errors, runtime errors, etc. Those exceptions are 
propagated to the compiler user in the form of error messages. 
Delete the last semicolon and rerun it. The compiler now spews:


/home/me/arsd/script.d(2164): Error: uncaught CTFE exception 
arsd.script.ScriptCompileException("4: Parse error, unexpected 
end of input when reading expression, expecting ;")
d.d(4):        called from here: interpret("\x0a\x09\x09var a = 
5;\x0a\x09\x09a += 2;\x0a\x09\x09a\x0a\x09", var(cast(Type)0, 
Payload(null, , , , , , )).this(null), null)
d.d(9):        while evaluating pragma(msg, a)


d.d line 4 is the interpret function call. The error message from 
the interpreter is listed above that and lists line 4 of its 
input.


So it isn't exactly nice to read.... but probably could be. Keep 
in mind that I didn't write this thing for compile time, it is a 
runtime script interpreter that just happens to also work at 
compile time. If I wrote a thing that caught that exception and 
nicely formatted it at compile time, it could be made prettier.


But still, that message *does* contain the information you need 
because exceptions from the external parser are returned to the D 
compiler.




THAT SAID, I do know what you're asking and I've kinda wanted it 
before. It would be nice to have a .sourceof for an alias 
function (and I think that's been implemented in a pull request 
before, it isn't impossible to do). I just don't feel it would be 
all that important, since what we have now really goes a very 
long way.
Sep 06 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/6/2015 1:38 PM, Adam D. Ruppe wrote:

 jsvar.d and script.d can be found here:
 https://github.com/adamdruppe/arsd


I'd always thought Javascript was an ideal extension language for a text editor.
Sep 06 2015
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 6 September 2015 at 23:33:17 UTC, Walter Bright wrote:

 I'd always thought Javascript was an ideal extension language 
 for a text editor.


Well, I don't think *ideal*, but indeed, it wouldn't be bad. And 
my little thing isn't quite JS, I borrow some ideas from D too. 
So it has string slicing for example, and the ~= operator instead 
of += for concatenation.

I think it'd be ok for a text editor. And heck, I might be 
writing one kinda soonish (I'm kinda reminded of the scene from 
Star Wars where Luke builds his own lightsaber, a rite of passage 
for a Jedi... every master programmer writes her or his own text 
editor at some point too...)

I'm in no rush though, with all the other stuff going on, 
"soonish" probably means "in the next three years".
Sep 06 2015
parent "Sergei Nosov" <sergei.nosov gmail.com> writes:
On Monday, 7 September 2015 at 02:50:06 UTC, Adam D. Ruppe wrote:

 On Sunday, 6 September 2015 at 23:33:17 UTC, Walter Bright 
 wrote:

 I'd always thought Javascript was an ideal extension language 
 for a text editor.


 Well, I don't think *ideal*, but indeed, it wouldn't be bad.


C'mon, kind sirs! Haven't you heard anything about Emacs Lisp? =)
Sep 07 2015
prev sibling parent "Prudence" <Pursuit Happyness.All> writes:
On Sunday, 6 September 2015 at 20:22:23 UTC, Zoadian wrote:

 On Sunday, 6 September 2015 at 19:32:58 UTC, Prudence wrote:

 template X(Y)
 {
 	string X = Y.stringof;
 }

 [...]



 as you'd have to write a parser for other languages why not 
 just use strings? you can already do this:

 template X(string Y)
 {
 	enum X = Y;
 }

 auto s = X!q{int 3;};

 obviously X has to be a compiletime js->d compiler.


Seriously? Is that all you got?
Sep 06 2015
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-09-06 21:32, Prudence wrote:

 template X(Y)
 {
      string X = Y.stringof;
 }

 auto s = X({int 3;})

 Of course, doesn't work!!


You might be interested in this [1].

[1] http://wiki.dlang.org/DIP50

-- 
/Jacob Carlborg
Sep 07 2015