www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP 50 - AST macros

reply Jacob Carlborg <doob me.com> writes:
I've been thinking quite long of how AST macros could look like in D. 
I've been posting my vision of AST macros here in the newsgroup a couple 
of times already. I've now been asked to create a DIP out of it, so here 
it is:

http://wiki.dlang.org/DIP50

-- 
/Jacob Carlborg
Nov 10 2013
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Jacob Carlborg:


 http://wiki.dlang.org/DIP50


I suggest to add some more use cases (possibly with their 
implementation).

Bye,
bearophile
Nov 10 2013
next sibling parent reply "Rob T" <alanb ucora.com> writes:
On Sunday, 10 November 2013 at 22:33:34 UTC, bearophile wrote:

 Jacob Carlborg:


 http://wiki.dlang.org/DIP50


 I suggest to add some more use cases (possibly with their 
 implementation).

 Bye,
 bearophile


A scalable and elegant solution to the "inspection" problem may 
be a AST use case.

Discussed here:
http://forum.dlang.org/thread/qconpedgdkyeawmdzect forum.dlang.org

--rt
Nov 10 2013
next sibling parent reply Timothee Cour <thelastmammoth gmail.com> writes:
<[ .. ]> syntax looks very foreign. How about m{...} (with q{...} as a
precedent)?


On Sun, Nov 10, 2013 at 5:08 PM, Rob T <alanb ucora.com> wrote:


 On Sunday, 10 November 2013 at 22:33:34 UTC, bearophile wrote:


 Jacob Carlborg:

  http://wiki.dlang.org/DIP50


 I suggest to add some more use cases (possibly with their implementation).

 Bye,
 bearophile


 A scalable and elegant solution to the "inspection" problem may be a AST
 use case.

 Discussed here:
 http://forum.dlang.org/thread/qconpedgdkyeawmdzect forum.dlang.org

 --rt





 Nov 10 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 02:40, Timothee Cour wrote:

 <[ .. ]> syntax looks very foreign. How about m{...} (with q{...} as a
 precedent)?


As the DIP says, that just one suggestion of the syntax. If we even need 
a syntax. An alternative, without any new syntax is the "ast" macro:

http://wiki.dlang.org/DIP50#The_AST_Macro

This is similar to the "refiy" macro in Scala.

-- 
/Jacob Carlborg



 Nov 10 2013



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 02:08, Rob T wrote:


 A scalable and elegant solution to the "inspection" problem may be a AST
 use case.

 Discussed here:
 http://forum.dlang.org/thread/qconpedgdkyeawmdzect forum.dlang.org


Please, feel free to add it as an example, if your interested. After 
all, it is a wiki.

-- 
/Jacob Carlborg



 Nov 10 2013



prev sibling next sibling parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-10 23:33, bearophile wrote:

 Jacob Carlborg:


 http://wiki.dlang.org/DIP50


 I suggest to add some more use cases (possibly with their implementation).


I could try and do that. The problem is I have not given much though on 
how the reflection API and the API used to build syntax trees should 
look like.

If this get any serious interest and we start to think about the 
implementation I was hoping that someone we more experience could help 
out with designing the API.

-- 
/Jacob Carlborg



 Nov 10 2013



prev sibling  parent reply "simendsjo" <simendsjo gmail.com>  writes:


On Sunday, 10 November 2013 at 22:33:34 UTC, bearophile wrote:

 Jacob Carlborg:


 http://wiki.dlang.org/DIP50


 I suggest to add some more use cases (possibly with their 
 implementation).

 Bye,
 bearophile


I agree examples would help a lot. Trying to define what 
information actually exists within these types would also help a 
lot.

In the first example, would Ast!(bool) be something like this?
   opBinary!"=="
     left = opBinary!"+"
              left = Literal
                type = int
                value = 1
              right = Literal
                type = int
                value = 2
     right = Literal
       type = int
       value = 4

Would there be helpers for matching part of the structure?
The same applies to the other types used - what information 
should they have?

As for examples, here's a couple of suggestions:
* Expression to prefix notation
* Expression to SQL

* Number intervals, like "int i = int[10..20];" where only 10 to 
20 are legal values
* Discriminated union
* Pattern matching



 Nov 11 2013



 next sibling parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 09:28, simendsjo wrote:


 I agree examples would help a lot. Trying to define what information
 actually exists within these types would also help a lot.


I can tell you right now that I haven't figured out everything, far from 
everything.


 In the first example, would Ast!(bool) be something like this?
    opBinary!"=="
      left = opBinary!"+"
               left = Literal
                 type = int
                 value = 1
               right = Literal
                 type = int
                 value = 2
      right = Literal
        type = int
        value = 4

 Would there be helpers for matching part of the structure?


I don't know. As I said above, I haven't figured out everything. I don't 
see any support for this in the language. But of course, there can be 
library functions that help with this.


 The same applies to the other types used - what information should they
 have?


Suggestions are welcome. Also, see my reply to bearophile

http://forum.dlang.org/thread/l5otb1$1dhi$1 digitalmars.com#post-l5q1p5:242bs4:241:40digitalmars.com


 As for examples, here's a couple of suggestions:
 * Expression to prefix notation
 * Expression to SQL



Would that be something like the attribute macro example?

http://wiki.dlang.org/DIP50#Attribute_macros


 * Number intervals, like "int i = int[10..20];" where only 10 to 20 are
 legal values
 * Discriminated union
 * Pattern matching


Pattern matching usually requires new syntax, which this proposal does 
not support.

-- 
/Jacob Carlborg



 Nov 11 2013



prev sibling  parent  dennis luehring <dl.soluz gmx.net>  writes:


Am 11.11.2013 09:28, schrieb simendsjo:

 On Sunday, 10 November 2013 at 22:33:34 UTC, bearophile wrote:

 Jacob Carlborg:


 http://wiki.dlang.org/DIP50


 I suggest to add some more use cases (possibly with their
 implementation).

 Bye,
 bearophile


 I agree examples would help a lot. Trying to define what
 information actually exists within these types would also help a
 lot.

 In the first example, would Ast!(bool) be something like this?
     opBinary!"=="
       left = opBinary!"+"
                left = Literal
                  type = int
                  value = 1
                right = Literal
                  type = int
                  value = 2
       right = Literal
         type = int
         value = 4

 Would there be helpers for matching part of the structure?
 The same applies to the other types used - what information
 should they have?

 As for examples, here's a couple of suggestions:
 * Expression to prefix notation
 * Expression to SQL

 * Number intervals, like "int i = int[10..20];" where only 10 to
 20 are legal values
 * Discriminated union
 * Pattern matching


can you add the example to the DIP?



 Nov 11 2013



prev sibling next sibling parent reply "bearophile" <bearophileHUGS lycos.com>  writes:


Jacob Carlborg:


 http://wiki.dlang.org/DIP50



http://tomasp.net/blog/2013/computation-zoo-padl/index.html

Bye,
bearophile



 Nov 10 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 02:46, bearophile wrote:



 http://tomasp.net/blog/2013/computation-zoo-padl/index.html


I'll do that. I've been looking at several languages, mainly Scala. But 


-- 
/Jacob Carlborg



 Nov 10 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/10/13 11:46 PM, Jacob Carlborg wrote:

 On 2013-11-11 02:46, bearophile wrote:



 http://tomasp.net/blog/2013/computation-zoo-padl/index.html


 I'll do that. I've been looking at several languages, mainly Scala. But



There is significant regret about the way macros are defined in Scala. 
Probably not an example to follow.

Andrei



 Nov 11 2013



 next sibling parent  "deadalnix" <deadalnix gmail.com>  writes:


On Tuesday, 12 November 2013 at 02:37:03 UTC, Andrei Alexandrescu 
wrote:

 On 11/10/13 11:46 PM, Jacob Carlborg wrote:

 On 2013-11-11 02:46, bearophile wrote:



 http://tomasp.net/blog/2013/computation-zoo-padl/index.html


 I'll do that. I've been looking at several languages, mainly 
 Scala. But



 There is significant regret about the way macros are defined in 
 Scala. Probably not an example to follow.


Do you have some article/documentation on that ?

BTW, scala implemented macros as a plugin called macro paradise, 
which allowed people to play with it without crippling the core 
language. We should probably introduce new features that way.



 Nov 11 2013



prev sibling next sibling parent reply "Rob T" <alanb ucora.com>  writes:


On Tuesday, 12 November 2013 at 02:37:03 UTC, Andrei Alexandrescu 
wrote:

 On 11/10/13 11:46 PM, Jacob Carlborg wrote:

 On 2013-11-11 02:46, bearophile wrote:



 http://tomasp.net/blog/2013/computation-zoo-padl/index.html


 I'll do that. I've been looking at several languages, mainly 
 Scala. But



 There is significant regret about the way macros are defined in 
 Scala. Probably not an example to follow.

 Andrei


Knowing what went wrong is important so as not to repeat past 
mistakes, is there more info somewhere to read on what went wrong?

--rt



 Nov 11 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/11/13 9:16 PM, Rob T wrote:

 On Tuesday, 12 November 2013 at 02:37:03 UTC, Andrei Alexandrescu wrote:

 On 11/10/13 11:46 PM, Jacob Carlborg wrote:

 On 2013-11-11 02:46, bearophile wrote:



 http://tomasp.net/blog/2013/computation-zoo-padl/index.html


 I'll do that. I've been looking at several languages, mainly Scala. But



 There is significant regret about the way macros are defined in Scala.
 Probably not an example to follow.

 Andrei


 Knowing what went wrong is important so as not to repeat past mistakes,
 is there more info somewhere to read on what went wrong?


Not yet. Martin mentioned to me in 2012 he doesn't think they took the 
right turns with defining macros. I don't know whether more information 
is available at the moment.

Andrei



 Nov 11 2013



  parent  "deadalnix" <deadalnix gmail.com>  writes:


On Tuesday, 12 November 2013 at 05:59:46 UTC, Andrei Alexandrescu 
wrote:

 Not yet. Martin mentioned to me in 2012 he doesn't think they 
 took the right turns with defining macros. I don't know whether 
 more information is available at the moment.

 Andrei


Loosely related, but worth watching : 
http://www.infoq.com/presentations/data-types-issues



 Nov 11 2013



prev sibling next sibling parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 03:37, Andrei Alexandrescu wrote:


 There is significant regret about the way macros are defined in Scala.
 Probably not an example to follow.


Unfortunately it's not easy to do anything about it if you cannot give 
us any more details to why. So far I haven't found anything.

-- 
/Jacob Carlborg



 Nov 11 2013



prev sibling  parent  "jerro" <a a.com>  writes:


 There is significant regret about the way macros are defined in 
 Scala. Probably not an example to follow.

 Andrei


Not an example to follow, but it would definitely
be worth looking into how they defined macros and
what aspect of that caused problems, so we can avoid
making the same mistakes.



 Nov 12 2013



prev sibling next sibling parent reply Timothee Cour <thelastmammoth gmail.com>  writes:


People have shunned proposals to have  mixin functions because it wouldn't
be obvious at call site that some statement is executed under a mixin
(which could access all variables in scope etc).

The same will happen here; I think it should be clear at call site that a
macro is used.
How about:

macro!myAssert(1 + 2 == 4);
instead of myAssert(1 + 2 == 4);



On Sun, Nov 10, 2013 at 1:20 PM, Jacob Carlborg <doob me.com> wrote:


 I've been thinking quite long of how AST macros could look like in D. I've
 been posting my vision of AST macros here in the newsgroup a couple of
 times already. I've now been asked to create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50

 --
 /Jacob Carlborg





 Nov 10 2013



 next sibling parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/11/2013 02:49 AM, Timothee Cour wrote:

 People have shunned proposals to have  mixin functions because it
 wouldn't be obvious at call site that some statement is executed under a
 mixin (which could access all variables in scope etc).

 The same will happen here; ...


macros are hygienic.



 Nov 10 2013



prev sibling next sibling parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 02:49, Timothee Cour wrote:

 People have shunned proposals to have  mixin functions because it
 wouldn't be obvious at call site that some statement is executed under a
 mixin (which could access all variables in scope etc).

 The same will happen here; I think it should be clear at call site that
 a macro is used.
 How about:

 macro!myAssert(1 + 2 == 4);
 instead of myAssert(1 + 2 == 4);


I would really prefer if we not had to do that.

-- 
/Jacob Carlborg



 Nov 10 2013



prev sibling  parent reply "Dicebot" <public dicebot.lv>  writes:


On Monday, 11 November 2013 at 01:49:45 UTC, Timothee Cour wrote:

 People have shunned proposals to have  mixin functions because 
 it wouldn't
 be obvious at call site that some statement is executed under a 
 mixin
 (which could access all variables in scope etc).

 The same will happen here; I think it should be clear at call 
 site that a
 macro is used.
 How about:

 macro!myAssert(1 + 2 == 4);
 instead of myAssert(1 + 2 == 4);


If macros are supposed to access outer scope, I agree, this is a 
necessary restriction.



 Nov 11 2013



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/11/2013 01:55 PM, Dicebot wrote:

 On Monday, 11 November 2013 at 01:49:45 UTC, Timothee Cour wrote:

 People have shunned proposals to have  mixin functions because it
 wouldn't
 be obvious at call site that some statement is executed under a mixin
 (which could access all variables in scope etc).

 The same will happen here; I think it should be clear at call site that a
 macro is used.
 How about:

 macro!myAssert(1 + 2 == 4);
 instead of myAssert(1 + 2 == 4);


 If macros are supposed to access outer scope, I agree, this is a
 necessary restriction.


I don't agree. (The argument against implicit mixin features was a 
complete lack of hygiene, not the mere possibility of accessing the 
caller scope.)



 Nov 11 2013



 next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Monday, 11 November 2013 at 16:28:17 UTC, Timon Gehr wrote:

 I don't agree. (The argument against implicit mixin features 
 was a complete lack of hygiene, not the mere possibility of 
 accessing the caller scope.)


How would you expect it to be hygienic in any way if it has 
implicit outer scope access? Reasoning is pretty much the same as 
with mixins.



 Nov 11 2013



  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/11/2013 05:47 PM, Dicebot wrote:

 On Monday, 11 November 2013 at 16:28:17 UTC, Timon Gehr wrote:

 I don't agree. (The argument against implicit mixin features was a
 complete lack of hygiene, not the mere possibility of accessing the
 caller scope.)


 How would you expect it to be hygienic in any way if it has implicit
 outer scope access? Reasoning is pretty much the same as with mixins.


With mixins _every_ identifier is by _default_ inserted/looked up in the 
so-called 'outer scope', whereas declarations and normal symbol lookups 
within macros are hygienic by default. Macros are as much about 
reflection as about code generation. The trade-offs are simly not the 
same. Implicit outer scope access does not imply it behaves in a 
counter-intuitive manner. (It could just reflect some debug information 
after all, or perform some additional checks on the context for more 
safety, etc.) I will be really unhappy with any proposal that does not 
allow to interchange function calls with macro calls transparently, even 
though such a scheme allows more abusive code to be written as well. If 
in some code base macros should be clearly marked, a naming convention 
(eg. prefix macro names with 'macro') can be used. (Enforcing such a 
convention is trivial and amounts to eg. a few lines of additional code 
in a modified D front end.)



 Nov 11 2013



  parent  Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/11/2013 10:23 PM, Timon Gehr wrote:

  simly


simply



 Nov 11 2013



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 17:28, Timon Gehr wrote:


 I don't agree. (The argument against implicit mixin features was a
 complete lack of hygiene, not the mere possibility of accessing the
 caller scope.)


The macros system need to provide both hygienic and non-hygienic macros.

-- 
/Jacob Carlborg



 Nov 11 2013



prev sibling next sibling parent reply Shammah Chancellor <anonymous coward.com>  writes:


On 2013-11-10 21:20:30 +0000, Jacob Carlborg said:


 I've been thinking quite long of how AST macros could look like in D. 
 I've been posting my vision of AST macros here in the newsgroup a 
 couple of times already. I've now been asked to create a DIP out of it, 
 so here it is:
 
 http://wiki.dlang.org/DIP50


Not a huge fan of the syntax, but I think it's a step in the right 
direction away from string mixins.

R/
Shammah



 Nov 10 2013



 next sibling parent  Shammah Chancellor <anonymous coward.com>  writes:


On 2013-11-11 03:04:34 +0000, Shammah Chancellor said:


 On 2013-11-10 21:20:30 +0000, Jacob Carlborg said:
 

 I've been thinking quite long of how AST macros could look like in D. 
 I've been posting my vision of AST macros here in the newsgroup a 
 couple of times already. I've now been asked to create a DIP out of it, 
 so here it is:
 
 http://wiki.dlang.org/DIP50


 
 Not a huge fan of the syntax, but I think it's a step in the right 
 direction away from string mixins.
 
 R/
 Shammah


I think this would gain a lot more traction if someone wrote an 
std.expressions which produced workable code from a AST.   At that 
point, it's just for the D-frontend to parse code into said 
std.expressions.

The .NET expressions framework might servce as some insperation.

R/
Shammah



 Nov 10 2013



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 04:04, Shammah Chancellor wrote:


 Not a huge fan of the syntax, but I think it's a step in the right
 direction away from string mixins.


To you mean the <[ ]> syntax or something else? See my reply to Timothee:

http://forum.dlang.org/thread/l5otb1$1dhi$1 digitalmars.com?page=2#post-l5q1vh:242c2k:241:40digitalmars.com

-- 
/Jacob Carlborg



 Nov 10 2013



prev sibling next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg wrote:

 I've been thinking quite long of how AST macros could look like 
 in D. I've been posting my vision of AST macros here in the 
 newsgroup a couple of times already. I've now been asked to 
 create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50


One of our targets for AST macros should be the ability to 


An example syntax for use with AST could be:

auto data = [5, 7, 9];
int[] data2;
query {
  from value in data
  where value >= 6
  add to data2
}

Could be unwrapped to:

auto data = [5, 7, 9];
int[] data2;
foreach(value; data) {
  if (value >= 6) data2 ~= value;
}

This isn't a thought out design but it should at least be a 
target or a possibility.

Example:

getset {
  public int id;
  private bool exit;
}

Would translate to:

private int id;
private bool exit;
 property {
  void id(int v) {this.id = v;}
  void exit(bool v) { this.exit = v; }
  int id() { return this.id; }
  bool exit() { return this.exit; }
}

This would definitely open new possibilities up.

features.

At current point I think the DIP does have the necessary features 
to implement this. However it would be nice for safety to be able 
to get all scope variables of where the macro was initiated from. 
Being able to check for if a variable exists could provide much 
needed compile safety and better error messages.



 Nov 10 2013



 next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 08:46, Rikki Cattermole wrote:


 One of our targets for AST macros should be the ability to replicate


 An example syntax for use with AST could be:

 auto data = [5, 7, 9];
 int[] data2;
 query {
   from value in data
   where value >= 6
   add to data2
 }

 Could be unwrapped to:

 auto data = [5, 7, 9];
 int[] data2;
 foreach(value; data) {
   if (value >= 6) data2 ~= value;
 }

 This isn't a thought out design but it should at least be a target or a
 possibility.


Absolutely. One of my favorite examples is the database query:

auto person = Person.where(e => e.name == "John");

Which translates to the following SQL:

select * from person where name = 'John'



 Example:

 getset {
   public int id;
   private bool exit;
 }

 Would translate to:

 private int id;
 private bool exit;
  property {
   void id(int v) {this.id = v;}
   void exit(bool v) { this.exit = v; }
   int id() { return this.id; }
   bool exit() { return this.exit; }
 }

 This would definitely open new possibilities up.



That's quite similar one of the examples, I like to call it "property 
shortcut":

http://wiki.dlang.org/DIP50#Attribute_macros


 At current point I think the DIP does have the necessary features to
 implement this. However it would be nice for safety to be able to get
 all scope variables of where the macro was initiated from. Being able to
 check for if a variable exists could provide much needed compile safety
 and better error messages.


Why not? There's quite a lot that is not specified in this DIP. Mostly 
because I haven't decided/figured out how it should work exactly. Of 
course, any help is always appreciated.

-- 
/Jacob Carlborg



 Nov 11 2013



 next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 08:12:59 UTC, Jacob Carlborg wrote:

 On 2013-11-11 08:46, Rikki Cattermole wrote:


 One of our targets for AST macros should be the ability to 
 replicate


 An example syntax for use with AST could be:

 auto data = [5, 7, 9];
 int[] data2;
 query {
  from value in data
  where value >= 6
  add to data2
 }

 Could be unwrapped to:

 auto data = [5, 7, 9];
 int[] data2;
 foreach(value; data) {
  if (value >= 6) data2 ~= value;
 }

 This isn't a thought out design but it should at least be a 
 target or a
 possibility.


 Absolutely. One of my favorite examples is the database query:

 auto person = Person.where(e => e.name == "John");

 Which translates to the following SQL:

 select * from person where name = 'John'



 Example:

 getset {
  public int id;
  private bool exit;
 }

 Would translate to:

 private int id;
 private bool exit;
  property {
  void id(int v) {this.id = v;}
  void exit(bool v) { this.exit = v; }
  int id() { return this.id; }
  bool exit() { return this.exit; }
 }

 This would definitely open new possibilities up.

 features.


 That's quite similar one of the examples, I like to call it 
 "property shortcut":

 http://wiki.dlang.org/DIP50#Attribute_macros


 At current point I think the DIP does have the necessary 
 features to
 implement this. However it would be nice for safety to be able 
 to get
 all scope variables of where the macro was initiated from. 
 Being able to
 check for if a variable exists could provide much needed 
 compile safety
 and better error messages.


 Why not? There's quite a lot that is not specified in this DIP. 
 Mostly because I haven't decided/figured out how it should work 
 exactly. Of course, any help is always appreciated.


Theres a few other things I think would need to be cleared up.
For example take this code:

shader {
  program
  vertex {
   #version 150
   void main() {
   }
  }
  fragment {
   #version 150
   void main() {
   }
  }
}

At this point the vertex / fragment segments would be treated as 
strings instead of calling their macros. So perhaps a rule to 
identify macros inside macros calls? It would save a lot of time 
for parsing reasons.

Also can you alias a macro?

alias fragment = tostring;

That would make things a lot simpler creating nice structures of 
them.

Perhaps a condition on a macro like the if's we got for templates 
would be useful in the sense of being able to say:

if (lexer.compareLine(0, "[a-zA-Z]{1}[a-zA-Z_0-9]*") && 
lexer.isSymbol(1, SymbolTypes.Macro, vertex))

This would require us to develop a lexer library. But if done 
right I don't see why it wouldn't be usable for more than just D 
macro checks. Preferably also for e.g. c/c++ wink wink for when 
D's front end is in D.



 Nov 11 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 09:31, Rikki Cattermole wrote:


 Theres a few other things I think would need to be cleared up.
 For example take this code:

 shader {
   program
   vertex {
    #version 150
    void main() {
    }
   }
   fragment {
    #version 150
    void main() {
    }
   }
 }

 At this point the vertex / fragment segments would be treated as strings
 instead of calling their macros. So perhaps a rule to identify macros
 inside macros calls? It would save a lot of time for parsing reasons.


They way I see that is the AST of the whole block would be passed to 
"shader". I have though a bit about macros inside macros, but I haven't 
come to a conclusion. The easiest way from a design point of view seems 
to be that "shader" to basically return the whole AST it receives plus 
any addition it needs to do. The the "vertex" and "fragment" macros are 
expanded recursively.


 Also can you alias a macro?

 alias fragment = tostring;

 That would make things a lot simpler creating nice structures of them.


I don't see why not.


 Perhaps a condition on a macro like the if's we got for templates would
 be useful in the sense of being able to say:

 if (lexer.compareLine(0, "[a-zA-Z]{1}[a-zA-Z_0-9]*") &&
 lexer.isSymbol(1, SymbolTypes.Macro, vertex))


I don't think I understand how this should be used.


 This would require us to develop a lexer library. But if done right I
 don't see why it wouldn't be usable for more than just D macro checks.
 Preferably also for e.g. c/c++ wink wink for when D's front end is in D.



-- 
/Jacob Carlborg



 Nov 11 2013



  parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 09:24:31 UTC, Jacob Carlborg wrote:

 On 2013-11-11 09:31, Rikki Cattermole wrote:


 Theres a few other things I think would need to be cleared up.
 For example take this code:

 shader {
  program
  vertex {
   #version 150
   void main() {
   }
  }
  fragment {
   #version 150
   void main() {
   }
  }
 }

 At this point the vertex / fragment segments would be treated 
 as strings
 instead of calling their macros. So perhaps a rule to identify 
 macros
 inside macros calls? It would save a lot of time for parsing 
 reasons.


 They way I see that is the AST of the whole block would be 
 passed to "shader". I have though a bit about macros inside 
 macros, but I haven't come to a conclusion. The easiest way 
 from a design point of view seems to be that "shader" to 
 basically return the whole AST it receives plus any addition it 
 needs to do. The the "vertex" and "fragment" macros are 
 expanded recursively.


 Also can you alias a macro?

 alias fragment = tostring;

 That would make things a lot simpler creating nice structures 
 of them.


 I don't see why not.


 Perhaps a condition on a macro like the if's we got for 
 templates would
 be useful in the sense of being able to say:

 if (lexer.compareLine(0, "[a-zA-Z]{1}[a-zA-Z_0-9]*") &&
 lexer.isSymbol(1, SymbolTypes.Macro, vertex))


 I don't think I understand how this should be used.


 This would require us to develop a lexer library. But if done 
 right I
 don't see why it wouldn't be usable for more than just D macro 
 checks.
 Preferably also for e.g. c/c++ wink wink for when D's front 
 end is in D.




An example of this might be:

macro foo (Context context, Ast!(string) str)
if (lexer.isSymbol(0, SymbolTypes.Class))
{
     return str;
}

class Bar {
  int i;
}

foo {
  Bar(7);
}
The above code would succeed however the below code will give a 
compiler error stating that the given macro is not found.

struct Haz {
  int i;
}

foo {
  Haz(9);
}

The above code is based on the assumption of a D lexer in phobos.
This is simpler set of code which won't require a lexer.

macro foo (Context context, Ast!(string) str)
if (str == "working")
{
     return "";
}

foo {fails}
foo {working}
In these cases its a simple string test to determine if the text 
given is specified value.

In the original example I gave, I was using regex to make the 
point of validation for a given line passed to the macro.



 Nov 11 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 10:38, Rikki Cattermole wrote:


 An example of this might be:

 macro foo (Context context, Ast!(string) str)
 if (lexer.isSymbol(0, SymbolTypes.Class))
 {
      return str;
 }

 class Bar {
   int i;
 }

 foo {
   Bar(7);
 }
 The above code would succeed however the below code will give a compiler
 error stating that the given macro is not found.

 struct Haz {
   int i;
 }

 foo {
   Haz(9);
 }

 The above code is based on the assumption of a D lexer in phobos.
 This is simpler set of code which won't require a lexer.

 macro foo (Context context, Ast!(string) str)
 if (str == "working")
 {
      return "";
 }

 foo {fails}
 foo {working}
 In these cases its a simple string test to determine if the text given
 is specified value.

 In the original example I gave, I was using regex to make the point of
 validation for a given line passed to the macro.


My idea of handling errors in macros is more something like triggering a 
real compile error. That could be done via the context parameter, 
something like:

macro foo (Context context, Ast!(string) str)
{
     if (str.eval() == "bar")
         context.error("Illegal value 'bar'"); // this will trigger the 
compile error
}

-- 
/Jacob Carlborg



 Nov 11 2013



  parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 10:04:31 UTC, Jacob Carlborg wrote:

 On 2013-11-11 10:38, Rikki Cattermole wrote:


 An example of this might be:

 macro foo (Context context, Ast!(string) str)
 if (lexer.isSymbol(0, SymbolTypes.Class))
 {
     return str;
 }

 class Bar {
  int i;
 }

 foo {
  Bar(7);
 }
 The above code would succeed however the below code will give 
 a compiler
 error stating that the given macro is not found.

 struct Haz {
  int i;
 }

 foo {
  Haz(9);
 }

 The above code is based on the assumption of a D lexer in 
 phobos.
 This is simpler set of code which won't require a lexer.

 macro foo (Context context, Ast!(string) str)
 if (str == "working")
 {
     return "";
 }

 foo {fails}
 foo {working}
 In these cases its a simple string test to determine if the 
 text given
 is specified value.

 In the original example I gave, I was using regex to make the 
 point of
 validation for a given line passed to the macro.


 My idea of handling errors in macros is more something like 
 triggering a real compile error. That could be done via the 
 context parameter, something like:

 macro foo (Context context, Ast!(string) str)
 {
     if (str.eval() == "bar")
         context.error("Illegal value 'bar'"); // this will 
 trigger the compile error
 }


For errors I was thinking a new pragma.
pragma(error, "You did x wrong");
This would fire an error and display the text given. Returning 
error code as expected. It is after all in the same league as msg.
It would also remove our current static asserts in our code.
So other words it could be replicated by:

pragma(msg, "You did x wrong");
static assert(0);

Just without the whole failed assert message.

Although what I was showing was the idea of overloading of macros 
like we have in templates.



 Nov 11 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 11:11, Rikki Cattermole wrote:


 For errors I was thinking a new pragma.
 pragma(error, "You did x wrong");
 This would fire an error and display the text given. Returning error
 code as expected. It is after all in the same league as msg.
 It would also remove our current static asserts in our code.
 So other words it could be replicated by:

 pragma(msg, "You did x wrong");
 static assert(0);

 Just without the whole failed assert message.


I like the context parameter better. I'm imagine the context parameter 
containing a lot of information and functionality to interact with the 
compiler. A pragma seems not be general enough.


 Although what I was showing was the idea of overloading of macros like
 we have in templates.


Ok, I see.

-- 
/Jacob Carlborg



 Nov 11 2013



  parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 10:19:49 UTC, Jacob Carlborg wrote:

 On 2013-11-11 11:11, Rikki Cattermole wrote:


 For errors I was thinking a new pragma.
 pragma(error, "You did x wrong");
 This would fire an error and display the text given. Returning 
 error
 code as expected. It is after all in the same league as msg.
 It would also remove our current static asserts in our code.
 So other words it could be replicated by:

 pragma(msg, "You did x wrong");
 static assert(0);

 Just without the whole failed assert message.


 I like the context parameter better. I'm imagine the context 
 parameter containing a lot of information and functionality to 
 interact with the compiler. A pragma seems not be general 
 enough.


I can understand wanting to have a high level version of it in 
context. I'm just considering a high reuse language feature that 
could back it. Really it should have been added a while ago given 
the static assert trick is used a bit.
I do actually like the idea of having the ability to call it on 
context with a bunch of extra information given to it for you. 
e.g. line number of initiation ext.



 Nov 11 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 11:28, Rikki Cattermole wrote:


 I can understand wanting to have a high level version of it in context.
 I'm just considering a high reuse language feature that could back it.
 Really it should have been added a while ago given the static assert
 trick is used a bit.
 I do actually like the idea of having the ability to call it on context
 with a bunch of extra information given to it for you. e.g. line number
 of initiation ext.


I think we want to have as much as possible of this collected in the 
same place. I don't think pragmas are a good fit for this.

-- 
/Jacob Carlborg



 Nov 11 2013



  parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 10:47:37 UTC, Jacob Carlborg wrote:

 On 2013-11-11 11:28, Rikki Cattermole wrote:


 I can understand wanting to have a high level version of it in 
 context.
 I'm just considering a high reuse language feature that could 
 back it.
 Really it should have been added a while ago given the static 
 assert
 trick is used a bit.
 I do actually like the idea of having the ability to call it 
 on context
 with a bunch of extra information given to it for you. e.g. 
 line number
 of initiation ext.


 I think we want to have as much as possible of this collected 
 in the same place. I don't think pragmas are a good fit for 
 this.


Yes I do agree with that. I'm just more worried about what backs 
it then pleasantries that we can give the developer.
Also if you're not looking towards pragma's to hook into stopping 
compilation what are your ideas and thoughts around this issue? 
Because at this point I only know of static assert to do it.



 Nov 11 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 11:51, Rikki Cattermole wrote:


 Yes I do agree with that. I'm just more worried about what backs it then
 pleasantries that we can give the developer.
 Also if you're not looking towards pragma's to hook into stopping
 compilation what are your ideas and thoughts around this issue? Because
 at this point I only know of static assert to do it.


A function call on the context parameter as I've already showed. This 
would just call "error" which is already implemented in the compiler. 
Although I don't know how to actually do the connection between the 
compiler internals the user code. I think this is one of the big 
challenges with this DIP. For this function I guess it could be done 
like an intrinsic (is that the correct word?).

-- 
/Jacob Carlborg



 Nov 11 2013



  parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 10:59:28 UTC, Jacob Carlborg wrote:

 A function call on the context parameter as I've already 
 showed. This would just call "error" which is already 
 implemented in the compiler. Although I don't know how to 
 actually do the connection between the compiler internals the 
 user code. I think this is one of the big challenges with this 
 DIP. For this function I guess it could be done like an 
 intrinsic (is that the correct word?).


Ok pragmas essentially call functions like error in the compiler 
[0].
What I am thinking error will do is instead of outputting like 
msg it'll call error[1]. This is exactly what you want.

[0] 
https://github.com/D-Programming-Language/dmd/blob/master/src/statement.c#L2862
[1] 
https://github.com/D-Programming-Language/dmd/blob/master/src/statement.c#L2957



 Nov 11 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 12:04, Rikki Cattermole wrote:


 Ok pragmas essentially call functions like error in the compiler [0].
 What I am thinking error will do is instead of outputting like msg it'll
 call error[1]. This is exactly what you want.

 [0]
 https://github.com/D-Programming-Language/dmd/blob/master/src/statement.c#L2862

 [1]
 https://github.com/D-Programming-Language/dmd/blob/master/src/statement.c#L2957


Yes, I still don't understand why you would want it as a pragma. Be 
usable outside of macros?

-- 
/Jacob Carlborg



 Nov 11 2013



  parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 12:30:07 UTC, Jacob Carlborg wrote:

 Yes, I still don't understand why you would want it as a 
 pragma. Be usable outside of macros?


Yes outside of macros would be useful. For example code like this 
would become redundant:
pragma(msg, "Support for x is not implemented on platform y");
static assert(0);

Becoming:
pragma(error, "Support for x is not implemented on platform y");

Also pragma's core responsibility is to cause the compiler to do 
something. In this case to say we hit an error during compilation 
please tell the user/dev and die.

It is a hook to the compilers workings.

Currently working on getting this implemented. Nearly done with 
it. Just got some extra spaces that shouldn't be in output.



 Nov 11 2013



 next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 13:36, Rikki Cattermole wrote:


 Yes outside of macros would be useful. For example code like this would
 become redundant:
 pragma(msg, "Support for x is not implemented on platform y");
 static assert(0);

 Becoming:
 pragma(error, "Support for x is not implemented on platform y");

 Also pragma's core responsibility is to cause the compiler to do
 something. In this case to say we hit an error during compilation please
 tell the user/dev and die.

 It is a hook to the compilers workings.

 Currently working on getting this implemented. Nearly done with it. Just
 got some extra spaces that shouldn't be in output.


I just don't want to add a bunch of pragmas. I don't know what's the 
best solution.

-- 
/Jacob Carlborg



 Nov 11 2013



  parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 13:16:23 UTC, Jacob Carlborg wrote:

 I just don't want to add a bunch of pragmas. I don't know 
 what's the best solution.


I am only suggesting one :)
We only need one that'll output any text we want it to and make 
the compiler consider it an error.
On the macro side of things we can build a string which then can 
be given to the pragma.

All of the wrapping giving the goodness would go into the context.

Current what I have enables this:

void main() {
	U u;
}

alias T!("hi", "bye") U;

struct T(string a, string b) {
	pragma(error, a ~ "\n" ~ b);
}

test.d(8): Error: hi
bye

test.d(5): Error: template instance test.T!("hi", "bye") error 
instantiating

Not really in pull state but thats the best I can do. Really that 
final error should be omitted.



 Nov 11 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 14:32, Rikki Cattermole wrote:


 I am only suggesting one :)


What about warnings? And no, the current pragma(msg) isn't the same.


 We only need one that'll output any text we want it to and make the
 compiler consider it an error.
 On the macro side of things we can build a string which then can be
 given to the pragma.

 All of the wrapping giving the goodness would go into the context.

 Current what I have enables this:

 void main() {
      U u;
 }

 alias T!("hi", "bye") U;

 struct T(string a, string b) {
      pragma(error, a ~ "\n" ~ b);
 }

 test.d(8): Error: hi
 bye

 test.d(5): Error: template instance test.T!("hi", "bye") error
 instantiating

 Not really in pull state but thats the best I can do. Really that final
 error should be omitted.


I wouldn't say no to this if macros are completely off the table. 
Perhaps it's a good addition regardless.

-- 
/Jacob Carlborg



 Nov 11 2013



  parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 13:36:14 UTC, Jacob Carlborg wrote:

 What about warnings? And no, the current pragma(msg) isn't the 
 same.


I'm not doing one for warnings. See how the error one goes. The 
warning one won't stop compilation and pragma msg can be used to 
fake it.


 I wouldn't say no to this if macros are completely off the 
 table. Perhaps it's a good addition regardless.


Personally I think it was already needed even without macros. 
Just this has started me off on actually looking into it. We'll 
see what the compiler devs think though.



 Nov 11 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 14:40, Rikki Cattermole wrote:


 I'm not doing one for warnings. See how the error one goes. The warning
 one won't stop compilation and pragma msg can be used to fake it.


Yes, but I want warnings to be available in the context parameter as 
well. That's what I've been talking about all along. It's not just 
errors, it's all of the stuff that can be added.

Warnings can be turned off, pragma(msg) cannot (at least not as far as I 
know).


 Personally I think it was already needed even without macros. Just this
 has started me off on actually looking into it. We'll see what the
 compiler devs think though.


I see. I'm not saying that we don't need it.

-- 
/Jacob Carlborg



 Nov 11 2013



  parent  "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 14:10:32 UTC, Jacob Carlborg wrote:

 On 2013-11-11 14:40, Rikki Cattermole wrote:


 I'm not doing one for warnings. See how the error one goes. 
 The warning
 one won't stop compilation and pragma msg can be used to fake 
 it.


 Yes, but I want warnings to be available in the context 
 parameter as well. That's what I've been talking about all 
 along. It's not just errors, it's all of the stuff that can be 
 added.

 Warnings can be turned off, pragma(msg) cannot (at least not as 
 far as I know).


Ah ok, I'll see what I can drum up in that regard then. I guess I 
was wrong about being able to fake it with pragma msg.



 Nov 11 2013



prev sibling next sibling parent reply dennis luehring <dl.soluz gmx.net>  writes:


Am 11.11.2013 13:36, schrieb Rikki Cattermole:

 On Monday, 11 November 2013 at 12:30:07 UTC, Jacob Carlborg wrote:

 Yes, I still don't understand why you would want it as a
 pragma. Be usable outside of macros?


 Yes outside of macros would be useful. For example code like this
 would become redundant:
 pragma(msg, "Support for x is not implemented on platform y");
 static assert(0);

 Becoming:
 pragma(error, "Support for x is not implemented on platform y");

 Also pragma's core responsibility is to cause the compiler to do
 something. In this case to say we hit an error during compilation
 please tell the user/dev and die.

 It is a hook to the compilers workings.

 Currently working on getting this implemented. Nearly done with
 it. Just got some extra spaces that shouldn't be in output.


but in macros the context information is much more interesting then 
anything else - so #pragma(error,...) won't fully fit the needs to 
context error returning - it could be much more then just a message

example: to help the compiler forming better error messages - or maybe 
recover from deep macro expansion etc...)

what you want is just #pragma(error,...) -> break compiliation now
Jacob is talking about the feedback for the compiler thing...



 Nov 11 2013



  parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 13:22:34 UTC, dennis luehring 
wrote:

 example: to help the compiler forming better error messages - 
 or maybe recover from deep macro expansion etc...)

 what you want is just #pragma(error,...) -> break compiliation 
 now
 Jacob is talking about the feedback for the compiler thing...


Jacob mentioned originally he wanted to create a compiler error. 
Or at least thats is how I took it. Hence pragma(error, "text");
We can already message out at both compile and runtime its simply 
does the compiler recognise it as an error part.



 Nov 11 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 14:36, Rikki Cattermole wrote:


 Jacob mentioned originally he wanted to create a compiler error. Or at
 least thats is how I took it. Hence pragma(error, "text");
 We can already message out at both compile and runtime its simply does
 the compiler recognise it as an error part.


"error" would be just a single function that is available in the context 
parameter. There would be many others. From the DIP:

* The arguments used when the compiler was invoked

* Functions for emitting messages of various verbosity level, like 
error, warning and info

* Functions for querying various types of settings/options, like which 
versions are defined, is "debug" or "release" defined and so on

* In general providing as much as possible of what the compiler knows 
about the compile run

* The context should have an associative array with references to all 
scoped variables at initiation point.

-- 
/Jacob Carlborg



 Nov 11 2013



 next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 14:13:31 UTC, Jacob Carlborg wrote:

 On 2013-11-11 14:36, Rikki Cattermole wrote:


 Jacob mentioned originally he wanted to create a compiler 
 error. Or at
 least thats is how I took it. Hence pragma(error, "text");
 We can already message out at both compile and runtime its 
 simply does
 the compiler recognise it as an error part.


 "error" would be just a single function that is available in 
 the context parameter. There would be many others. From the DIP:

 * The arguments used when the compiler was invoked

 * Functions for emitting messages of various verbosity level, 
 like error, warning and info

 * Functions for querying various types of settings/options, 
 like which versions are defined, is "debug" or "release" 
 defined and so on

 * In general providing as much as possible of what the compiler 
 knows about the compile run

 * The context should have an associative array with references 
 to all scoped variables at initiation point.


Is there anything specific for info that I should look at? or 
does pragma msg do everything for you?



 Nov 11 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 16:24, Rikki Cattermole wrote:


 Is there anything specific for info that I should look at? or does
 pragma msg do everything for you?


Sorry, I don't understand what you're meaning.

-- 
/Jacob Carlborg



 Nov 11 2013



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/11/2013 03:13 PM, Jacob Carlborg wrote:

 "error" would be just a single function that is available in the context
 parameter. There would be many others. From the DIP:


"error" should also be able to specify where the error occurred in the 
input, so that it is displayed at the call site.



 Nov 11 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 17:23, Timon Gehr wrote:


 "error" should also be able to specify where the error occurred in the
 input, so that it is displayed at the call site.


That's a good point.

-- 
/Jacob Carlborg



 Nov 11 2013



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/11/2013 01:36 PM, Rikki Cattermole wrote:

 Yes outside of macros would be useful. For example code like this would
 become redundant:
 pragma(msg, "Support for x is not implemented on platform y");
 static assert(0);

 Becoming:
 pragma(error, "Support for x is not implemented on platform y");


static assert(0, "Support for x is not implemented on platform y");



 Nov 11 2013



  parent  "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 16:22:40 UTC, Timon Gehr wrote:

 On 11/11/2013 01:36 PM, Rikki Cattermole wrote:

 Yes outside of macros would be useful. For example code like 
 this would
 become redundant:
 pragma(msg, "Support for x is not implemented on platform y");
 static assert(0);

 Becoming:
 pragma(error, "Support for x is not implemented on platform 
 y");


 static assert(0, "Support for x is not implemented on platform 
 y");


Hmm wasn't aware of that syntax. Needs an example I think on docs.
I will still work on pull request as the point of an assert and 
pragma message is different, atleast I believe.



 Nov 11 2013



prev sibling  parent reply "bearophile" <bearophileHUGS lycos.com>  writes:


Jacob Carlborg:


 One of my favorite examples is the database query:

 auto person = Person.where(e => e.name == "John");

 Which translates to the following SQL:

 select * from person where name = 'John'


Can't you do the same thing with functions similar (same API but 
different semantics) to std.algorithm ones that generate 
expression templates?

auto person = persons.filter!(e => e.name == "John");

--------

simendsjo:


 * Number intervals, like "int i = int[10..20];" where only 10 
 to 20 are legal values


What's wrong with this syntax that doesn't reqiire macros? It's 
more uniform with the rest of the language:

Ranged!(int, 10, 20) i;

Bye,
bearophile



 Nov 11 2013



 next sibling parent reply "simendsjo" <simendsjo gmail.com>  writes:


On Monday, 11 November 2013 at 09:02:01 UTC, bearophile wrote:

 Jacob Carlborg:


 One of my favorite examples is the database query:

 auto person = Person.where(e => e.name == "John");

 Which translates to the following SQL:

 select * from person where name = 'John'


 Can't you do the same thing with functions similar (same API 
 but different semantics) to std.algorithm ones that generate 
 expression templates?

 auto person = persons.filter!(e => e.name == "John");


The problem here is that a library need to know that it has to 
create
   SELECT * FROM persons WHERE name = 'John';
and not
   SELECT * FROM persons;
and filter it locally.
So it needs a way to inspect the body of the delegate and extract 
"name" "==" and "John".


 --------

 simendsjo:


 * Number intervals, like "int i = int[10..20];" where only 10 
 to 20 are legal values


 What's wrong with this syntax that doesn't reqiire macros? It's 
 more uniform with the rest of the language:

 Ranged!(int, 10, 20) i;

 Bye,
 bearophile


Nothing wrong with it, I was just trying to come up with some 
examples. I don't say they're necessarily good :)



 Nov 11 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 10:23, simendsjo wrote:

 On Monday, 11 November 2013 at 09:02:01 UTC, bearophile wrote:



 auto person = persons.filter!(e => e.name == "John");


 The problem here is that a library need to know that it has to create
    SELECT * FROM persons WHERE name = 'John';
 and not
    SELECT * FROM persons;
 and filter it locally.
 So it needs a way to inspect the body of the delegate and extract "name"
 "==" and "John".


This particular example would work. "e" would be a proxy which 
implements opDispatch which returns another proxy which overloads 
opEquals. The problem is that != and all the comparison operators won't 
work. See my reply to bearophile:

http://forum.dlang.org/thread/l5otb1$1dhi$1 digitalmars.com?page=3#post-l5q892:242kpf:241:40digitalmars.com

-- 
/Jacob Carlborg



 Nov 11 2013



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 10:02, bearophile wrote:


 Can't you do the same thing with functions similar (same API but
 different semantics) to std.algorithm ones that generate expression
 templates?

 auto person = persons.filter!(e => e.name == "John");


I have been thinking about that, a solution where "e" is a proxy that 
implements opDispatch and which returns a new proxy which overloads 
"==". The problem with this is that you cannot separately overload the 
equal and comparison operators. In D the all the comparisons operators 
are implement with the single overload, opCmp. You cannot know from 
inside of opCmp if it's the "<" or the ">" operator that is being 
called. Same problem with opEquals.

-- 
/Jacob Carlborg



 Nov 11 2013



prev sibling  parent reply dennis luehring <dl.soluz gmx.net>  writes:


 One of our targets for AST macros should be the ability to


 An example syntax for use with AST could be:

 auto data = [5, 7, 9];
 int[] data2;
 query {
    from value in data
    where value >= 6
    add to data2
 }

 Could be unwrapped to:

 auto data = [5, 7, 9];
 int[] data2;
 foreach(value; data) {
    if (value >= 6) data2 ~= value;
 }


could you add the example to the DIP wiki page



 Nov 11 2013



  parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 09:01:31 UTC, dennis luehring 
wrote:

 One of our targets for AST macros should be the ability to


 An example syntax for use with AST could be:

 auto data = [5, 7, 9];
 int[] data2;
 query {
   from value in data
   where value >= 6
   add to data2
 }

 Could be unwrapped to:

 auto data = [5, 7, 9];
 int[] data2;
 foreach(value; data) {
   if (value >= 6) data2 ~= value;
 }


 could you add the example to the DIP wiki page


Adding use case section with Linq example.
Please remove / modify as required as is first time editing wiki.



 Nov 11 2013



  parent reply Timothee Cour <thelastmammoth gmail.com>  writes:


I would really like to use AST macros for the following use case:

myAssert( x < y);
//will print, on failure, a message, along with all values appearing inside
macro myAssert:
x<y failed: x=..., y=...

myAssert( fun(x,y)==z1+z2)
//likewise, but nesting down to all individual variables appearing inside
the macro:
x=..., y=..., fun(x,y)=..., z1=..., z2=..., bar(z1+z2)=...

This would advantageously replace the plethora of unittest helpers found in
other languages, eg: CHECK_EQ, CHECK_LEQ, etc.
Invaluable for debugging or informative unittests and logs.





On Mon, Nov 11, 2013 at 1:52 AM, Rikki Cattermole
<alphaglosined gmail.com>wrote:


 On Monday, 11 November 2013 at 09:01:31 UTC, dennis luehring wrote:


 One of our targets for AST macros should be the ability to



 An example syntax for use with AST could be:

 auto data = [5, 7, 9];
 int[] data2;
 query {
   from value in data
   where value >= 6
   add to data2
 }

 Could be unwrapped to:

 auto data = [5, 7, 9];
 int[] data2;
 foreach(value; data) {
   if (value >= 6) data2 ~= value;
 }


 could you add the example to the DIP wiki page


 Adding use case section with Linq example.
 Please remove / modify as required as is first time editing wiki.





 Nov 11 2013



 next sibling parent  "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 10:10:33 UTC, Timothee Cour wrote:

 I would really like to use AST macros for the following use 
 case:

 myAssert( x < y);
 //will print, on failure, a message, along with all values 
 appearing inside
 macro myAssert:
 x<y failed: x=..., y=...

 myAssert( fun(x,y)==z1+z2)
 //likewise, but nesting down to all individual variables 
 appearing inside
 the macro:
 x=..., y=..., fun(x,y)=..., z1=..., z2=..., bar(z1+z2)=...

 This would advantageously replace the plethora of unittest 
 helpers found in
 other languages, eg: CHECK_EQ, CHECK_LEQ, etc.
 Invaluable for debugging or informative unittests and logs.





 On Mon, Nov 11, 2013 at 1:52 AM, Rikki Cattermole
 <alphaglosined gmail.com>wrote:


 On Monday, 11 November 2013 at 09:01:31 UTC, dennis luehring 
 wrote:


 One of our targets for AST macros should be the ability to



 An example syntax for use with AST could be:

 auto data = [5, 7, 9];
 int[] data2;
 query {
   from value in data
   where value >= 6
   add to data2
 }

 Could be unwrapped to:

 auto data = [5, 7, 9];
 int[] data2;
 foreach(value; data) {
   if (value >= 6) data2 ~= value;
 }


 could you add the example to the DIP wiki page


 Adding use case section with Linq example.
 Please remove / modify as required as is first time editing 
 wiki.




Perhaps an alternative would be:

asserts {
     x < y
     z == z
}

Where the macro would do something like this once converted:

if (!(x < y)) writeln("x < y failed with x=", x, " y= ", y);
if (!(z == z)) writeln("z == z failed with z=", z);

This would also have the benefit of allowing for multiple assert 
statements for one block.



 Nov 11 2013



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 11:10, Timothee Cour wrote:

 I would really like to use AST macros for the following use case:

 myAssert( x < y);
 //will print, on failure, a message, along with all values appearing
 inside macro myAssert:
 x<y failed: x=..., y=...

 myAssert( fun(x,y)==z1+z2)
 //likewise, but nesting down to all individual variables appearing
 inside the macro:
 x=..., y=..., fun(x,y)=..., z1=..., z2=..., bar(z1+z2)=...

 This would advantageously replace the plethora of unittest helpers found
 in other languages, eg: CHECK_EQ, CHECK_LEQ, etc.
 Invaluable for debugging or informative unittests and logs.


Agree. That's the first example in the DIP. Although a very simplified 
version.

-- 
/Jacob Carlborg



 Nov 11 2013



  parent reply Timothee Cour <thelastmammoth gmail.com>  writes:


On Mon, Nov 11, 2013 at 2:21 AM, Jacob Carlborg <doob me.com> wrote:


 On 2013-11-11 11:10, Timothee Cour wrote:


 I would really like to use AST macros for the following use case:

 myAssert( x < y);
 //will print, on failure, a message, along with all values appearing
 inside macro myAssert:
 x<y failed: x=..., y=...

 myAssert( fun(x,y)==z1+z2)
 //likewise, but nesting down to all individual variables appearing
 inside the macro:
 x=..., y=..., fun(x,y)=..., z1=..., z2=..., bar(z1+z2)=...

 This would advantageously replace the plethora of unittest helpers found
 in other languages, eg: CHECK_EQ, CHECK_LEQ, etc.
 Invaluable for debugging or informative unittests and logs.


 Agree. That's the first example in the DIP. Although a very simplified
 version.



yes, I think your initial example becomes more interesting with this, as in
its current form it can already be done in current D.

This would make error messages self-documenting:

myAssert(!file.exists );
// "!file. exists" failed: dump of AST:
file: string ="foobar.d"
|_file.exists: bool = false
  |_!file. exists: bool = false

In many cases, this would be so much more useful than an out of date /
incomplete string error message, esp w a lot of variables involved.



 Nov 11 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 11:39, Timothee Cour wrote:


 yes, I think your initial example becomes more interesting with this, as
 in its current form it can already be done in current D.


The current example prints out the exact expression what was passed to 
the assert macro. Not the result of evaluating the expression. I don't 
see how this can currently be done in D


 This would make error messages self-documenting:

 myAssert(!file.exists );
 // "!file. exists" failed: dump of AST:
 file: string ="foobar.d"
 |_file.exists: bool = false
    |_!file. exists: bool = false

 In many cases, this would be so much more useful than an out of date /
 incomplete string error message, esp w a lot of variables involved.


I agree.

-- 
/Jacob Carlborg



 Nov 11 2013



 next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 10:55:23 UTC, Jacob Carlborg wrote:

 The current example prints out the exact expression what was 
 passed to the assert macro. Not the result of evaluating the 
 expression. I don't see how this can currently be done in D


I made a suggestion regarding a macro able to get scoped 
variables from initiation point with Linq. And why that would be 
a good thing. This is another use case for it. Perhaps add it as 
a use case for having that feature on the DIP?



 Nov 11 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 11:58, Rikki Cattermole wrote:


 I made a suggestion regarding a macro able to get scoped variables from
 initiation point with Linq. And why that would be a good thing. This is
 another use case for it. Perhaps add it as a use case for having that
 feature on the DIP?


I already have an example with the assert (not among the use cases but 
at the top). Or were you referring to something else?

-- 
/Jacob Carlborg



 Nov 11 2013



  parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Monday, 11 November 2013 at 11:01:41 UTC, Jacob Carlborg wrote:

 I already have an example with the assert (not among the use 
 cases but at the top). Or were you referring to something else?


I was referring to the use case as to why we should have the 
ability to get scoped variables from the initiation point.
Example:

void func(int i) {
     bool b;
     macr {}
}

macro foo (Context context, Ast!(string) str)
{
     writeln(context.scopeVariables!int("i"));
     writeln(context.scopeVariables!bool("b"));
     return "";
}



 Nov 11 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 12:08, Rikki Cattermole wrote:


 I was referring to the use case as to why we should have the ability to
 get scoped variables from the initiation point.
 Example:

 void func(int i) {
      bool b;
      macr {}
 }

 macro foo (Context context, Ast!(string) str)
 {
      writeln(context.scopeVariables!int("i"));
      writeln(context.scopeVariables!bool("b"));
      return "";
 }


Ok, I see. Please add additional examples to the DIP if you think it's 
missing.

-- 
/Jacob Carlborg



 Nov 11 2013



prev sibling  parent reply Timothee Cour <thelastmammoth gmail.com>  writes:


On Mon, Nov 11, 2013 at 2:55 AM, Jacob Carlborg <doob me.com> wrote:


 On 2013-11-11 11:39, Timothee Cour wrote:

  yes, I think your initial example becomes more interesting with this, as

 in its current form it can already be done in current D.


 The current example prints out the exact expression what was passed to the
 assert macro. Not the result of evaluating the expression. I don't see how
 this can currently be done in D



I'm using a modified assert that uses the ugly import(file)[line] trick, so
that can be done (but is really ugly and has impact either runtime or
compile time). But we're essentially in agreement.



  This would make error messages self-documenting:

 myAssert(!file.exists );
 // "!file. exists" failed: dump of AST:
 file: string ="foobar.d"
 |_file.exists: bool = false
    |_!file. exists: bool = false

 In many cases, this would be so much more useful than an out of date /
 incomplete string error message, esp w a lot of variables involved.


 I agree.

 --
 /Jacob Carlborg





 Nov 11 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 12:06, Timothee Cour wrote:


 I'm using a modified assert that uses the ugly import(file)[line] trick,
 so that can be done (but is really ugly and has impact either runtime or
 compile time). But we're essentially in agreement.


Right, that ugly trick :)

-- 
/Jacob Carlborg



 Nov 11 2013



prev sibling next sibling parent reply luka8088 <luka8088 owave.net>  writes:


On 10.11.2013. 22:20, Jacob Carlborg wrote:

 I've been thinking quite long of how AST macros could look like in D.
 I've been posting my vision of AST macros here in the newsgroup a couple
 of times already. I've now been asked to create a DIP out of it, so here
 it is:
 
 http://wiki.dlang.org/DIP50
 


Thumbs up!



 Nov 11 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 09:08, luka8088 wrote:


 Thumbs up!


Thanks.

-- 
/Jacob Carlborg



 Nov 11 2013



prev sibling next sibling parent reply "Dejan Lekic" <dejan.lekic gmail.com>  writes:


On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg wrote:

 I've been thinking quite long of how AST macros could look like 
 in D. I've been posting my vision of AST macros here in the 
 newsgroup a couple of times already. I've now been asked to 
 create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50


I like it, Jacob. With or without <[ ]> it is a good proposal, 
and I see several places where I would use it.

+1



 Nov 11 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 10:07, Dejan Lekic wrote:


 I like it, Jacob. With or without <[ ]> it is a good proposal, and I see
 several places where I would use it.

 +1


Thank you.

-- 
/Jacob Carlborg



 Nov 11 2013



prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


11-Nov-2013 01:20, Jacob Carlborg пишет:

 I've been thinking quite long of how AST macros could look like in D.
 I've been posting my vision of AST macros here in the newsgroup a couple
 of times already. I've now been asked to create a DIP out of it, so here
 it is:

 http://wiki.dlang.org/DIP50


I have to say I like it.

I find it nice that it would allow one to finally ensure that inlining 
is always done by making critical primitives macros instead of 
functions. This is vastly superior compared to the _only_ current option 
of generating unrolled code with string mixins.

I'd call attribute macros - declaration macros in line with statement 
macros, then describe how they look - a lot like attributes.

Things to note though:
  - Can't implement scope(exit/success/failure) because that would need 
to capture the outer scope to pack it into try/catch/finally.
  - Can't implement foreach if only because of the syntax of the latter.
  - There is no definition of Ast!(T) construct - I take it's a struct 
or a class of sorts and the whole thing works with the CTFE engine. Then 
it also needs specification of Statements, Declarations.
  - It seems like I can use AST literal in R-T code as well? That would 
be useful but only if some library actually accepted these AST nodes.

-- 
Dmitry Olshansky



 Nov 11 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 12:34, Dmitry Olshansky wrote:


 I have to say I like it.


Cool :)


 I find it nice that it would allow one to finally ensure that inlining
 is always done by making critical primitives macros instead of
 functions. This is vastly superior compared to the _only_ current option
 of generating unrolled code with string mixins.

 I'd call attribute macros - declaration macros in line with statement
 macros, then describe how they look - a lot like attributes.


Ok, I can change that.


 Things to note though:
   - Can't implement scope(exit/success/failure) because that would need
 to capture the outer scope to pack it into try/catch/finally.


Right. scope(exit) could perhaps be implement with a custom struct and 
calling a block in the destructor.


   - Can't implement foreach if only because of the syntax of the latter.


I would think something like this:

macro foreach (T) (Context context, Ast!(Symbol) var, Ast!(T) 
collection, Statement block)
{
     if (isRange(collection))
     {
         return <[
             for (auto __r = $collection; !__r.empty; __r.popFront())
             {
                 auto $var = __r.front;
                 block;
             }
         ]>;
     }

     else if (hasOpApply(collection))
     {
         ...
     }

     else ...
}

Perhaps not all of the existing syntax can fit in a macro.


   - There is no definition of Ast!(T) construct - I take it's a struct
 or a class of sorts and the whole thing works with the CTFE engine. Then
 it also needs specification of Statements, Declarations.


Yes, something like that. I don't know exactly how the Ast type need to 
look like. See one of my replies to bearophile:

http://forum.dlang.org/thread/l5otb1$1dhi$1 digitalmars.com


   - It seems like I can use AST literal in R-T code as well? That would
 be useful but only if some library actually accepted these AST nodes.


Actually, I haven't thought about this. I think my original idea was 
that <[ ]> should only work inside macros. But that it would be possible 
to pass Ast nodes to other functions.

-- 
/Jacob Carlborg



 Nov 11 2013



prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de>  writes:


Am Sun, 10 Nov 2013 22:20:30 +0100
schrieb Jacob Carlborg <doob me.com>:


 I've been thinking quite long of how AST macros could look like in D. 
 I've been posting my vision of AST macros here in the newsgroup a couple 
 of times already. I've now been asked to create a DIP out of it, so here 
 it is:
 
 http://wiki.dlang.org/DIP50


Wow, some topics really explode recently. Is the D user base
growing or are AST macros drawing so much attention?

-- 
Marco



 Nov 11 2013



 next sibling parent  "Chris Cain" <clcain uncg.edu>  writes:


On Monday, 11 November 2013 at 12:13:54 UTC, Marco Leise wrote:

 Wow, some topics really explode recently. Is the D user base
 growing or are AST macros drawing so much attention?


I'd like to say both. AST macros are one of those few things that 
I wouldn't mind seeing breaking changes for and I'm in the camp 
of "avoid breaking changes at all cost". AST macros are to code 
what templates are to data. Despite it being possible to go 
without templates, I'm not at all tempted to switch to Go.



 Nov 11 2013



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 13:13, Marco Leise wrote:


 Wow, some topics really explode recently. Is the D user base
 growing or are AST macros drawing so much attention?


AST macros are drawing so much attension. For some, it's like the holy 
grail.

-- 
/Jacob Carlborg



 Nov 11 2013



  parent  "bearophile" <bearophileHUGS lycos.com>  writes:


Jacob Carlborg:


 AST macros are drawing so much attension. For some, it's like 
 the holy grail.


That's a bad approach to technology design. You first need to 
create a good list of things that you want to do, then you have 
to try to do them without new features, and then if something 
important is not possible without the new feature (or it becomes 
too much ugly, too much long, too much bug-prone, etc), you have 
to see if the complexity of introducing the new feature is paid 
back. Macros are a significant feature, so they have a 
significant cost in code complexity, language dis-uniformity, ecc.

Bye,
bearophile



 Nov 11 2013



prev sibling next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg wrote:

 I've been thinking quite long of how AST macros could look like 
 in D. I've been posting my vision of AST macros here in the 
 newsgroup a couple of times already. I've now been asked to 
 create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50


I don't have time to investigate this in details but I'd like to 
mention that this is one of few possible new features where ROI 
is really worth it. It removes lot of burden from core language 
and allows to replace some very arcane template hacks with 
cleaner code. I'd say it is a bit too early to go for it (really, 
we need at least to figure out how to make releases without 
issues first :)) but it is a good future goal even in context of 
general feature freezing.



 Nov 11 2013



  parent  "deadalnix" <deadalnix gmail.com>  writes:


On Monday, 11 November 2013 at 13:26:37 UTC, Dicebot wrote:

 On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg 
 wrote:

 I've been thinking quite long of how AST macros could look 
 like in D. I've been posting my vision of AST macros here in 
 the newsgroup a couple of times already. I've now been asked 
 to create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50


 I don't have time to investigate this in details but I'd like 
 to mention that this is one of few possible new features where 
 ROI is really worth it. It removes lot of burden from core 
 language and allows to replace some very arcane template hacks 
 with cleaner code. I'd say it is a bit too early to go for it 
 (really, we need at least to figure out how to make releases 
 without issues first :)) but it is a good future goal even in 
 context of general feature freezing.


Can you stop reading my mind, please ?



 Nov 11 2013



prev sibling next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu>  writes:


On 11/10/2013 01:20 PM, Jacob Carlborg wrote:

 I've been thinking quite long of how AST macros could look like in D.
 I've been posting my vision of AST macros here in the newsgroup a couple
 of times already. I've now been asked to create a DIP out of it, so here
 it is:

 http://wiki.dlang.org/DIP50


For macros that generate macros, I think you need a way to escape the 
splicing and maybe define how splicing works in an inner quasi quote.

I guess you want <[ <[ $exp ]> ]> to turn into the ast <[ 1 ]>

Then if you wanted ast <[ <[ $exp ]> ]> with the splice associated with 
the inner quasi quote, you'd have to do something like

<[ <[ <[ \\\$exp ]> ]> ]>

maybe a way to associate a splice with a quasi quote?

<a[ <b[ <c[ $(a, exp) + $(b, exp) + $(c, exp)  ]> ]> ]>

a's exp is A
b's exp is B
c's exp is C

then splice the quasi quote N times:

1: ast <b[ <c[ A + $(b, exp) + $(c, exp) ]> ]>
2: ast <c[ A + B + $(c, exp) ]>
3: ast A + B + C

just dinking around here



 Nov 11 2013



 next sibling parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-11 20:23, Ellery Newcomer wrote:


 For macros that generate macros, I think you need a way to escape the
 splicing and maybe define how splicing works in an inner quasi quote.

 I guess you want <[ <[ $exp ]> ]> to turn into the ast <[ 1 ]>

 Then if you wanted ast <[ <[ $exp ]> ]> with the splice associated with
 the inner quasi quote, you'd have to do something like

 <[ <[ <[ \\\$exp ]> ]> ]>

 maybe a way to associate a splice with a quasi quote?

 <a[ <b[ <c[ $(a, exp) + $(b, exp) + $(c, exp)  ]> ]> ]>

 a's exp is A
 b's exp is B
 c's exp is C

 then splice the quasi quote N times:

 1: ast <b[ <c[ A + $(b, exp) + $(c, exp) ]> ]>
 2: ast <c[ A + B + $(c, exp) ]>
 3: ast A + B + C

 just dinking around here


This would require some thought.

-- 
/Jacob Carlborg



 Nov 11 2013



prev sibling  parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Monday, 11 November 2013 at 19:23:21 UTC, Ellery Newcomer 
wrote:

 On 11/10/2013 01:20 PM, Jacob Carlborg wrote:

 I've been thinking quite long of how AST macros could look 
 like in D.
 I've been posting my vision of AST macros here in the 
 newsgroup a couple
 of times already. I've now been asked to create a DIP out of 
 it, so here
 it is:

 http://wiki.dlang.org/DIP50


 For macros that generate macros, I think you need a way to 
 escape the splicing and maybe define how splicing works in an 
 inner quasi quote.

 I guess you want <[ <[ $exp ]> ]> to turn into the ast <[ 1 ]>

 Then if you wanted ast <[ <[ $exp ]> ]> with the splice 
 associated with the inner quasi quote, you'd have to do 
 something like

 <[ <[ <[ \\\$exp ]> ]> ]>

 maybe a way to associate a splice with a quasi quote?

 <a[ <b[ <c[ $(a, exp) + $(b, exp) + $(c, exp)  ]> ]> ]>

 a's exp is A
 b's exp is B
 c's exp is C

 then splice the quasi quote N times:

 1: ast <b[ <c[ A + $(b, exp) + $(c, exp) ]> ]>
 2: ast <c[ A + B + $(c, exp) ]>
 3: ast A + B + C

 just dinking around here


$ refers to the enclosing scope. so $$foo should refers to $foo 
in the enclosing scope. No need for special rule or label.



 Nov 11 2013



  parent  Ellery Newcomer <ellery-newcomer utulsa.edu>  writes:


On 11/11/2013 12:06 PM, deadalnix wrote:

 On Monday, 11 November 2013 at 19:23:21 UTC, Ellery Newcomer wrote:

 On 11/10/2013 01:20 PM, Jacob Carlborg wrote:

 I've been thinking quite long of how AST macros could look like in D.
 I've been posting my vision of AST macros here in the newsgroup a couple
 of times already. I've now been asked to create a DIP out of it, so here
 it is:

 http://wiki.dlang.org/DIP50


 For macros that generate macros, I think you need a way to escape the
 splicing and maybe define how splicing works in an inner quasi quote.

 I guess you want <[ <[ $exp ]> ]> to turn into the ast <[ 1 ]>

 Then if you wanted ast <[ <[ $exp ]> ]> with the splice associated
 with the inner quasi quote, you'd have to do something like

 <[ <[ <[ \\\$exp ]> ]> ]>

 maybe a way to associate a splice with a quasi quote?

 <a[ <b[ <c[ $(a, exp) + $(b, exp) + $(c, exp)  ]> ]> ]>

 a's exp is A
 b's exp is B
 c's exp is C

 then splice the quasi quote N times:

 1: ast <b[ <c[ A + $(b, exp) + $(c, exp) ]> ]>
 2: ast <c[ A + B + $(c, exp) ]>
 3: ast A + B + C

 just dinking around here


 $ refers to the enclosing scope. so $$foo should refers to $foo in the
 enclosing scope. No need for special rule or label.


so if I have to splice my ast N times, then I have to generate M $'s 
depending on when I want it to expand?



 Nov 11 2013



prev sibling next sibling parent  "Jesse Phillips" <Jesse.K.Phillips+D gmail.com>  writes:


On Sunday, 10 November 2013 at 21:20:34 UTC, Jacob Carlborg wrote:

 I've been thinking quite long of how AST macros could look like 
 in D. I've been posting my vision of AST macros here in the 
 newsgroup a couple of times already. I've now been asked to 
 create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50


Just a quick comment related to the 2007 proposal:

http://youtu.be/FRfTk44nuWE?t=1h5m38s
Page 45: http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf

Based on my read though, yours is more powerful. The 2007 
proposal is basically the "Quasi-Quoting" portions of your 
proposal (which is such a small part of your proposal).



 Nov 11 2013



prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/10/2013 1:20 PM, Jacob Carlborg wrote:

 I've been thinking quite long of how AST macros could look like in D. I've been
 posting my vision of AST macros here in the newsgroup a couple of times
already.
 I've now been asked to create a DIP out of it, so here it is:

 http://wiki.dlang.org/DIP50



I confess I have some serious reservations about AST macros in general:

1. I've seen very heavy use of such macros in macro assemblers. What happens is 
people use it to invent their own (very baroque) language on top of the
existing 
assembler one. Anyone trying to read the code has to learn this new unique 
language, and given the limitations of the macro capability, it often comes
with 
unfixable bizarre corner cases and limitations.

This got so bad that in some cases I know of, the poor sap who is given the job 
of making improvements resorted to running the assembler to generate an object 
file, then disassembling the object file back into source code! Something went 
very wrong for this to be necessary. I know in my own assembler work I have 
abandoned all use of macros.

(Note that macros in typical assemblers are far more powerful than C's macro 
language, which stands out for being pathetically underpowered.)

2. Macros are something that should be used very sparingly. However, this is
not 
what happens. I used to be perplexed that even top shelf C/C++ programmers tend 
to write what I not so humbly consider to be pretty abusive use of macros. Now 
it just saddens me.

3. Lisp is a language that encourages users to write macros to pretty much 
invent a custom language for the task at hand. Heavy use of such makes the code 
unrecognizable as being Lisp code. I believe a language ought to have some 
"anchors" that the person reading the code can reliably recognize. (GO has
taken 
this idea pretty far.)

4. AST macros, pretty much by definition, manipulate the AST. However, if you 
examine the semantic routines in DMD, a *lot* more is required to do more 
significant things. The symbol table must be consulted, etc. I just don't see 
how one could hope to implement something like function overloading with AST 
macros. Or template type deduction. Or closure semantics, attribute inference, 
etc. Sure, some forms of foreach can be rewritten into for statements by 
manipulating the AST, but not the more interesting forms, such as the ones that 
deal with opApply().

There are some statements in the DIP about the Context parameter and it'll 
provide semantic information, but I don't see how this can work without making 
it an ever-expanding collection of arbitrary methods.

5. We've said "no" in the past to things like user-defined tokens, which are 
closely related.

6. Note that D has a limited form of AST macros with mixin templates.


To sum up, any AST macro proposal has some tough barriers to get over. Within 
its necessarily severe semantic limitations, it has to deliver a pretty 
compelling set of solutions, while avoiding the morass of incomprehensibility 
that far too many macro systems become in practice.

For example, the Linq example in the DIP is not compelling, as aesthetically 
nicer code can be written using D's ranges and algorithms:

     auto data = arr.filter!(x => x > 5).array;


blunt 
it's like the old:

     #define BEGIN {
     #define END }

macros used in old C code to make it look like Pascal.



 Nov 12 2013



 next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


I forgot to mention that "expression templates" can be used in D in an 
equivalent manner that they are used in C++, and can cover much of the 
functionality used in AST macros.

Eric Anderton who wrote a regex expression template module a few years back:

http://www.dsource.org/projects/ddl/browser/trunk/meta/regex.d



 Nov 12 2013



 next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/12/2013 1:55 AM, Walter Bright wrote:

 I forgot to mention that "expression templates" can be used in D in an
 equivalent manner that they are used in C++, and can cover much of the
 functionality used in AST macros.

 Eric Anderton who wrote a regex expression template module a few years back:

 http://www.dsource.org/projects/ddl/browser/trunk/meta/regex.d


Here's a project to add Linq support to C++:

http://www.codeproject.com/Articles/17844/CLinq-LINQ-support-for-the-C-CLI-language

There's also:

http://pfultz2.github.io/Linq/

https://github.com/hjiang/linqxx/wiki



 Nov 12 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 11:03, Walter Bright wrote:


 Here's a project to add Linq support to C++:

 http://www.codeproject.com/Articles/17844/CLinq-LINQ-support-for-the-C-CLI-language


 There's also:

 http://pfultz2.github.io/Linq/

 https://github.com/hjiang/linqxx/wiki


Again, operator overloading in D is too limiting to implement something 
Linq like.

-- 
/Jacob Carlborg



 Nov 13 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 12:03 AM, Jacob Carlborg wrote:

 Again, operator overloading in D is too limiting to implement something Linq
like.


Ok, let's set aside the opEquals and opCmp issue for the moment.

Can AST macros do anything that expression templates cannot?



 Nov 13 2013



 next sibling parent reply "Don" <x nospam.com>  writes:


On Wednesday, 13 November 2013 at 08:45:13 UTC, Walter Bright 
wrote:

 On 11/13/2013 12:03 AM, Jacob Carlborg wrote:

 Again, operator overloading in D is too limiting to implement 
 something Linq like.


 Ok, let's set aside the opEquals and opCmp issue for the moment.

 Can AST macros do anything that expression templates cannot?


With an expression template you still can't create a statement. 
It has to revolve to a declaration.

But as I said before, it's primarily syntax sugar. Expression 
templates are just a mass of boilerplate code. I came to the 
conclusion that the code for expression templates was no less 
ugly than for string mixins.

What is true, though, is everything an AST macro can do, can 
already be done with a string mixin. The syntax is just ugly.

                  calling syntax power  implementation
string mixin       ugly         high      ugly
expr template       ok          low       ugly
AST macro          good          ?         ?



 Nov 13 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 1:08 AM, Don wrote:

 On Wednesday, 13 November 2013 at 08:45:13 UTC, Walter Bright wrote:

 On 11/13/2013 12:03 AM, Jacob Carlborg wrote:

 Again, operator overloading in D is too limiting to implement something Linq
 like.


 Ok, let's set aside the opEquals and opCmp issue for the moment.

 Can AST macros do anything that expression templates cannot?


 With an expression template you still can't create a statement. It has to
 revolve to a declaration.


Actually, there is a way to do this. I wrote an article years back on how to 
write "statement" workalikes using lazy function parameters. I suppose I should 
dig that up if I can find it.



 But as I said before, it's primarily syntax sugar. Expression templates are
just
 a mass of boilerplate code. I came to the conclusion that the code for
 expression templates was no less ugly than for string mixins.

 What is true, though, is everything an AST macro can do, can already be done
 with a string mixin. The syntax is just ugly.

                   calling syntax power  implementation
 string mixin       ugly         high      ugly
 expr template       ok          low       ugly
 AST macro          good          ?         ?


This implies to me that we ought to explore the limits of expression templates, 
which already exist, in preference to creating a wholesale new feature.



 Nov 13 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 11:24 AM, Walter Bright wrote:

 Actually, there is a way to do this. I wrote an article years back on how to
 write "statement" workalikes using lazy function parameters. I suppose I should
 dig that up if I can find it.


Ah, found the code:

void ifthen(bool cond, lazy void dg)
{
     if (cond)
         dg();
}

void ifthen(bool cond, lazy void dgthen, lazy void dgelse)
{
     if (cond)
         dgthen();
     else
         dgelse();
}

void dotimes(int i, lazy int dg)
{
     for (int j = 0; j < i; j++)
         dg();
}

void switcher(bool delegate()[] cases...)
{
     foreach (c; cases)
     {
         if (c())
             break;
     }
}

bool scase(bool b, lazy void dg)
{
     if (b)
     {
         dg();
         return true;
     }
     return false;
}

bool sdefault(lazy void dg)
{
     dg();
     return true;
}

void whiler(lazy bool cond, lazy void bdy)
{
     while (cond())
         bdy();
}

void test1()
{
     int x = 3;
     dotimes(5, printf("%d\n", ++x));

     ifthen(true, printf("yes\n"));
     ifthen(false, printf("no\n"));

     ifthen(true, printf("yes\n"), printf("no\n"));
     ifthen(false, printf("yes\n"), printf("no\n"));

     int v = 2;
     switcher(
         scase(v == 1, printf("it is 1\n")),
         scase(v == 2, printf("it is 2\n")),
         scase(v == 3, printf("it is 3\n")),
         sdefault( printf("it is default\n"))
     );

     whiler( x < 100,
         (printf("%d\n", x), x *= 2)
     );
}



 Nov 13 2013



 next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/13/2013 08:25 PM, Walter Bright wrote:


 Ah, found the code:

 void ifthen(bool cond, lazy void dg)
 {
      if (cond)
          dg();
 }


int foo(int x){
     ifthen(!x, return 2); // uh oh
     return 3;
}



 Nov 13 2013



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 12:50 PM, Timon Gehr wrote:

 On 11/13/2013 08:25 PM, Walter Bright wrote:


 Ah, found the code:

 void ifthen(bool cond, lazy void dg)
 {
      if (cond)
          dg();
 }


 int foo(int x){
      ifthen(!x, return 2); // uh oh
      return 3;
 }


This approach definitely hews to the single-entry/single-exit paradigm.



 Nov 13 2013



prev sibling  parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Wednesday, 13 November 2013 at 19:25:18 UTC, Walter Bright 
wrote:

 On 11/13/2013 11:24 AM, Walter Bright wrote:

 Actually, there is a way to do this. I wrote an article years 
 back on how to
 write "statement" workalikes using lazy function parameters. I 
 suppose I should
 dig that up if I can find it.


 Ah, found the code:

 void ifthen(bool cond, lazy void dg)
 {
     if (cond)
         dg();
 }

 void ifthen(bool cond, lazy void dgthen, lazy void dgelse)
 {
     if (cond)
         dgthen();
     else
         dgelse();
 }

 void dotimes(int i, lazy int dg)
 {
     for (int j = 0; j < i; j++)
         dg();
 }

 void switcher(bool delegate()[] cases...)
 {
     foreach (c; cases)
     {
         if (c())
             break;
     }
 }

 bool scase(bool b, lazy void dg)
 {
     if (b)
     {
         dg();
         return true;
     }
     return false;
 }

 bool sdefault(lazy void dg)
 {
     dg();
     return true;
 }

 void whiler(lazy bool cond, lazy void bdy)
 {
     while (cond())
         bdy();
 }

 void test1()
 {
     int x = 3;
     dotimes(5, printf("%d\n", ++x));

     ifthen(true, printf("yes\n"));
     ifthen(false, printf("no\n"));

     ifthen(true, printf("yes\n"), printf("no\n"));
     ifthen(false, printf("yes\n"), printf("no\n"));

     int v = 2;
     switcher(
         scase(v == 1, printf("it is 1\n")),
         scase(v == 2, printf("it is 2\n")),
         scase(v == 3, printf("it is 3\n")),
         sdefault( printf("it is default\n"))
     );

     whiler( x < 100,
         (printf("%d\n", x), x *= 2)
     );
 }


Was is missing is the capability to look into the lazy 
parameter's ast and act accordingly.

Here you can wrap something around code, not generation code 
depending on passed code.



 Nov 13 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 2:18 PM, deadalnix wrote:

 Was is missing is the capability to look into the lazy parameter's ast and act
 accordingly.

 Here you can wrap something around code, not generation code depending on
passed
 code.


That's correct. But you can look into the AST with expression templates. So, 
expression templates combined with lazy arguments and mixin templates may fill 
the problem space.



 Nov 13 2013



  parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Wednesday, 13 November 2013 at 22:34:02 UTC, Walter Bright 
wrote:

 On 11/13/2013 2:18 PM, deadalnix wrote:

 Was is missing is the capability to look into the lazy 
 parameter's ast and act
 accordingly.

 Here you can wrap something around code, not generation code 
 depending on passed
 code.


 That's correct. But you can look into the AST with expression 
 templates. So, expression templates combined with lazy 
 arguments and mixin templates may fill the problem space.


Expression templates are limited to expression and require to use 
its own set of types. Still the idea has some merits.



 Nov 13 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 3:58 PM, deadalnix wrote:

 On Wednesday, 13 November 2013 at 22:34:02 UTC, Walter Bright wrote:

 On 11/13/2013 2:18 PM, deadalnix wrote:

 Was is missing is the capability to look into the lazy parameter's ast and act
 accordingly.

 Here you can wrap something around code, not generation code depending on
passed
 code.


 That's correct. But you can look into the AST with expression templates. So,
 expression templates combined with lazy arguments and mixin templates may fill
 the problem space.


 Expression templates are limited to expression and require to use its own set
of
 types.


Yes. But that's a good thing. I'd be pretty skeptical of the value of an AST 
macro that took 3+4 and changed it so it did something other than compute 7.



 Still the idea has some merits.


It has the merit of being implemented and available right now :-)

I do understand that expression templates aren't very sexy, and are kinda hard 
to wrap one's head around. But they're not so bad from the perspective of the 
user of the expression templates, and I really think that the limits of
existing 
features should be explored before inventing wholesale new ones that may turn 
out to be redundant.



 Nov 13 2013



 next sibling parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Thursday, 14 November 2013 at 00:16:45 UTC, Walter Bright 
wrote:

 I do understand that expression templates aren't very sexy, and 
 are kinda hard to wrap one's head around. But they're not so 
 bad from the perspective of the user of the expression 
 templates, and I really think that the limits of existing 
 features should be explored before inventing wholesale new ones 
 that may turn out to be redundant.


My very first project in D involved quite a lot of them. So I 
very aware of what they are.

The whole idea is to be able to extends that to statement and 
declarations. Making what expression templates can do is simply a 
side effect so to speak.



 Nov 13 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 5:14 PM, deadalnix wrote:

 My very first project in D involved quite a lot of them. So I very aware of
what
 they are.

 The whole idea is to be able to extends that to statement and declarations.


My question about that is just what problem is this trying to solve, and why 
that problem cannot be reformulated in terms of expression templates?

I.e. we shouldn't be adding things to D just because they are cool. They need
to 
solve a problem that is fairly commonplace and too painful to solve with 
existing mechanisms. The larger the proposed language enhancement is, the more 
commonplace and and painful the problems must be that it purports to solve.



 Making what expression templates can do is simply a side effect so to speak.





 Nov 13 2013



  parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Thursday, 14 November 2013 at 01:38:29 UTC, Walter Bright 
wrote:

 On 11/13/2013 5:14 PM, deadalnix wrote:

 My very first project in D involved quite a lot of them. So I 
 very aware of what
 they are.

 The whole idea is to be able to extends that to statement and 
 declarations.


 My question about that is just what problem is this trying to 
 solve, and why that problem cannot be reformulated in terms of 
 expression templates?

 I.e. we shouldn't be adding things to D just because they are 
 cool. They need to solve a problem that is fairly commonplace 
 and too painful to solve with existing mechanisms. The larger 
 the proposed language enhancement is, the more commonplace and 
 and painful the problems must be that it purports to solve.


I think the whole point of macro is to NOT add too much feature 
to the language.

See for instance the example I gave before to create generator. 

an example). But with macro, the feature can be implemented as a 
library.

It is NOT achievable with expression templates.

I can say with certainty that the async/await mechanism is 
heavily used in some codebases and with great benefit. It is 
being added to all languages one after another.

The idea behind macros is that instead of adding new feature to 
the language, the language should propose a toolkit to build 
these features as library.

I don't think the time has come for macros in D. As discussed 
before, it seems like filling the gap in existing feature is more 
important than adding new ones. This should however be considered 
very seriously when the gaps has been filled.



 Nov 13 2013



 next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-14 02:56, deadalnix wrote:


 I think the whole point of macro is to NOT add too much feature to the
 language.

 See for instance the example I gave before to create generator. This can

 with macro, the feature can be implemented as a library.

 It is NOT achievable with expression templates.

 I can say with certainty that the async/await mechanism is heavily used
 in some codebases and with great benefit. It is being added to all
 languages one after another.

 The idea behind macros is that instead of adding new feature to the
 language, the language should propose a toolkit to build these features
 as library.


Agree, good points.


 I don't think the time has come for macros in D. As discussed before, it
 seems like filling the gap in existing feature is more important than
 adding new ones. This should however be considered very seriously when
 the gaps has been filled.


Unfortunately I don't think we will have a feature freeze from now until 
those gaps has been filled. Most likely there will be added a couple of 
small features that could have been solved by AST macros. The problem 
with that is that each of these features, in themselves, are way to 
small to add AST macros for. So instead we get new features and even 
less chance of adding AST macros.

Example. There's a bug report about adding support for C++ namespaces. 
That should be solveable with library code. It's just mangling and we 
already have pragma(mangle).

pragma(mangle, namespace("foo::bar")) extern(C++) void x ();

The problem here is that "x" needs to be mangled as well.

pragma(mangle, namespace("foo::bar", x)) extern(C++) void x ();

The above doesn't work due to forward reference errors.

pragma(mangle, namespace("foo::bar", void function (), "x")) extern(C++) 
void x ();

The problem with the above is that we now need to repeat the signature 
and the name of the function.

With AST macros:

macro namespace (Context context, Ast!(string) name, Declaration 
declaration) { ... }

 namespace("foo::bar") extern (C++) void x ();

This works because the macro receives the whole declaration that is "x" 
and "x" is replaced with whatever the macro returns:

pragma(mangle, "mangled_name_of_x") extern (C++) void x ();

Since it uses the same syntax as UDA's you it can look like real 
namespaces in C++:

 namespace("foo::bar") extern (C++)
{
     void x ();
     void y ();
}

-- 
/Jacob Carlborg



 Nov 13 2013



  parent reply dennis luehring <dl.soluz gmx.net>  writes:


Am 14.11.2013 08:50, schrieb Jacob Carlborg:

 On 2013-11-14 02:56, deadalnix wrote:


 I think the whole point of macro is to NOT add too much feature to the
 language.

 See for instance the example I gave before to create generator. This can

 with macro, the feature can be implemented as a library.

 It is NOT achievable with expression templates.

 I can say with certainty that the async/await mechanism is heavily used
 in some codebases and with great benefit. It is being added to all
 languages one after another.

 The idea behind macros is that instead of adding new feature to the
 language, the language should propose a toolkit to build these features
 as library.


 Agree, good points.


 I don't think the time has come for macros in D. As discussed before, it
 seems like filling the gap in existing feature is more important than
 adding new ones. This should however be considered very seriously when
 the gaps has been filled.


 Unfortunately I don't think we will have a feature freeze from now until
 those gaps has been filled. Most likely there will be added a couple of
 small features that could have been solved by AST macros. The problem
 with that is that each of these features, in themselves, are way to
 small to add AST macros for. So instead we get new features and even
 less chance of adding AST macros.

 Example. There's a bug report about adding support for C++ namespaces.
 That should be solveable with library code. It's just mangling and we
 already have pragma(mangle).

 pragma(mangle, namespace("foo::bar")) extern(C++) void x ();

 The problem here is that "x" needs to be mangled as well.

 pragma(mangle, namespace("foo::bar", x)) extern(C++) void x ();

 The above doesn't work due to forward reference errors.

 pragma(mangle, namespace("foo::bar", void function (), "x")) extern(C++)
 void x ();

 The problem with the above is that we now need to repeat the signature
 and the name of the function.

 With AST macros:

 macro namespace (Context context, Ast!(string) name, Declaration
 declaration) { ... }

  namespace("foo::bar") extern (C++) void x ();

 This works because the macro receives the whole declaration that is "x"
 and "x" is replaced with whatever the macro returns:

 pragma(mangle, "mangled_name_of_x") extern (C++) void x ();

 Since it uses the same syntax as UDA's you it can look like real
 namespaces in C++:

  namespace("foo::bar") extern (C++)
 {
       void x ();
       void y ();
 }


perfect for the DIP example section - more of these please :)



 Nov 13 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-14 08:54, dennis luehring wrote:


 perfect for the DIP example section - more of these please :)


Done: http://wiki.dlang.org/DIP50#C.2B.2B_Namespaces_.28issue_7961.29

-- 
/Jacob Carlborg



 Nov 14 2013



  parent  dennis luehring <dl.soluz gmx.net>  writes:


Am 14.11.2013 09:40, schrieb Jacob Carlborg:

 On 2013-11-14 08:54, dennis luehring wrote:


 perfect for the DIP example section - more of these please :)


 Done: http://wiki.dlang.org/DIP50#C.2B.2B_Namespaces_.28issue_7961.29


it would be also nice to have (always) an (if possible) string mixin 
and/or expression-template variant of the example around to show the 
dirtyness, give can't-work information very fast



 Nov 14 2013



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 5:56 PM, deadalnix wrote:

 I think the whole point of macro is to NOT add too much feature to the
language.

 See for instance the example I gave before to create generator. This can be

 macro, the feature can be implemented as a library.

 It is NOT achievable with expression templates.

 I can say with certainty that the async/await mechanism is heavily used in some
 codebases and with great benefit. It is being added to all languages one after
 another.


I agree that async/await has to eventually be added to D. I'm not convinced it 
can or should be done with AST macros.



 The idea behind macros is that instead of adding new feature to the language,
 the language should propose a toolkit to build these features as library.


I do understand this. I've not only extensively used macro systems, I've 
designed two successful ones and implemented other peoples' designs.

I've also seen the sheer awfulness of what happens with these systems over the 
long term. The fascinating thing about this awfulness is people are well aware 
of it in everyone's macro libraries but their own.



 I don't think the time has come for macros in D. As discussed before, it seems
 like filling the gap in existing feature is more important than adding new
ones.
 This should however be considered very seriously when the gaps has been filled.





 Nov 14 2013



 next sibling parent reply dennis luehring <dl.soluz gmx.net>  writes:


Am 14.11.2013 10:12, schrieb Walter Bright:

 I've also seen the sheer awfulness of what happens with these systems over the
 long term. The fascinating thing about this awfulness is people are well aware
 of it in everyone's macro libraries but their own.


the same was said about templates AND string mixins before - but 
currently only the wise guys building good/great templates/mixins which 
are in use by everyone - no terror from the deep


 I don't think the time has come for macros in D.




the DIP needs to grow by many examples and scenarios - some sort
of long term dicussion DIP



 Nov 14 2013



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/14/2013 1:25 AM, dennis luehring wrote:

 Am 14.11.2013 10:12, schrieb Walter Bright:

 I've also seen the sheer awfulness of what happens with these systems over the
 long term. The fascinating thing about this awfulness is people are well aware
 of it in everyone's macro libraries but their own.


 the same was said about templates AND string mixins before - but currently only
 the wise guys building good/great templates/mixins which are in use by everyone
 - no terror from the deep


I don't buy that these are analogous. They are very different features.



 Nov 14 2013



prev sibling next sibling parent reply "Chris Cain" <clcain uncg.edu>  writes:


On Thursday, 14 November 2013 at 09:12:38 UTC, Walter Bright 
wrote:

 I agree that async/await has to eventually be added to D. I'm 
 not convinced it can or should be done with AST macros.


Out of curiousity, how would that be implemented without macros? 
I could see it being done with macros with a rewrite and using 
some of the standard library (like std.parallelism), maybe. But 
with a language feature, you couldn't use the standard library. 
It seems like some parallelism code would have to leak into the 
runtime to get it to work as people might expect.

I guess I would have thought the exact opposite from you... that 
this is precisely something that *should* be done with a 
library-defined macro.



 Nov 14 2013



 next sibling parent  "deadalnix" <deadalnix gmail.com>  writes:


On Thursday, 14 November 2013 at 10:07:31 UTC, Chris Cain wrote:

 On Thursday, 14 November 2013 at 09:12:38 UTC, Walter Bright 
 wrote:

 I agree that async/await has to eventually be added to D. I'm 
 not convinced it can or should be done with AST macros.


 Out of curiousity, how would that be implemented without 
 macros? I could see it being done with macros with a rewrite 
 and using some of the standard library (like std.parallelism), 
 maybe. But with a language feature, you couldn't use the 
 standard library. It seems like some parallelism code would 
 have to leak into the runtime to get it to work as people might 
 expect.

 I guess I would have thought the exact opposite from you... 
 that this is precisely something that *should* be done with a 
 library-defined macro.


async await do not create parallelism.



 Nov 14 2013



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/14/2013 2:07 AM, Chris Cain wrote:

 I guess I would have thought the exact opposite from you... that this is
 precisely something that *should* be done with a library-defined macro.


Again, I reiterate what experience shows happens with macro systems in the long 
term.



 Nov 14 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/14/13 2:17 AM, Walter Bright wrote:

 On 11/14/2013 2:07 AM, Chris Cain wrote:

 I guess I would have thought the exact opposite from you... that this is
 precisely something that *should* be done with a library-defined macro.


 Again, I reiterate what experience shows happens with macro systems in
 the long term.


FWIW Walter talked me back around 2005-2006 into abandoning my own ideas 
about languages with configurable syntax.

There are quite a few failings about comparing programming languages 
against natural languages, but here's one that I think does have value: 
fixed syntax is ingrained into people's notion of language, and swapping 
syntax within an otherwise identical linguistic context is extremely 
taxing on the brain. Anyone who's read with a C or C++ codebase full of 
macros and #if-driven code can attest how unbelievably difficult 
juxtapositions of the normal syntax with preprocessor syntax can quickly 
become. That's part of why "static if" has been so successful in D - it 
drives new semantics but within the same syntax.

Natural languages are "humans complete" because they are the one vehicle 
we use to describe and manipulate our understanding of the entire 
reality. If configurable syntax was something necessary to model the 
world, it would have inevitably occurred in natural languages one way or 
another. Instead, all human languages (with no exception I know of) have 
fixed syntax and prefer to add modeling power in other ways. There must 
be something about this.


Andrei



 Nov 14 2013



 next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-14 19:23, Andrei Alexandrescu wrote:


 FWIW Walter talked me back around 2005-2006 into abandoning my own ideas
 about languages with configurable syntax.

 There are quite a few failings about comparing programming languages
 against natural languages, but here's one that I think does have value:
 fixed syntax is ingrained into people's notion of language, and swapping
 syntax within an otherwise identical linguistic context is extremely
 taxing on the brain. Anyone who's read with a C or C++ codebase full of
 macros and #if-driven code can attest how unbelievably difficult
 juxtapositions of the normal syntax with preprocessor syntax can quickly
 become. That's part of why "static if" has been so successful in D - it
 drives new semantics but within the same syntax.


AST macros cannot change the syntax or introduce new syntax. C 
preprocessor macros and AST macros are not the same. I'm starting to 
regret that I called it "macros".

-- 
/Jacob Carlborg



 Nov 14 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/14/2013 11:18 AM, Jacob Carlborg wrote:

 AST macros cannot change the syntax or introduce new syntax. C preprocessor
 macros and AST macros are not the same. I'm starting to regret that I called it
 "macros".


If it is powerful enough to do async/await but look like normal D syntax, then 
is going to suffer from these faults.

(A lot of the C++ macro abuse looks like normal syntax to the user, but it 
behaves in a way that the casual user would find completely unexpected. In the 
same vein, macro assembler abuse of macros did not introduce new syntax - the 
syntax for the macros was straightforward. It was what those macros *did* that 
made them incomprehensible.)



 Nov 14 2013



 next sibling parent  dennis luehring <dl.soluz gmx.net>  writes:


Am 14.11.2013 21:54, schrieb Walter Bright:

 (A lot of the C++ macro abuse looks like normal syntax to the user, but it
 behaves in a way that the casual user would find completely unexpected.


same can be said about Boost Spirit templates



 Nov 14 2013



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-14 21:54, Walter Bright wrote:


 If it is powerful enough to do async/await but look like normal D
 syntax, then is going to suffer from these faults.


I don't think so. The idea is to have it look consistent within the 
language. Take a look at __traits. It looks just like a function call, 
but it's far from being a function call. Classes and structs look very 
similar, only the keyword is different, but they are very different.

-- 
/Jacob Carlborg



 Nov 14 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/14/2013 11:38 PM, Jacob Carlborg wrote:

 On 2013-11-14 21:54, Walter Bright wrote:


 If it is powerful enough to do async/await but look like normal D
 syntax, then is going to suffer from these faults.


 I don't think so. The idea is to have it look consistent within the language.
 Take a look at __traits. It looks just like a function call, but it's far from
 being a function call. Classes and structs look very similar, only the keyword
 is different, but they are very different.


Think about the desired feature mentioned earlier in this thread of being able 
to insert a "return;" statement, causing what looks like a function call to 
return from the caller.

That's a major step increase in obfuscation, not just a detail.



 Nov 17 2013



 next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-17 09:58, Walter Bright wrote:


 Think about the desired feature mentioned earlier in this thread of
 being able to insert a "return;" statement, causing what looks like a
 function call to return from the caller.


I'm not entirely sure what you're saying. Could you please post a link 
to the post mentioning this or show an example.


 That's a major step increase in obfuscation, not just a detail.


-- 
/Jacob Carlborg



 Nov 17 2013



  parent reply =?UTF-8?B?U2ltZW4gS2rDpnLDpXM=?= <simen.kjaras gmail.com>  writes:


On 2013-11-17 11:37, Jacob Carlborg wrote:

 On 2013-11-17 09:58, Walter Bright wrote:


 Think about the desired feature mentioned earlier in this thread of
 being able to insert a "return;" statement, causing what looks like a
 function call to return from the caller.


 I'm not entirely sure what you're saying. Could you please post a link
 to the post mentioning this or show an example.


I believe it is basically this:

int bar() {
     foo(return 3);
     return 5;
}


If that program is allowed to compile, and to return 3.



 Nov 17 2013



 next sibling parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Sunday, 17 November 2013 at 13:36:32 UTC, Simen Kjærås wrote:

 On 2013-11-17 11:37, Jacob Carlborg wrote:

 On 2013-11-17 09:58, Walter Bright wrote:


 Think about the desired feature mentioned earlier in this 
 thread of
 being able to insert a "return;" statement, causing what 
 looks like a
 function call to return from the caller.


 I'm not entirely sure what you're saying. Could you please 
 post a link
 to the post mentioning this or show an example.


 I believe it is basically this:

 int bar() {
     foo(return 3);
     return 5;
 }


 If that program is allowed to compile, and to return 3.


That never was in the DIP.



 Nov 17 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/17/2013 10:47 AM, deadalnix wrote:

 On Sunday, 17 November 2013 at 13:36:32 UTC, Simen Kjærås wrote:

 On 2013-11-17 11:37, Jacob Carlborg wrote:

 On 2013-11-17 09:58, Walter Bright wrote:


 Think about the desired feature mentioned earlier in this thread of
 being able to insert a "return;" statement, causing what looks like a
 function call to return from the caller.


 I'm not entirely sure what you're saying. Could you please post a link
 to the post mentioning this or show an example.


 I believe it is basically this:

 int bar() {
     foo(return 3);
     return 5;
 }


 If that program is allowed to compile, and to return 3.


 That never was in the DIP.


It was this by Timon Gehr essentially posting why lazy parameters weren't good 
enough:


 On 11/13/2013 08:25 PM, Walter Bright wrote:


 Ah, found the code:

 void ifthen(bool cond, lazy void dg)
 {
      if (cond)
          dg();
 }


 int foo(int x){
     ifthen(!x, return 2); // uh oh
     return 3;
 }





 Nov 17 2013



 next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/17/2013 09:48 PM, Walter Bright wrote:

 It was this by Timon Gehr essentially posting why lazy parameters
 weren't good enough:


 On 11/13/2013 08:25 PM, Walter Bright wrote:


 Ah, found the code:

 void ifthen(bool cond, lazy void dg)
 {
      if (cond)
          dg();
 }


 int foo(int x){
     ifthen(!x, return 2); // uh oh
     return 3;
 }




This code was just supposed to demonstrate that ifthen is not actually a 
"statement" work-alike as claimed in a previous post.



 Nov 17 2013



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/17/2013 1:58 PM, Timon Gehr wrote:

 This code was just supposed to demonstrate that ifthen is not actually a
 "statement" work-alike as claimed in a previous post.


Ok, my apologies.



 Nov 17 2013



prev sibling  parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Sunday, 17 November 2013 at 20:48:15 UTC, Walter Bright wrote:

 That never was in the DIP.


 It was this by Timon Gehr essentially posting why lazy 
 parameters weren't good enough:


 On 11/13/2013 08:25 PM, Walter Bright wrote:


 Ah, found the code:

 void ifthen(bool cond, lazy void dg)
 {
     if (cond)
         dg();
 }


 int foo(int x){
    ifthen(!x, return 2); // uh oh
    return 3;
 }




As I understand it, Timon choosed that syntax simply to 
demonstrate the limitation of your proposal using a similar 
syntax. Not to propose a syntax.



 Nov 17 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/17/2013 7:14 PM, deadalnix wrote:

 As I understand it, Timon choosed that syntax simply to demonstrate the
 limitation of your proposal using a similar syntax. Not to propose a syntax.


Ok, then I'm not seeing what AST macros do that lazy parameters / template 
overloading / mixin templates do not?



 Nov 17 2013



 next sibling parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Monday, 18 November 2013 at 05:05:02 UTC, Walter Bright wrote:

 On 11/17/2013 7:14 PM, deadalnix wrote:

 As I understand it, Timon choosed that syntax simply to 
 demonstrate the
 limitation of your proposal using a similar syntax. Not to 
 propose a syntax.


 Ok, then I'm not seeing what AST macros do that lazy parameters 
 / template overloading / mixin templates do not?


2 things. First, they can act on statement or declaration. Simply 
not with the proposed syntax. Second they can reflect what is 
passed as argument and act accordingly, when the lazy expression 
solution cannot.



 Nov 17 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/17/2013 9:10 PM, deadalnix wrote:

 On Monday, 18 November 2013 at 05:05:02 UTC, Walter Bright wrote:

 On 11/17/2013 7:14 PM, deadalnix wrote:

 As I understand it, Timon choosed that syntax simply to demonstrate the
 limitation of your proposal using a similar syntax. Not to propose a syntax.


 Ok, then I'm not seeing what AST macros do that lazy parameters / template
 overloading / mixin templates do not?


 2 things. First, they can act on statement or declaration. Simply not with the
 proposed syntax.


If they can insert a statement or a declaration uplevel, then they are doing 
what I suggested with the return statement.


 Second they can reflect what is passed as argument and act
 accordingly, when the lazy expression solution cannot.


Expression templates can.



 Nov 18 2013



  parent  "deadalnix" <deadalnix gmail.com>  writes:


On Monday, 18 November 2013 at 09:25:22 UTC, Walter Bright wrote:

 On 11/17/2013 9:10 PM, deadalnix wrote:

 On Monday, 18 November 2013 at 05:05:02 UTC, Walter Bright 
 wrote:

 On 11/17/2013 7:14 PM, deadalnix wrote:

 As I understand it, Timon choosed that syntax simply to 
 demonstrate the
 limitation of your proposal using a similar syntax. Not to 
 propose a syntax.


 Ok, then I'm not seeing what AST macros do that lazy 
 parameters / template
 overloading / mixin templates do not?


 2 things. First, they can act on statement or declaration. 
 Simply not with the
 proposed syntax.


 If they can insert a statement or a declaration uplevel, then 
 they are doing what I suggested with the return statement.


 Second they can reflect what is passed as argument and act
 accordingly, when the lazy expression solution cannot.


 Expression templates can.



*Expression* templates.



 Nov 18 2013



prev sibling  parent  luka8088 <luka8088 owave.net>  writes:


On 18.11.2013. 6:05, Walter Bright wrote:

 On 11/17/2013 7:14 PM, deadalnix wrote:
 
 Ok, then I'm not seeing what AST macros do that lazy parameters /
 template overloading / mixin templates do not?



Well, this is just a small piece of the puzzle but I would like to be
able to write (as syntax sugar):

query q{
  select data;
}

a compiler to rewrite it to:

mixin(query!(q{
  select data;
}));

if the following exists:

mixin statement query (string code) {
  // ...
}

Maybe I am the only one but I found that string mixins + code generation
are used this way even i Phobos and I find the first syntax much more
cleaner.

Or this is not in the general interest?



 Nov 19 2013



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-17 14:36, Simen Kjærås wrote:


 I believe it is basically this:

 int bar() {
      foo(return 3);
      return 5;
 }


 If that program is allowed to compile, and to return 3.


I see, I guess that should work with with the DIP.

-- 
/Jacob Carlborg



 Nov 18 2013



prev sibling next sibling parent  "Dicebot" <public dicebot.lv>  writes:


 Think about the desired feature mentioned earlier in this 
 thread of being able to insert a "return;" statement, causing 
 what looks like a function call to return from the caller.

 That's a major step increase in obfuscation, not just a detail.


You do realize that, as per more strict proposal, it won't change 
a _single_ thing about semantics of actual D function but allows 
to generate another one with semantics defined by macro?

It has _zero_ added ofbuscation over existing template + traits 
combo. You have not shown a single example which proves otherwise 
and keep resorting to personal experience argument instead of 
doing actual comparison. You have full right to do so but it does 
not sound like a decision made on behalf of careful reasoning.



 Nov 17 2013



prev sibling  parent  "Rob T" <alanb ucora.com>  writes:


Walter,

what do you think about allowing mixins to work with parameter 
list construction?

Currently you cannot generate code unless it is syntactically 
complete on its own, this disallows parameter list construction 
even for syntactically correct parameter lists.

The other failure point of mixins is when stringing together 
function calls with the dot, eg A.mixin(MYFUNC).foo;

A whole lot more could be done with mixins if restrictions were 
relaxed just a bit more in a few key areas. I don't see too much 
obfuscation happening so long as the allowable constructs are 
carefully limited and cannot be manipulated in ways that factor 
into the concerns you mentioned previously.

To make it clear we could have a special parammixin() and 
dotmixin() or whatever to ensure there's no possibility of 
confusion.

--rt



 Nov 17 2013



prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Thursday, 14 November 2013 at 18:23:20 UTC, Andrei 
Alexandrescu wrote:

 On 11/14/13 2:17 AM, Walter Bright wrote:

 On 11/14/2013 2:07 AM, Chris Cain wrote:

 I guess I would have thought the exact opposite from you... 
 that this is
 precisely something that *should* be done with a 
 library-defined macro.


 Again, I reiterate what experience shows happens with macro 
 systems in
 the long term.


 FWIW Walter talked me back around 2005-2006 into abandoning my 
 own ideas about languages with configurable syntax.


This is not proposing to configure syntax.


 There are quite a few failings about comparing programming 
 languages against natural languages, but here's one that I 
 think does have value: fixed syntax is ingrained into people's 
 notion of language, and swapping syntax within an otherwise 
 identical linguistic context is extremely taxing on the brain. 
 Anyone who's read with a C or C++ codebase full of macros and 
 #if-driven code can attest how unbelievably difficult 
 juxtapositions of the normal syntax with preprocessor syntax 
 can quickly become. That's part of why "static if" has been so 
 successful in D - it drives new semantics but within the same 
 syntax.


C macro can introduce new syntax. This cannot.


 Natural languages are "humans complete" because they are the 
 one vehicle we use to describe and manipulate our understanding 
 of the entire reality. If configurable syntax was something 
 necessary to model the world, it would have inevitably occurred 
 in natural languages one way or another. Instead, all human 
 languages (with no exception I know of) have fixed syntax and 
 prefer to add modeling power in other ways. There must be 
 something about this.


 Andrei


I think this very post pretty much show that you missed the point 
of the DIP.



 Nov 14 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/14/2013 11:40 AM, deadalnix wrote:

 This is not proposing to configure syntax.


Right, it is about inserting arbitrarily different meaning into existing syntax.

For example, I have a friend from the middle east who worked at Microsoft. His 
english was excellent. But sometimes he'd get tripped up - for example, he'd 
take his idea to his boss, who'd discuss it, and then say:

     "fine"

He took that to mean "my boss approves of my great idea!" Unfortunately, "fine" 
has undergone a meaning reversal colloquially, and actually means "your idea 
sux". This wound up getting him into trouble. There were a few other phrases 
like this with unintentionally hilarious misunderstandings.

(Also, a popular way of encrypting communications is to have private meanings 
for phrases like "my cat ate my homework" meaning "initiate Operation X".)

Now, let's take the AST macros, and pass it "4+3". Apparently is a feature that 
the macro can transform this to produce "hello" as the result. The result is 
that although the language looks the same, there is no "anchor" a person can 
rely on to indicate the meaning. The meaning of "fine" is reversed, so to speak.

The riposte is, of course, every feature can be abused and good programmers 
won't do that. Unfortunately, it's IN THE VERY NATURE of macros to be abusive, 
and even the very best programmers can and will take it to the limit. They will 
see and decry the abuse in everyone else's code but their own, which they will 
justify. As I said, I see this over and over again.

On the other hand, with string mixins, it is immediately obvious that one is 
dealing with different syntax, and so expect something different to happen.



 Nov 14 2013



 next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx>  writes:


On Thu, Nov 14, 2013 at 01:12:11PM -0800, Walter Bright wrote:

 On 11/14/2013 11:40 AM, deadalnix wrote:

This is not proposing to configure syntax.


 
 Right, it is about inserting arbitrarily different meaning into
 existing syntax.


[...]

 Now, let's take the AST macros, and pass it "4+3". Apparently is a
 feature that the macro can transform this to produce "hello" as the
 result. The result is that although the language looks the same,
 there is no "anchor" a person can rely on to indicate the meaning.
 The meaning of "fine" is reversed, so to speak.


[...]

 On the other hand, with string mixins, it is immediately obvious
 that one is dealing with different syntax, and so expect something
 different to happen.


What if macros adopted an overtly different syntax?


T

-- 
English has the lovely word "defenestrate", meaning "to execute by
throwing someone out a window", or more recently "to remove Windows from
a computer and replace it with something useful". :-) -- John Cowan



 Nov 14 2013



 next sibling parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/14/2013 1:34 PM, H. S. Teoh wrote:

 What if macros adopted an overtly different syntax?


That would be better, but I don't think enough better.



 Nov 17 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-17 10:00, Walter Bright wrote:


 That would be better, but I don't think enough better.


Then why to we have UDA's with the same syntax as language attributes 
and why do we have operator overloading.

You can hide arbitrary code behind operator overloading.

-- 
/Jacob Carlborg



 Nov 17 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/17/2013 2:38 AM, Jacob Carlborg wrote:

 On 2013-11-17 10:00, Walter Bright wrote:


 That would be better, but I don't think enough better.


 Then why to we have UDA's with the same syntax as language attributes and why
do
 we have operator overloading.

 You can hide arbitrary code behind operator overloading.


1. I don't believe we can decide on language features by analogy. D is complex 
enough that one can use analogy to justify anything.

2. You cannot do anything behind a function call - the 'return' discussed 
earlier, and async/await for another, i.e. operator overloading cannot
introduce 
control flow, cannot introduce variables into the current scope, etc.



 Nov 17 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-17 21:52, Walter Bright wrote:


 1. I don't believe we can decide on language features by analogy. D is
 complex enough that one can use analogy to justify anything.

 2. You cannot do anything behind a function call - the 'return'
 discussed earlier, and async/await for another, i.e. operator
 overloading cannot introduce control flow, cannot introduce variables
 into the current scope, etc.


You can turn this:

BigInt(3) + 7;

Into something that is not 10.

You cannot do anything behind a macro. You cannot introduce new syntax.

-- 
/Jacob Carlborg



 Nov 18 2013



 next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/18/2013 09:22 AM, Jacob Carlborg wrote:

 On 2013-11-17 21:52, Walter Bright wrote:


 1. I don't believe we can decide on language features by analogy. D is
 complex enough that one can use analogy to justify anything.

 2. You cannot do anything behind a function call - the 'return'
 discussed earlier, and async/await for another, i.e. operator
 overloading cannot introduce control flow, cannot introduce variables
 into the current scope, etc.


 You can turn this:

 BigInt(3) + 7;

 Into something that is not 10.

 You cannot do anything behind a macro. You cannot introduce new syntax.


Well, you could do:

 screwUp int foo(){ return 3+7; }

assert(foo()!=10);



 Nov 18 2013



 next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-18 10:12, Timon Gehr wrote:


 Well, you could do:

  screwUp int foo(){ return 3+7; }

 assert(foo()!=10);


Sure, what I'm saying is that you can do stupid things with other 
features than macros.

-- 
/Jacob Carlborg



 Nov 18 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/18/2013 1:14 AM, Jacob Carlborg wrote:

 Sure, what I'm saying is that you can do stupid things with other features than
 macros.


Yes, you can write terrible code with all features.

My point is not that you can do stupid things with macros. It is that macros 
encourage this to a high degree, and in fact the flexibility that macros have 
over lazy parameters / expression templates / mixin templates are exactly those 
features that are abused to the point it is hard to see much good come from 
them. I am not making this up, this is what happens with macro systems again
and 
again. I know this is a difficult position to argue from and convince people
about.

D has a deliberate lack of capability in some areas because long experience 
shows that little good comes from them. Examples include lack of ability to 
overload && and ||, and lack of expression support for version statements. This 
is not because these features are hard - they're easy to implement. I've seen 
attempts to replace version statements with static if in order to get back the 
ugliness, and the result is ugly and unnecessary and damned hard to figure out 
what was actually intended and why it didn't work.

I fear that implementing these things with the idea that they'll somehow be
used 
with restraint is a pipe dream and that D will degenerate into an ugly snarl 
that everyone hates, and then some D-Go will be invented that everyone will 
cheer because it won't allow such code to be written.

Case in point - again and again I see utterly wretched use of macros in C++
from 
the top level of C++ programmers who really ought to know better. Want to see a 
lovely example? Take a look at the C header files for openssl:

https://github.com/D-Programming-Deimos/openssl/tree/master/C

This is not unusual.

I try to put my money where my mouth is. If you look at older parts of dmd, 
you'll see macro abuse that I cleverly wrote. The newer bits contain fewer and 
fewer use of any macros, and I've made an ongoing effort to try and eliminate 
all of it (yebblies has campaigned to get rid of it, too, even crap I tried to 
hang on to). This has been done this without compromise on performance and 
usability and readability of the code - and I hope you'll agree that it looks 
nice without the usual rat's nest of preprocessor abuse one sees in typical C++ 
code.



 Nov 18 2013



 next sibling parent reply "IgorStepanov" <wazar mail.ru>  writes:


On Monday, 18 November 2013 at 09:55:42 UTC, Walter Bright wrote:

 On 11/18/2013 1:14 AM, Jacob Carlborg wrote:

 Sure, what I'm saying is that you can do stupid things with 
 other features than
 macros.


 Yes, you can write terrible code with all features.

 My point is not that you can do stupid things with macros. It 
 is that macros encourage this to a high degree, and in fact the 
 flexibility that macros have over lazy parameters / expression 
 templates / mixin templates are exactly those features that are 
 abused to the point it is hard to see much good come from them. 
 I am not making this up, this is what happens with macro 
 systems again and again. I know this is a difficult position to 
 argue from and convince people about.

 D has a deliberate lack of capability in some areas because 
 long experience shows that little good comes from them. 
 Examples include lack of ability to overload && and ||, and 
 lack of expression support for version statements. This is not 
 because these features are hard - they're easy to implement. 
 I've seen attempts to replace version statements with static if 
 in order to get back the ugliness, and the result is ugly and 
 unnecessary and damned hard to figure out what was actually 
 intended and why it didn't work.

 I fear that implementing these things with the idea that 
 they'll somehow be used with restraint is a pipe dream and that 
 D will degenerate into an ugly snarl that everyone hates, and 
 then some D-Go will be invented that everyone will cheer 
 because it won't allow such code to be written.

 Case in point - again and again I see utterly wretched use of 
 macros in C++ from the top level of C++ programmers who really 
 ought to know better. Want to see a lovely example? Take a look 
 at the C header files for openssl:

 https://github.com/D-Programming-Deimos/openssl/tree/master/C

 This is not unusual.

 I try to put my money where my mouth is. If you look at older 
 parts of dmd, you'll see macro abuse that I cleverly wrote. The 
 newer bits contain fewer and fewer use of any macros, and I've 
 made an ongoing effort to try and eliminate all of it (yebblies 
 has campaigned to get rid of it, too, even crap I tried to hang 
 on to). This has been done this without compromise on 
 performance and usability and readability of the code - and I 
 hope you'll agree that it looks nice without the usual rat's 
 nest of preprocessor abuse one sees in typical C++ code.




to SQL WHERE condition. For example s => s.Number > 2 && s.Number 
< 20 can be translated to ... WHERE s.Number > 2 AND s.Number < 20

It is interesting function, because it allow to write hard code 

this code hides poorly standardized SQL for the well-standardized 
language.


limitations for ast macros. May be they allowed only for lambda?

This is a general macro-feature which I had like to see in D.



 Nov 18 2013



  parent reply "Kapps" <opantm2+spam gmail.com>  writes:


On Monday, 18 November 2013 at 16:03:54 UTC, IgorStepanov wrote:



 to SQL WHERE condition. For example s => s.Number > 2 && 
 s.Number < 20 can be translated to ... WHERE s.Number > 2 AND 
 s.Number < 20

 It is interesting function, because it allow to write hard code 

 Also this code hides poorly standardized SQL for the 
 well-standardized language.


 some limitations for ast macros. May be they allowed only for 
 lambda?

 This is a general macro-feature which I had like to see in D.



expression trees. Essentially, it allows the compiler to generate 
an expression tree of the lambda expression which your library 
parses to do some work. It doesn't inject any code, and it's done 
at runtime. I could be wrong about the exact details, but 
essentially it comes down to the compiler passing in an 
expression tree into your method, and then your method parsing it 
at runtime and doing something with the results; it can't be used 
to generate new code.



 Nov 18 2013



 next sibling parent reply "IgorStepanov" <wazar mail.ru>  writes:


On Monday, 18 November 2013 at 19:21:35 UTC, Kapps wrote:

 On Monday, 18 November 2013 at 16:03:54 UTC, IgorStepanov wrote:



 expression to SQL WHERE condition. For example s => s.Number > 
 2 && s.Number < 20 can be translated to ... WHERE s.Number > 2 
 AND s.Number < 20

 It is interesting function, because it allow to write hard 

 (SQL). Also this code hides poorly standardized SQL for the 
 well-standardized language.


 some limitations for ast macros. May be they allowed only for 
 lambda?

 This is a general macro-feature which I had like to see in D.



 expression trees. Essentially, it allows the compiler to 
 generate an expression tree of the lambda expression which your 
 library parses to do some work. It doesn't inject any code, and 
 it's done at runtime. I could be wrong about the exact details, 
 but essentially it comes down to the compiler passing in an 
 expression tree into your method, and then your method parsing 
 it at runtime and doing something with the results; it can't be 
 used to generate new code.


May be this way is good for D? And CTFE allow to parse this tree
at compile-time.
Something like:

 property Collection where(alias cond, Collection)(Collection c)
{
      enum ast = cond.astof;
      static if(is(Collection : DBResultSet))
      {
         string where_cond = generateCondition!(ast, Collection);
         return c.doWhere(where_cond);
      }
      else
      {
         ...
      }
}

...

auto result = db.myItems.where!(x => x.Count > 10);



 Nov 18 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-18 22:08, IgorStepanov wrote:


 May be this way is good for D? And CTFE allow to parse this tree
 at compile-time.
 Something like:

  property Collection where(alias cond, Collection)(Collection c)
 {
       enum ast = cond.astof;
       static if(is(Collection : DBResultSet))
       {
          string where_cond = generateCondition!(ast, Collection);
          return c.doWhere(where_cond);
       }
       else
       {
          ...
       }
 }

 ...

 auto result = db.myItems.where!(x => x.Count > 10);


The idea is to have something more general. Linq is just one example of 
what can be implemented.

-- 
/Jacob Carlborg



 Nov 18 2013



prev sibling  parent  Ellery Newcomer <ellery-newcomer utulsa.edu>  writes:


On 11/18/2013 11:21 AM, Kapps wrote:

 On Monday, 18 November 2013 at 16:03:54 UTC, IgorStepanov wrote:
 it can't be
 used to generate new code.


Sure it can. Generate lambda expression tree; invoke Compile(). You just 
can't rewrite existing code.



 Nov 19 2013



prev sibling  parent  "Doodoo" <barf barf.com>  writes:


 Case in point - again and again I see utterly wretched use of 
 macros in C++ from the top level of C++ programmers who really 
 ought to know better. Want to see a lovely example? Take a look 
 at the C header files for openssl:

 https://github.com/D-Programming-Deimos/openssl/tree/master/C

 This is not unusual.

 I try to put my money where my mouth is. If you look at older 
 parts of dmd, you'll see macro abuse that I cleverly wrote. The 
 newer bits contain fewer and fewer use of any macros, and I've 
 made an ongoing effort to try and eliminate all of it (yebblies 
 has campaigned to get rid of it, too, even crap I tried to hang 
 on to). This has been done this without compromise on 
 performance and usability and readability of the code - and I 
 hope you'll agree that it looks nice without the usual rat's 
 nest of preprocessor abuse one sees in typical C++ code.


  Now, now Walter, that is C not C++. In C++ we would use Boost 
Preprocessor:)

  Also C macros are just crappy textural replacement, I would 
assume an AST macro would be along the lines of lisp, which 
doesn't have much of anything to do with the shitty macro system 
in C despite using the word "macro".

  Not that I care so whatevars!



 Nov 18 2013



prev sibling  parent reply "Dicebot" <public dicebot.lv>  writes:


On Monday, 18 November 2013 at 09:12:19 UTC, Timon Gehr wrote:

 Well, you could do:

  screwUp int foo(){ return 3+7; }

 assert(foo()!=10);


Anything that allows it routinely should be banned whenever 
macros are accepted or not. Modifying existing "normal" symbols = 
no way.



 Nov 18 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-18 16:59, Dicebot wrote:


 Anything that allows it routinely should be banned whenever macros are
 accepted or not. Modifying existing "normal" symbols = no way.


The whole point of macros is to modify symbols. Turning this:

auto p = Person.where(q => q.name == "John");

To an SQL query.

-- 
/Jacob Carlborg



 Nov 18 2013



  parent reply "Dicebot" <public dicebot.lv>  writes:


On Monday, 18 November 2013 at 21:09:26 UTC, Jacob Carlborg wrote:

 On 2013-11-18 16:59, Dicebot wrote:


 Anything that allows it routinely should be banned whenever 
 macros are
 accepted or not. Modifying existing "normal" symbols = no way.


 The whole point of macros is to modify symbols. Turning this:

 auto p = Person.where(q => q.name == "John");

 To an SQL query.


You don't need to modify actual input lambda for this to become 
SQL query. I fully support macros as convenient way for AST 
reflection and related code generation but not a single step 
further. Changing semantics of existing valid piece of code is 
never worth it.



 Nov 18 2013



 next sibling parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/18/2013 11:09 PM, Dicebot wrote:

 On Monday, 18 November 2013 at 21:09:26 UTC, Jacob Carlborg wrote:

 On 2013-11-18 16:59, Dicebot wrote:


 Anything that allows it routinely should be banned whenever macros are
 accepted or not. Modifying existing "normal" symbols = no way.


 The whole point of macros is to modify symbols. Turning this:

 auto p = Person.where(q => q.name == "John");

 To an SQL query.


 You don't need to modify actual input lambda for this to become SQL
 query. I fully support macros as convenient way for AST reflection and
 related code generation but not a single step further. Changing
 semantics of existing valid piece of code is never worth it.


So you oppose macro attributes?



 Nov 18 2013



  parent reply "Dicebot" <public dicebot.lv>  writes:


On Monday, 18 November 2013 at 22:47:59 UTC, Timon Gehr wrote:

 So you oppose macro attributes?


Yes. For me proper "macro attribute" is just an UDA that can be 
used by macro to tweak code generation. Actually I think I'd 
oppose a very large part of this specific DIP when it comes to 
exact syntax and usage models - I am more interested in "AST" 
than in "macro" part.



 Nov 19 2013



  parent  "Yota" <yotaxp thatGoogleMailThing.com>  writes:


On Tuesday, 19 November 2013 at 13:10:44 UTC, Dicebot wrote:

 On Monday, 18 November 2013 at 22:47:59 UTC, Timon Gehr wrote:

 So you oppose macro attributes?


 Yes. For me proper "macro attribute" is just an UDA that can be 
 used by macro to tweak code generation. Actually I think I'd 
 oppose a very large part of this specific DIP when it comes to 
 exact syntax and usage models - I am more interested in "AST" 
 than in "macro" part.


This topic really took off.  I'm afraid to say that I've only 
gotten through about half of it so far.

I noticed computation expressions were mentioned, but has anyone 

exactly like what is being proposed here.
http://msdn.microsoft.com/en-us/library/vstudio/dd233212.aspx




The LINQ sample in the DIP kinda rubs me the wrong way.  LINQ 

does not generate imperative code, but rather a chain of 
extension methods.  More importantly, it does not eagerly chomp 
through all the data.  It runs on IEnumerable, and is therefore 
lazy. (Like D ranges.)  I wouldn't mind, however, if I saw these 
macros generate state machines specifically made for a given 
query.



 Nov 19 2013



prev sibling next sibling parent  "SomeDude" <lovelydear mailmetrash.com>  writes:


On Monday, 18 November 2013 at 22:09:35 UTC, Dicebot wrote:

 On Monday, 18 November 2013 at 21:09:26 UTC, Jacob Carlborg 
 wrote:

 On 2013-11-18 16:59, Dicebot wrote:


 Anything that allows it routinely should be banned whenever 
 macros are
 accepted or not. Modifying existing "normal" symbols = no way.


 The whole point of macros is to modify symbols. Turning this:

 auto p = Person.where(q => q.name == "John");

 To an SQL query.


 You don't need to modify actual input lambda for this to become 
 SQL query. I fully support macros as convenient way for AST 
 reflection and related code generation but not a single step 
 further. Changing semantics of existing valid piece of code is 
 never worth it.


And that's exactly the kind of thing Walter doesn't want to see.



 Nov 18 2013



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-18 23:09, Dicebot wrote:


 You don't need to modify actual input lambda for this to become SQL
 query. I fully support macros as convenient way for AST reflection and
 related code generation but not a single step further. Changing
 semantics of existing valid piece of code is never worth it.


Well, it depends on how you look at it. You don't need to modify the 
lambda, but you're changing the meaning of it.

-- 
/Jacob Carlborg



 Nov 18 2013



  parent reply "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 19 November 2013 at 07:23:52 UTC, Jacob Carlborg 
wrote:

 Well, it depends on how you look at it. You don't need to 
 modify the lambda, but you're changing the meaning of it.


Not really. You won't say that this snippet changes the meaning 
of lambda, do you?

```
foo(a => a*2);
// ...
bool foo(T)(T func)
{
     // does not even call `func` and does something unrelated
     return is(ReturnType!T == int);
}
```

This is pretty much the same. Functions and templates can do 
arbitrary things unrelated to their input and we do not call it 
"changing the meaning" of the input. Macros are not much 
different here. Though I'd still prefer requirement for explicit 
"macro!" syntax or something similar, just to make clear that 
code gen is incoming.



 Nov 19 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-19 14:16, Dicebot wrote:


 Not really. You won't say that this snippet changes the meaning of
 lambda, do you?

 ```
 foo(a => a*2);
 // ...
 bool foo(T)(T func)
 {
      // does not even call `func` and does something unrelated
      return is(ReturnType!T == int);
 }
 ```


No, I guess not.

-- 
/Jacob Carlborg



 Nov 19 2013



prev sibling  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/18/2013 12:22 AM, Jacob Carlborg wrote:

 On 2013-11-17 21:52, Walter Bright wrote:


 1. I don't believe we can decide on language features by analogy. D is
 complex enough that one can use analogy to justify anything.

 2. You cannot do anything behind a function call - the 'return'
 discussed earlier, and async/await for another, i.e. operator
 overloading cannot introduce control flow, cannot introduce variables
 into the current scope, etc.


 You can turn this:

 BigInt(3) + 7;

 Into something that is not 10.


That is not at all what I was talking about. I was talking about:

    3+7

being transformed by AST manipulation into something that is not 10.


 You cannot do anything behind a macro. You cannot introduce new syntax.





 Nov 18 2013



prev sibling  parent  "Doodoo" <barf barf.com>  writes:


HS Teoh why u always gotta go and break the forum



 Nov 17 2013



prev sibling next sibling parent  "Dicebot" <public dicebot.lv>  writes:


On Thursday, 14 November 2013 at 21:12:11 UTC, Walter Bright 
wrote:

 On the other hand, with string mixins, it is immediately 
 obvious that one is dealing with different syntax, and so 
 expect something different to happen.


This is exactly the reason why I have been asking to make 
denoting macro usage with explicit keyword mandatory, like it is 
done with mixins.

It does not change much about the concept itself and is not worth 
fixating upon as a blocker.



 Nov 14 2013



prev sibling next sibling parent  "Zsombor Barna" <belzurix sch.bme.hu>  writes:


On Thursday, 14 November 2013 at 21:12:11 UTC, Walter Bright 
wrote:

 On 11/14/2013 11:40 AM, deadalnix wrote:

 This is not proposing to configure syntax.


 Right, it is about inserting arbitrarily different meaning into 
 existing syntax.

 For example, I have a friend from the middle east who worked at 
 Microsoft. His english was excellent. But sometimes he'd get 
 tripped up - for example, he'd take his idea to his boss, who'd 
 discuss it, and then say:

     "fine"

 He took that to mean "my boss approves of my great idea!" 
 Unfortunately, "fine" has undergone a meaning reversal 
 colloquially, and actually means "your idea sux". This wound up 
 getting him into trouble. There were a few other phrases like 
 this with unintentionally hilarious misunderstandings.

 (Also, a popular way of encrypting communications is to have 
 private meanings for phrases like "my cat ate my homework" 
 meaning "initiate Operation X".)

 Now, let's take the AST macros, and pass it "4+3". Apparently 
 is a feature that the macro can transform this to produce 
 "hello" as the result. The result is that although the language 
 looks the same, there is no "anchor" a person can rely on to 
 indicate the meaning. The meaning of "fine" is reversed, so to 
 speak.

 The riposte is, of course, every feature can be abused and good 
 programmers won't do that. Unfortunately, it's IN THE VERY 
 NATURE of macros to be abusive, and even the very best 
 programmers can and will take it to the limit. They will see 
 and decry the abuse in everyone else's code but their own, 
 which they will justify. As I said, I see this over and over 
 again.

 On the other hand, with string mixins, it is immediately 
 obvious that one is dealing with different syntax, and so 
 expect something different to happen.


You have a point with the example, but just as with templates ( 
the distinctive binary ! operator), making macros not to look 
first class language elements when called does great job with 

very foreign.

#loop ( ) {
   -- code --
}

Maybe for macros that return macros it would be ugly:


But just as in C where the -> operator automatically dereferences 
pointers, it could be solved.

(Just for fun)



 Nov 14 2013



prev sibling  parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Thursday, 14 November 2013 at 21:12:11 UTC, Walter Bright 
wrote:

 On 11/14/2013 11:40 AM, deadalnix wrote:

 This is not proposing to configure syntax.


 Right, it is about inserting arbitrarily different meaning into 
 existing syntax.


Yes, but EXPLICITELY.


 (Also, a popular way of encrypting communications is to have 
 private meanings for phrases like "my cat ate my homework" 
 meaning "initiate Operation X".)


Note that you do not pretend that sentence with "THIS IS AN 
ENCRYPTED SENTENCE, HERE TO DECRYPT IT DO XXX".


 Now, let's take the AST macros, and pass it "4+3". Apparently 
 is a feature that the macro can transform this to produce 
 "hello" as the result. The result is that although the language 
 looks the same, there is no "anchor" a person can rely on to 
 indicate the meaning. The meaning of "fine" is reversed, so to 
 speak.


mymacro(4+3); // <= can be "hello"
myfunction(4+3); // <= can be "hello as well"

Quite frankly, this isn't really convincing.


 The riposte is, of course, every feature can be abused and good 
 programmers won't do that. Unfortunately, it's IN THE VERY 
 NATURE of macros to be abusive, and even the very best 
 programmers can and will take it to the limit. They will see 
 and decry the abuse in everyone else's code but their own, 
 which they will justify. As I said, I see this over and over 
 again.


I can find article on the internet saying the exact same thing 
for any feature ever invented. You'll find quite easily some 
about templates or operator overloading, but if you go in 
Functionnalland, you'll find the same about virtual dispatch and 
inheritance. If you go in OOPland, you'll find the exact same 
argument for monads.

You are qualifying the nature of macro here, that is not an 
argument. "Macros are bad, so I conclude macro are bad."


 On the other hand, with string mixins, it is immediately 
 obvious that one is dealing with different syntax, and so 
 expect something different to happen.


string mixins + a CTFE usable lexer+parser+D codegen from AST 
would achieve something close to the macros proposed here. This 
would obviously require quite a lot of work and is doomed to be 
dramatically slow and resources hungry, less hygienic (as 
identifier are implicitly resolved in the mixed in scope rather 
than explicitly) but quite close in term of capabilities.

You can't argue that string mixin is good while AST macro is bad.



 Nov 14 2013



 next sibling parent reply "SomeDude" <lovelydear mailmetrash.com>  writes:


On Thursday, 14 November 2013 at 22:06:06 UTC, deadalnix wrote:

 mymacro(4+3); // <= can be "hello"
 myfunction(4+3); // <= can be "hello as well"

 Quite frankly, this isn't really convincing.


True but you will hardly use myfunction as an extension to the 
language.

If I understand the issues, I quite understand Walter's 
reluctance to add such a feature. Once you add it, you add the 
capability to change the language, or to create an entirely new 
language. If you let it, no matter how loud you cry, people 
*will* abuse the feature, and they will do it from day one. The 
C++ templates experiment shows what happens when you do that. We 
don't want to repeat the same mistakes.

Basically, people want to change the language, but without having 
to discuss their own extensions in the newsgroups. It's sometimes 
handy, but you'll end up with crappy features all over the place.



 Nov 17 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-17 09:18, SomeDude wrote:


 True but you will hardly use myfunction as an extension to the language.

 If I understand the issues, I quite understand Walter's reluctance to
 add such a feature. Once you add it, you add the capability to change
 the language, or to create an entirely new language. If you let it, no
 matter how loud you cry, people *will* abuse the feature, and they will
 do it from day one. The C++ templates experiment shows what happens when
 you do that. We don't want to repeat the same mistakes.


We already have templates and operator overloading. Perhaps we should 
remove those, we don't want to take the chance of people abusing them.


 Basically, people want to change the language, but without having to
 discuss their own extensions in the newsgroups. It's sometimes handy,
 but you'll end up with crappy features all over the place.


No, people want a general solution that doesn't require the language to 
be extended each time they come up with a useful feature.

-- 
/Jacob Carlborg



 Nov 17 2013



  parent reply "SomeDude" <lovelydear mailmetrash.com>  writes:


On Sunday, 17 November 2013 at 10:41:46 UTC, Jacob Carlborg wrote:

 We already have templates and operator overloading. Perhaps we 
 should remove those, we don't want to take the chance of people 
 abusing them.


D templates have been designed with the past experience of C++ 
templates in mind, so the mistakes have been avoided. With 
macros, we don't have much experience; Lisp macros can't count as 
the language is too different.


 Basically, people want to change the language, but without 
 having to
 discuss their own extensions in the newsgroups. It's sometimes 
 handy,
 but you'll end up with crappy features all over the place.


 No, people want a general solution that doesn't require the 
 language to be extended each time they come up with a useful 
 feature.


Sorry but I tend to value the feedback of people who have already 
put some serious effort in expression templates and mixins like 
Don.



 Nov 18 2013



  parent  "deadalnix" <deadalnix gmail.com>  writes:


On Monday, 18 November 2013 at 21:38:08 UTC, SomeDude wrote:

 No, people want a general solution that doesn't require the 
 language to be extended each time they come up with a useful 
 feature.


 Sorry but I tend to value the feedback of people who have 
 already put some serious effort in expression templates and 
 mixins like Don.


The very first code I wrote in D used expression templates to do 
vector/matrices computation. I'm really well aware of they are 
and what they can do (and what error messages you get when you do 
something wrong).



 Nov 18 2013



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/14/2013 2:06 PM, deadalnix wrote:

 You can't argue that string mixin is good while AST macro is bad.


True, but you also cannot argue that since D has string mixins it must also
have 
AST macros.

What macros do is bless, encourage, and make inevitable a particularly abusive 
style of programming. D already has super powerful metaprogramming abilities 
that we are far from exploring the limits of. I don't think we should be adding 
on even more that experience has shown goes quickly out of control.



 Nov 17 2013



  parent  "Zsombor Barna" <belzurix sch.bme.hu>  writes:


On Sunday, 17 November 2013 at 09:13:04 UTC, Walter Bright wrote:

 On 11/14/2013 2:06 PM, deadalnix wrote:

 You can't argue that string mixin is good while AST macro is 
 bad.


 True, but you also cannot argue that since D has string mixins 
 it must also have AST macros.

 What macros do is bless, encourage, and make inevitable a 
 particularly abusive style of programming. D already has super 
 powerful metaprogramming abilities that we are far from 
 exploring the limits of. I don't think we should be adding on 
 even more that experience has shown goes quickly out of control.


Yeah, you are right, but I doubt anybody would use macros in 
production code without considering much less powerful constructs 
of the language to solve the problem. In hobby projects, I don't 
mind if create spaghetti code while experimenting.



 Nov 17 2013



prev sibling next sibling parent reply "Zsombor Barna" <belzurix sch.bme.hu>  writes:


On Thursday, 14 November 2013 at 18:23:20 UTC, Andrei 
Alexandrescu wrote:

 On 11/14/13 2:17 AM, Walter Bright wrote:

 On 11/14/2013 2:07 AM, Chris Cain wrote:

 I guess I would have thought the exact opposite from you... 
 that this is
 precisely something that *should* be done with a 
 library-defined macro.


 Again, I reiterate what experience shows happens with macro 
 systems in
 the long term.


 FWIW Walter talked me back around 2005-2006 into abandoning my 
 own ideas about languages with configurable syntax.

 There are quite a few failings about comparing programming 
 languages against natural languages, but here's one that I 
 think does have value: fixed syntax is ingrained into people's 
 notion of language, and swapping syntax within an otherwise 
 identical linguistic context is extremely taxing on the brain. 
 Anyone who's read with a C or C++ codebase full of macros and 
 #if-driven code can attest how unbelievably difficult 
 juxtapositions of the normal syntax with preprocessor syntax 
 can quickly become. That's part of why "static if" has been so 
 successful in D - it drives new semantics but within the same 
 syntax.

 Natural languages are "humans complete" because they are the 
 one vehicle we use to describe and manipulate our understanding 
 of the entire reality. If configurable syntax was something 
 necessary to model the world, it would have inevitably occurred 
 in natural languages one way or another. Instead, all human 
 languages (with no exception I know of) have fixed syntax and 
 prefer to add modeling power in other ways. There must be 
 something about this.


 Andrei


D's syntax remains the same ( statements, expressions, function 
calling, numbers etc. ). These AST manipulation tools just define 
new words or language constructs. Human languages tend to be 
altered as times passes and the needs change. That's why new 
words appear. Even the grammar is not the same as the one my 
grandparents used.



 Nov 14 2013



  parent  "SomeDude" <lovelydear mailmetrash.com>  writes:


On Thursday, 14 November 2013 at 19:42:36 UTC, Zsombor Barna 
wrote:

 D's syntax remains the same ( statements, expressions, function 
 calling, numbers etc. ). These AST manipulation tools just 
 define new words or language constructs. Human languages tend 
 to be altered as times passes and the needs change. That's why 
 new words appear. Even the grammar is not the same as the one 
 my grandparents used.


Which shows all the problems raised by Walter:
1. you need an entry in the dictionary that you need to look up 
for each new word that someone has introduced,
2. with natural language, you don't need to know its *exact* 
semantics when you use a new word, while in programming it is 
mandatory,
3. you *don't* want your programming language to arbitrarily 
change over time. That's probably the biggest issue.



 Nov 17 2013



prev sibling  parent  "Rob T" <alanb ucora.com>  writes:


On Thursday, 14 November 2013 at 18:23:20 UTC, Andrei 
Alexandrescu wrote:
[...]

 Natural languages are "humans complete" because they are the 
 one vehicle we use to describe and manipulate our understanding 
 of the entire reality. If configurable syntax was something 
 necessary to model the world, it would have inevitably occurred 
 in natural languages one way or another. Instead, all human 
 languages (with no exception I know of) have fixed syntax and 
 prefer to add modeling power in other ways. There must be 
 something about this.


 Andrei


Not an unreasonable assumption to make, so assuming it's true, 
then we need a fixed way to deal with a certain set of problems 
that so far cannot be done reasonably (or at all) in current D.

One case, which I also so happen to want to solve, is the 
async/await type of problem. There's simply no reasonable 
solution available other than through extensions to the language, 
which may or may not ever materialize. It's very frustrating when 
you want to explore possibilities today rather than wait for an 
extension that may never arrive.

Adding in the async/await extensions can help solve only that 
specific problem, but how many other types of problems that are 
similar to this one will programmers still want to solve? A more 
general solution would seem to be far more useful.

The other simple problem I encountered, at first seems like a 
mixin can deal with or perhaps a template, but current D simply 
cannot do it in a reasonable way that I'm aware of.

I want to be able to pass an argument to an "inspect" function, 
that will log or display the symbol name of the top-level 
argument and it's corresponding value.

Example:

int a = 300;
inspect( a ); // or inspect!a;

The only way I can do this is through double entry of the item to 
inspect
eg

inspect( a.stringof, a ); // error prone and irritatingly 
redundant

I can get some success using an alias template parameter, but it 
does not work in general, only in a few cases will it work, so 
templates are not a solution without some additional enhancements.

May be by adding in something like __ARGS__, or expanding 
on__traits can solve this type of problem, but there's no doubt 
many other scenarios that will require similar extensions that 
users of the language cannot implement. There's no end to this 
without much better access to reflection.

--rt



 Nov 14 2013



prev sibling  parent reply "Meta" <jared771 gmail.com>  writes:


On Thursday, 14 November 2013 at 09:12:38 UTC, Walter Bright 
wrote:

 I do understand this. I've not only extensively used macro 
 systems, I've designed two successful ones and implemented 
 other peoples' designs.

 I've also seen the sheer awfulness of what happens with these 
 systems over the long term. The fascinating thing about this 
 awfulness is people are well aware of it in everyone's macro 
 libraries but their own.


What kind of macro systems are you talking about? C-like macros? 
Assembly macros? Lisp-like macros? It seems that Jacob's proposed 
system is similar to none of these.



 Nov 14 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/14/2013 9:08 AM, Meta wrote:

 What kind of macro systems are you talking about? C-like macros? Assembly
 macros? Lisp-like macros? It seems that Jacob's proposed system is similar to
 none of these.


See my first comment in this thread - it's an ancestor to this one.



 Nov 14 2013



  parent reply "Meta" <jared771 gmail.com>  writes:


On Thursday, 14 November 2013 at 17:55:57 UTC, Walter Bright 
wrote:

 On 11/14/2013 9:08 AM, Meta wrote:

 What kind of macro systems are you talking about? C-like 
 macros? Assembly
 macros? Lisp-like macros? It seems that Jacob's proposed 
 system is similar to
 none of these.


 See my first comment in this thread - it's an ancestor to this 
 one.


I'm using the website forum, so I see posts in a continuous 
format, not a threaded view.

Anyway, it seems like assembly macros are actually quite similar 
to Lisp macros. Again, it seems like what Jacob proposes is not 
really much like Lisp macros, nor C macros.



 Nov 14 2013



  parent reply "simendsjo" <simendsjo gmail.com>  writes:


On Thursday, 14 November 2013 at 19:22:08 UTC, Meta wrote:

 On Thursday, 14 November 2013 at 17:55:57 UTC, Walter Bright 
 wrote:

 On 11/14/2013 9:08 AM, Meta wrote:

 What kind of macro systems are you talking about? C-like 
 macros? Assembly
 macros? Lisp-like macros? It seems that Jacob's proposed 
 system is similar to
 none of these.


 See my first comment in this thread - it's an ancestor to this 
 one.


 I'm using the website forum, so I see posts in a continuous 
 format, not a threaded view.

 Anyway, it seems like assembly macros are actually quite 
 similar to Lisp macros. Again, it seems like what Jacob 
 proposes is not really much like Lisp macros, nor C macros.


At the top of the page you should have "view mode" links which 
supports a threaded view.



 Nov 14 2013



  parent  "Meta" <jared771 gmail.com>  writes:


On Thursday, 14 November 2013 at 19:27:42 UTC, simendsjo wrote:

 At the top of the page you should have "view mode" links which 
 supports a threaded view.


Neat. I have my browser set up to auto-login, so I rarely, if 
ever, have to look up there.



 Nov 14 2013



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-14 01:16, Walter Bright wrote:


 Yes. But that's a good thing. I'd be pretty skeptical of the value of an
 AST macro that took 3+4 and changed it so it did something other than
 compute 7.


You can still do stupid things like that with operator overloading. Not 
on built-in types, but on user defined types. Every language allows you 
to do stupid things, one way or another.

-- 
/Jacob Carlborg



 Nov 13 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 11:37 PM, Jacob Carlborg wrote:

 On 2013-11-14 01:16, Walter Bright wrote:


 Yes. But that's a good thing. I'd be pretty skeptical of the value of an
 AST macro that took 3+4 and changed it so it did something other than
 compute 7.


 You can still do stupid things like that with operator overloading. Not on
 built-in types, but on user defined types. Every language allows you to do
 stupid things, one way or another.


Sure, but the issue is that expression templates are not for "int+int", but for 
"T+int". My question is what value would there be in a rewrite of "int+int" to 
mean something different.



 Nov 14 2013



  parent reply dennis luehring <dl.soluz gmx.net>  writes:


Am 14.11.2013 09:53, schrieb Walter Bright:

 On 11/13/2013 11:37 PM, Jacob Carlborg wrote:

 On 2013-11-14 01:16, Walter Bright wrote:


 Yes. But that's a good thing. I'd be pretty skeptical of the value of an
 AST macro that took 3+4 and changed it so it did something other than
 compute 7.


 You can still do stupid things like that with operator overloading. Not on
 built-in types, but on user defined types. Every language allows you to do
 stupid things, one way or another.


 Sure, but the issue is that expression templates are not for "int+int", but for
 "T+int". My question is what value would there be in a rewrite of "int+int" to
 mean something different.


"int + int" could be part of an PC<->GPU kombination that generates 
CUDA- and D-interchange code at compiletime



 Nov 14 2013



  parent reply dennis luehring <dl.soluz gmx.net>  writes:


Am 14.11.2013 10:06, schrieb dennis luehring:

 Am 14.11.2013 09:53, schrieb Walter Bright:

 On 11/13/2013 11:37 PM, Jacob Carlborg wrote:

 On 2013-11-14 01:16, Walter Bright wrote:


 Yes. But that's a good thing. I'd be pretty skeptical of the value of an
 AST macro that took 3+4 and changed it so it did something other than
 compute 7.


 You can still do stupid things like that with operator overloading. Not on
 built-in types, but on user defined types. Every language allows you to do
 stupid things, one way or another.


 Sure, but the issue is that expression templates are not for "int+int", but for
 "T+int". My question is what value would there be in a rewrite of "int+int" to
 mean something different.


 "int + int" could be part of an PC<->GPU kombination that generates
 CUDA- and D-interchange code at compiletime


or like Don Blade Engine with string mixins

could produce "high performance x87 asm" code for normal D expressions



 Nov 14 2013



  parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/14/2013 1:08 AM, dennis luehring wrote:

 Am 14.11.2013 10:06, schrieb dennis luehring:

 Am 14.11.2013 09:53, schrieb Walter Bright:

 On 11/13/2013 11:37 PM, Jacob Carlborg wrote:

 On 2013-11-14 01:16, Walter Bright wrote:


 Yes. But that's a good thing. I'd be pretty skeptical of the value of an
 AST macro that took 3+4 and changed it so it did something other than
 compute 7.


 You can still do stupid things like that with operator overloading. Not on
 built-in types, but on user defined types. Every language allows you to do
 stupid things, one way or another.


 Sure, but the issue is that expression templates are not for "int+int", but for
 "T+int". My question is what value would there be in a rewrite of "int+int" to
 mean something different.


 "int + int" could be part of an PC<->GPU kombination that generates
 CUDA- and D-interchange code at compiletime


 or like Don Blade Engine with string mixins

 could produce "high performance x87 asm" code for normal D expressions


Good example, but still quite doable using type T instead.



 Nov 14 2013



prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Wednesday, 13 November 2013 at 08:45:13 UTC, Walter Bright 
wrote:

 On 11/13/2013 12:03 AM, Jacob Carlborg wrote:

 Again, operator overloading in D is too limiting to implement 
 something Linq like.


 Ok, let's set aside the opEquals and opCmp issue for the moment.

 Can AST macros do anything that expression templates cannot?


Yes, they can reflect the ast provided and act accordingly.



 Nov 13 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 1:15 AM, deadalnix wrote:

 On Wednesday, 13 November 2013 at 08:45:13 UTC, Walter Bright wrote:

 On 11/13/2013 12:03 AM, Jacob Carlborg wrote:

 Again, operator overloading in D is too limiting to implement something Linq
 like.


 Ok, let's set aside the opEquals and opCmp issue for the moment.

 Can AST macros do anything that expression templates cannot?


 Yes, they can reflect the ast provided and act accordingly.


The reflection ability is not something specific to an AST, it could be added
so 
that any expression node can be reflected. We already do much of that with
__traits.



 Nov 13 2013



 next sibling parent  "deadalnix" <deadalnix gmail.com>  writes:


On Wednesday, 13 November 2013 at 19:19:32 UTC, Walter Bright 
wrote:

 The reflection ability is not something specific to an AST, it 
 could be added so that any expression node can be reflected. We 
 already do much of that with __traits.


Yes, that is why Dicebot was mentioning the parallel with compile 
time reflection.



 Nov 13 2013



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 20:19, Walter Bright wrote:


 The reflection ability is not something specific to an AST, it could be
 added so that any expression node can be reflected. We already do much
 of that with __traits.


Yes, I been thinking of ways to expand __traits. When there's enough of 
__traits it's not much left during it into real AST macros. Just the 
part that passes the AST instead of the evaluated result of the 
arguments to a function.

BTW, is it complicated to return a more structured values, like a struct 
or a class, than just plain values like strings?

-- 
/Jacob Carlborg



 Nov 13 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 11:53 PM, Jacob Carlborg wrote:

 On 2013-11-13 20:19, Walter Bright wrote:


 The reflection ability is not something specific to an AST, it could be
 added so that any expression node can be reflected. We already do much
 of that with __traits.


 Yes, I been thinking of ways to expand __traits. When there's enough of
__traits
 it's not much left during it into real AST macros. Just the part that passes
the
 AST instead of the evaluated result of the arguments to a function.

 BTW, is it complicated to return a more structured values, like a struct or a
 class, than just plain values like strings?



No, but then you're face with the issue of where those structs are defined.



 Nov 14 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-14 10:13, Walter Bright wrote:


 No, but then you're face with the issue of where those structs are defined.


In druntime, just like MouldeInfo, TypeInfo and similar classes.

-- 
/Jacob Carlborg



 Nov 14 2013



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 09:45, Walter Bright wrote:


 Ok, let's set aside the opEquals and opCmp issue for the moment.

 Can AST macros do anything that expression templates cannot?


I need to read up on what expression templates can do.

-- 
/Jacob Carlborg



 Nov 13 2013



  parent reply "Rob T" <alanb ucora.com>  writes:


On Wednesday, 13 November 2013 at 12:21:22 UTC, Jacob Carlborg 
wrote:


 I need to read up on what expression templates can do.


But where to read? There's no such thing as "expression template" 
explicitly mentioned in here.

http://dlang.org/template.html

--rt



 Nov 13 2013



 next sibling parent  Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 3:27 PM, Rob T wrote:

 But where to read? There's no such thing as "expression template" explicitly
 mentioned in here.



This may help:

http://www.dsource.org/projects/ddl/browser/trunk/meta/regex.d



 Nov 13 2013



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-14 00:27, Rob T wrote:

 On Wednesday, 13 November 2013 at 12:21:22 UTC, Jacob Carlborg wrote:


 I need to read up on what expression templates can do.


 But where to read? There's no such thing as "expression template"
 explicitly mentioned in here.

 http://dlang.org/template.html


I guess everything about C++ expression templates would be a start.

http://en.wikipedia.org/wiki/Expression_templates

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling  parent reply "Don" <x nospam.com>  writes:


On Tuesday, 12 November 2013 at 09:55:20 UTC, Walter Bright wrote:

 I forgot to mention that "expression templates" can be used in 
 D in an equivalent manner that they are used in C++,


They are crippled compared to C++, because you have no control 
over the return type of opCmp and opEquals, which means that you 
can't have expression templates involving comparisons. That's 
extremely limiting.


My feeling with AST macros is:

(1) AST macros are basically syntax sugar for string mixins. The 
one and only thing that string mixins have in their favour, is 
that they have almost no syntax. But that is actually a *huge* 
plus.

Because it's syntax sugar, an AST macro syntax would be most 
convincing if you took some existing string mixins from various 
real projects, and show how beautiful they would become with 
macros.


(2) The big functionality we don't have, is comprehensive 
reflection. I'd like to see more thought in that area. It seems 
challenging to define reflection in a way that doesn't expose 
compiler internals.

The question I find particularly interesting is, "if we had 
macros, would reflection look different?"



 Nov 12 2013



 next sibling parent reply "Rob T" <alanb ucora.com>  writes:


On Tuesday, 12 November 2013 at 11:06:17 UTC, Don wrote:

 On Tuesday, 12 November 2013 at 09:55:20 UTC, Walter Bright 
 wrote:

 I forgot to mention that "expression templates" can be used in 
 D in an equivalent manner that they are used in C++,


 They are crippled compared to C++, because you have no control 
 over the return type of opCmp and opEquals, which means that 
 you can't have expression templates involving comparisons. 
 That's extremely limiting.


 My feeling with AST macros is:

 (1) AST macros are basically syntax sugar for string mixins. 
 The one and only thing that string mixins have in their favour, 
 is that they have almost no syntax. But that is actually a 
 *huge* plus.

 Because it's syntax sugar, an AST macro syntax would be most 
 convincing if you took some existing string mixins from various 
 real projects, and show how beautiful they would become with 
 macros.


 (2) The big functionality we don't have, is comprehensive 
 reflection. I'd like to see more thought in that area. It seems 
 challenging to define reflection in a way that doesn't expose 
 compiler internals.

 The question I find particularly interesting is, "if we had 
 macros, would reflection look different?"


My personal main need for macros at this point, is to make up for 
a lack of better reflection, but as has been pointed out, it's 
not clear how macros would help anyway without access to the sort 
of reflection that I'm currently lacking.

What we seem to be discussing, is the need to unify within D the 
abilities to perform better reflection along with a means to 
automate code generation during compile time. The current way 
reflection and mixins are done, are totally different, almost 
like two different languages with D, so it would be nice to have 
a more unified system that more closely resembles D itself.

BTW, I agree with Walters concerns about AST macros. It would be 
ugly to work with code that is not comprehensible due to over use 
of macros that attempt to redefine the language into something 
else. OTOH there's also significant advantages to be able to add 
certain things into the language that are currently lacking. For 
example, I've experimented with a method of implementing 
co-routines that use switch statements, but it's very ugly to do 
without macro support to the point of being impractical.

--rt



 Nov 12 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 19:01, Rob T wrote:


 My personal main need for macros at this point, is to make up for a lack
 of better reflection, but as has been pointed out, it's not clear how
 macros would help anyway without access to the sort of reflection that
 I'm currently lacking.


Macros would need a reflection API to be usable.


 What we seem to be discussing, is the need to unify within D the
 abilities to perform better reflection along with a means to automate
 code generation during compile time. The current way reflection and
 mixins are done, are totally different, almost like two different
 languages with D, so it would be nice to have a more unified system that
 more closely resembles D itself.

 BTW, I agree with Walters concerns about AST macros. It would be ugly to
 work with code that is not comprehensible due to over use of macros that
 attempt to redefine the language into something else. OTOH there's also
 significant advantages to be able to add certain things into the
 language that are currently lacking. For example, I've experimented with
 a method of implementing co-routines that use switch statements, but
 it's very ugly to do without macro support to the point of being
 impractical.

 --rt



-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling next sibling parent reply "Joseph Cassman" <jc7919 outlook.com>  writes:


On Tuesday, 12 November 2013 at 11:06:17 UTC, Don wrote:

 On Tuesday, 12 November 2013 at 09:55:20 UTC, Walter Bright 
 wrote:

 I forgot to mention that "expression templates" can be used in 
 D in an equivalent manner that they are used in C++,


 They are crippled compared to C++, because you have no control 
 over the return type of opCmp and opEquals, which means that 
 you can't have expression templates involving comparisons. 
 That's extremely limiting.


 My feeling with AST macros is:

 (1) AST macros are basically syntax sugar for string mixins. 
 The one and only thing that string mixins have in their favour, 
 is that they have almost no syntax. But that is actually a 
 *huge* plus.

 Because it's syntax sugar, an AST macro syntax would be most 
 convincing if you took some existing string mixins from various 
 real projects, and show how beautiful they would become with 
 macros.


 (2) The big functionality we don't have, is comprehensive 
 reflection. I'd like to see more thought in that area. It seems 
 challenging to define reflection in a way that doesn't expose 
 compiler internals.

 The question I find particularly interesting is, "if we had 
 macros, would reflection look different?"


After using string mixins for a while I have come to feel much 
like your statement in (1). I like the power and flexibility of 
string mixins. Text and strings are understandable and 
straightforward to manipulate. This is a clear win in my opinion.

There are also some things I find difficult at the moment.

  - One is the process of what to do when things do not go as 
expected. To debug I currently use pragma statements and try to 
visualize what is going on in my mind. This is do-able. But I 
feel it could be better.

  - It would be nice to be able to use some more D language 
concepts to construct the string mixin inside the template. For 
example, sometimes I would like to use a loop to construct the 
result.

If these two items could be addressed I would be pretty satisfied 
with using string mixins as they are.

One win that the AST macro concept provides that string mixins 
does not is manipulation of code syntax trees (either at compile 
or run time). It would be nice to combine the various code 
manipulation ideas that have been tossed around before into a 
single module, say std.meta or std.reflection. In addition to 
combining the various current code manipulation functionality 
into a single api (e.g. __traits, std.traits), it could provide 
functionality to work with code-in-use. For example, it would be 
pretty cool to be able to do the following (the actual syntax for 
constructing the code is not that important to me, but the 
underlying functionality for manipulating code is).

sample.d

     import std.meta;
     void main(string[] args) {
         auto e = construct("a + %s", args[1]);
         writefln("%s", e(5));
         auto f = construct(e + "b");
         writefln("%s", f(?, 10).toDebugString);
         writefln("%s", f(5, 10));
     }

-----


 ./sample 2
 7
 Numerical Expression: a + 2 + 10
 17


Implementing the "expression tree" side of AST macros seems like 
it could be done as a library, which would be a win for 
maintainability and cross-compiler portability. There is probably 
required, compiler-related functionality that I cannot see making 
this unfeasable or possibly difficult to do. Still, looks good to 
me on paper at least. Perhaps going this route could gain the 
good functionality of AST macros while also making use of the 
already implemented features?

Seems to me like some of these ideas tie into previous 
discussions. Here are two I remember. I am sure there are others.

http://forum.dlang.org/thread/juf7sk$16rl$1 digitalmars.com
http://forum.dlang.org/thread/mailman.1716.1340388570.24740.digitalmars-d puremagic.com

Joseph



 Nov 12 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 21:03, Joseph Cassman wrote:


 After using string mixins for a while I have come to feel much like your
 statement in (1). I like the power and flexibility of string mixins.
 Text and strings are understandable and straightforward to manipulate.
 This is a clear win in my opinion.

 There are also some things I find difficult at the moment.

   - One is the process of what to do when things do not go as expected.
 To debug I currently use pragma statements and try to visualize what is
 going on in my mind. This is do-able. But I feel it could be better.


If we got macros we should really have a good way to debug them. At 
least a flag to shows how the macros are lowered.


   - It would be nice to be able to use some more D language concepts to
 construct the string mixin inside the template. For example, sometimes I
 would like to use a loop to construct the result.

 If these two items could be addressed I would be pretty satisfied with
 using string mixins as they are.

 One win that the AST macro concept provides that string mixins does not
 is manipulation of code syntax trees (either at compile or run time). It
 would be nice to combine the various code manipulation ideas that have
 been tossed around before into a single module, say std.meta or
 std.reflection. In addition to combining the various current code
 manipulation functionality into a single api (e.g. __traits,
 std.traits), it could provide functionality to work with code-in-use.
 For example, it would be pretty cool to be able to do the following (the
 actual syntax for constructing the code is not that important to me, but
 the underlying functionality for manipulating code is).

 sample.d

      import std.meta;
      void main(string[] args) {
          auto e = construct("a + %s", args[1]);
          writefln("%s", e(5));
          auto f = construct(e + "b");
          writefln("%s", f(?, 10).toDebugString);
          writefln("%s", f(5, 10));
      }


Yes, but the problem with string mixins is that you need to reparse a 
lot of code. With string mixins it becomes:

1. The compiler parses the code and creates an AST (fine so far)
2. Does type checking an other stuff
3. Library code parses a string into an AST
4. Library code manipulates the AST
5. Library code converts the AST back to a string
6. The string is interested with a string mixin at the calling context
7. The compiler parses the code and creates an AST
8. Does type checking an other stuff

With AST macaros you can avoid many of these steps:

1. The compiler parses the code and creates an AST
2. Does type checking an other stuff
3. Library code (macro) manipulates the AST
4. Compiler inserts the manipulate AST into the calling context
5. Does type checking an other stuff

Instead of eight steps we get five steps. We remove the unnecessary 
conversions between AST's and strings.

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/12/2013 3:06 AM, Don wrote:

 On Tuesday, 12 November 2013 at 09:55:20 UTC, Walter Bright wrote:

 I forgot to mention that "expression templates" can be used in D in an
 equivalent manner that they are used in C++,


 They are crippled compared to C++, because you have no control over the return
 type of opCmp and opEquals, which means that you can't have expression
templates
 involving comparisons. That's extremely limiting.


It is limiting, but I don't know about extremely limiting. Eric Anderton's
regex 
engine didn't need them. And, if you really do need them, you can always define 
the templates as regular names:

    lessThan(a,b)
    equals(a,b)

etc. Not the greatest, but not unworkable.



 Nov 12 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 00:26, Walter Bright wrote:


 It is limiting, but I don't know about extremely limiting. Eric
 Anderton's regex engine didn't need them. And, if you really do need
 them, you can always define the templates as regular names:

     lessThan(a,b)
     equals(a,b)

 etc. Not the greatest, but not unworkable.


I don't know what's so special about the regex engine. It takes a regex 
as a string and matches that against another string. Seems to be just 
like any other regular expression library I've used.

-- 
/Jacob Carlborg



 Nov 13 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 12:17 AM, Jacob Carlborg wrote:

 On 2013-11-13 00:26, Walter Bright wrote:


 It is limiting, but I don't know about extremely limiting. Eric
 Anderton's regex engine didn't need them. And, if you really do need
 them, you can always define the templates as regular names:

     lessThan(a,b)
     equals(a,b)

 etc. Not the greatest, but not unworkable.


 I don't know what's so special about the regex engine. It takes a regex as a
 string and matches that against another string. Seems to be just like any other
 regular expression library I've used.


What's special about it was its use of expression templates. It made for a nice 
demo of how to do such.



 Nov 13 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 09:38, Walter Bright wrote:


 What's special about it was its use of expression templates. It made for
 a nice demo of how to do such.


Is there an actual point in doing that, except a nice exercise in 
expression templates?

-- 
/Jacob Carlborg



 Nov 13 2013



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 11/13/2013 4:25 AM, Jacob Carlborg wrote:

 On 2013-11-13 09:38, Walter Bright wrote:


 What's special about it was its use of expression templates. It made for
 a nice demo of how to do such.


 Is there an actual point in doing that, except a nice exercise in expression
 templates?


It actually generated a very fast regex engine, though Dmitry's work has since 
eclipsed it.



 Nov 13 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 20:16, Walter Bright wrote:


 It actually generated a very fast regex engine, though Dmitry's work has
 since eclipsed it.


Right, a specially tuned engine for that particular regex.

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling next sibling parent  "Wyatt" <wyatt.epp gmail.com>  writes:


On Tuesday, 12 November 2013 at 09:20:07 UTC, Walter Bright wrote:

 5. We've said "no" in the past to things like user-defined 
 tokens, which are closely related.


Regarding this point, I get that arbitrary expansion is out, but 
would you be amenable to adding a small, _strictly-defined_ set 
of operators that are compile-time errors unless overloaded?


     #define BEGIN {
     #define END }

 macros used in old C code to make it look like Pascal.


A plague o' their houses!

-Wyatt



 Nov 12 2013



prev sibling next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 10:20, Walter Bright wrote:


 For example, the Linq example in the DIP is not compelling, as
 aesthetically nicer code can be written using D's ranges and algorithms:

      auto data = arr.filter!(x => x > 5).array;


I was actually not the one that added the Linq examples. I like the 
above better. I've used a similar example as well:

auto person = Person.where(e => e.name == "John");

Translates to:

select * from person where name = 'John'

But I don't see how that can currently be done in D. Operator 
overloading in D isn't flexible enough. For example, when implementing 
opEquals, you don't know if it's "==" or "!=" that's being called. Same 
with the comparison operators.



 be blunt it's like the old:

      #define BEGIN {
      #define END }

 macros used in old C code to make it look like Pascal.


I agree with this.

-- 
/Jacob Carlborg



 Nov 12 2013



  parent reply "John Colvin" <john.loughran.colvin gmail.com>  writes:


On Tuesday, 12 November 2013 at 13:50:49 UTC, Jacob Carlborg 
wrote:

 auto person = Person.where(e => e.name == "John");

 Translates to:

 select * from person where name = 'John'


for those of us entirely unfamiliar with linq, what is this 
supposed to do? Select people with name "John" from a collection 
of people, like in sql? It seems trivial to do this using filter, 
or am I missing something...?



 Nov 12 2013



 next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 12 November 2013 at 14:38:44 UTC, John Colvin wrote:

 for those of us entirely unfamiliar with linq, what is this 
 supposed to do? Select people with name "John" from a 
 collection of people, like in sql? It seems trivial to do this 
 using filter, or am I missing something...?


I don't know much about linq but in macro context I'd expect it 
to generate an actual SQL query to database (and execute it) 
based on in-language filter statement (via reflection)



 Nov 12 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 16:20, Dicebot wrote:


 I don't know much about linq but in macro context I'd expect it to
 generate an actual SQL query to database (and execute it) based on
 in-language filter statement (via reflection)


Exactly, spot on. The interesting thing there, in the context of macros, 
is converting the AST of the lambda to an string containing SQL code.

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu>  writes:


On 11/12/2013 06:38 AM, John Colvin wrote:

 On Tuesday, 12 November 2013 at 13:50:49 UTC, Jacob Carlborg wrote:

 auto person = Person.where(e => e.name == "John");

 Translates to:

 select * from person where name = 'John'


 for those of us entirely unfamiliar with linq, what is this supposed to
 do? Select people with name "John" from a collection of people, like in
 sql? It seems trivial to do this using filter, or am I missing
 something...?


linq provides an interface to query any collection, but what is 
interesting in this case is

Person.where(e => e.name == "John")

invokes an engine that builds the necessary sql to issue an equivalent 
query to your sql database and assembles the results into a range of 
Person. And it can do this for any arbitrary predicate (well, almost. It 
doesn't handle function calls to arbitrary code too well).

the .NET framework can do this because it exposes an api for querying, 
building, and compiling asts.

D cannot do this because it doesn't. (and I have tried to make it work)



 Nov 12 2013



 next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com>  writes:


On Tuesday, 12 November 2013 at 15:21:16 UTC, Ellery Newcomer 
wrote:

 On 11/12/2013 06:38 AM, John Colvin wrote:

 On Tuesday, 12 November 2013 at 13:50:49 UTC, Jacob Carlborg 
 wrote:

 auto person = Person.where(e => e.name == "John");

 Translates to:

 select * from person where name = 'John'


 for those of us entirely unfamiliar with linq, what is this 
 supposed to
 do? Select people with name "John" from a collection of 
 people, like in
 sql? It seems trivial to do this using filter, or am I missing
 something...?


 linq provides an interface to query any collection, but what is 
 interesting in this case is

 Person.where(e => e.name == "John")

 invokes an engine that builds the necessary sql to issue an 
 equivalent query to your sql database and assembles the results 
 into a range of Person. And it can do this for any arbitrary 
 predicate (well, almost. It doesn't handle function calls to 
 arbitrary code too well).

 the .NET framework can do this because it exposes an api for 
 querying, building, and compiling asts.

 D cannot do this because it doesn't. (and I have tried to make 
 it work)


oh, I see. Would AST macros really be enough to make this work in 
D? "Arbitrary code" is a huge feature space in D, including much 
that doesn't map well to anything outside of a relatively 
low-level language, let alone SQL.
I can see it quickly becoming a nightmare that would be worse 
than just issuing the predicate as an sql string or some generic 
equivalent.



 Nov 12 2013



 next sibling parent reply "Ellery Newcomer" <ellery-newcomer utulsa.edu>  writes:


On Tuesday, 12 November 2013 at 16:14:57 UTC, John Colvin wrote:

 On Tuesday, 12 November 2013 at 15:21:16 UTC, Ellery Newcomer 
 wrote:

 On 11/12/2013 06:38 AM, John Colvin wrote:

 On Tuesday, 12 November 2013 at 13:50:49 UTC, Jacob Carlborg 
 wrote:

 auto person = Person.where(e => e.name == "John");

 Translates to:

 select * from person where name = 'John'


 for those of us entirely unfamiliar with linq, what is this 
 supposed to
 do? Select people with name "John" from a collection of 
 people, like in
 sql? It seems trivial to do this using filter, or am I missing
 something...?


 linq provides an interface to query any collection, but what 
 is interesting in this case is

 Person.where(e => e.name == "John")

 invokes an engine that builds the necessary sql to issue an 
 equivalent query to your sql database and assembles the 
 results into a range of Person. And it can do this for any 
 arbitrary predicate (well, almost. It doesn't handle function 
 calls to arbitrary code too well).

 the .NET framework can do this because it exposes an api for 
 querying, building, and compiling asts.

 D cannot do this because it doesn't. (and I have tried to make 
 it work)


 oh, I see. Would AST macros really be enough to make this work 
 in D? "Arbitrary code" is a huge feature space in D, including 
 much that doesn't map well to anything outside of a relatively 
 low-level language, let alone SQL.
 I can see it quickly becoming a nightmare that would be worse 
 than just issuing the predicate as an sql string or some 
 generic equivalent.


Actually,

s/arbitrary code/simple to moderately complex expressions that 
don't call functions that aren't explicitly handled by the 
translation engine/

and this is a better representation of what linq can do in e.g. 
entity framework.

and you could go the route of django models, e.g.

Place.objects.filter(name="Bob's Cafe")

as long as you had enough metadata set up. you could probably get 
this to work in D, but

A. it looks like crap
B. it in no way resembles sql
C. it would look like even worse crap in D (no named parameters!)
D. it doesn't readily permit you to make queries with complex 
expression trees (idk, maybe something like linq's

Places.Where(p => p.Elev < 200 && Addresses.Any(a => a.State == 
p.State))

why am I arguing against this? nobody wants this, nobody asked 
for this.

Also, sql is to databases what assembly is to cpus. It's low 
level, [moderately] untyped, and it differs on every single 
platform. provide access to it, sure, but please don't use it to 
build your abstractions.



 Nov 12 2013



  parent reply =?ISO-8859-15?Q?Simen_Kj=E6r=E5s?= <simen.kjaras gmail.com>  writes:


On 12.11.2013 18:53, Ellery Newcomer wrote:



 Place.objects.filter(name="Bob's Cafe")


[snip]

 C. it would look like even worse crap in D (no named parameters!)


It's perfectly possible in D to make this work:

     Place.objects.filter(args.name = "Bob", args.state = "unemployed");


Proof of concept:

import std.stdio : writeln;

struct NamedArg(string name, Args...) {
     Args value;
     alias value this;
}

struct args {
      property
     static auto opDispatch(string name, Args...)(Args args) {
         return NamedArg!(name, Args)(args);
     }
}

void foo(NamedArg!("name", string) a) {
     writeln(a);
}

void main() {
     foo(args.name = "Bob");
}

And since the name is statically known, you can use this for e.g. lookup 
on objects.

No, it's nowhere near as pretty, and the receiving code is horrible. If 
you were to use this for some kind of magic behind the scenes, it might 
be okay. Might. Please don't do it.



 D. it doesn't readily permit you to make queries with complex expression
 trees (idk, maybe something like linq's

 Places.Where(p => p.Elev < 200 && Addresses.Any(a => a.State == p.State))


Sorry, I got nuffin'. :p


Waaait...

   Places.where(args.elev.lessThan(200) & args.state =
         (state)=>Addresses.any(state));

I *can* make that work. I'm not going to.

--
   Simen



 Nov 13 2013



  parent reply "Ellery Newcomer" <ellery-newcomer utulsa.edu>  writes:


On Wednesday, 13 November 2013 at 10:51:48 UTC, Simen Kjærås 
wrote:

 On 12.11.2013 18:53, Ellery Newcomer wrote:

 It's perfectly possible in D to make this work:


hey, cool impl

*comprehends code*

I mean Ewww


 I *can* make that work. I'm not going to.

 --
   Simen


I concur with the second part.



 Nov 13 2013



 next sibling parent reply =?UTF-8?B?U2ltZW4gS2rDpnLDpXM=?= <simen.kjaras gmail.com>  writes:


On 13.11.2013 20:05, Ellery Newcomer wrote:

 On Wednesday, 13 November 2013 at 10:51:48 UTC, Simen Kjærås wrote:

 On 12.11.2013 18:53, Ellery Newcomer wrote:

 It's perfectly possible in D to make this work:


 hey, cool impl

 *comprehends code*

 I mean Ewww


 I *can* make that work. I'm not going to.

 --
   Simen


 I concur with the second part.


I decided to abandon sanity. Luckily, only for the named parameters. I 
now have this code working:


void test(int a, int b = 2, string c = "foo", int d = 14) {
}

alias test2 = nameify!test;

void main() {
     test2(1, Args.d = 4, Args.c = "bar");
}

With reasonable error messages for duplicate and missing parameters, as 
well as type mismatches.

Source is attached. I hope God forgives me.

-- 
   Simen



 Nov 17 2013



 next sibling parent  "deadalnix" <deadalnix gmail.com>  writes:


On Sunday, 17 November 2013 at 19:41:40 UTC, Simen Kjærås wrote:

 On 13.11.2013 20:05, Ellery Newcomer wrote:

 On Wednesday, 13 November 2013 at 10:51:48 UTC, Simen Kjærås 
 wrote:

 On 12.11.2013 18:53, Ellery Newcomer wrote:

 It's perfectly possible in D to make this work:


 hey, cool impl

 *comprehends code*

 I mean Ewww


 I *can* make that work. I'm not going to.

 --
   Simen


 I concur with the second part.


 I decided to abandon sanity. Luckily, only for the named 
 parameters. I
 now have this code working:


 void test(int a, int b = 2, string c = "foo", int d = 14) {
 }

 alias test2 = nameify!test;

 void main() {
      test2(1, Args.d = 4, Args.c = "bar");
 }

 With reasonable error messages for duplicate and missing 
 parameters, as
 well as type mismatches.

 Source is attached. I hope God forgives me.


That is a really cool idea !



 Nov 17 2013



prev sibling next sibling parent reply Ellery Newcomer <ellery-newcomer utulsa.edu>  writes:


On 11/17/2013 11:41 AM, Simen Kjærås wrote:

 Source is attached. I hope God forgives me.


I suppose you'd have to do something like

x.Where(OR( NOT(Args.thing = "thing"), Args.sing = "sing"))

for nesting and negation and all that.

I'd wait for walter to relax the restrictions on ==, &&, etc operator 
overloading. Then you probably could do

x.Where(X => X.thing != "thing" || X.sing == "sing")

let's see here. If X's type is QueryExp!Entity and the return type is 
SomeRange!Entity then querying looks doable.

how about composing expressions? take the result of that lambda EXP,
EXP = EXP && X.cling == 1; // should work fine?

how about decomposing expressions?
EXP = EXP.lhs; // should work fine?

EXP.rhs.rhs.value // returns "sing" - should work fine? the lambda 
should create a closure, so we should be able to get out any value we put in

you won't be able to do

x.Where(X => upper(X.thing) == "THING")

without ast macros but will have to be satisfied with

x.Where(X => X.thing.upper() == "THING")

will you be able to do

x.Where(X => X.thing == "THING" && x.Any(X2 => X2.thing == X.thing && 
X2.id != X.id))

?

either "Any" would have to be only for template expressions, or X would 
have to be an outer context like so:

x.Where(X => X.it.thing == "THING" && X.x.Any(X2 => X2.it.thing == 
X.it.thing && X2.it.id != X.it.id))

because x.Any(...) should return bool, but inside the query expression 
it should return QueryExp. In both cases it would take a param of

QueryExp delegate(QueryExp)

Anybody else have any other ideas?



 Nov 17 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-17 22:15, Ellery Newcomer wrote:

 On 11/17/2013 11:41 AM, Simen Kjærås wrote:

 Source is attached. I hope God forgives me.


 I suppose you'd have to do something like

 x.Where(OR( NOT(Args.thing = "thing"), Args.sing = "sing"))

 for nesting and negation and all that.

 I'd wait for walter to relax the restrictions on ==, &&, etc operator
 overloading. Then you probably could do

 x.Where(X => X.thing != "thing" || X.sing == "sing")

 let's see here. If X's type is QueryExp!Entity and the return type is
 SomeRange!Entity then querying looks doable.

 how about composing expressions? take the result of that lambda EXP,
 EXP = EXP && X.cling == 1; // should work fine?

 how about decomposing expressions?
 EXP = EXP.lhs; // should work fine?

 EXP.rhs.rhs.value // returns "sing" - should work fine? the lambda
 should create a closure, so we should be able to get out any value we
 put in

 you won't be able to do

 x.Where(X => upper(X.thing) == "THING")


That works with the Rails plugin Squeel. Although you need to prefix 
"upper" with "X.".

https://github.com/ernie/squeel

-- 
/Jacob Carlborg



 Nov 18 2013



prev sibling  parent reply Timon Gehr <timon.gehr gmx.ch>  writes:


On 11/17/2013 08:41 PM, Simen Kjærås wrote:

 I decided to abandon sanity. Luckily, only for the named parameters. I
 now have this code working:


 void test(int a, int b = 2, string c = "foo", int d = 14) {
 }

 alias test2 = nameify!test;

 void main() {
      test2(1, Args.d = 4, Args.c = "bar");
 }

 With reasonable error messages for duplicate and missing parameters, as
 well as type mismatches.
 ...


Still missing overload resolution. :o)



 Nov 17 2013



  parent  =?UTF-8?B?U2ltZW4gS2rDpnLDpXM=?= <simen.kjaras gmail.com>  writes:


On 17.11.2013 23:21, Timon Gehr wrote:

 On 11/17/2013 08:41 PM, Simen Kjærås wrote:

 I decided to abandon sanity. Luckily, only for the named parameters. I
 now have this code working:


 void test(int a, int b = 2, string c = "foo", int d = 14) {
 }

 alias test2 = nameify!test;

 void main() {
      test2(1, Args.d = 4, Args.c = "bar");
 }

 With reasonable error messages for duplicate and missing parameters, as
 well as type mismatches.
 ...


 Still missing overload resolution. :o)


Please don't tempt me. :p

-- 
   Simen



 Nov 17 2013



prev sibling  parent reply Philippe Sigaud <philippe.sigaud gmail.com>  writes:


On Sun, Nov 17, 2013 at 8:41 PM, Simen Kjærås <simen.kjaras gmail.com> wrote:


 Source is attached. I hope God forgives me.


Nice ideas. I particularly like this one:

final abstract class Args {
     property
    static auto opDispatch(string name, Arg)(Arg arg) {
        return NamedArg!(name, Arg)(arg);
    }
}

Which allows you to have Args.c = 1 (and thus storing the name and the value).

Half of your implementation is in fact generic enough to be in Phobos
(staticFilter, staticZip). I know I use my own...

Btw, with DMD 2.064.2, we can now write:

template staticToString(T...) {
    enum staticToString = T.stringof;
}

as:

alias staticToString(T...) = T.stringof;

Which is both easier to understand and easier on the eyes.



 Nov 17 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-17 21:04, Philippe Sigaud wrote:


 alias staticToString(T...) = T.stringof;

 Which is both easier to understand and easier on the eyes.


Wouldn't that be enum instead:

enum staticToString(T...) = T.stringof;

-- 
/Jacob Carlborg



 Nov 18 2013



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 17:14, John Colvin wrote:


 oh, I see. Would AST macros really be enough to make this work in D?
 "Arbitrary code" is a huge feature space in D, including much that
 doesn't map well to anything outside of a relatively low-level language,
 let alone SQL.
 I can see it quickly becoming a nightmare that would be worse than just
 issuing the predicate as an sql string or some generic equivalent.


Person.where(e => e.name == "John")

I'm thinking that we only need to convert the part that is prefixed 
with, in this example, "e". Any other code should be executed in the 
context of the caller. It should be possible to do this as well:

auto foo = "John";
auto result = Person.where(e => e.name == foo);

Which will result in the same SQL query.

I'm using a pluign to Ruby on Rails that does something similar but by 
overloading operators. The problem with this approach, in Ruby, is that 
you cannot overload operators like || and &&, so instead they overload | 
and & resulting in new problems like operator precedence. Example:

Person.where{ |e| (e.name == "John") & (e.address == "Main street") }

-- 
/Jacob Carlborg



 Nov 13 2013



 next sibling parent reply luka8088 <luka8088 owave.net>  writes:


On 13.11.2013. 9:26, Jacob Carlborg wrote:

 On 2013-11-12 17:14, John Colvin wrote:
 

 oh, I see. Would AST macros really be enough to make this work in D?
 "Arbitrary code" is a huge feature space in D, including much that
 doesn't map well to anything outside of a relatively low-level language,
 let alone SQL.
 I can see it quickly becoming a nightmare that would be worse than just
 issuing the predicate as an sql string or some generic equivalent.


 
 Person.where(e => e.name == "John")
 
 I'm thinking that we only need to convert the part that is prefixed
 with, in this example, "e". Any other code should be executed in the
 context of the caller. It should be possible to do this as well:
 
 auto foo = "John";
 auto result = Person.where(e => e.name == foo);
 
 Which will result in the same SQL query.
 
 I'm using a pluign to Ruby on Rails that does something similar but by
 overloading operators. The problem with this approach, in Ruby, is that
 you cannot overload operators like || and &&, so instead they overload |
 and & resulting in new problems like operator precedence. Example:
 
 Person.where{ |e| (e.name == "John") & (e.address == "Main street") }
 


What about something like this?

class Person {

  macro where (Context context, Statement statement) {
    // ...
  }

}

auto foo = "John";
auto result = Person.where(e => e.name == foo);

// is replaced by
auto foo = "John";
auto result = Person.query("select * from person where person.name = " ~
sqlQuote(foo) ~ ";");



 Nov 13 2013



 next sibling parent  dennis luehring <dl.soluz gmx.net>  writes:


Am 13.11.2013 09:34, schrieb luka8088:

 What about something like this?

 class Person {

    macro where (Context context, Statement statement) {
      // ...
    }


it is not generic - and that is the biggest goal to reach



 Nov 13 2013



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 09:34, luka8088 wrote:


 What about something like this?

 class Person {

    macro where (Context context, Statement statement) {
      // ...
    }

 }

 auto foo = "John";
 auto result = Person.where(e => e.name == foo);

 // is replaced by
 auto foo = "John";
 auto result = Person.query("select * from person where person.name = " ~
 sqlQuote(foo) ~ ";");


That's basically what would happen.

-- 
/Jacob Carlborg



 Nov 13 2013



  parent reply luka8088 <luka8088 owave.net>  writes:


On 13.11.2013. 13:26, Jacob Carlborg wrote:

 On 2013-11-13 09:34, luka8088 wrote:
 

 What about something like this?

 class Person {

    macro where (Context context, Statement statement) {
      // ...
    }

 }

 auto foo = "John";
 auto result = Person.where(e => e.name == foo);

 // is replaced by
 auto foo = "John";
 auto result = Person.query("select * from person where person.name = " ~
 sqlQuote(foo) ~ ";");


 
 That's basically what would happen.
 


May/Should I add such example to wiki?

--
Luka



 Nov 13 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 15:47, luka8088 wrote:


 May/Should I add such example to wiki?


Yes, "should". Sure, add it to the DIP.

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling  parent reply "Chris Nicholson-Sauls" <ibisbasenji gmail.com>  writes:


On Wednesday, 13 November 2013 at 08:26:52 UTC, Jacob Carlborg 
wrote:

 I'm using a pluign to Ruby on Rails that does something similar 
 but by overloading operators. The problem with this approach, 
 in Ruby, is that you cannot overload operators like || and &&, 
 so instead they overload | and & resulting in new problems like 
 operator precedence. Example:

 Person.where{ |e| (e.name == "John") & (e.address == "Main 
 street") }


Is this in any way better than the basic Ruby

   where( name: 'John', address: 'Main street' )

?  Or was it just something quick and contrived to show the 
behavior, and not the utility, of the plugin?  Just curious.



 Nov 13 2013



  parent  "Jacob Carlborg" <doob me.com>  writes:


On Wednesday, 13 November 2013 at 17:13:11 UTC, Chris 
Nicholson-Sauls wrote:


 Is this in any way better than the basic Ruby

   where( name: 'John', address: 'Main street' )

 ?  Or was it just something quick and contrived to show the 
 behavior, and not the utility, of the plugin?  Just curious.


It was mostly to show what happens when the you have limited 
operator overloading.

It has it advantages over the hash-syntax. You can do much more, 
not equal, less then, greater then, like and so on.

--
/Jacob Carlborg



 Nov 13 2013



prev sibling  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/12/13 7:21 AM, Ellery Newcomer wrote:

 the .NET framework can do this because it exposes an api for querying,
 building, and compiling asts.

 D cannot do this because it doesn't. (and I have tried to make it work)


Maybe the problem needs to be reformulated for D. I think an SQL mixin 
that either stays unchanged (for DB engines) or translates to a D 
expression (for native D data types) would be doable, nontrivial, 
interesting, and instantly usable for people who already know SQL 
without any extra learning. In other words... actually better than Linq.

Andrei



 Nov 12 2013



 next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 12 November 2013 at 16:39:18 UTC, Andrei Alexandrescu 
wrote:

 Maybe the problem needs to be reformulated for D. I think an 
 SQL mixin that either stays unchanged (for DB engines) or 
 translates to a D expression (for native D data types) would be 
 doable, nontrivial, interesting, and instantly usable for 
 people who already know SQL without any extra learning. In 
 other words... actually better than Linq.


It does not matter that much if it is macro or mixin. Key feature 
is AST reflection which allows to augment valid D code with 
additional semantics when appropriate. It is very similar to how 
UDA's augment declarative approach, but applied to 
imperative/procedural instead. We can't do that in D right now. 
Only way is to import own source file and parse it which does not 
really sound sane.



 Nov 12 2013



  parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/12/13 8:47 AM, Dicebot wrote:

 On Tuesday, 12 November 2013 at 16:39:18 UTC, Andrei Alexandrescu wrote:

 Maybe the problem needs to be reformulated for D. I think an SQL mixin
 that either stays unchanged (for DB engines) or translates to a D
 expression (for native D data types) would be doable, nontrivial,
 interesting, and instantly usable for people who already know SQL
 without any extra learning. In other words... actually better than Linq.


 It does not matter that much if it is macro or mixin. Key feature is AST
 reflection which allows to augment valid D code with additional
 semantics when appropriate.


I don't see how AST reflection is needed for generating correct D code 
from an SQL expression. Dmitry did exactly that with ctRegex. Please 
illuminate.

Andrei



 Nov 12 2013



  parent reply "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 12 November 2013 at 16:54:19 UTC, Andrei Alexandrescu 
wrote:

 I don't see how AST reflection is needed for generating correct 
 D code from an SQL expression. Dmitry did exactly that with 
 ctRegex. Please illuminate.


Other way around. Generating arbitrary output (including but not 
limited to SQL expressions) from D code. As I have already 
mentioned, it is the very same thing we do now with __traits and 
UDA's but not limited to only declarations.



 Nov 12 2013



 next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/12/13 9:03 AM, Dicebot wrote:

 On Tuesday, 12 November 2013 at 16:54:19 UTC, Andrei Alexandrescu wrote:

 I don't see how AST reflection is needed for generating correct D code
 from an SQL expression. Dmitry did exactly that with ctRegex. Please
 illuminate.


 Other way around. Generating arbitrary output (including but not limited
 to SQL expressions) from D code.


Yah. That I agree with. That's why I said we should reformulate the 
problem that LINQ solves.


 As I have already mentioned, it is the
 very same thing we do now with __traits and UDA's but not limited to
 only declarations.


Interesting.


Andrei



 Nov 12 2013



 next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 12 November 2013 at 17:20:20 UTC, Andrei Alexandrescu 
wrote:

 Other way around. Generating arbitrary output (including but 
 not limited
 to SQL expressions) from D code.


 Yah. That I agree with. That's why I said we should reformulate 
 the problem that LINQ solves.


I had an impression it was exactly the context in which linq was 
originally mentioned, no idea why discussion has moved from that 
to expressing DSL's (which is not really a problem in D) :)


 As I have already mentioned, it is the
 very same thing we do now with __traits and UDA's but not 
 limited to
 only declarations.


 Interesting.


Reason why I am doing this comparison is that some of the cases 
why UDA's are so useful applies to AST reflection capabilities 
too. Most importantly, it allows to have the code which is both 
valid (and used) D code on its own and used to generate some 
co-product from it in automatic mode. Similar declarative 
approach has been used to huge success in vibe.d code (rest, 
forms) and I see no reasons why it can't be so for imperative one.



 Nov 12 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 21:25, Dicebot wrote:


 I had an impression it was exactly the context in which linq was
 originally mentioned, no idea why discussion has moved from that to
 expressing DSL's (which is not really a problem in D) :)


My example is a DSL:

Person.where(e => e.name == "John");

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling  parent reply "deadalnix" <deadalnix gmail.com>  writes:


On Tuesday, 12 November 2013 at 17:20:20 UTC, Andrei Alexandrescu 
wrote:

 As I have already mentioned, it is the
 very same thing we do now with __traits and UDA's but not 
 limited to
 only declarations.


 Interesting.


The thing I'd like to be able to do it to create a 
async/await/yield like mechanism, purely as library.

 generator
uint foo() {
     writeln("generating !");
     return 0;
     return 1;

     writeln("and the last one");
     return 2;
}

=>

auto foo() {
     enum States {
         S0,
         S1,
         S2,
     }

     struct Generator {
         State s;
         uint current;

          property front() {
             return current;
         }

         void popFront() {
             final switch(s) with(State) {
                 case S0:
                     current = 1;
                     s = S1;
                     return;
                 case S1:
                     writeln("and the last one");
                     current = 2;
                     s = S2;
                     return;
                 case S2:
                     assert(0, "Nothing more in here");
             }
         }

          property empty() {
             return s == State.S2;
         }
     }

     auto g = Generator();

     writeln("generating !");
     g.current = 0;
     g.state = State.S0;

     return g;
}



 Nov 12 2013



 next sibling parent reply Martin Nowak <code dawg.eu>  writes:


On 11/12/2013 09:36 PM, deadalnix wrote:

 The thing I'd like to be able to do it to create a async/await/yield
 like mechanism, purely as library.


That's a prime example for AST macros.



 Nov 12 2013



  parent  "Zsombor Barna" <belzurix sch.bme.hu>  writes:


On Tuesday, 12 November 2013 at 23:34:38 UTC, Martin Nowak wrote:

 On 11/12/2013 09:36 PM, deadalnix wrote:

 The thing I'd like to be able to do it to create a 
 async/await/yield
 like mechanism, purely as library.


 That's a prime example for AST macros.


First of all: I haven't used macros extensively yet. Its merely a 
theoretical viewpoint, so don't take the arguments 
well-established.

Macros could be used as in ScalaCL ( I mean the latest version, 
which is experimental : https://github.com/ochafik/ScalaCL ). I 
like the idea using the host language ( or a subset of it) to 
program GPUs, not C, like PyOpenCL does. It reduces the 
complexity of the problem when you switch target of the algorithm 
from CPU to GPU and back.

Macros could also be used in (re)implementing some of the basic 
syntax - that's what developers of Nemerle did. Somebody can 
argue whether it was a good idea or not, but it would be 
interesting to disable syntax by disabling the macro that 
transforms it by a single line in the file ( not importing the 
macro ). ( Could be useful for microcontroller platforms to 
control the runtime's memory usage by disabling certain features 
- not compiling them into the executable )

In AOT compilation, there isn't any difference between macros and 
CTFE functions which do something with AST. That would be a way 
to implement them - creating a compiler API which could only be 
used by compile-time functions.

Yeah, and I know, it's crippling when code is ambigous. But I 
also think macros are the best way to define interfaces for 
modules. Using them can obfuscate code very easily when not 
enough care is taken handling them and it can make the learning 
curve much steeper, but there always existed ill-designed API-s 
and we could still use the good old tools to get the API done :) 
. Maybe it wouldn't be as good as a macro-ed API would be, but 
still works. If we choose the right tool, it can lower the 
complexity of the API.

Let's have a look at the feature: pointers. A very powerful, be 
also dangerous weapon that cal blow up the entire program 
suddenly. With some techniques used to fend off the problems it 
can cause, it can be much safer. Rust made a trial with their 
pointer system. I don't think it is a solution, but ignoring the 
problem isn't either.

The second example: the first cannons blew up, killing people. 
But time showed technology can evolve.
The area of macros - in my opinion - is not explored yet. Only 
the use of them can reveal the ways to make them better. 
Theoretical scientists cannot create good solutions for everyday 
use in the first place. It's a trial-and-error process.

By allowing usage of macros only in certain cases, language 
desginer can make the burden of the coders lighter ( although I 
like statement macros, they can be very ambigous ). Also, if 
designer make macros visually very distinct from other tokens, 
users can read the code even easier, but sense is needed when 

alien from D, in my opinion ).

Personally I wouldn't like hacking on the compiler or creating a 
whole new parser to do the same as we could do with macros, and I 
suggest a trial.
Another option is to  wait until somebody comes up with an idea 
and when time proved if the solution is right, include it - doing 
this saves a lot of work. That's how cutting-edge technologies 
work ( even if AST macros are an old idea ).



 Nov 12 2013



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 21:36, deadalnix wrote:


 The thing I'd like to be able to do it to create a async/await/yield
 like mechanism, purely as library.


Please add this as an example to the DIP.

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling  parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


12-Nov-2013 21:03, Dicebot пишет:

 On Tuesday, 12 November 2013 at 16:54:19 UTC, Andrei Alexandrescu wrote:

 I don't see how AST reflection is needed for generating correct D code
 from an SQL expression. Dmitry did exactly that with ctRegex. Please
 illuminate.


 Other way around. Generating arbitrary output (including but not limited
 to SQL expressions) from D code. As I have already mentioned, it is the
 very same thing we do now with __traits and UDA's but not limited to
 only declarations.


Actually I couldn't shake off the feeling that macros are just CTFE 
functions on Ast objects. How objects are created and converted back to 
source code is a separate question.

If we just had:

//this would invoke compiler's parser at CTFE
auto ast = "x = y;".astof

and have it work at CTFE to return sensible AST (a big if btw).

And then (after some manipulations):
ast.toString() //get back a string of D code

It may help code generation of D --> DSL.

Alternatively one can implement D parser that works at CTFE and watch it 
crawl :)

-- 
Dmitry Olshansky



 Nov 12 2013



 next sibling parent reply "Dicebot" <public dicebot.lv>  writes:


On Tuesday, 12 November 2013 at 20:56:34 UTC, Dmitry Olshansky 
wrote:

 If we just had:

 //this would invoke compiler's parser at CTFE
 auto ast = "x = y;".astof

 and have it work at CTFE to return sensible AST (a big if btw).

 And then (after some manipulations):
 ast.toString() //get back a string of D code

 It may help code generation of D --> DSL.


Add there "foo.codeof" in the toolset and it will pretty much 
give core needed stuff.

However, it will destroy compilation times if not coupled with 
compiler internal parsing/semantic phase.

Actual "macro" keyword syntax sugar is hardly important to me.



 Nov 12 2013



 next sibling parent  Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


13-Nov-2013 01:16, Dicebot пишет:

 On Tuesday, 12 November 2013 at 20:56:34 UTC, Dmitry Olshansky wrote:

 If we just had:

 //this would invoke compiler's parser at CTFE
 auto ast = "x = y;".astof

 and have it work at CTFE to return sensible AST (a big if btw).

 And then (after some manipulations):
 ast.toString() //get back a string of D code

 It may help code generation of D --> DSL.


 Add there "foo.codeof" in the toolset and it will pretty much give core
 needed stuff.

 However, it will destroy compilation times if not coupled with compiler
 internal parsing/semantic phase.


For me it feels a lot like implementing something as a soft-core or 
emulator in software vs producing a bare-metal chip. It's a good idea to 
test waters and prototype a soft-thingy before getting dirty. When it 
comes to manufacture ASICs the idea should be well tested and could be 
actually sold as is already ;)

If we consider that compiler is sort of "hardware" accelerator for 
getting ast's of strings and functions alike. Ditto with toString of 
ast. Then if somebody comes up with soft-core implementation of "macros" 
and shows:
a) Cool benefits of manipulating on ASTs compared to cracking some strings
b) Limitations of this implementation (unhyginic is one)
c) Quantify obvious performance problems
d) Prepare a solid file of use cases

Then we might get people to manufacture ASICs (pay costs, implement in 
compiler/spec).

Paraphrasing somebody from C++ ISO committee (I think Herb):

Boost C++ Lambda is a prime reason we had to add lambdas as a core 
feature in C++11. If it took these top-notch experts that much of coding 
to get only this far and it's still brittle, limited and etc.
... we have no choice but to add it to the language.


 Actual "macro" keyword syntax sugar is hardly important to me.



-- 
Dmitry Olshansky



 Nov 13 2013



prev sibling  parent reply Martin Nowak <code dawg.eu>  writes:


On 11/12/2013 10:16 PM, Dicebot wrote:

 Add there "foo.codeof" in the toolset and it will pretty much give core
 needed stuff.


Well, codeof is nice but it lacks the parser.



 Nov 13 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 23:29, Martin Nowak wrote:


 Well, codeof is nice but it lacks the parser.


Yes, and the you need to mixin it again. Several unnecessary steps, see 
the bottom of:

http://forum.dlang.org/thread/l5otb1$1dhi$1 digitalmars.com?page=13#post-l5vcct:242lit:241:40digitalmars.com

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling next sibling parent reply Philippe Sigaud <philippe.sigaud gmail.com>  writes:


On Tue, Nov 12, 2013 at 9:55 PM, Dmitry Olshansky <dmitry.olsh gmail.com> wrote:


 If we just had:

 //this would invoke compiler's parser at CTFE
 auto ast = "x = y;".astof

 and have it work at CTFE to return sensible AST (a big if btw).

 And then (after some manipulations):
 ast.toString() //get back a string of D code


Which you still have to mixin, btw. Which means any tree manipulation
you do must be done at compile-time.
Unless, of course,  you then put the resulting code into another file,
to be compiled and loaded afterwards.



 It may help code generation of D --> DSL.

 Alternatively one can implement D parser that works at CTFE and watch it crawl
:)


*cough cough*

But, as other have said, these would be very un-hygienic macros, with
no knowledge of the surrounding environment.
Unless said CT-compatible parser could also somehow determine the
local symbols (you could pass it 'by hand', but who would do that,
realistically?)



 Nov 12 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 22:55, Philippe Sigaud wrote:


 Which you still have to mixin, btw. Which means any tree manipulation
 you do must be done at compile-time.
 Unless, of course,  you then put the resulting code into another file,
 to be compiled and loaded afterwards.


It would be at compile time with macros anyway. The DIP even says that 
the macro functions are removed in the code generating phase.


 *cough cough*

 But, as other have said, these would be very un-hygienic macros, with
 no knowledge of the surrounding environment.
 Unless said CT-compatible parser could also somehow determine the
 local symbols (you could pass it 'by hand', but who would do that,
 realistically?)


Exactly.

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling next sibling parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 21:55, Dmitry Olshansky wrote:


 Actually I couldn't shake off the feeling that macros are just CTFE
 functions on Ast objects. How objects are created and converted back to
 source code is a separate question.


Yes, and reflection.


 If we just had:

 //this would invoke compiler's parser at CTFE
 auto ast = "x = y;".astof

 and have it work at CTFE to return sensible AST (a big if btw).

 And then (after some manipulations):
 ast.toString() //get back a string of D code

 It may help code generation of D --> DSL.

 Alternatively one can implement D parser that works at CTFE and watch it
 crawl :)


You would still need to do strign mixin. Which results in a some 
unnecessary conversion between AST's and strings. See the list at the 
bottom of:

http://forum.dlang.org/thread/l5otb1$1dhi$1 digitalmars.com?page=13#post-l5vcct:242lit:241:40digitalmars.com

It also looks ugly.

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling  parent reply Martin Nowak <code dawg.eu>  writes:


 //this would invoke compiler's parser at CTFE
 auto ast = "x = y;".astof
 ast.toString() //get back a string of D code


Right, that's the basic functionality we want.
What's not clear is how to get the type of y.



 Nov 13 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 23:29, Martin Nowak wrote:


 Right, that's the basic functionality we want.
 What's not clear is how to get the type of y.


That's what the reflection API is for. I guess many people here know a 
lot more about building AST's than me. But here's a simple API:

auto ast = "x = y".astof;

auto type = ast.right.type;
auto name = type.name;

assert(ast.left.type == type);

-- 
/Jacob Carlborg



 Nov 14 2013



prev sibling next sibling parent reply dennis luehring <dl.soluz gmx.net>  writes:


Am 12.11.2013 17:39, schrieb Andrei Alexandrescu:

 Maybe the problem needs to be reformulated for D. I think an SQL mixin
 that either stays unchanged (for DB engines) or translates to a D
 expression (for native D data types) would be doable, nontrivial,
 interesting, and instantly usable for people who already know SQL
 without any extra learning. In other words... actually better than Linq.


"actually better than Linq" good statement without knowing it deep

linq allows construction and querieng of non table-like hierarchical 
data, so its more an object-hierarchy store/retrival system which can 
"also" work with the lower just-tables-like world of sql results



 Nov 12 2013



  parent  Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>  writes:


On 11/12/13 8:56 AM, dennis luehring wrote:

 Am 12.11.2013 17:39, schrieb Andrei Alexandrescu:

 Maybe the problem needs to be reformulated for D. I think an SQL mixin
 that either stays unchanged (for DB engines) or translates to a D
 expression (for native D data types) would be doable, nontrivial,
 interesting, and instantly usable for people who already know SQL
 without any extra learning. In other words... actually better than Linq.


 "actually better than Linq" good statement without knowing it deep

 linq allows construction and querieng of non table-like hierarchical
 data, so its more an object-hierarchy store/retrival system which can
 "also" work with the lower just-tables-like world of sql results


Yah, it was an exaggeration. Linq is very powerful.

Andrei



 Nov 12 2013



prev sibling  parent  "Rikki Cattermole" <alphaglosined gmail.com>  writes:


On Tuesday, 12 November 2013 at 16:39:18 UTC, Andrei Alexandrescu 
wrote:

 On 11/12/13 7:21 AM, Ellery Newcomer wrote:

 the .NET framework can do this because it exposes an api for 
 querying,
 building, and compiling asts.

 D cannot do this because it doesn't. (and I have tried to make 
 it work)


 Maybe the problem needs to be reformulated for D. I think an 
 SQL mixin that either stays unchanged (for DB engines) or 
 translates to a D expression (for native D data types) would be 
 doable, nontrivial, interesting, and instantly usable for 
 people who already know SQL without any extra learning. In 
 other words... actually better than Linq.

 Andrei


I was using linq as an example of what we could do with this DIP 
or what it should be able to do. It wasn't a suggestion of how we 
should implement such a library.

I also identified with it that having scoped variables from the 
initiation call is required for safety as well as pragmas for 
error. Later Jacob alerted me to warning as well. Will submit 
pull once warning is done.

I also suggested a generic lexer with rules to make it well 
defined. With it also if's like templates have for macro over 
loading but nobody was keen for this hence not on wiki.

Yes this can be done already in theory however it doesnt make it 
not a valid use case.



 Nov 12 2013



prev sibling next sibling parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 15:38, John Colvin wrote:


 for those of us entirely unfamiliar with linq, what is this supposed to
 do? Select people with name "John" from a collection of people, like in
 sql? It seems trivial to do this using filter, or am I missing
 something...?


The idea of that example is, as Dicebot said, to convert the AST of the 
lambda to a string containing SQL code then query the database and 
return the result. The interesting part here, in the context of macros, 
is to convert the lambda to an SQL string.

The filter is not applied on the result, it's converted to SQL and 
performed in the database.

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling  parent reply "Regan Heath" <regan netmail.co.nz>  writes:


On Tue, 12 Nov 2013 14:38:43 -0000, John Colvin  
<john.loughran.colvin gmail.com> wrote:


 On Tuesday, 12 November 2013 at 13:50:49 UTC, Jacob Carlborg wrote:

 auto person = Person.where(e => e.name == "John");

 Translates to:

 select * from person where name = 'John'


 for those of us entirely unfamiliar with linq, what is this supposed to  
 do? Select people with name "John" from a collection of people, like in  
 sql? It seems trivial to do this using filter, or am I missing  
 something...?


Linq is often confused with LinqToSQL, the above was a description of what  
happens in the latter.  If Person was an object representing a table from  
a SQL database, then calling 'where' on it would translate into an  
IQueryable<Person> object which when iterated upon using foreach would  
execute the SQL statement shown above and return the resulting rows as  
<someperson> objects one by one to the foreach body.

Pure Linq is a DSL, an alternate syntax, which looks a lot like SQL, which  
can translate to SQL but is not limited to SQL.  It could translate to any  
alternate database syntax, or simply access a container supporting the  
required operations.

The motivation is the same as for "foreach", it is a general syntax for  
accessing a wide range of containers including databases with a common  
syntax.  We could, equally, go the other way, use the IMO nicer  
composition syntax X.where(..).foo(...).etc and translate that to SQL,  
without the need to the SQL like DSL inside D.

For some LINQ examples, see:
http://code.msdn.microsoft.com/101-LINQ-Samples-3fb9811b

For a specific where example using a simple container, see "Where - simple  
1"

R

-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/



 Nov 13 2013



 next sibling parent reply dennis luehring <dl.soluz gmx.net>  writes:


Am 13.11.2013 10:58, schrieb Regan Heath:

 Linq is often confused with LinqToSQL, the above was a description of what
 happens in the latter.  If Person was an object representing a table from
 a SQL database, then calling 'where' on it would translate into an
 IQueryable<Person> object which when iterated upon using foreach would
 execute the SQL statement shown above and return the resulting rows as
 <someperson> objects one by one to the foreach body.

 Pure Linq is a DSL, an alternate syntax, which looks a lot like SQL, which
 can translate to SQL but is not limited to SQL.  It could translate to any
 alternate database syntax, or simply access a container supporting the
 required operations.


linq allows construction AND querying of non table-like hierarchical
data, so its more an (basetypes|object)-hierarchy store/retrival system 
which can "also" work with the lower just-tables-like world of sql results



 Nov 13 2013



  parent  dennis luehring <dl.soluz gmx.net>  writes:


Am 13.11.2013 11:41, schrieb dennis luehring:

 Am 13.11.2013 10:58, schrieb Regan Heath:

 Linq is often confused with LinqToSQL, the above was a description of what
 happens in the latter.  If Person was an object representing a table from
 a SQL database, then calling 'where' on it would translate into an
 IQueryable<Person> object which when iterated upon using foreach would
 execute the SQL statement shown above and return the resulting rows as
 <someperson> objects one by one to the foreach body.

 Pure Linq is a DSL, an alternate syntax, which looks a lot like SQL, which
 can translate to SQL but is not limited to SQL.  It could translate to any
 alternate database syntax, or simply access a container supporting the
 required operations.


 linq allows construction AND querying of non table-like hierarchical
 data, so its more an (basetypes|object)-hierarchy store/retrival system
 which can "also" work with the lower just-tables-like world of sql results


but linq isn't good in graph operations like neo4j
http://www.neo4j.org/learn/cypher

which woulds be also very nice as an macro feature for in D reference 
traversing



 Nov 13 2013



prev sibling  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 10:58, Regan Heath wrote:


 The motivation is the same as for "foreach", it is a general syntax for
 accessing a wide range of containers including databases with a common
 syntax.  We could, equally, go the other way, use the IMO nicer
 composition syntax X.where(..).foo(...).etc and translate that to SQL,
 without the need to the SQL like DSL inside D.


My original example was not Linq. I tried to make it look as much as 
ranges as possible. Yes, I'm tinking something like what you're suggesting:

Post.where(e => e.title == "Foo" && e.comments.title == "Bar").join(e => 
e.comments);

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling next sibling parent  Martin Nowak <code dawg.eu>  writes:


On 11/12/2013 10:20 AM, Walter Bright wrote:

 I confess I have some serious reservations about AST macros in general:

 1. I've seen very heavy use of such macros in macro assemblers. What
 happens is people use it to invent their own (very baroque) language on
 top of the existing assembler one. Anyone trying to read the code has to
 learn this new unique language, and given the limitations of the macro
 capability, it often comes with unfixable bizarre corner cases and
 limitations.


Yes, this is a big issue.
I have similar concerns about UDA, but it's easier to put the most 
needed UDAs into phobos making them more standardized.


 4. AST macros, pretty much by definition, manipulate the AST. However,
 if you examine the semantic routines in DMD, a *lot* more is required to
 do more significant things. The symbol table must be consulted, etc. I
 just don't see how one could hope to implement something like function
 overloading with AST macros.  Or template type deduction. Or closure
 semantics, attribute inference, etc. Sure, some forms of foreach can be
 rewritten into for statements by manipulating the AST, but not the more
 interesting forms, such as the ones that deal with opApply().


That's the question we'd have to answer.
I think what we should target for is that constructs like async/await or 
SIMDifying of code can be implemented using AST macros.


 For example, the Linq example in the DIP is not compelling, as
 aesthetically nicer code can be written using D's ranges and algorithms:

      auto data = arr.filter!(x => x > 5).array;


 be blunt it's like the old:

      #define BEGIN {
      #define END }

 macros used in old C code to make it look like Pascal.


True



 Nov 12 2013



prev sibling next sibling parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-12 10:20, Walter Bright wrote:


 I confess I have some serious reservations about AST macros in general:

 1. I've seen very heavy use of such macros in macro assemblers. What
 happens is people use it to invent their own (very baroque) language on
 top of the existing assembler one. Anyone trying to read the code has to
 learn this new unique language, and given the limitations of the macro
 capability, it often comes with unfixable bizarre corner cases and
 limitations.


There's a word for that, it's called DSL (Domain Specific Language) :)


 This got so bad that in some cases I know of, the poor sap who is given
 the job of making improvements resorted to running the assembler to
 generate an object file, then disassembling the object file back into
 source code! Something went very wrong for this to be necessary. I know
 in my own assembler work I have abandoned all use of macros.


DSL's are so popular so many languages are specifically designed to make 
it easy to create DLS's.


 (Note that macros in typical assemblers are far more powerful than C's
 macro language, which stands out for being pathetically underpowered.)

 2. Macros are something that should be used very sparingly. However,
 this is not what happens. I used to be perplexed that even top shelf
 C/C++ programmers tend to write what I not so humbly consider to be
 pretty abusive use of macros. Now it just saddens me.


It depends on what kind of macros we're talking about. For C 
preprocessor macros, absolutely. But just because to different features 
have the same word "macro", you shouldn't put the in the same box. I 
guess I shouldn't have used the word "macro" at all in the DIP.


 3. Lisp is a language that encourages users to write macros to pretty
 much invent a custom language for the task at hand. Heavy use of such
 makes the code unrecognizable as being Lisp code. I believe a language
 ought to have some "anchors" that the person reading the code can
 reliably recognize. (GO has taken this idea pretty far.)


Lisp doesn't have any syntax to begin with.


 4. AST macros, pretty much by definition, manipulate the AST. However,
 if you examine the semantic routines in DMD, a *lot* more is required to
 do more significant things. The symbol table must be consulted, etc. I
 just don't see how one could hope to implement something like function
 overloading with AST macros. Or template type deduction. Or closure
 semantics, attribute inference, etc. Sure, some forms of foreach can be
 rewritten into for statements by manipulating the AST, but not the more
 interesting forms, such as the ones that deal with opApply().


Macros are not for implementing function overloading or template type 
deduction, we already have those features. I don't consider these easy 
to implement and they're not lowered in the same way as "scope" or 
"foreach" are.


 There are some statements in the DIP about the Context parameter and
 it'll provide semantic information, but I don't see how this can work
 without making it an ever-expanding collection of arbitrary methods.


We'll just put in what we want and need there. At the worst case you'll 
get access to the whole compiler. Which is what we want any way, at 
least the front end, to be usable as a library.


 5. We've said "no" in the past to things like user-defined tokens, which
 are closely related.


No, they're not.


 6. Note that D has a limited form of AST macros with mixin templates.


Yes, I do know. Note that they are too limiting compared to AST macros.


 To sum up, any AST macro proposal has some tough barriers to get over.
 Within its necessarily severe semantic limitations, it has to deliver a
 pretty compelling set of solutions, while avoiding the morass of
 incomprehensibility that far too many macro systems become in practice.


Note that AST macros (according to the DIP) is not anything like C 
preprocessor macros. Basically the only thing they have in common is the 
name, "macro". It's not a valid comparison.

C macros are textual macros. They can create completely new syntax. AST 
macros cannot. They work at the semantic level in the compiler, just as 
templates. They can add new semantic meaning to existing syntax. Just 
like templates, macros need to lex and parse correctly. Templates do not 
need to be semantically correct if they're not instantiated.

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling  parent  "Dejan Lekic" <dejan.lekic gmail.com>  writes:


 1. I've seen very heavy use of such macros in macro assemblers. 
 What happens is people use it to invent their own (very 
 baroque) language on top of the existing assembler one. Anyone 
 trying to read the code has to learn this new unique language, 
 and given the limitations of the macro capability, it often 
 comes with unfixable bizarre corner cases and limitations.


Metaprogramming (almost) does the same. At least that is my 
impression. Sometimes I have to go through hundreds of lines of 
templated D code to understand what a single line does...



 Nov 14 2013



prev sibling next sibling parent  Martin Nowak <code dawg.eu>  writes:


On 11/10/2013 10:20 PM, Jacob Carlborg wrote:

 I've been thinking quite long of how AST macros could look like in D.
 I've been posting my vision of AST macros here in the newsgroup a couple
 of times already. I've now been asked to create a DIP out of it, so here
 it is:

 http://wiki.dlang.org/DIP50


Thanks, looks like an insightful discussion.



 Nov 12 2013



prev sibling  parent reply luka8088 <luka8088 owave.net>  writes:


On 10.11.2013. 22:20, Jacob Carlborg wrote:

 I've been thinking quite long of how AST macros could look like in D.
 I've been posting my vision of AST macros here in the newsgroup a couple
 of times already. I've now been asked to create a DIP out of it, so here
 it is:
 
 http://wiki.dlang.org/DIP50
 


I took a look at it as here is my conclusion for now:

Statement and attribute macro examples look great. But I don't like Linq
example. I don't think code like the following should be allowed.

query {
  from element in array
  where element > 2
  add element to data
}


From my point of view this whole idea is great as it makes it easier
what is already possible. For example, with current behavior if I wanted
to write.

foo {
  writeln("foo");
  writeln("foo again");
}

I would have to write:

mixin(foo!(q{
  writeln("foo");
  writeln("foo again");
}));

So the proposed behavior looks much nicer, and I agree with it as the
content of foo block is actually written in D and I think whoever is
reading it would be comfortable with it.


However, for other, non-D syntax-es I would prefer something like:

query q{
  from element in array
  where element > 2
  add element to data
}

Which can be handled by:

macro query (Context context, string dsl) {
    return domainSpecificLanguageToD(dsl);
}

This in terms is already possible by writing the following, it only
allows to be written in a more readable way. And the q{ ... } notation
clearly points out that there is something special going on. Also by
passing such content as string user can implement custom (or call one of
the predefined) tokenizer/lexer/parser.

mixin(query!(q{
  from element in array
  where element > 2
  add element to data
}));


I also don't like the <[ ... ]> syntax because:
1. Like others have said, it looks very foreign.
2. I don't think there is a need to add a new syntax.

I think that string concatenation is enough (at least for now), and if
you want another syntax for templates you can write a macro for that.

For example:

macro myAssert (Context context, Ast!(bool) val, Ast!(string) str = null) {
  auto message = str ? "Assertion failure: " ~ str.eval : val.toString();
  auto msgExpr = literal(constant(message));

  return "
    if (!" ~ val ~ ")
      throw new AssertError(" ~ msgExpr ~ ");
  ";

  // or
  return astTemplate q{
    if (!$val)
      throw new AssertError($msgExpr);
  };
}

void main () {
    myAssert(1 + 2 == 4);
}


What do you guys think?

--
Luka



 Nov 13 2013



 next sibling parent reply "Rikki Cattermole" <alphaglosined gmail.com>  writes:


 From my point of view this whole idea is great as it makes it 
 easier
 what is already possible. For example, with current behavior if 
 I wanted
 to write.

 foo {
   writeln("foo");
   writeln("foo again");
 }


Currently this would be sent as a string. I have suggested 
previously (given a lexer implementation) a lexer would be 
included that can work by line and symbol. This is fine but it 
would mean the lexer would need to parse it before calling the 
macro itself. Perhaps it should be explored a forced parsing and 
validation of macro values?


 I would have to write:

 mixin(foo!(q{
   writeln("foo");
   writeln("foo again");
 }));


AST macros currently suggested do not work at the same stage as 
mixins, as far as I am aware. They work at a much earlier stage. 
Mixins have limits because of there lateness in lexing process.


 So the proposed behavior looks much nicer, and I agree with it 
 as the
 content of foo block is actually written in D and I think 
 whoever is
 reading it would be comfortable with it.


 However, for other, non-D syntax-es I would prefer something 
 like:

 query q{
   from element in array
   where element > 2
   add element to data
 }


I find it intriguing to add a keyword to distinguish a D code 
block from a string. As currently they are parsed the same way.


 Which can be handled by:

 macro query (Context context, string dsl) {
     return domainSpecificLanguageToD(dsl);
 }

 This in terms is already possible by writing the following, it 
 only
 allows to be written in a more readable way. And the q{ ... } 
 notation
 clearly points out that there is something special going on. 
 Also by
 passing such content as string user can implement custom (or 
 call one of
 the predefined) tokenizer/lexer/parser.

 mixin(query!(q{
   from element in array
   where element > 2
   add element to data
 }));


Perhaps a different way of identifying such code is required?

query "{
    from element in array
    where element > 2
    add element to data
}

Although it would act as a triple quoted string, except only 
valid for macros maybe? This would signify that the following 
block of text was to be pushed into a macro. Otherwise it would 
be assumed as valid D code?

Assuming a lexer is available it could check that any code given 
to a macro is correct. I will be submitting a pull request for 
pragma error and warning so if it is not valid the macro could 
fire a compile error.



 Nov 13 2013



  parent  Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 12:49, Rikki Cattermole wrote:


 AST macros currently suggested do not work at the same stage as mixins,
 as far as I am aware. They work at a much earlier stage. Mixins have
 limits because of there lateness in lexing process.


I think macros would work on the same stage as mixins.


 Assuming a lexer is available it could check that any code given to a
 macro is correct. I will be submitting a pull request for pragma error
 and warning so if it is not valid the macro could fire a compile error.


All macro invocations are already checked. They're lexed and parsed. 
Marcos work at the semantic phase. Then when the macro is returned the 
resulting AST is typed checked as usual.

-- 
/Jacob Carlborg



 Nov 13 2013



prev sibling next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-13 09:15, luka8088 wrote:


 I took a look at it as here is my conclusion for now:

 Statement and attribute macro examples look great. But I don't like Linq
 example. I don't think code like the following should be allowed.


I'm not a fan of the Linq example either. Or rather if I put it like 
this. I think the Linq example is quite a good example what can be done 
with AST macros. But I don't think it's a good fit in D.


 query {
    from element in array
    where element > 2
    add element to data
 }


  From my point of view this whole idea is great as it makes it easier
 what is already possible. For example, with current behavior if I wanted
 to write.

 foo {
    writeln("foo");
    writeln("foo again");
 }

 I would have to write:

 mixin(foo!(q{
    writeln("foo");
    writeln("foo again");
 }));

 So the proposed behavior looks much nicer, and I agree with it as the
 content of foo block is actually written in D and I think whoever is
 reading it would be comfortable with it.


 However, for other, non-D syntax-es I would prefer something like:


Well, the macros in the DIP only supports the D syntax. The macros work 
at the semantic level. You cannot create new syntax but you can give 
existing syntax new meaning. Compared with current D, macros need to lex 
and parse correctly. But they don't need to be semantically correct. 
That's the same thing as with templates, they can be semantically 
invalid as long as they're not instantiated.

The Linq example is lexically correct. I'm not sure if the parser will 
reject it or not.


 query q{
    from element in array
    where element > 2
    add element to data
 }

 Which can be handled by:

 macro query (Context context, string dsl) {
      return domainSpecificLanguageToD(dsl);
 }

 This in terms is already possible by writing the following, it only
 allows to be written in a more readable way. And the q{ ... } notation
 clearly points out that there is something special going on. Also by
 passing such content as string user can implement custom (or call one of
 the predefined) tokenizer/lexer/parser.

 mixin(query!(q{
    from element in array
    where element > 2
    add element to data
 }));


 I also don't like the <[ ... ]> syntax because:
 1. Like others have said, it looks very foreign.
 2. I don't think there is a need to add a new syntax.


As it says in the DIP, this is one suggestion of the syntax, if new 
syntax is needed at all. It's just easier to show examples like that 
than building a complete AST with function calls.

As you can see I have an alternative that doesn't require new syntax:

http://wiki.dlang.org/DIP50#The_AST_Macro

Instead of "<[ a + b ]>" you would do "ast(a + b)" or "ast { a + b }".


 I think that string concatenation is enough (at least for now), and if
 you want another syntax for templates you can write a macro for that.


Strings are far from enough. Then you have missed the whole idea. It's 
not supposed to be syntax sugar for string mixins.

-- 
/Jacob Carlborg



 Nov 13 2013



  parent reply luka8088 <luka8088 owave.net>  writes:


On 13.11.2013. 13:47, Jacob Carlborg wrote:

 On 2013-11-13 09:15, luka8088 wrote:

 I think that string concatenation is enough (at least for now), and if
 you want another syntax for templates you can write a macro for that.


 
 Strings are far from enough. Then you have missed the whole idea. It's
 not supposed to be syntax sugar for string mixins.
 


Oh, I see. It seems that I indeed missed the point.

It seems to me that this DIP could to be granulated into: AST
reflection, AST manipulation and AST template. The reason for this, as
far as I see, is that although they overlap in some cases, in other they
could be used independently and it could help with understanding.

Regarding AST reflection and AST manipulation, I have been thinking and
found a few situations that bugs me:

In the example of

auto person = Person.where(e => e.name == "John");

what would be the response of:

auto f = e => e.name == "John";
auto person = Person.where(f);

I guess it should be a compiler error because f could be modified at
runtime and f's body could be hidden. So basically AST macros are
something that look like D but actually are not D. This seems to me like
an example that look good as a toy example but fails on the larger scale
so I agree with Water in this matter (unless better
examples/explanations are provided). Or maybe I am not seeing it clear
enough.

But! Regarding AST templates, I think something like the following would
be a great syntax sugar:

int i = 5;

Ast a = t{ // t as template
  int j = $i;
  int k = $i + $i;
};

// ... alter ast here if necessary

mixin(a);

Where syntax (and semantic) in template must be strictly D. The only
difference would be $ sign which would allow reference to symbols
outside the template.

So if you would want to have conditional template you could write:

int i = 5;
int j;
bool b = false;

mixin(t{
  $j = $i;
  static if ($b) {
    $j++;
  }
});



 Nov 19 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-19 19:32, luka8088 wrote:


 Oh, I see. It seems that I indeed missed the point.

 It seems to me that this DIP could to be granulated into: AST
 reflection, AST manipulation and AST template. The reason for this, as
 far as I see, is that although they overlap in some cases, in other they
 could be used independently and it could help with understanding.

 Regarding AST reflection and AST manipulation, I have been thinking and
 found a few situations that bugs me:

 In the example of

 auto person = Person.where(e => e.name == "John");

 what would be the response of:

 auto f = e => e.name == "John";
 auto person = Person.where(f);

 I guess it should be a compiler error because f could be modified at
 runtime and f's body could be hidden.


I haven't thought about that. But AST macros work at compile time


 So basically AST macros are
 something that look like D but actually are not D. This seems to me like
 an example that look good as a toy example but fails on the larger scale
 so I agree with Water in this matter (unless better
 examples/explanations are provided). Or maybe I am not seeing it clear
 enough.

 But! Regarding AST templates, I think something like the following would
 be a great syntax sugar:

 int i = 5;

 Ast a = t{ // t as template
    int j = $i;
    int k = $i + $i;
 };

 // ... alter ast here if necessary

 mixin(a);


So you don't like that it's not a "mixin" there with AST macros?


 Where syntax (and semantic) in template must be strictly D. The only
 difference would be $ sign which would allow reference to symbols
 outside the template.


If the semantics need to be valid this is very limited. Even more 
limiting than what we have now with string mixins. Note that with AST 
macros, the input needs to syntactically valid. The resulting AST of the 
macro needs to both be syntactically and semantically valid D code.

-- 
/Jacob Carlborg



 Nov 19 2013



  parent reply luka8088 <luka8088 owave.net>  writes:


On 19.11.2013. 21:32, Jacob Carlborg wrote:

 On 2013-11-19 19:32, luka8088 wrote:
 

 Oh, I see. It seems that I indeed missed the point.

 It seems to me that this DIP could to be granulated into: AST
 reflection, AST manipulation and AST template. The reason for this, as
 far as I see, is that although they overlap in some cases, in other they
 could be used independently and it could help with understanding.

 Regarding AST reflection and AST manipulation, I have been thinking and
 found a few situations that bugs me:

 In the example of

 auto person = Person.where(e => e.name == "John");

 what would be the response of:

 auto f = e => e.name == "John";
 auto person = Person.where(f);

 I guess it should be a compiler error because f could be modified at
 runtime and f's body could be hidden.


 
 I haven't thought about that. But AST macros work at compile time


Well, do think about that :)

auto f = e => e.name == "John";
if (true)
  f = e => e.name == "Jack";

auto person = Person.where(f);

I can think of a many use cases where conditional query generation is
required. But I don't see how this could be done using AST macros.


 

 So basically AST macros are
 something that look like D but actually are not D. This seems to me like
 an example that look good as a toy example but fails on the larger scale
 so I agree with Water in this matter (unless better
 examples/explanations are provided). Or maybe I am not seeing it clear
 enough.

 But! Regarding AST templates, I think something like the following would
 be a great syntax sugar:

 int i = 5;

 Ast a = t{ // t as template
    int j = $i;
    int k = $i + $i;
 };

 // ... alter ast here if necessary

 mixin(a);


 
 So you don't like that it's not a "mixin" there with AST macros?


Um, sorry. I don't understand the question.

This example (and suggestion) was suppose to show that we could allow
AST mixins as well as string mixins. It should behave like string mixins
but the main difference is that AST is structured so it much cleaner to
manipulate.


 

 Where syntax (and semantic) in template must be strictly D. The only
 difference would be $ sign which would allow reference to symbols
 outside the template.


 
 If the semantics need to be valid this is very limited. Even more
 limiting than what we have now with string mixins. Note that with AST
 macros, the input needs to syntactically valid. The resulting AST of the
 macro needs to both be syntactically and semantically valid D code.
 


When I first started programing I was introduced to static typing. Then
I discovered dynamic typing and dynamic structures (in php mostly). And
for me at that time it was much better and easier to use. However, over
time I learned that sometimes limitations are good and now like static
typing much more :) My point here is that limits are sometimes good,
although that don't seem that way.

I think that we should think of a more complex (real world) examples and
than real issues will reveal themselves.



 Nov 19 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-19 21:54, luka8088 wrote:


 Well, do think about that :)

 auto f = e => e.name == "John";
 if (true)
    f = e => e.name == "Jack";

 auto person = Person.where(f);

 I can think of a many use cases where conditional query generation is
 required. But I don't see how this could be done using AST macros.


Using the Rails plugin, which I've got this idea from, I would not make 
the lambda conditional but the whole statement, translated to D:

auto person = Person.scoped();

if (true)
     person = person.where(e => e.name == "Jack");

else
     person = person.where(e => e.name == "John");


 Um, sorry. I don't understand the question.

 This example (and suggestion) was suppose to show that we could allow
 AST mixins as well as string mixins. It should behave like string mixins
 but the main difference is that AST is structured so it much cleaner to
 manipulate.


What I was trying to say is, what don't you like about my suggestion. Is 
it that the "mixin" keyword isn't used.

-- 
/Jacob Carlborg



 Nov 20 2013



  parent reply luka8088 <luka8088 owave.net>  writes:


On 20.11.2013. 9:04, Jacob Carlborg wrote:

 On 2013-11-19 21:54, luka8088 wrote:
 

 Well, do think about that :)

 auto f = e => e.name == "John";
 if (true)
    f = e => e.name == "Jack";

 auto person = Person.where(f);

 I can think of a many use cases where conditional query generation is
 required. But I don't see how this could be done using AST macros.


 
 Using the Rails plugin, which I've got this idea from, I would not make
 the lambda conditional but the whole statement, translated to D:
 
 auto person = Person.scoped();
 
 if (true)
     person = person.where(e => e.name == "Jack");
 
 else
     person = person.where(e => e.name == "John");


Ok, I see. Yes, I tried to think of a case where this could not work but
I was unable to find one.


 

 Um, sorry. I don't understand the question.

 This example (and suggestion) was suppose to show that we could allow
 AST mixins as well as string mixins. It should behave like string mixins
 but the main difference is that AST is structured so it much cleaner to
 manipulate.


 
 What I was trying to say is, what don't you like about my suggestion. Is
 it that the "mixin" keyword isn't used.
 


If I understood you correctly, the issue with current way DSLs are
implemented is that code needs to be parsed two times. First time DSL
author parses it and creates D code from it, and second time D compiler
parses that D code and compiles it. What I would suggest in this case is
that instead of intercepting the compiler and "fixing" the semantics
before it is verified we allow the user to build D Ast and give it to
the compiler. That is why I used the mixin in my example.

Ast dCodeAst = dslToD!(q{
  if true
    do()
  else
    dont()
    dont2()
  whatever()
});

// manipulate dCodeAst

mixin(dCodeAst);

The point of this example is that mixin() accepts Ast instead of string.
That way, we parse our DSL to D Ast and give it straight to the compiler
and everything is done only once!



 Nov 20 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-20 23:25, luka8088 wrote:


 If I understood you correctly, the issue with current way DSLs are
 implemented is that code needs to be parsed two times. First time DSL
 author parses it and creates D code from it, and second time D compiler
 parses that D code and compiles it. What I would suggest in this case is
 that instead of intercepting the compiler and "fixing" the semantics
 before it is verified we allow the user to build D Ast and give it to
 the compiler. That is why I used the mixin in my example.

 Ast dCodeAst = dslToD!(q{
    if true
      do()
    else
      dont()
      dont2()
    whatever()
 });

 // manipulate dCodeAst

 mixin(dCodeAst);

 The point of this example is that mixin() accepts Ast instead of string.
 That way, we parse our DSL to D Ast and give it straight to the compiler
 and everything is done only once!


So how is this different except for my proposal, except the use of q{} 
and "mixin"?

-- 
/Jacob Carlborg



 Nov 20 2013



 next sibling parent reply luka8088 <luka8088 owave.net>  writes:


On 21.11.2013. 8:28, Jacob Carlborg wrote:

 On 2013-11-20 23:25, luka8088 wrote:
 

 If I understood you correctly, the issue with current way DSLs are
 implemented is that code needs to be parsed two times. First time DSL
 author parses it and creates D code from it, and second time D compiler
 parses that D code and compiles it. What I would suggest in this case is
 that instead of intercepting the compiler and "fixing" the semantics
 before it is verified we allow the user to build D Ast and give it to
 the compiler. That is why I used the mixin in my example.

 Ast dCodeAst = dslToD!(q{
    if true
      do()
    else
      dont()
      dont2()
    whatever()
 });

 // manipulate dCodeAst

 mixin(dCodeAst);

 The point of this example is that mixin() accepts Ast instead of string.
 That way, we parse our DSL to D Ast and give it straight to the compiler
 and everything is done only once!


 
 So how is this different except for my proposal, except the use of q{}
 and "mixin"?
 


When using q{} compiler treats the contents as a regular string, and you
have to parse it and give it to the compiler using "mixin". So basically
you can say to the compiler: this part of code is my DSL and I will
parse it and check the semantics instead of you, then when I am done I
will give you D AST represantation of it.

The interpretation of the content of q{} is up to you, where in other
proposals compiler builds AST based on, I would say, guessing.

Take a look at this last example. And lets say that it is from python.
In python dont2() belong inside else block. So how else would you make
sure that compiler behaves accordingly?



 Nov 21 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-21 09:01, luka8088 wrote:


 When using q{} compiler treats the contents as a regular string, and you
 have to parse it and give it to the compiler using "mixin". So basically
 you can say to the compiler: this part of code is my DSL and I will
 parse it and check the semantics instead of you, then when I am done I
 will give you D AST represantation of it.

 The interpretation of the content of q{} is up to you, where in other
 proposals compiler builds AST based on, I would say, guessing.

 Take a look at this last example. And lets say that it is from python.
 In python dont2() belong inside else block. So how else would you make
 sure that compiler behaves accordingly?


Sorry, I meant your original suggestion of using t{}.

-- 
/Jacob Carlborg



 Nov 21 2013



  parent reply luka8088 <luka8088 owave.net>  writes:


On 21.11.2013. 9:31, Jacob Carlborg wrote:

 On 2013-11-21 09:01, luka8088 wrote:
 

 When using q{} compiler treats the contents as a regular string, and you
 have to parse it and give it to the compiler using "mixin". So basically
 you can say to the compiler: this part of code is my DSL and I will
 parse it and check the semantics instead of you, then when I am done I
 will give you D AST represantation of it.

 The interpretation of the content of q{} is up to you, where in other
 proposals compiler builds AST based on, I would say, guessing.

 Take a look at this last example. And lets say that it is from python.
 In python dont2() belong inside else block. So how else would you make
 sure that compiler behaves accordingly?


 
 Sorry, I meant your original suggestion of using t{}.
 


Um, my it's suppose to be the same as <[ ... ]> but I liked t{ ... }
syntax better as it looked more consistent with what D already has. But
I should have used <[ ... ]> , my mistake sorry.



 Nov 22 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-22 10:27, luka8088 wrote:


 Um, my it's suppose to be the same as <[ ... ]> but I liked t{ ... }
 syntax better as it looked more consistent with what D already has. But
 I should have used <[ ... ]> , my mistake sorry.


I thought you argued that the t{ } need to contain semantically valid code?

-- 
/Jacob Carlborg



 Nov 22 2013



  parent reply luka8088 <luka8088 owave.net>  writes:


On 22.11.2013. 11:17, Jacob Carlborg wrote:

 On 2013-11-22 10:27, luka8088 wrote:
 

 Um, my it's suppose to be the same as <[ ... ]> but I liked t{ ... }
 syntax better as it looked more consistent with what D already has. But
 I should have used <[ ... ]> , my mistake sorry.


 
 I thought you argued that the t{ } need to contain semantically valid code?
 


Yes. I still do. And I think that <[ ... ]> should contain semantically
valid code. In my opinion, if you wish to be able to write anything
else, the way to go would be: write it using q{ ... }, parse it
yourself, build a valid D AST yourself, give D AST to the compiler using
mixin().

Currently mixin() only accepts a string that contains a valid D code. So
if you have a custom parser for your DSL you need to generate a D code
from that DSL and then pass it to the compiler using mixin() in order
for the compiler to parse it again. Double parsing could be skipped if
mixin() would accept already built D AST and on the other hand parsing
DSL and building D AST yourself would allow maximum flexibility.



 Nov 22 2013



  parent reply "Jacob Carlborg" <doob me.com>  writes:


On Friday, 22 November 2013 at 23:43:26 UTC, luka8088 wrote:


 Yes. I still do. And I think that <[ ... ]> should contain 
 semantically
 valid code. In my opinion, if you wish to be able to write 
 anything
 else, the way to go would be: write it using q{ ... }, parse it
 yourself, build a valid D AST yourself, give D AST to the 
 compiler using
 mixin().


Then q{ } or <[ ]> would be very limited. Not even templets need 
to contain semantically valid code, as long as it's not 
instantiated. Usually the idea with AST macros is to take code 
that is not valid (semantically) and create a meaning for it 
(make it valid). If it's already valid in the first place why use 
AST macros at all? Just use the code as is.

Hmm, it could be useful to only allow semantically valid code and 
use macros to add new things to the code. But again, very 
limiting.


 Currently mixin() only accepts a string that contains a valid D 
 code.


The AST returned from a macro need to be valid D code as well.


 So if you have a custom parser for your DSL you need to 
 generate a D code
 from that DSL and then pass it to the compiler using mixin() in 
 order
 for the compiler to parse it again. Double parsing could be 
 skipped if
 mixin() would accept already built D AST and on the other hand 
 parsing
 DSL and building D AST yourself would allow maximum flexibility.


You can already do that today with string mixins, although you 
would need to convert the AST back to a string first.

--
/Jacob Carlborg



 Nov 23 2013



  parent  luka8088 <luka8088 owave.net>  writes:


On 23.11.2013. 11:40, Jacob Carlborg wrote:

 On Friday, 22 November 2013 at 23:43:26 UTC, luka8088 wrote:
 
 Then q{ } or <[ ]> would be very limited. Not even templets need to
 contain semantically valid code, as long as it's not instantiated.
 Usually the idea with AST macros is to take code that is not valid
 (semantically) and create a meaning for it (make it valid). If it's
 already valid in the first place why use AST macros at all? Just use the
 code as is.
 
 Hmm, it could be useful to only allow semantically valid code and use
 macros to add new things to the code. But again, very limiting.
 


I don't think so. What I was proposing is to split the problem into
sub-problems and then find the best method for each one. Templates are
one of them and they solve the problem of not so pretty string
concatenation and that is way they should contain only valid D code.

I don't think that writing a non valid code and then making in valid in
the compilation process should be allowed. Only transforming one valid
code into another valid code but hygienically. I don't have a concrete
example why that is a bad idea, I can only speak from experience. So
this is only my opinion, not an argument.


 Currently mixin() only accepts a string that contains a valid D code.


 
 The AST returned from a macro need to be valid D code as well.
 

 So if you have a custom parser for your DSL you need to generate a D code
 from that DSL and then pass it to the compiler using mixin() in order
 for the compiler to parse it again. Double parsing could be skipped if
 mixin() would accept already built D AST and on the other hand parsing
 DSL and building D AST yourself would allow maximum flexibility.


 
 You can already do that today with string mixins, although you would
 need to convert the AST back to a string first.
 


Yeah. I was referring to
http://forum.dlang.org/post/l5vcct$2lit$1 digitalmars.com

So for example, the problem from referred post could be addressed by
defining as AST struct type that is accessible to user and making
mixin() accept both string and AST struct. That is also why I wrote
http://forum.dlang.org/post/l6n91e$29hd$1 digitalmars.com


It seems to me now that we don't understand each other so well :) Maybe
we should put in more examples?



 Nov 23 2013



prev sibling  parent reply luka8088 <luka8088 owave.net>  writes:


On 21.11.2013. 8:28, Jacob Carlborg wrote:

 On 2013-11-20 23:25, luka8088 wrote:

 The point of this example is that mixin() accepts Ast instead of string.
 That way, we parse our DSL to D Ast and give it straight to the compiler
 and everything is done only once!


 
 So how is this different except for my proposal, except the use of q{}
 and "mixin"?
 


What we currently have:


custom syntax ---> custom parser ---> custom code generator ---> D
syntax (as string)


D syntax (as string) ---> mixin ---
                                   \
D syntax (read by compiler) -----------> D AST (held by the compiler,
unreachable) ---> compilation to binary (unreachable)


The way I see it we should allow to following:


custom syntax ---> custom parser -----
                                      \
D syntax inside <[ ... ]> template -------> D AST (as struct)


D AST (as struct) ---> mixin ---
                                \
D syntax (read by compiler) --------> D AST (held by compiler)


D AST (held by compiler, reachable) ---> reflection ---> D AST (as struct)

D AST (held by compiler) ---> compilation to binary



 Nov 22 2013



  parent reply Jacob Carlborg <doob me.com>  writes:


On 2013-11-22 10:44, luka8088 wrote:


 What we currently have:


I should have been t{ }. I do understand the difference between t{ } and 
q{ }. But not between <[ ]> and t{ }. Is it just two different syntaxes 
for the same?

-- 
/Jacob Carlborg



 Nov 22 2013



  parent  luka8088 <luka8088 owave.net>  writes:


On 22.11.2013. 11:19, Jacob Carlborg wrote:

 On 2013-11-22 10:44, luka8088 wrote:
 

 What we currently have:


 
 I should have been t{ }. I do understand the difference between t{ } and
 q{ }. But not between <[ ]> and t{ }. Is it just two different syntaxes
 for the same?
 


Yes. t{ } and <[ ]> are the same.



 Nov 22 2013



prev sibling  parent  Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


13-Nov-2013 12:15, luka8088 пишет:

 On 10.11.2013. 22:20, Jacob Carlborg wrote:


[snip]

 foo {
    writeln("foo");
    writeln("foo again");
 }

 I would have to write:

 mixin(foo!(q{
    writeln("foo");
    writeln("foo again");
 }));


Actually even now it's can be just
foo!q{
  ...
};

For instance this is compilable:
{
  import std.regex;
  auto r =  ctRegex!"how do you think that works?";
}

See ctRegex template in std.regex.

-- 
Dmitry Olshansky



 Nov 13 2013