www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - New syntax for string mixins

reply Jacob Carlborg <doob me.com> writes:
This is an idea I've been thinking of for a while, it's not a really 
suggestion (at least not yet) I just wanted to here what people think 
about it.

If we take a step back and look at what string mixins actually do or 
rather what they're used for, that would be: inserting a piece/block of 
code where the mixin expression is used. If we then take a look at how a 
block of code is represented in D (how you store it in variables and how 
you pass it around). It's not as a string which is used by the mixin 
expression, instead delegates are used to represent a block of code in D 
that can be passed around. Therefore this is my idea:

Add a new mixin operator " " (1). When that operator is put in front of 
a function call it would behave as a string mixin. The function that is 
called needs to be CTFE and return a delegate or an array of delegates:

class Foo
{
      get_set("int", "bar");
}

The above code would be the same as the following code:

class Foo
{
     mixin(get_set("int", "bar"));
}

If we now move to the declaration of "get_set" this is how it could look 
like:

void delegate () get_set (string type, string name)
{
     return
     {
          type _ name;

          type  name ()
         {
             return _ name;
         }

          type  name ( type  name)
         {
             return _ name =  name;
         }
     };
}

In the above code when " " is used in the delegate literal it basically 
behaves like string interpolation, so " type" would be replaced with the 
content of the "type" variable (2).

When the "get_set" function is called with the mixin symbol the content 
of the returned delegate is inserted where the call is made. If the 
function returns an array of delegates then the array would be unfolded 
and the content of all the delegates would be inserted.



Taking it one step further:

Allow the mixin syntax to be placed in front of most of the declarations 
and drop the need for string literals, commas and parentheses, basically 
allowing the following syntax:

class Foo
{
      get_set int bar;
}

Would be translated into this:

class Foo
{
      get_set!(int)("bar");
}

Maybe one could do something like this as well:

 singleton class Foo
{

}

void delegate () singleton (string name, void delegate () classBody)
{
     return { // a simple singleton implementation
         class  name
         {
             static  name instance;

             static this ()
             {
                 instance = new  name;
             }

             private this () {}

              classBody;
         }
     };
}

Above, in the last example,  classBody would insert the content of the 
delegate, basically the class body.

1. I will use " " in this example because I think it looks good but it 
would probably conflict with  nothrow,  property and others.

2. I have no idea if this usage of the mixin symbol, " ", will conflict 
with the first usage.

-- 
/Jacob Carlborg
Dec 13 2010
next sibling parent reply Graham St Jack <Graham.StJack internode.on.net> writes:
I have done a fair bit of mixin coding using recursive templates 
(inspired by std.typecons). It was an amazing taste of what you can do 
in D, and I am delighted with the result - HEAPS of boiler-plate coding 
vanished before my eyes. However, the template code is virtually 
impossible to understand.

What you are suggesting here seems to be a way to dramatically reduce 
the complexity of code that generates source-code and mixes it in. I 
think something like that is needed before this mind-bogglingly powerful 
feature of D can realise its potential.

There is of course the worry that it could get so easy that everyone 
starts doing it, and we have (relatively) impenetrable code everywhere 
instead of just deep in the bowels of framework libraries.


On 14/12/10 07:07, Jacob Carlborg wrote:
 This is an idea I've been thinking of for a while, it's not a really 
 suggestion (at least not yet) I just wanted to here what people think 
 about it.

 If we take a step back and look at what string mixins actually do or 
 rather what they're used for, that would be: inserting a piece/block 
 of code where the mixin expression is used. If we then take a look at 
 how a block of code is represented in D (how you store it in variables 
 and how you pass it around). It's not as a string which is used by the 
 mixin expression, instead delegates are used to represent a block of 
 code in D that can be passed around. Therefore this is my idea:

 Add a new mixin operator " " (1). When that operator is put in front 
 of a function call it would behave as a string mixin. The function 
 that is called needs to be CTFE and return a delegate or an array of 
 delegates:

 class Foo
 {
      get_set("int", "bar");
 }

 The above code would be the same as the following code:

 class Foo
 {
     mixin(get_set("int", "bar"));
 }

 If we now move to the declaration of "get_set" this is how it could 
 look like:

 void delegate () get_set (string type, string name)
 {
     return
     {
          type _ name;

          type  name ()
         {
             return _ name;
         }

          type  name ( type  name)
         {
             return _ name =  name;
         }
     };
 }

 In the above code when " " is used in the delegate literal it 
 basically behaves like string interpolation, so " type" would be 
 replaced with the content of the "type" variable (2).

 When the "get_set" function is called with the mixin symbol the 
 content of the returned delegate is inserted where the call is made. 
 If the function returns an array of delegates then the array would be 
 unfolded and the content of all the delegates would be inserted.



 Taking it one step further:

 Allow the mixin syntax to be placed in front of most of the 
 declarations and drop the need for string literals, commas and 
 parentheses, basically allowing the following syntax:

 class Foo
 {
      get_set int bar;
 }

 Would be translated into this:

 class Foo
 {
      get_set!(int)("bar");
 }

 Maybe one could do something like this as well:

  singleton class Foo
 {

 }

 void delegate () singleton (string name, void delegate () classBody)
 {
     return { // a simple singleton implementation
         class  name
         {
             static  name instance;

             static this ()
             {
                 instance = new  name;
             }

             private this () {}

              classBody;
         }
     };
 }

 Above, in the last example,  classBody would insert the content of the 
 delegate, basically the class body.

 1. I will use " " in this example because I think it looks good but it 
 would probably conflict with  nothrow,  property and others.

 2. I have no idea if this usage of the mixin symbol, " ", will 
 conflict with the first usage.
-- Graham St Jack
Dec 13 2010
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack  
<Graham.StJack internode.on.net> wrote:

 There is of course the worry that it could get so easy that everyone  
 starts doing it, and we have (relatively) impenetrable code everywhere  
 instead of just deep in the bowels of framework libraries.
TBH, I'm more excited by AST macros which I understood are planned for D3, as mentioned here: http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf They seem to promise the power of mixins, but without the mess. -- Best regards, Vladimir mailto:vladimir thecybershadow.net
Dec 14 2010
next sibling parent reply Graham St Jack <Graham.StJack internode.on.net> writes:
On 14/12/10 20:33, Vladimir Panteleev wrote:
 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:

 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.
TBH, I'm more excited by AST macros which I understood are planned for D3, as mentioned here: http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf They seem to promise the power of mixins, but without the mess.
I took a look at the pdf, but couldn't see how the AST macros could come close to the kinds of things that are possible (but difficult) with mixins. Is there more information somewhere? Jacob, I can see how your proposed syntax would make simple mixins easier and clearer, but how would it do something more complex? For example taking a classname and a bunch of field types and names, and turning it into a class definition complete with constructor from field values, constructor from an input stream, a method to write to an output stream, and const getters. Or maybe std.typecons.AutoImplement. -- Graham St Jack
Dec 14 2010
next sibling parent reply Don <nospam nospam.com> writes:
Graham St Jack wrote:
 On 14/12/10 20:33, Vladimir Panteleev wrote:
 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:

 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.
TBH, I'm more excited by AST macros which I understood are planned for D3, as mentioned here: http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf They seem to promise the power of mixins, but without the mess.
I took a look at the pdf, but couldn't see how the AST macros could come close to the kinds of things that are possible (but difficult) with mixins.
That fact was recognized at the conference, on the following day. As a result, AST macros were dropped from D2. They need to roughly match string mixins in power. At this stage, there is no proposal for how they should work.
Dec 14 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-14 13:05, Don wrote:
 Graham St Jack wrote:
 On 14/12/10 20:33, Vladimir Panteleev wrote:
 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:

 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.
TBH, I'm more excited by AST macros which I understood are planned for D3, as mentioned here: http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf They seem to promise the power of mixins, but without the mess.
I took a look at the pdf, but couldn't see how the AST macros could come close to the kinds of things that are possible (but difficult) with mixins.
That fact was recognized at the conference, on the following day. As a result, AST macros were dropped from D2.
Do you have an example that would work with string mixins but not with AST macros?
 They need to roughly match string mixins in power. At this stage, there
 is no proposal for how they should work.
I think someone, Nick Sabalausky perhaps, suggested to have something like the hygiene macros in Nemerle: http://nemerle.org/wiki/index.php?title=Macros -- /Jacob Carlborg
Dec 14 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:ie8dpq$kfg$2 digitalmars.com...
 On 2010-12-14 13:05, Don wrote:
 Graham St Jack wrote:
 On 14/12/10 20:33, Vladimir Panteleev wrote:
 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:

 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.
TBH, I'm more excited by AST macros which I understood are planned for D3, as mentioned here: http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf They seem to promise the power of mixins, but without the mess.
I took a look at the pdf, but couldn't see how the AST macros could come close to the kinds of things that are possible (but difficult) with mixins.
That fact was recognized at the conference, on the following day. As a result, AST macros were dropped from D2.
Do you have an example that would work with string mixins but not with AST macros?
 They need to roughly match string mixins in power. At this stage, there
 is no proposal for how they should work.
I think someone, Nick Sabalausky perhaps, suggested to have something like the hygiene macros in Nemerle: http://nemerle.org/wiki/index.php?title=Macros
Though I'm a huge D/systems-language/native-compiled guy, Nemerle's macros and pattern matching are two things I'm very jealous of and have made it very tempting to put up with the CLR for certain things.
Dec 14 2010
prev sibling parent reply Don <nospam nospam.com> writes:
Jacob Carlborg wrote:
 On 2010-12-14 13:05, Don wrote:
 Graham St Jack wrote:
 On 14/12/10 20:33, Vladimir Panteleev wrote:
 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:

 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.
TBH, I'm more excited by AST macros which I understood are planned for D3, as mentioned here: http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf They seem to promise the power of mixins, but without the mess.
I took a look at the pdf, but couldn't see how the AST macros could come close to the kinds of things that are possible (but difficult) with mixins.
That fact was recognized at the conference, on the following day. As a result, AST macros were dropped from D2.
Do you have an example that would work with string mixins but not with AST macros?
Well, it's a bit hard without a formal definition of AST macros. But the stuff I talked about at the conference, I have no idea how to do with AST macros. There's code like this, which generates an asm instruction. ------ mixin( opToSSE[operations[done+1]] ~ suffix ~ " " ~ XMM(numOnStack-1) ~ ", " ~ indexedSSEVector(ranklist, operations[done], vectorsize)); ------- using the functions: ============ const char [][5] vectorRegister = ["ECX", "EDX", "EBX", "ESI", "EDI"]; char [] indexedSSEVector(char [] ranklist, char var, char [] vecsize) { return "[" ~ vectorRegister[vectorNum(ranklist, var)] ~ " + " ~ vecsize ~"*EAX]"; } char [] XMM(int k) { return "XMM"~ itoa(k); } char [][char] opToSSE() { return ['*':"mulp"[], '+': "addp", '-': "subp", '/': "divp"]; } int vectorNum(char [] ranklist, char var) { int numVecs=0; for (int i=0; i<var-'A'; ++i) { if (ranklist[i]=='1') ++numVecs; } return numVecs; } ============
 They need to roughly match string mixins in power. At this stage, there
 is no proposal for how they should work.
I think someone, Nick Sabalausky perhaps, suggested to have something like the hygiene macros in Nemerle: http://nemerle.org/wiki/index.php?title=Macros
From an implementation point of view, the differences between Nemerle macros and string mixins are mostly syntactic. The one thing about them that I find really impressive is the IDE integration, especially that they got syntax highlighting to work. I don't know they've done that.
Dec 14 2010
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-15 08:57, Don wrote:
 Jacob Carlborg wrote:
 On 2010-12-14 13:05, Don wrote:
 Graham St Jack wrote:
 On 14/12/10 20:33, Vladimir Panteleev wrote:
 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:

 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code
 everywhere
 instead of just deep in the bowels of framework libraries.
TBH, I'm more excited by AST macros which I understood are planned for D3, as mentioned here: http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf They seem to promise the power of mixins, but without the mess.
I took a look at the pdf, but couldn't see how the AST macros could come close to the kinds of things that are possible (but difficult) with mixins.
That fact was recognized at the conference, on the following day. As a result, AST macros were dropped from D2.
Do you have an example that would work with string mixins but not with AST macros?
Well, it's a bit hard without a formal definition of AST macros. But the stuff I talked about at the conference, I have no idea how to do with AST macros. There's code like this, which generates an asm instruction. ------ mixin( opToSSE[operations[done+1]] ~ suffix ~ " " ~ XMM(numOnStack-1) ~ ", " ~ indexedSSEVector(ranklist, operations[done], vectorsize)); ------- using the functions: ============ const char [][5] vectorRegister = ["ECX", "EDX", "EBX", "ESI", "EDI"]; char [] indexedSSEVector(char [] ranklist, char var, char [] vecsize) { return "[" ~ vectorRegister[vectorNum(ranklist, var)] ~ " + " ~ vecsize ~"*EAX]"; } char [] XMM(int k) { return "XMM"~ itoa(k); } char [][char] opToSSE() { return ['*':"mulp"[], '+': "addp", '-': "subp", '/': "divp"]; } int vectorNum(char [] ranklist, char var) { int numVecs=0; for (int i=0; i<var-'A'; ++i) { if (ranklist[i]=='1') ++numVecs; } return numVecs; } ============
I can't quite visualize how the final code will look like and as you say it's hard without a formal definition of AST macros. But I think somehow it would be possible, I mean instead of combining strings one combine expressions/syntax. But I don't know if it would be possible to combine incomplete expressions.
 They need to roughly match string mixins in power. At this stage, there
 is no proposal for how they should work.
I think someone, Nick Sabalausky perhaps, suggested to have something like the hygiene macros in Nemerle: http://nemerle.org/wiki/index.php?title=Macros
From an implementation point of view, the differences between Nemerle macros and string mixins are mostly syntactic. The one thing about them that I find really impressive is the IDE integration, especially that they got syntax highlighting to work. I don't know they've done that.
As far as I can see the content of a macro in Nemerle is just code. But if you're referring to the syntax expression/statement which adds new syntax to the language then I agree. -- /Jacob Carlborg
Dec 16 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:iednio$2vj5$1 digitalmars.com...
 I can't quite visualize how the final code will look like and as you say 
 it's hard without a formal definition of AST macros. But I think somehow 
 it would be possible, I mean instead of combining strings one combine 
 expressions/syntax. But I don't know if it would be possible to combine 
 incomplete expressions.
One parallel that may or may not be applicable, but might be worth considering, is dynamically building XHTML: Using a string-template system is generally found to work very well, but building it via a DOM (essentially an AST) is often considered a bit of a pain. I guess the simplest take-away would be that string-based approaches may be easier get working well. FWIW.
Dec 16 2010
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2010-12-16 21:13, Nick Sabalausky wrote:
 "Jacob Carlborg"<doob me.com>  wrote in message
 news:iednio$2vj5$1 digitalmars.com...
 I can't quite visualize how the final code will look like and as you say
 it's hard without a formal definition of AST macros. But I think somehow
 it would be possible, I mean instead of combining strings one combine
 expressions/syntax. But I don't know if it would be possible to combine
 incomplete expressions.
One parallel that may or may not be applicable, but might be worth considering, is dynamically building XHTML: Using a string-template system is generally found to work very well, but building it via a DOM (essentially an AST) is often considered a bit of a pain. I guess the simplest take-away would be that string-based approaches may be easier get working well. FWIW.
I the case of XML I think it can be quite easy if you use the right libraries/tools. I think the easiest library I've used for building XML files is the Ruby library Bulilder, a code example using Builder can look like this: xml = Builder::XmlMarkup.new xml.person do xml.first_name "John" xml.last_name "Doe" xml.phone "5484654", :type => "mobile" end Which will generate this: <person> <first_name>John</first_name> <last_name>Doe</last_name> <phone type="mobile">5484654</phone> </person Now I don't think that this library is a DOM library, which allows you to manipulate a DOM tree, it's something simpler that just generates XML. Link: http://builder.rubyforge.org/ -- /Jacob Carlborg
Dec 16 2010
prev sibling parent Lutger Blijdestijn <lutger.blijdestijn gmail.com> writes:
Nick Sabalausky wrote:

 "Jacob Carlborg" <doob me.com> wrote in message
 news:iednio$2vj5$1 digitalmars.com...
 I can't quite visualize how the final code will look like and as you say
 it's hard without a formal definition of AST macros. But I think somehow
 it would be possible, I mean instead of combining strings one combine
 expressions/syntax. But I don't know if it would be possible to combine
 incomplete expressions.
One parallel that may or may not be applicable, but might be worth considering, is dynamically building XHTML: Using a string-template system is generally found to work very well, but building it via a DOM (essentially an AST) is often considered a bit of a pain. I guess the simplest take-away would be that string-based approaches may be easier get working well. FWIW.
The newest vb.net has this feature, if I understand you correctly, via xml literals and linq expressions. It is quite convenient. http://msdn.microsoft.com/en-us/library/bb384460.aspx
Dec 17 2010
prev sibling parent reply VladD2 <vc rsdn.ru> writes:
Hi,

I'm one of Nemerle developers.

 From an implementation point of view, the differences between Nemerle
 macros and string mixins are mostly syntactic.
You are deeply mistaken! I'd advise to study the Nemerle macro-system. You missed: pattern matching, quasi-quotation, running of fully functional Nemerle code in compile time, access to compiler API in macros and IntelliSense support.
The one thing about them that I find really impressive is the IDE
integration, especially that they got syntax highlighting to work.
I don't know they've done that.
"syntax highlighting" ? :) We have full IntelliSense support: syntax highlighting, code completion, macro expansion (in tooltips). We simply use the Nemerle compiler to achive that.
Dec 17 2010
parent reply Don <nospam nospam.com> writes:
VladD2 wrote:
 Hi,
 
 I'm one of Nemerle developers.
Cool!
 From an implementation point of view, the differences between Nemerle
 macros and string mixins are mostly syntactic.
You are deeply mistaken! I'd advise to study the Nemerle macro-system. You missed: pattern matching,
Yes, you've got me there. I've assumed that pattern matching, while a major feature, is not fundamental to the Nemerle macro-system, but I may be mistaken. One obvious difference which *is* fundamental is that Nemerle macros allow new syntax. Am I correct in thinking that Nemerle always requires complete ASTs? That is, given a name "x", can you access variables "x1", "x2", "x3" ?
 quasi-quotation,
No, that's present in D. It's the primary reason I say that the differences are mostly syntactic, since I see this as THE fundamental feature. You need to be able to reenter the macro system after you have left it. Once you can do that, you can do pretty much anything. BTW if you argued that D's current syntax is quite horrible, I couldn't disagree with you.
 running of fully functional Nemerle
 code in compile time, 
Yes, but that's a different issue. In D, running code at compile time is regarded as an aspect of constant-folding, and is not restricted to macros.
 access to compiler API in macros
In practice, has to be a library, right? Otherwise the compiler internals would be exposed? (This is an issue we're struggling with). Another one of the big differences is that D doesn't allow compile-time code to call external functions. Although it could certainly be done, it raises some big issues. Eg, we cannot assume that the target CPU is the same as the one we're running on. With a JIT compiler, you don't have that problem.
 and IntelliSense support.
I did mention that...
 The one thing about them that I find really impressive is the IDE
 integration, especially that they got syntax highlighting to work.
 I don't know they've done that.
"syntax highlighting" ? :) We have full IntelliSense support: syntax highlighting, code completion, macro expansion (in tooltips).
I meant Intellisense, not just syntax highlighting. Though note that you can't do syntax highlighting *perfectly* unless the IDE understands the code.
 We simply use the Nemerle compiler to achive that.
Doesn't leave me any less impressed. <g>
Dec 17 2010
parent reply foobar <foo bar.com> writes:
Don Wrote:

 VladD2 wrote:
 Hi,
 
 I'm one of Nemerle developers.
Cool!
 From an implementation point of view, the differences between Nemerle
 macros and string mixins are mostly syntactic.
You are deeply mistaken! I'd advise to study the Nemerle macro-system. You missed: pattern matching,
Yes, you've got me there. I've assumed that pattern matching, while a major feature, is not fundamental to the Nemerle macro-system, but I may be mistaken. One obvious difference which *is* fundamental is that Nemerle macros allow new syntax. Am I correct in thinking that Nemerle always requires complete ASTs? That is, given a name "x", can you access variables "x1", "x2", "x3" ? > quasi-quotation, No, that's present in D. It's the primary reason I say that the differences are mostly syntactic, since I see this as THE fundamental feature. You need to be able to reenter the macro system after you have left it. Once you can do that, you can do pretty much anything. BTW if you argued that D's current syntax is quite horrible, I couldn't disagree with you.
 running of fully functional Nemerle
 code in compile time, 
Yes, but that's a different issue. In D, running code at compile time is regarded as an aspect of constant-folding, and is not restricted to macros.
 access to compiler API in macros
In practice, has to be a library, right? Otherwise the compiler internals would be exposed? (This is an issue we're struggling with). Another one of the big differences is that D doesn't allow compile-time code to call external functions. Although it could certainly be done, it raises some big issues. Eg, we cannot assume that the target CPU is the same as the one we're running on. With a JIT compiler, you don't have that problem.
Don, can you please elaborate on this point? Here's my understanding: The D compiler is run once to both 'execute' compile time code which you refer above as constant folding AND to generate the binary to execute at run-time. Nemerle separates this into two distinct steps: 1. you compile regular code inside macro definitions into a compiler plugin. 2. when compiling the intended run-time code you need to load the compiled macros from step 1 above on the command line of the compiler. Since you're talking above about cross-compilation and let's say we run the compiler on X and compile for Y, I see no problem to load precompiled macros for X in order to compile the code for Y. The only limit as far as I can see is that it won't be possible to load macros compiled by the such a cross compiler on X unless as you say JIT is employed. Perhaps it makes sense to disable macro definition in a cross-compilation scenario and only allow usage. Is there something else that I'm missing here?
 
 and IntelliSense support.
I did mention that...
 The one thing about them that I find really impressive is the IDE
 integration, especially that they got syntax highlighting to work.
 I don't know they've done that.
"syntax highlighting" ? :) We have full IntelliSense support: syntax highlighting, code completion, macro expansion (in tooltips).
I meant Intellisense, not just syntax highlighting. Though note that you can't do syntax highlighting *perfectly* unless the IDE understands the code.
 We simply use the Nemerle compiler to achive that.
Doesn't leave me any less impressed. <g>
Dec 17 2010
parent reply Don <nospam nospam.com> writes:
foobar wrote:
 Don Wrote:
 Another one of the big differences is that D doesn't allow compile-time 
 code to call external functions. Although it could certainly be done, it 
 raises some big issues. Eg, we cannot assume that the target CPU is the 
 same as the one we're running on. With a JIT compiler, you don't have 
 that problem.
Don, can you please elaborate on this point? Here's my understanding: The D compiler is run once to both 'execute' compile time code which you refer above as constant folding AND to generate the binary to execute at run-time.
 
 Nemerle separates this into two distinct steps:
 1. you compile regular code inside macro definitions into a compiler plugin. 
 2. when compiling the intended run-time code you need to load the compiled
macros from step 1 above on the command line of the compiler.
 
 Since you're talking above about cross-compilation and let's say we run the
compiler on X and compile for Y, I see no problem to load precompiled macros
for X in order to compile the code for Y. The only limit as far as I can see is
that it won't be possible to load macros compiled by the such a cross compiler
on X unless as you say JIT is employed. Perhaps it makes sense to disable macro
definition in a cross-compilation scenario and only allow usage.
Suppose the pre-compiled code, when run, asks what CPU it's on. What's the answer? Is it X or Y?
Dec 17 2010
parent reply VladD2 <vc rsdn.ru> writes:
Don Wrote:
 Suppose the pre-compiled code, when run, asks what CPU it's on. What's 
 the answer? Is it X or Y?
Current: X Target: Y Macro - a plugin to the compiler. It works on the same platform as the compiler, but generates code through the API which abstracts the macro from the target platform. If you need generate platform specific code, you should worry about it in macro logic. In any case macro is a meta-programm wich generate or/and transform code.
Dec 17 2010
parent reply Don <nospam nospam.com> writes:
VladD2 wrote:
 Don Wrote:
 Suppose the pre-compiled code, when run, asks what CPU it's on. What's 
 the answer? Is it X or Y?
Current: X Target: Y Macro - a plugin to the compiler. It works on the same platform as the compiler, but generates code through the API which abstracts the macro from the target platform. If you need generate platform specific code, you should worry about it in macro logic. In any case macro is a meta-programm wich generate or/and transform code.
Yes. But in D there's no distinction between code which is destined for a macro, versus any other function. You can call a function once at compile time, and the same function at compile time. My understanding of Nemerle (which is quite likely to be wrong!) is that at least some functions are callable only at compile-time. I'm also scared of the implications of allowing arbitrary code execution during compilation. Make a typo in your program, and then compilation may wipe files from your hard disk, or corrupt an external database, etc... On some platforms you may be able to sandbox it, but since it's running as part of the compilation process, rather than with the permissions it will eventually have, it just seems like a security nightmare. All these problems would be reduced somewhat if it were only permissible to call pure functions at compile time. Although even that has issues, since not all compiler platforms may allow calling of shared libraries. If we build that into the language, we'd be cutting ourselves off from those platforms. I'm not certain that there are any unsurmountable problems, but these issues make me really cautious.
Dec 18 2010
parent reply foobar <foo bar.com> writes:
Don Wrote:

 VladD2 wrote:
 Don Wrote:
 Suppose the pre-compiled code, when run, asks what CPU it's on. What's 
 the answer? Is it X or Y?
Current: X Target: Y Macro - a plugin to the compiler. It works on the same platform as the compiler, but generates code through the API which abstracts the macro from the target platform. If you need generate platform specific code, you should worry about it in macro logic. In any case macro is a meta-programm wich generate or/and transform code.
Yes. But in D there's no distinction between code which is destined for a macro, versus any other function. You can call a function once at compile time, and the same function at compile time. My understanding of Nemerle (which is quite likely to be wrong!) is that at least some functions are callable only at compile-time.
I don't see how there needs to be different code to accomplish the above use case. You have a function that tests the hardware it's being run on. When this function is called in the compiler context it would return X, when it's called from the target executable it returns Y.
 I'm also scared of the implications of allowing arbitrary code execution 
 during compilation. Make a typo in your program, and then compilation 
 may wipe files from your hard disk, or corrupt an external database, 
 etc... On some platforms you may be able to sandbox it, but since it's 
 running as part of the compilation process, rather than with the 
 permissions it will eventually have, it just seems like a security 
 nightmare.
 
This is a void argument since templates are Turing complete. It's *already* possible to do all of the above in D at compile time, it's just a matter of how much code is required to accomplish this.
 All these problems would be reduced somewhat if it were only permissible 
 to call pure functions at compile time. Although even that has issues, 
 since not all compiler platforms may allow calling of shared libraries. 
 If we build that into the language, we'd be cutting ourselves off from 
 those platforms.
 
 I'm not certain that there are any unsurmountable problems, but these 
 issues make me really cautious.
Dec 18 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"foobar" <foo bar.com> wrote in message 
news:ieijt6$21mh$1 digitalmars.com...
 Don Wrote:

 VladD2 wrote:
 Don Wrote:
 Suppose the pre-compiled code, when run, asks what CPU it's on. What's
 the answer? Is it X or Y?
Current: X Target: Y Macro - a plugin to the compiler. It works on the same platform as the compiler, but generates code through the API which abstracts the macro from the target platform. If you need generate platform specific code, you should worry about it in macro logic. In any case macro is a meta-programm wich generate or/and transform code.
Yes. But in D there's no distinction between code which is destined for a macro, versus any other function. You can call a function once at compile time, and the same function at compile time.
I think you mean "You can call a function once at compile time, and the same function at **runtime**."
 My understanding of
 Nemerle (which is quite likely to be wrong!) is that at least some
 functions are callable only at compile-time.
I'd be surprised. I would think that all you would have to do to use the same Nemerle function at both runtime and compile-time would be to include its module in both the "compile the compiler-plugin step" and in the "load compiler-plugins and compile the app" step.
 I don't see how there needs to be different code to accomplish the above 
 use case. You have a function that tests the hardware it's being run on. 
 When this function is called in the compiler context it would return X, 
 when it's called from the target executable it returns Y.
The problem with that is, what if you're generating target-platform-specific code at compile-time? You'd be generating code for the wrong platform. I think VladD2 is right: You need to keep track of both "current" system and "target" system. Unfortunately, there is some information about the "target" system the compile-time code wouldn't be able discern without giving it the ability to run code (RPC? Virtualization? Really, really good emulator?) on the target system, but then again, that's a limitation with any cross-compiling scenario.
 I'm also scared of the implications of allowing arbitrary code execution
 during compilation. Make a typo in your program, and then compilation
 may wipe files from your hard disk, or corrupt an external database,
 etc... On some platforms you may be able to sandbox it, but since it's
 running as part of the compilation process, rather than with the
 permissions it will eventually have, it just seems like a security
 nightmare.
That's an interesting point. OTOH, it's common for software to come with its own build script or makefile, and those can certainly do literally anything like you describe above - but I haven't seen that as being a real problem.
 This is a void argument since templates are Turing complete.
 It's *already* possible to do all of the above in D at compile time, it's 
 just a matter of how much code is required to accomplish this.
Not true. This is a frequent misconception about Turning-completeness. Just because something is Turing complete does *not* mean it can do anything that any other Turing complete system can do (contrary to how many college profs explain it). It *does* mean that it can *calculate* anything that any other Turing complete system can calculate. But it doesn't necessarily have *access* to everything that any other Turing complete system has access to. And D's compile-time system, whether CTFE or templates, does not currently provide any way to access any I/O, launch any processes, or do any direct memory access.
Dec 18 2010
parent reply Don <nospam nospam.com> writes:
Nick Sabalausky wrote:
 "foobar" <foo bar.com> wrote in message 
 news:ieijt6$21mh$1 digitalmars.com...
 Don Wrote:

 VladD2 wrote:
 Don Wrote:
 Suppose the pre-compiled code, when run, asks what CPU it's on. What's
 the answer? Is it X or Y?
Current: X Target: Y Macro - a plugin to the compiler. It works on the same platform as the compiler, but generates code through the API which abstracts the macro from the target platform. If you need generate platform specific code, you should worry about it in macro logic. In any case macro is a meta-programm wich generate or/and transform code.
Yes. But in D there's no distinction between code which is destined for a macro, versus any other function. You can call a function once at compile time, and the same function at compile time.
I think you mean "You can call a function once at compile time, and the same function at **runtime**."
 My understanding of
 Nemerle (which is quite likely to be wrong!) is that at least some
 functions are callable only at compile-time.
I'd be surprised. I would think that all you would have to do to use the same Nemerle function at both runtime and compile-time would be to include its module in both the "compile the compiler-plugin step" and in the "load compiler-plugins and compile the app" step.
 I don't see how there needs to be different code to accomplish the above 
 use case. You have a function that tests the hardware it's being run on. 
 When this function is called in the compiler context it would return X, 
 when it's called from the target executable it returns Y.
The problem with that is, what if you're generating target-platform-specific code at compile-time? You'd be generating code for the wrong platform. I think VladD2 is right: You need to keep track of both "current" system and "target" system. Unfortunately, there is some information about the "target" system the compile-time code wouldn't be able discern without giving it the ability to run code (RPC? Virtualization? Really, really good emulator?) on the target system, but then again, that's a limitation with any cross-compiling scenario.
Note that for this to work at all, the compiler needs to be able to generate exectuable code for platform X as well as for Y -- that is, it needs to include two back-ends.
 I'm also scared of the implications of allowing arbitrary code execution
 during compilation. Make a typo in your program, and then compilation
 may wipe files from your hard disk, or corrupt an external database,
 etc... On some platforms you may be able to sandbox it, but since it's
 running as part of the compilation process, rather than with the
 permissions it will eventually have, it just seems like a security
 nightmare.
That's an interesting point. OTOH, it's common for software to come with its own build script or makefile, and those can certainly do literally anything like you describe above - but I haven't seen that as being a real problem.
I don't think it's quite the same. In a makefile, every executable is listed, and so you can have some degree of control over it. But in this scenario, the compiler is making calls to arbitrary shared libraries with arbitrary parameters. It means the compiler cannot be trusted *at all*.
Dec 18 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"Don" <nospam nospam.com> wrote in message 
news:iej7eu$b1l$1 digitalmars.com...
 Nick Sabalausky wrote:
 "foobar" <foo bar.com> wrote in message 
 news:ieijt6$21mh$1 digitalmars.com...
 I don't see how there needs to be different code to accomplish the above 
 use case. You have a function that tests the hardware it's being run on. 
 When this function is called in the compiler context it would return X, 
 when it's called from the target executable it returns Y.
The problem with that is, what if you're generating target-platform-specific code at compile-time? You'd be generating code for the wrong platform. I think VladD2 is right: You need to keep track of both "current" system and "target" system. Unfortunately, there is some information about the "target" system the compile-time code wouldn't be able discern without giving it the ability to run code (RPC? Virtualization? Really, really good emulator?) on the target system, but then again, that's a limitation with any cross-compiling scenario.
Note that for this to work at all, the compiler needs to be able to generate exectuable code for platform X as well as for Y -- that is, it needs to include two back-ends.
But if the compiler doesn't have both backends then the whole question of "how does the user's compile-time code get handled if it's being cross-compiled?" becomes irrelevent - you'd have to compile it on the target plaform anyway, so X == Y. Or am I missing your point?
 I'm also scared of the implications of allowing arbitrary code 
 execution
 during compilation. Make a typo in your program, and then compilation
 may wipe files from your hard disk, or corrupt an external database,
 etc... On some platforms you may be able to sandbox it, but since it's
 running as part of the compilation process, rather than with the
 permissions it will eventually have, it just seems like a security
 nightmare.
That's an interesting point. OTOH, it's common for software to come with its own build script or makefile, and those can certainly do literally anything like you describe above - but I haven't seen that as being a real problem.
I don't think it's quite the same. In a makefile, every executable is listed, and so you can have some degree of control over it. But in this scenario, the compiler is making calls to arbitrary shared libraries with arbitrary parameters. It means the compiler cannot be trusted *at all*.
I suppose that's a reasonable point.
Dec 18 2010
prev sibling next sibling parent reply VladD2 <vc rsdn.ru> writes:
Don Wrote:

 I 
 think VladD2 is right: You need to keep track of both "current" system and 
 "target" system. Unfortunately, there is some information about the "target" 
 system the compile-time code wouldn't be able discern without giving it the 
 ability to run code (RPC? Virtualization? Really, really good emulator?) on 
 the target system, but then again, that's a limitation with any 
 cross-compiling scenario.
Note that for this to work at all, the compiler needs to be able to generate exectuable code for platform X as well as for Y -- that is, it needs to include two back-ends.
If the macros have been compiled and are in binary (executable) form, the compiler must only be able to generate code for platform X, and run macros (execute code from DLL). This is exactly what makes Nemerle compiler. In this case, compiling of the same macros looks like any other compilation process (on the platform X for the platform Y).
 I don't think it's quite the same. In a makefile, every executable is 
 listed, and so you can have some degree of control over it. 
Trust to rmdir ... lol! And what about NAnt or MSBuild which can have binary extensions? I think, you are completely wrong.
 But in this 
 scenario, the compiler is making calls to arbitrary shared libraries 
 with arbitrary parameters.
 It means the compiler cannot be trusted *at all*.
The experience of Lisp (50 years!) and Nemerel (about 6 years) shows that the ability to access any library - is not a problem. This is a huge advantage. And limit the possibility of a macro, you can simply forbidding them to use some libraries.
Dec 19 2010
parent Don <nospam nospam.com> writes:
VladD2 wrote:
 Don Wrote:
 
 I 
 think VladD2 is right: You need to keep track of both "current" system and 
 "target" system. Unfortunately, there is some information about the "target" 
 system the compile-time code wouldn't be able discern without giving it the 
 ability to run code (RPC? Virtualization? Really, really good emulator?) on 
 the target system, but then again, that's a limitation with any 
 cross-compiling scenario.
Note that for this to work at all, the compiler needs to be able to generate exectuable code for platform X as well as for Y -- that is, it needs to include two back-ends.
If the macros have been compiled and are in binary (executable) form, the compiler must only be able to generate code for platform X,
Yes, but it's not a compiler for platform X! It's only a compiler for platform Y.
 and run macros (execute code from DLL). This is exactly what makes Nemerle
compiler.
The .NET system always has a compiler for the platform it's running on. That's not necessarily true for D compilers.
 In this case, compiling of the same macros looks like any other compilation
process (on the platform X for the platform Y).
 
 
 I don't think it's quite the same. In a makefile, every executable is 
 listed, and so you can have some degree of control over it. 
Trust to rmdir ... lol! And what about NAnt or MSBuild which can have binary extensions? I think, you are completely wrong.
 But in this 
 scenario, the compiler is making calls to arbitrary shared libraries 
 with arbitrary parameters.
 It means the compiler cannot be trusted *at all*.
The experience of Lisp (50 years!) and Nemerel (about 6 years) shows that the ability to access any library - is not a problem.
I don't think Nemerle has been sufficiently widely used, to be able to draw strong conclusions from it. The Lisp argument is strong though.
 This is a huge advantage.
 
 And limit the possibility of a macro, you can simply forbidding them to use
some libraries. 
I hope you're right, because it's indeed a powerful feature. But I'd want to hear the opinion of a security expert. In particular, if it can be shown that it's exactly the same as Lisp, I would be convinced.
Dec 20 2010
prev sibling parent reply "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"Don" <nospam nospam.com> wrote:
 I don't think it's quite the same. In a makefile, every executable is 
 listed, and so you can have some degree of control over it. But in this 
 scenario, the compiler is making calls to arbitrary shared libraries with 
 arbitrary parameters.
 It means the compiler cannot be trusted *at all*.
You are right only partially - it's unsafe for browser language where code is taken from untrusted source. But this feature gives so much power to the macro sysrem - that I think is worth considering it. IMO, usually compiled code is run just after compilation (with the same prermissions as compiler) - so compiled code can make dangerous things and can't be trusted at all, but no one is worry about that. Yes compiler can't be *trusted* with this features, but if one knows what he is doing, why to prevent him - add option --enable-ctfe-DANGEROUS-features to allow potentially dangerous features then it wouldn't be so unexpected. Are those features hard to add to the current implementation?
Dec 19 2010
parent reply Don <nospam nospam.com> writes:
Alex_Dovhal wrote:
 "Don" <nospam nospam.com> wrote:
 I don't think it's quite the same. In a makefile, every executable is 
 listed, and so you can have some degree of control over it. But in this 
 scenario, the compiler is making calls to arbitrary shared libraries with 
 arbitrary parameters.
 It means the compiler cannot be trusted *at all*.
You are right only partially - it's unsafe for browser language where code is taken from untrusted source. But this feature gives so much power to the macro sysrem - that I think is worth considering it. IMO, usually compiled code is run just after compilation (with the same prermissions as compiler) - so compiled code can make dangerous things and can't be trusted at all, but no one is worry about that. Yes compiler can't be *trusted* with this features, but if one knows what he is doing, why to prevent him - add option --enable-ctfe-DANGEROUS-features to allow potentially dangerous features then it wouldn't be so unexpected. Are those features hard to add to the current implementation?
In order for CTFE code to call pre-compiled code, three things are required: (1) the compiler needs to be able to find the file (.obj/.lib/shared library) containing the compiled code; (2) the compiler needs to be able to load the module and call it. This requires some form of dynamic linking. (3) We need a marshalling step, to convert from compiler literal to compiled data, and back. Step (3) is straightforward. The challenge is step(2), although note that it's a general "allow the compiler to load a plugin" problem, and doesn't have much to do with CTFE.
Dec 20 2010
next sibling parent "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"Don" <nospam nospam.com> wrote:
 In order for CTFE code to call pre-compiled code, three things are 
 required:
 (1) the compiler needs to be able to find the file (.obj/.lib/shared 
 library) containing the compiled code;
 (2) the compiler needs to be able to load the module and call it. This 
 requires some form of dynamic linking.
 (3) We need a marshalling step, to convert from compiler literal to 
 compiled data, and back.


 Step (3) is straightforward. The challenge is step(2), although note that 
 it's a general "allow the compiler to load a plugin" problem, and doesn't 
 have much to do with CTFE.
Understand. So, it should be dynamic loaded, compiler should know which D library to load for used function and this function's name mangling, also then phobos should be dynamic library to call it's functions in macro. This is non trivial stuff, and compiler itselt is written in C++ so this plugin architecture should be working in C++ too. Also when cross-compile it's neeaded compiler for both X and Y architectures or two compilers, communicating among them. So that compiler for Y when finds macro should call compiler X and dynamically load to itself produced function. OK, IMO it's too complex and experimental to be of any priority in nearest future.
Dec 20 2010
prev sibling parent reply "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"Don" <nospam nospam.com> wrote:
 In order for CTFE code to call pre-compiled code, three things are 
 required:
 (1) the compiler needs to be able to find the file (.obj/.lib/shared 
 library) containing the compiled code;
 (2) the compiler needs to be able to load the module and call it. This 
 requires some form of dynamic linking.
 (3) We need a marshalling step, to convert from compiler literal to 
 compiled data, and back.


 Step (3) is straightforward. The challenge is step(2), although note that 
 it's a general "allow the compiler to load a plugin" problem, and doesn't 
 have much to do with CTFE.
I thought it over, and got: (1) can be solved by adding compiler option: --macro-libs=<list_of_libs>, so the compiler knows in which dynamic libraries to search for the plugins ; (2) plugin library functions should be *stdcall*, so C++ compiler can load them. That library should implement function like that: PluginInfo* getFunctionsInfo () ; where: typedef struct _PluginInfo{ struct _PluginInfo * nextPlugin ; char* fcnName ; // short name of a function, e.g. "fcn" char* params ; // param list, e.g. "(int, float, char*)" char* mangledName;//name of funct. in the library, e.g. "_fcn 12" char* returnType ; // e.g. "int" bool isNothrow ; // if useful? bool isPure ; // ditto //... etc ... } PluginInfo ; And also should implement all the functions, info about which is returned by getFunctionsInfo(). D compiler calls getFunctionsInfo from each plugin library - and then loads all the implemented functions with full info about them. If one plugin function name found in two or more libraries - D throws compiler error. Not a perfect solution, but at least straightforward one and solves given problem. One more note: this approach makes CTFE functions calling such plugins uncallable at runtime, which IMO is OK, as one should not call it in any case. Is that suitable?
Dec 21 2010
parent reply Don <nospam nospam.com> writes:
Alex_Dovhal wrote:
 "Don" <nospam nospam.com> wrote:
 In order for CTFE code to call pre-compiled code, three things are 
 required:
 (1) the compiler needs to be able to find the file (.obj/.lib/shared 
 library) containing the compiled code;
 (2) the compiler needs to be able to load the module and call it. This 
 requires some form of dynamic linking.
 (3) We need a marshalling step, to convert from compiler literal to 
 compiled data, and back.


 Step (3) is straightforward. The challenge is step(2), although note that 
 it's a general "allow the compiler to load a plugin" problem, and doesn't 
 have much to do with CTFE.
I thought it over, and got: (1) can be solved by adding compiler option: --macro-libs=<list_of_libs>, so the compiler knows in which dynamic libraries to search for the plugins ; (2) plugin library functions should be *stdcall*, so C++ compiler can load them. That library should implement function like that: PluginInfo* getFunctionsInfo () ; where: typedef struct _PluginInfo{ struct _PluginInfo * nextPlugin ; char* fcnName ; // short name of a function, e.g. "fcn" char* params ; // param list, e.g. "(int, float, char*)" char* mangledName;//name of funct. in the library, e.g. "_fcn 12" char* returnType ; // e.g. "int" bool isNothrow ; // if useful? bool isPure ; // ditto //... etc ... } PluginInfo ; And also should implement all the functions, info about which is returned by getFunctionsInfo(). D compiler calls getFunctionsInfo from each plugin library - and then loads all the implemented functions with full info about them. If one plugin function name found in two or more libraries - D throws compiler error. Not a perfect solution, but at least straightforward one and solves given problem. One more note: this approach makes CTFE functions calling such plugins uncallable at runtime, which IMO is OK, as one should not call it in any case. Is that suitable?
It's not that complicated. Once you can load the library and call *one* function in it, the problem is solved. But at a deeper level, I'm not sure what you'd hope to achieve with this. I mean (excluding some not-yet implemented features), CTFE allows you to execute any pure + safe function which you have source code for. So, if you had such a plugin, it could do things like make database queries. Everything else you can do already.
Dec 21 2010
parent "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"Don" <nospam nospam.com> wrote:
 It's not that complicated. Once you can load the library and call *one* 
 function in it, the problem is solved.
 But at a deeper level, I'm not sure what you'd hope to achieve with this.
 I mean (excluding some not-yet implemented features), CTFE allows you to 
 execute any pure + safe function which you have source code for.

 So, if you had such a plugin, it could do things like make database 
 queries. Everything else you can do already.
May be you are right, I don't know. So yes - adding plugins is questionable feature now. I thought on it in general, and found several things they are suitable for, not sure if so useful: (1) To call external tools, e.g. SQL-syntax validator, DSL code parser ; (2) To load and handle resources at CT, e.g. (*) load XML, JSON, etc. generated by IDE GUI and convert it to D code; (*) load resources as handled data in inner program format, like: auto imgData = mixin (loadJPEG ("./img/photo1.jpeg")) ; (that seems useful) (3) make impure CTFE functions, e.g. one macro configures the state, and some/all other depend on it (like optimisation options, hardware resources, etc.) or cache result from call to call like fibonacci numbers. (4) Workaround CTFE limitations if really needed (questionable, as you pointed above).
Dec 22 2010
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-14 12:42, Graham St Jack wrote:
 On 14/12/10 20:33, Vladimir Panteleev wrote:
 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:

 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.
TBH, I'm more excited by AST macros which I understood are planned for D3, as mentioned here: http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf They seem to promise the power of mixins, but without the mess.
I took a look at the pdf, but couldn't see how the AST macros could come close to the kinds of things that are possible (but difficult) with mixins. Is there more information somewhere? Jacob, I can see how your proposed syntax would make simple mixins easier and clearer, but how would it do something more complex?
I don't know, do you have an example ?
 For example taking a classname and a bunch of field types and names, and
 turning it into a class definition complete with constructor from field
 values, constructor from an input stream, a method to write to an output
 stream, and const getters. Or maybe std.typecons.AutoImplement.
Could you post an example of how that mixin would be used and the code it would generate then I can see if I can translate it to my syntax. AutoImplement seems to just contain template mixins which is something else. -- /Jacob Carlborg
Dec 14 2010
parent reply Graham St Jack <Graham.StJack internode.on.net> writes:
 I don't know, do you have an example ?

 For example taking a classname and a bunch of field types and names, and
 turning it into a class definition complete with constructor from field
 values, constructor from an input stream, a method to write to an output
 stream, and const getters. Or maybe std.typecons.AutoImplement.
Could you post an example of how that mixin would be used and the code it would generate then I can see if I can translate it to my syntax. AutoImplement seems to just contain template mixins which is something else.
I have attached my concurrency framework, which relies heavily on mixins, plus its unit test to show how it is used. I haven't included the various dependencies because I assume you just want the example code. Let me know if you want something buildable, or perhaps something more cut-down. What the code-generating template does is to create from something like this (you can have any number of Messages in a Protocol): alias Protocol!("Requests", Message!("job", string, "name")).code jobCode; mixin(jobCode); this code: class Requests { struct jobMsg { string name; this(string name) { this.name = name; } void read(InStream stream) { name = stream.get!string; } void write(OutStream stream) { stream(name); } } struct Message { uint kind; union { jobMsg job; } this(ref jobMsg msg) { kind = 0; job = msg; } this(InStream stream) { kind = stream.get!uint; switch(kind) { case 0: job.read(stream); break; default: assert(0, "Cannot read unsupported message kind"); } } void write(OutStream stream) { stream(kind); switch(kind) { case 0: job.write(stream); break; default: assert(0, "Cannot write unsupported message kind"); } } } private alias Channel!(Message) _Chan; private alias shared _Chan Chan; private Chan channel; this() { channel = new Chan(); } ChannelSelectable newSelectable() { return channel.newSelectable(); } void finalize() { channel.finalize; } interface IHandler { void job(string name); } void job(string name) { channel.add(Message(jobMsg(name))); } void receive(IHandler handler) { auto message = channel.remove; switch (message.kind) { case 0: handler.job(message.job.name); break; default: assert(0, "Cannot dispatch unsupported message kind"); } } } I use this for inter-thread communications, and I use the discriminated union to pass messages between processes. The manual mixin after the alias is a debugging aid - the compiler errors when doing mixins aren't helpful at all. I case you are wondering why I did all this, it was partly a learning experience, but mostly an attempt to do something properly thread-safe (!hasAliasing), using shared, const and immutable properly. -- Graham St Jack
Dec 14 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-15 00:19, Graham St Jack wrote:
 I don't know, do you have an example ?

 For example taking a classname and a bunch of field types and names, and
 turning it into a class definition complete with constructor from field
 values, constructor from an input stream, a method to write to an output
 stream, and const getters. Or maybe std.typecons.AutoImplement.
Could you post an example of how that mixin would be used and the code it would generate then I can see if I can translate it to my syntax. AutoImplement seems to just contain template mixins which is something else.
I have attached my concurrency framework, which relies heavily on mixins, plus its unit test to show how it is used. I haven't included the various dependencies because I assume you just want the example code. Let me know if you want something buildable, or perhaps something more cut-down. What the code-generating template does is to create from something like this (you can have any number of Messages in a Protocol): alias Protocol!("Requests", Message!("job", string, "name")).code jobCode; mixin(jobCode); this code: class Requests { struct jobMsg { string name; this(string name) { this.name = name; } void read(InStream stream) { name = stream.get!string; } void write(OutStream stream) { stream(name); } } struct Message { uint kind; union { jobMsg job; } this(ref jobMsg msg) { kind = 0; job = msg; } this(InStream stream) { kind = stream.get!uint; switch(kind) { case 0: job.read(stream); break; default: assert(0, "Cannot read unsupported message kind"); } } void write(OutStream stream) { stream(kind); switch(kind) { case 0: job.write(stream); break; default: assert(0, "Cannot write unsupported message kind"); } } } private alias Channel!(Message) _Chan; private alias shared _Chan Chan; private Chan channel; this() { channel = new Chan(); } ChannelSelectable newSelectable() { return channel.newSelectable(); } void finalize() { channel.finalize; } interface IHandler { void job(string name); } void job(string name) { channel.add(Message(jobMsg(name))); } void receive(IHandler handler) { auto message = channel.remove; switch (message.kind) { case 0: handler.job(message.job.name); break; default: assert(0, "Cannot dispatch unsupported message kind"); } } } I use this for inter-thread communications, and I use the discriminated union to pass messages between processes. The manual mixin after the alias is a debugging aid - the compiler errors when doing mixins aren't helpful at all. I case you are wondering why I did all this, it was partly a learning experience, but mostly an attempt to do something properly thread-safe (!hasAliasing), using shared, const and immutable properly.
I've attached a part of how concurrency.d could look like translated to my suggested syntax. It probably contains a lot of errors because did a quick translation and I had some trouble understanding the mixins. -- /Jacob Carlborg
Dec 16 2010
parent reply Graham St Jack <Graham.StJack internode.on.net> writes:
 I've attached a part of how concurrency.d could look like translated 
 to my suggested syntax. It probably contains a lot of errors because 
 did a quick translation and I had some trouble understanding the mixins.
Yes, even I couldn't understand them even a week later. Maintenance is a real problem. My initial reaction is that the proposed syntax helps a bit, but isn't really a game-changer. I will let you know if I change my mind on closer inspection. -- Graham St Jack
Dec 16 2010
parent Jacob Carlborg <doob me.com> writes:
On 2010-12-17 00:34, Graham St Jack wrote:
 I've attached a part of how concurrency.d could look like translated
 to my suggested syntax. It probably contains a lot of errors because
 did a quick translation and I had some trouble understanding the mixins.
Yes, even I couldn't understand them even a week later. Maintenance is a real problem. My initial reaction is that the proposed syntax helps a bit, but isn't really a game-changer. I will let you know if I change my mind on closer inspection.
No it's basically just syntactic sugar. -- /Jacob Carlborg
Dec 18 2010
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2010-12-14 11:03, Vladimir Panteleev wrote:
 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:

 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.
TBH, I'm more excited by AST macros which I understood are planned for D3, as mentioned here: http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf They seem to promise the power of mixins, but without the mess.
I would like to have AST macros too but I was think my suggestion was perhaps easier to implement, just some syntactic sugar. -- /Jacob Carlborg
Dec 14 2010
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Graham St Jack" <Graham.StJack internode.on.net> wrote in message 
news:ie76ig$b2v$1 digitalmars.com...
 What you are suggesting here seems to be a way to dramatically reduce the 
 complexity of code that generates source-code and mixes it in. I think 
 something like that is needed before this mind-bogglingly powerful feature 
 of D can realise its potential.
I think a decent string-template library could probably come very close to the proposal without needing any language changes at all: string get_set(T, string name)() { return q{ type _ name; type name () { return _ name; } type name ( type name) { return _ name = name; } }.replace( [" type": T.stringof, " name": name] ); } class Foo { mixin(get_set!(int, "bar")()); } There are definitely some things about the proposal that are better than with this, but I just wanted to point out that the value of the proposal should probably be evaluated against something roughly like the above rather than something that does a bunch of procedural string splicing.
Dec 14 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-14 19:13, Nick Sabalausky wrote:
 "Graham St Jack"<Graham.StJack internode.on.net>  wrote in message
 news:ie76ig$b2v$1 digitalmars.com...
 What you are suggesting here seems to be a way to dramatically reduce the
 complexity of code that generates source-code and mixes it in. I think
 something like that is needed before this mind-bogglingly powerful feature
 of D can realise its potential.
I think a decent string-template library could probably come very close to the proposal without needing any language changes at all: string get_set(T, string name)() { return q{ type _ name; type name () { return _ name; } type name ( type name) { return _ name = name; } }.replace( [" type": T.stringof, " name": name] ); } class Foo { mixin(get_set!(int, "bar")()); } There are definitely some things about the proposal that are better than with this, but I just wanted to point out that the value of the proposal should probably be evaluated against something roughly like the above rather than something that does a bunch of procedural string splicing.
The whole point of the idea was to get rid of the strings and the mixin expression (as it looks like to day). -- /Jacob Carlborg
Dec 14 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:ie8i8c$15f0$1 digitalmars.com...
 On 2010-12-14 19:13, Nick Sabalausky wrote:
 "Graham St Jack"<Graham.StJack internode.on.net>  wrote in message
 news:ie76ig$b2v$1 digitalmars.com...
 What you are suggesting here seems to be a way to dramatically reduce 
 the
 complexity of code that generates source-code and mixes it in. I think
 something like that is needed before this mind-bogglingly powerful 
 feature
 of D can realise its potential.
I think a decent string-template library could probably come very close to the proposal without needing any language changes at all: string get_set(T, string name)() { return q{ type _ name; type name () { return _ name; } type name ( type name) { return _ name = name; } }.replace( [" type": T.stringof, " name": name] ); } class Foo { mixin(get_set!(int, "bar")()); } There are definitely some things about the proposal that are better than with this, but I just wanted to point out that the value of the proposal should probably be evaluated against something roughly like the above rather than something that does a bunch of procedural string splicing.
The whole point of the idea was to get rid of the strings and the mixin expression (as it looks like to day).
While I'm not necessarily opposed to the idea of getting rid of the strings, I guess I don't really see what improvement your proposal provides other than not having to manually specify the mapping of "replace what identifier with what data". Getting rid of the "mixin" I see as a separate issue. We could just as easily say that given the function "string get_set(...)": get_set("int", "bar"); or get_set int bar; ...Is shorthand for, or is even the new syntax for: mixin(get_set("int", "bar"));
Dec 14 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-14 21:44, Nick Sabalausky wrote:
 "Jacob Carlborg"<doob me.com>  wrote in message
 news:ie8i8c$15f0$1 digitalmars.com...
 On 2010-12-14 19:13, Nick Sabalausky wrote:
 "Graham St Jack"<Graham.StJack internode.on.net>   wrote in message
 news:ie76ig$b2v$1 digitalmars.com...
 What you are suggesting here seems to be a way to dramatically reduce
 the
 complexity of code that generates source-code and mixes it in. I think
 something like that is needed before this mind-bogglingly powerful
 feature
 of D can realise its potential.
I think a decent string-template library could probably come very close to the proposal without needing any language changes at all: string get_set(T, string name)() { return q{ type _ name; type name () { return _ name; } type name ( type name) { return _ name = name; } }.replace( [" type": T.stringof, " name": name] ); } class Foo { mixin(get_set!(int, "bar")()); } There are definitely some things about the proposal that are better than with this, but I just wanted to point out that the value of the proposal should probably be evaluated against something roughly like the above rather than something that does a bunch of procedural string splicing.
The whole point of the idea was to get rid of the strings and the mixin expression (as it looks like to day).
While I'm not necessarily opposed to the idea of getting rid of the strings, I guess I don't really see what improvement your proposal provides other than not having to manually specify the mapping of "replace what identifier with what data". Getting rid of the "mixin" I see as a separate issue. We could just as easily say that given the function "string get_set(...)": get_set("int", "bar"); or get_set int bar; ...Is shorthand for, or is even the new syntax for: mixin(get_set("int", "bar"));
That was my idea as well, that get_set("int", "bar"); could be translated into mixin(get_set("int", "bar")); just like just like scope statements are translated into try/catch/finally. -- /Jacob Carlborg
Dec 15 2010
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:
 On 2010-12-14 21:44, Nick Sabalausky wrote:
 "Jacob Carlborg"<doob me.com>  wrote in message
 news:ie8i8c$15f0$1 digitalmars.com...
 
 On 2010-12-14 19:13, Nick Sabalausky wrote:
 "Graham St Jack"<Graham.StJack internode.on.net>   wrote in message
 news:ie76ig$b2v$1 digitalmars.com...
 
 What you are suggesting here seems to be a way to dramatically reduce
 the
 complexity of code that generates source-code and mixes it in. I think
 something like that is needed before this mind-bogglingly powerful
 feature
 of D can realise its potential.
I think a decent string-template library could probably come very close to the proposal without needing any language changes at all: string get_set(T, string name)() { return q{ type _ name; type name () { return _ name; } type name ( type name) { return _ name = name; } }.replace( [" type": T.stringof, " name": name] ); } class Foo { mixin(get_set!(int, "bar")()); } There are definitely some things about the proposal that are better than with this, but I just wanted to point out that the value of the proposal should probably be evaluated against something roughly like the above rather than something that does a bunch of procedural string splicing.
The whole point of the idea was to get rid of the strings and the mixin expression (as it looks like to day).
While I'm not necessarily opposed to the idea of getting rid of the strings, I guess I don't really see what improvement your proposal provides other than not having to manually specify the mapping of "replace what identifier with what data". Getting rid of the "mixin" I see as a separate issue. We could just as easily say that given the function "string get_set(...)": get_set("int", "bar"); or get_set int bar; ...Is shorthand for, or is even the new syntax for: mixin(get_set("int", "bar"));
That was my idea as well, that get_set("int", "bar"); could be translated into mixin(get_set("int", "bar")); just like just like scope statements are translated into try/catch/finally.
Honestly, I don't see much gain in using rather than mixin(). It's a little less typing, but that's it. And it precludes stuff like mixin("lhs " ~ op ~ " rhs") like happens all the time in overloaded operator functions. - Jonathan M Davis
Dec 15 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Jonathan M Davis" <jmdavisProg gmx.com> wrote in message 
news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...
 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:
 That was my idea as well, that

  get_set("int", "bar");

 could be translated into

 mixin(get_set("int", "bar")); just like

 just like scope statements are translated into try/catch/finally.
Honestly, I don't see much gain in using rather than mixin(). It's a little less typing, but that's it.
It does seem like a small difference, just replacing "mixin" with " " and removing one layer of parens. But I think that extra layer of parens, minor as it seems, makes a big difference in the readability (and "typeability") of mixin invocations. Those extra parens do get to be a real bother, major visual noise at least to my eyes.
 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.
I don't see why these shouldn't work: "int foo;"; return ("lhs " ~ op ~ " rhs"); At least with just the " " part of the proposal. Maybe the delegate thing might make it tricker, I dunno.
Dec 15 2010
next sibling parent reply =?ISO-8859-1?Q?Pelle_M=E5nsson?= <pelle.mansson gmail.com> writes:
On 12/15/2010 11:00 PM, Nick Sabalausky wrote:
 "Jonathan M Davis"<jmdavisProg gmx.com>  wrote in message
 news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...
 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:
 That was my idea as well, that

  get_set("int", "bar");

 could be translated into

 mixin(get_set("int", "bar")); just like

 just like scope statements are translated into try/catch/finally.
Honestly, I don't see much gain in using rather than mixin(). It's a little less typing, but that's it.
It does seem like a small difference, just replacing "mixin" with " " and removing one layer of parens. But I think that extra layer of parens, minor as it seems, makes a big difference in the readability (and "typeability") of mixin invocations. Those extra parens do get to be a real bother, major visual noise at least to my eyes.
I agree with this. Actually, just removing the parenthesis would be a huge gain for me.
 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.
I don't see why these shouldn't work: "int foo;"; return ("lhs " ~ op ~ " rhs"); At least with just the " " part of the proposal. Maybe the delegate thing might make it tricker, I dunno.
This could work, but I don't think anyone is suggesting completely replacing the mixin. I think could be a function/template thing, and have strings in a more explicit mixin""; Then again, inconsistency sucks.
Dec 16 2010
parent Jacob Carlborg <doob me.com> writes:
On 2010-12-16 11:18, Pelle Månsson wrote:
 On 12/15/2010 11:00 PM, Nick Sabalausky wrote:
 "Jonathan M Davis"<jmdavisProg gmx.com> wrote in message
 news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...
 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:
 That was my idea as well, that

  get_set("int", "bar");

 could be translated into

 mixin(get_set("int", "bar")); just like

 just like scope statements are translated into try/catch/finally.
Honestly, I don't see much gain in using rather than mixin(). It's a little less typing, but that's it.
It does seem like a small difference, just replacing "mixin" with " " and removing one layer of parens. But I think that extra layer of parens, minor as it seems, makes a big difference in the readability (and "typeability") of mixin invocations. Those extra parens do get to be a real bother, major visual noise at least to my eyes.
I agree with this. Actually, just removing the parenthesis would be a huge gain for me.
 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.
I don't see why these shouldn't work: "int foo;"; return ("lhs " ~ op ~ " rhs"); At least with just the " " part of the proposal. Maybe the delegate thing might make it tricker, I dunno.
This could work, but I don't think anyone is suggesting completely replacing the mixin. I think could be a function/template thing, and have strings in a more explicit mixin""; Then again, inconsistency sucks.
The whole point of the idea/suggestion was to get rid of the strings. Then if the syntax I've suggested is translated into the string mixin syntax we have now then I would be fine with that. -- /Jacob Carlborg
Dec 16 2010
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-15 23:00, Nick Sabalausky wrote:
 "Jonathan M Davis"<jmdavisProg gmx.com>  wrote in message
 news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...
 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:
 That was my idea as well, that

  get_set("int", "bar");

 could be translated into

 mixin(get_set("int", "bar")); just like

 just like scope statements are translated into try/catch/finally.
Honestly, I don't see much gain in using rather than mixin(). It's a little less typing, but that's it.
It does seem like a small difference, just replacing "mixin" with " " and removing one layer of parens. But I think that extra layer of parens, minor as it seems, makes a big difference in the readability (and "typeability") of mixin invocations. Those extra parens do get to be a real bother, major visual noise at least to my eyes.
^^ I completely agree.
 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.
I don't see why these shouldn't work: "int foo;"; return ("lhs " ~ op ~ " rhs"); At least with just the " " part of the proposal. Maybe the delegate thing might make it tricker, I dunno.
My idea was actually to get rid of the strings where the code to be mixed in is defined and to have a better syntax where it's used. The delegates are just a way of passing a block of code around. If you just use it in place then maybe one could do like this: (int foo;); return (lhs (op)rhs); -- /Jacob Carlborg
Dec 16 2010
next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:iedpbg$3i0$1 digitalmars.com...
 On 2010-12-15 23:00, Nick Sabalausky wrote:
 I don't see why these shouldn't work:

  "int foo;";
 return  ("lhs " ~ op ~ " rhs");

 At least with just the " " part of the proposal. Maybe the delegate thing
 might make it tricker, I dunno.
My idea was actually to get rid of the strings where the code to be mixed in is defined and to have a better syntax where it's used. The delegates are just a way of passing a block of code around. If you just use it in place then maybe one could do like this: (int foo;); return (lhs (op)rhs);
Yea, my point was just that the " ..." stuff could work either way, with the string-based system or with your delegate-based one. I don't mean to come across like I'm ignoring or against the idea of the whole delegate aspect, and I understand that the main point of the OP is to replace the strings with delegates, but with the q{...} syntax and string-templating, I'm still struggling to see a big enough benefit compared to the status quo. I see that using delegates instead of strings could probably be made to work, but my questions are "For what benefit(s)?" and "Would those benefits be sufficient to warrant the change?" I'm not necessarily saying the answer is "no", but I'm unconvinced so far. And here's another thing: Suppose we got a Ruby/PHP-like syntax for embedding code substitutions directly into a string (which would have other useful applications besides mixins): auto name = "Joe"; Joe++; Would that eliminate much (or all) of the benefit of the delegate approach?
Dec 16 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-16 21:35, Nick Sabalausky wrote:
 "Jacob Carlborg"<doob me.com>  wrote in message
 news:iedpbg$3i0$1 digitalmars.com...
 On 2010-12-15 23:00, Nick Sabalausky wrote:
 I don't see why these shouldn't work:

  "int foo;";
 return  ("lhs " ~ op ~ " rhs");

 At least with just the " " part of the proposal. Maybe the delegate thing
 might make it tricker, I dunno.
My idea was actually to get rid of the strings where the code to be mixed in is defined and to have a better syntax where it's used. The delegates are just a way of passing a block of code around. If you just use it in place then maybe one could do like this: (int foo;); return (lhs (op)rhs);
Yea, my point was just that the " ..." stuff could work either way, with the string-based system or with your delegate-based one. I don't mean to come across like I'm ignoring or against the idea of the whole delegate aspect, and I understand that the main point of the OP is to replace the strings with delegates, but with the q{...} syntax and string-templating, I'm still struggling to see a big enough benefit compared to the status quo. I see that using delegates instead of strings could probably be made to work, but my questions are "For what benefit(s)?" and "Would those benefits be sufficient to warrant the change?" I'm not necessarily saying the answer is "no", but I'm unconvinced so far. And here's another thing: Suppose we got a Ruby/PHP-like syntax for embedding code substitutions directly into a string (which would have other useful applications besides mixins): auto name = "Joe"; Joe++; Would that eliminate much (or all) of the benefit of the delegate approach?
I guess using q{...} with string interpolation is very similar to the delegate approach. It just feels wrong passing around strings to represent code. I haven't though much about it but with delegates one could at lest hope for better help from the compiler validating the code. I don't know how IDEs will treat q{...} but with delegates you would get the full benefit of the IDE like autocompletion and similar features. -- /Jacob Carlborg
Dec 16 2010
parent "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:iee4en$ttb$1 digitalmars.com...
 On 2010-12-16 21:35, Nick Sabalausky wrote:
 "Jacob Carlborg"<doob me.com>  wrote in message
 news:iedpbg$3i0$1 digitalmars.com...
 On 2010-12-15 23:00, Nick Sabalausky wrote:
 I don't see why these shouldn't work:

  "int foo;";
 return  ("lhs " ~ op ~ " rhs");

 At least with just the " " part of the proposal. Maybe the delegate 
 thing
 might make it tricker, I dunno.
My idea was actually to get rid of the strings where the code to be mixed in is defined and to have a better syntax where it's used. The delegates are just a way of passing a block of code around. If you just use it in place then maybe one could do like this: (int foo;); return (lhs (op)rhs);
Yea, my point was just that the " ..." stuff could work either way, with the string-based system or with your delegate-based one. I don't mean to come across like I'm ignoring or against the idea of the whole delegate aspect, and I understand that the main point of the OP is to replace the strings with delegates, but with the q{...} syntax and string-templating, I'm still struggling to see a big enough benefit compared to the status quo. I see that using delegates instead of strings could probably be made to work, but my questions are "For what benefit(s)?" and "Would those benefits be sufficient to warrant the change?" I'm not necessarily saying the answer is "no", but I'm unconvinced so far. And here's another thing: Suppose we got a Ruby/PHP-like syntax for embedding code substitutions directly into a string (which would have other useful applications besides mixins): auto name = "Joe"; Joe++; Would that eliminate much (or all) of the benefit of the delegate approach?
I guess using q{...} with string interpolation is very similar to the delegate approach. It just feels wrong passing around strings to represent code.
Well, code *is* text after all. But I know what you mean - after all, it does have more semantic structure than just ordinary generic strings.
 I haven't though much about it but with delegates one could at lest hope 
 for better help from the compiler validating the code. I don't know how 
 IDEs will treat q{...} but with delegates you would get the full benefit 
 of the IDE like autocompletion and similar features.
My editor (Programmer's Notepad 2, based off Scintilla) handles that fine. It doesn't know anything about q{}, so it assumes it's an identifier ("q") followed by a normal code block. And since it doesn't try to do any grammatical/semantic validation (only lexical, and only for the purpose of highlighting) it doesn't complain about "identifier { ... }" being invalid or any of the indentifiers-to-be-replaced inside of it being undeclared. But for fancier IDE's, like Eclipse with Descent or DDT, I don't know - that's a good question. OTOH, even with the delegate approach, I'm assuming that delegate would still get evaluated in a different context from where it's defined (which you'd probably want). So that might still cause some trouble with the more intelligent IDEs trying to tell you that identifierXYZ isn't accessable from within what it thinks is the delegate's scope.
Dec 16 2010
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, December 16, 2010 11:28:03 Jacob Carlborg wrote:
 On 2010-12-15 23:00, Nick Sabalausky wrote:
 "Jonathan M Davis"<jmdavisProg gmx.com>  wrote in message
 news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...
 
 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:
 That was my idea as well, that
 
  get_set("int", "bar");
 
 could be translated into
 
 mixin(get_set("int", "bar")); just like
 
 just like scope statements are translated into try/catch/finally.
Honestly, I don't see much gain in using rather than mixin(). It's a little less typing, but that's it.
It does seem like a small difference, just replacing "mixin" with " " and removing one layer of parens. But I think that extra layer of parens, minor as it seems, makes a big difference in the readability (and "typeability") of mixin invocations. Those extra parens do get to be a real bother, major visual noise at least to my eyes.
^^ I completely agree.
 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.
I don't see why these shouldn't work: "int foo;"; return ("lhs " ~ op ~ " rhs"); At least with just the " " part of the proposal. Maybe the delegate thing might make it tricker, I dunno.
My idea was actually to get rid of the strings where the code to be mixed in is defined and to have a better syntax where it's used. The delegates are just a way of passing a block of code around. If you just use it in place then maybe one could do like this: (int foo;); return (lhs (op)rhs);
I would have thought that template mixins would be the thing to use when you didn't want to deal with strings. string mixins are extremely powerful and flexible, and I'd really hate to lose them. And IIRC, Kenji Hara was working on a module to really help make dealing with complicated string mixins easier and less painful. Anything that you propose is going to have to have major benefits over the current string mixin situation for it to stand any chance of being accepted. - Jonathan M Davis
Dec 16 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-16 23:05, Jonathan M Davis wrote:
 On Thursday, December 16, 2010 11:28:03 Jacob Carlborg wrote:
 On 2010-12-15 23:00, Nick Sabalausky wrote:
 "Jonathan M Davis"<jmdavisProg gmx.com>   wrote in message
 news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...

 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:
 That was my idea as well, that

  get_set("int", "bar");

 could be translated into

 mixin(get_set("int", "bar")); just like

 just like scope statements are translated into try/catch/finally.
Honestly, I don't see much gain in using rather than mixin(). It's a little less typing, but that's it.
It does seem like a small difference, just replacing "mixin" with " " and removing one layer of parens. But I think that extra layer of parens, minor as it seems, makes a big difference in the readability (and "typeability") of mixin invocations. Those extra parens do get to be a real bother, major visual noise at least to my eyes.
^^ I completely agree.
 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.
I don't see why these shouldn't work: "int foo;"; return ("lhs " ~ op ~ " rhs"); At least with just the " " part of the proposal. Maybe the delegate thing might make it tricker, I dunno.
My idea was actually to get rid of the strings where the code to be mixed in is defined and to have a better syntax where it's used. The delegates are just a way of passing a block of code around. If you just use it in place then maybe one could do like this: (int foo;); return (lhs (op)rhs);
I would have thought that template mixins would be the thing to use when you didn't want to deal with strings. string mixins are extremely powerful and flexible, and I'd really hate to lose them. And IIRC, Kenji Hara was working on a module to really help make dealing with complicated string mixins easier and less painful. Anything that you propose is going to have to have major benefits over the current string mixin situation for it to stand any chance of being accepted. - Jonathan M Davis
Template mixins and string mixins are used for different things. There's a lot of things that string mixins can do that template mixins can't. I have no intention what so ever to suggest something that isn't as powerful as string mixins, just a new syntax. If it turns out that a having a powerful syntax without strings isn't possibles than I'll guess we have to live with the strings. Don't know if you read my first post put there I wrote that it wasn't a real suggestion (at least not yet) I just wanted the community's thoughts on the idea and see if we could turn it into something useful that could become a real suggestion, if people where interested. The ideas I wrote in my extended suggestion, "Taking it one step further", I think that those can have benefits over string mixins. Basically allowing you to pass the whole body of a class declaration to a function, as a delegate, with a syntax looking like Java annotations. -- /Jacob Carlborg
Dec 16 2010
parent "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:iee561$v5s$1 digitalmars.com...
 On 2010-12-16 23:05, Jonathan M Davis wrote:
 On Thursday, December 16, 2010 11:28:03 Jacob Carlborg wrote:
 On 2010-12-15 23:00, Nick Sabalausky wrote:
 "Jonathan M Davis"<jmdavisProg gmx.com>   wrote in message
 news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...

 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:
 That was my idea as well, that

  get_set("int", "bar");

 could be translated into

 mixin(get_set("int", "bar")); just like

 just like scope statements are translated into try/catch/finally.
Honestly, I don't see much gain in using rather than mixin(). It's a little less typing, but that's it.
It does seem like a small difference, just replacing "mixin" with " " and removing one layer of parens. But I think that extra layer of parens, minor as it seems, makes a big difference in the readability (and "typeability") of mixin invocations. Those extra parens do get to be a real bother, major visual noise at least to my eyes.
^^ I completely agree.
 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.
I don't see why these shouldn't work: "int foo;"; return ("lhs " ~ op ~ " rhs"); At least with just the " " part of the proposal. Maybe the delegate thing might make it tricker, I dunno.
My idea was actually to get rid of the strings where the code to be mixed in is defined and to have a better syntax where it's used. The delegates are just a way of passing a block of code around. If you just use it in place then maybe one could do like this: (int foo;); return (lhs (op)rhs);
I would have thought that template mixins would be the thing to use when you didn't want to deal with strings. string mixins are extremely powerful and flexible, and I'd really hate to lose them. And IIRC, Kenji Hara was working on a module to really help make dealing with complicated string mixins easier and less painful. Anything that you propose is going to have to have major benefits over the current string mixin situation for it to stand any chance of being accepted. - Jonathan M Davis
Template mixins and string mixins are used for different things. There's a lot of things that string mixins can do that template mixins can't. I have no intention what so ever to suggest something that isn't as powerful as string mixins, just a new syntax. If it turns out that a having a powerful syntax without strings isn't possibles than I'll guess we have to live with the strings. Don't know if you read my first post put there I wrote that it wasn't a real suggestion (at least not yet) I just wanted the community's thoughts on the idea and see if we could turn it into something useful that could become a real suggestion, if people where interested. The ideas I wrote in my extended suggestion, "Taking it one step further", I think that those can have benefits over string mixins. Basically allowing you to pass the whole body of a class declaration to a function, as a delegate, with a syntax looking like Java annotations.
It would seem to make sense to treat "chucks of code" the same way regardless of whether you're passing them around, mixing them in or instantiating them as a template.
Dec 16 2010
prev sibling next sibling parent VladD2 <vc rsdn.ru> writes:
Don Wrote:

 You missed: pattern matching, 
Yes, you've got me there. I've assumed that pattern matching, while a major feature, is not fundamental to the Nemerle macro-system, but I may be mistaken. One obvious difference which *is* fundamental is that Nemerle macros allow new syntax. Am I correct in thinking that Nemerle always requires complete ASTs? That is, given a name "x", can you access variables "x1", "x2", "x3" ? > quasi-quotation, No, that's present in D. It's the primary reason I say that the differences are mostly syntactic, since I see this as THE fundamental feature. You need to be able to reenter the macro system after you have left it. Once you can do that, you can do pretty much anything.
You not understand... pattern matching and quasi-quotation tightly bound. The quasi-quotation produce instances of Algebraic Data Type (AlgTD) which can be decomposed (and matched) by pattern matching (PM). We use PM to "read" code of a project. http://code.google.com/p/nemerle/source/browse/nemerle/trunk/snippets/csharp-parser/CSharpParser/Parser.n?r=9436 which made by PegGrammar macro: http://code.google.com/p/nemerle/source/browse/nemerle/trunk/snippets/peg-parser/Nemerle.Peg.Macros?r=9436 Or see our foreach macro:
 running of fully functional Nemerle
 code in compile time, 
Yes, but that's a different issue. In D, running code at compile time is regarded as an aspect of constant-folding, and is not restricted to macros.
The constant-folding is too limited solution. The power of of macros, including the ability to use are any libraries and any data sources. And ... you use enterpretation... slow way.
 access to compiler API in macros
In practice, has to be a library, right? Otherwise the compiler internals would be exposed? (This is an issue we're struggling with).
Yes. The Nemerle compiler is library. :) Unfortunately Nemerle API is not pretty clear. But I believe it's a right way. P.S. Sorry for my (Russian) English. :)
Dec 17 2010
prev sibling parent foobar <foo bar.com> writes:
Nick Sabalausky Wrote:

 
 The problem with that is, what if you're generating target-platform-specific 
 code at compile-time? You'd be generating code for the wrong platform. I 
 think VladD2 is right: You need to keep track of both "current" system and 
 "target" system. Unfortunately, there is some information about the "target" 
 system the compile-time code wouldn't be able discern without giving it the 
 ability to run code (RPC? Virtualization? Really, really good emulator?) on 
 the target system, but then again, that's a limitation with any 
 cross-compiling scenario.
 
 
I don't understand this. If you're talking about version blocks, obviously all the current D compile time features are incompatible with the macro system. They're also redundant cause you can (and should) simply write instead: macro (version) { if (version == Version.Linux) { ...} } Also, the macro dll is compiled for some platform Z and will only be loadable by a compiler that also runs on platform Z. I don't think that macros need to be cross-compiled for our (X,Y) scenario, they need to evaluate a TargetPlatform parameter at their run-time which the cross-compiler will provide (much like current D have predefined version identifiers defined by the compiler)
Dec 19 2010