www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - New syntax for string mixins

reply Jacob Carlborg <doob me.com> writes:
This is an idea I've been thinking of for a while, it's not a really 
suggestion (at least not yet) I just wanted to here what people think 
about it.

If we take a step back and look at what string mixins actually do or 
rather what they're used for, that would be: inserting a piece/block of 
code where the mixin expression is used. If we then take a look at how a 
block of code is represented in D (how you store it in variables and how 
you pass it around). It's not as a string which is used by the mixin 
expression, instead delegates are used to represent a block of code in D 
that can be passed around. Therefore this is my idea:

Add a new mixin operator " " (1). When that operator is put in front of 
a function call it would behave as a string mixin. The function that is 
called needs to be CTFE and return a delegate or an array of delegates:

class Foo
{
      get_set("int", "bar");
}

The above code would be the same as the following code:

class Foo
{
     mixin(get_set("int", "bar"));
}

If we now move to the declaration of "get_set" this is how it could look 
like:

void delegate () get_set (string type, string name)
{
     return
     {
          type _ name;

          type  name ()
         {
             return _ name;
         }

          type  name ( type  name)
         {
             return _ name =  name;
         }
     };
}

In the above code when " " is used in the delegate literal it basically 
behaves like string interpolation, so " type" would be replaced with the 
content of the "type" variable (2).

When the "get_set" function is called with the mixin symbol the content 
of the returned delegate is inserted where the call is made. If the 
function returns an array of delegates then the array would be unfolded 
and the content of all the delegates would be inserted.



Taking it one step further:

Allow the mixin syntax to be placed in front of most of the declarations 
and drop the need for string literals, commas and parentheses, basically 
allowing the following syntax:

class Foo
{
      get_set int bar;
}

Would be translated into this:

class Foo
{
      get_set!(int)("bar");
}

Maybe one could do something like this as well:

 singleton class Foo
{

}

void delegate () singleton (string name, void delegate () classBody)
{
     return { // a simple singleton implementation
         class  name
         {
             static  name instance;

             static this ()
             {
                 instance = new  name;
             }

             private this () {}

              classBody;
         }
     };
}

Above, in the last example,  classBody would insert the content of the 
delegate, basically the class body.

1. I will use " " in this example because I think it looks good but it 
would probably conflict with  nothrow,  property and others.

2. I have no idea if this usage of the mixin symbol, " ", will conflict 
with the first usage.

-- 
/Jacob Carlborg
Dec 13 2010
next sibling parent reply Graham St Jack <Graham.StJack internode.on.net> writes:
I have done a fair bit of mixin coding using recursive templates 
(inspired by std.typecons). It was an amazing taste of what you can do 
in D, and I am delighted with the result - HEAPS of boiler-plate coding 
vanished before my eyes. However, the template code is virtually 
impossible to understand.

What you are suggesting here seems to be a way to dramatically reduce 
the complexity of code that generates source-code and mixes it in. I 
think something like that is needed before this mind-bogglingly powerful 
feature of D can realise its potential.

There is of course the worry that it could get so easy that everyone 
starts doing it, and we have (relatively) impenetrable code everywhere 
instead of just deep in the bowels of framework libraries.


On 14/12/10 07:07, Jacob Carlborg wrote:

 This is an idea I've been thinking of for a while, it's not a really 
 suggestion (at least not yet) I just wanted to here what people think 
 about it.

 If we take a step back and look at what string mixins actually do or 
 rather what they're used for, that would be: inserting a piece/block 
 of code where the mixin expression is used. If we then take a look at 
 how a block of code is represented in D (how you store it in variables 
 and how you pass it around). It's not as a string which is used by the 
 mixin expression, instead delegates are used to represent a block of 
 code in D that can be passed around. Therefore this is my idea:

 Add a new mixin operator " " (1). When that operator is put in front 
 of a function call it would behave as a string mixin. The function 
 that is called needs to be CTFE and return a delegate or an array of 
 delegates:

 class Foo
 {
      get_set("int", "bar");
 }

 The above code would be the same as the following code:

 class Foo
 {
     mixin(get_set("int", "bar"));
 }

 If we now move to the declaration of "get_set" this is how it could 
 look like:

 void delegate () get_set (string type, string name)
 {
     return
     {
          type _ name;

          type  name ()
         {
             return _ name;
         }

          type  name ( type  name)
         {
             return _ name =  name;
         }
     };
 }

 In the above code when " " is used in the delegate literal it 
 basically behaves like string interpolation, so " type" would be 
 replaced with the content of the "type" variable (2).

 When the "get_set" function is called with the mixin symbol the 
 content of the returned delegate is inserted where the call is made. 
 If the function returns an array of delegates then the array would be 
 unfolded and the content of all the delegates would be inserted.



 Taking it one step further:

 Allow the mixin syntax to be placed in front of most of the 
 declarations and drop the need for string literals, commas and 
 parentheses, basically allowing the following syntax:

 class Foo
 {
      get_set int bar;
 }

 Would be translated into this:

 class Foo
 {
      get_set!(int)("bar");
 }

 Maybe one could do something like this as well:

  singleton class Foo
 {

 }

 void delegate () singleton (string name, void delegate () classBody)
 {
     return { // a simple singleton implementation
         class  name
         {
             static  name instance;

             static this ()
             {
                 instance = new  name;
             }

             private this () {}

              classBody;
         }
     };
 }

 Above, in the last example,  classBody would insert the content of the 
 delegate, basically the class body.

 1. I will use " " in this example because I think it looks good but it 
 would probably conflict with  nothrow,  property and others.

 2. I have no idea if this usage of the mixin symbol, " ", will 
 conflict with the first usage.



-- 
Graham St Jack
Dec 13 2010
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack  
<Graham.StJack internode.on.net> wrote:


 There is of course the worry that it could get so easy that everyone  
 starts doing it, and we have (relatively) impenetrable code everywhere  
 instead of just deep in the bowels of framework libraries.


TBH, I'm more excited by AST macros which I understood are planned for D3,  
as mentioned here:
http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf
They seem to promise the power of mixins, but without the mess.

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Dec 14 2010
next sibling parent reply Graham St Jack <Graham.StJack internode.on.net> writes:
On 14/12/10 20:33, Vladimir Panteleev wrote:

 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:


 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.


 TBH, I'm more excited by AST macros which I understood are planned for
 D3, as mentioned here:
 http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf
 They seem to promise the power of mixins, but without the mess.


I took a look at the pdf, but couldn't see how the AST macros could come
close to the kinds of things that are possible (but difficult) with 
mixins. Is there more information somewhere?

Jacob, I can see how your proposed syntax would make simple mixins 
easier and clearer, but how would it do something more complex?

For example taking a classname and a bunch of field types and names, and 
turning it into a class definition complete with constructor from field 
values, constructor from an input stream, a method to write to an output 
stream, and const getters. Or maybe std.typecons.AutoImplement.


-- 
Graham St Jack
Dec 14 2010
next sibling parent reply Don <nospam nospam.com> writes:
Graham St Jack wrote:

 On 14/12/10 20:33, Vladimir Panteleev wrote:

 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:


 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.


 TBH, I'm more excited by AST macros which I understood are planned for
 D3, as mentioned here:
 http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf
 They seem to promise the power of mixins, but without the mess.


 
 I took a look at the pdf, but couldn't see how the AST macros could come
 close to the kinds of things that are possible (but difficult) with 
 mixins. 


That fact was recognized at the conference, on the following day. As a 
result, AST macros were dropped from D2.

They need to roughly match string mixins in power.  At this stage, there 
is no proposal for how they should work.
Dec 14 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-14 13:05, Don wrote:

 Graham St Jack wrote:

 On 14/12/10 20:33, Vladimir Panteleev wrote:

 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:


 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.


 TBH, I'm more excited by AST macros which I understood are planned for
 D3, as mentioned here:
 http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf
 They seem to promise the power of mixins, but without the mess.


 I took a look at the pdf, but couldn't see how the AST macros could come
 close to the kinds of things that are possible (but difficult) with
 mixins.


 That fact was recognized at the conference, on the following day. As a
 result, AST macros were dropped from D2.


Do you have an example that would work with string mixins but not with 
AST macros?


 They need to roughly match string mixins in power. At this stage, there
 is no proposal for how they should work.


I think someone, Nick Sabalausky perhaps, suggested to have something 
like the hygiene macros in Nemerle: 
http://nemerle.org/wiki/index.php?title=Macros

-- 
/Jacob Carlborg
Dec 14 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:ie8dpq$kfg$2 digitalmars.com...

 On 2010-12-14 13:05, Don wrote:

 Graham St Jack wrote:

 On 14/12/10 20:33, Vladimir Panteleev wrote:

 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:


 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.


 TBH, I'm more excited by AST macros which I understood are planned for
 D3, as mentioned here:
 http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf
 They seem to promise the power of mixins, but without the mess.


 I took a look at the pdf, but couldn't see how the AST macros could come
 close to the kinds of things that are possible (but difficult) with
 mixins.


 That fact was recognized at the conference, on the following day. As a
 result, AST macros were dropped from D2.


 Do you have an example that would work with string mixins but not with AST 
 macros?


 They need to roughly match string mixins in power. At this stage, there
 is no proposal for how they should work.


 I think someone, Nick Sabalausky perhaps, suggested to have something like 
 the hygiene macros in Nemerle: 
 http://nemerle.org/wiki/index.php?title=Macros


Though I'm a huge D/systems-language/native-compiled guy, Nemerle's macros 
and pattern matching are two things I'm very jealous of and have made it 
very tempting to put up with the CLR for certain things.
Dec 14 2010
prev sibling parent reply Don <nospam nospam.com> writes:
Jacob Carlborg wrote:

 On 2010-12-14 13:05, Don wrote:

 Graham St Jack wrote:

 On 14/12/10 20:33, Vladimir Panteleev wrote:

 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:


 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.


 TBH, I'm more excited by AST macros which I understood are planned for
 D3, as mentioned here:
 http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf
 They seem to promise the power of mixins, but without the mess.


 I took a look at the pdf, but couldn't see how the AST macros could come
 close to the kinds of things that are possible (but difficult) with
 mixins.


 That fact was recognized at the conference, on the following day. As a
 result, AST macros were dropped from D2.


 
 Do you have an example that would work with string mixins but not with 
 AST macros?


Well, it's a bit hard without a formal definition of AST macros.
But the stuff I talked about at the conference, I have no idea how to do 
with AST macros.

There's code like this, which generates an asm instruction.

------
mixin( opToSSE[operations[done+1]] ~ suffix ~ " " ~ XMM(numOnStack-1) ~ 
", "  ~ indexedSSEVector(ranklist, operations[done], vectorsize));
-------

using the functions:
============
const char [][5] vectorRegister = ["ECX", "EDX", "EBX", "ESI", "EDI"];

char [] indexedSSEVector(char [] ranklist, char var, char [] vecsize)
{
     return "[" ~ vectorRegister[vectorNum(ranklist, var)] ~ " + " ~ 
vecsize ~"*EAX]";
}

char [] XMM(int k) { return "XMM"~ itoa(k); }

char [][char] opToSSE() {
     return ['*':"mulp"[], '+': "addp", '-': "subp", '/': "divp"]; }

int vectorNum(char [] ranklist, char var)
{
     int numVecs=0;
     for (int i=0; i<var-'A'; ++i) {
         if (ranklist[i]=='1') ++numVecs;
     }
     return numVecs;
}
============


 They need to roughly match string mixins in power. At this stage, there
 is no proposal for how they should work.


 
 I think someone, Nick Sabalausky perhaps, suggested to have something 
 like the hygiene macros in Nemerle: 
 http://nemerle.org/wiki/index.php?title=Macros


 From an implementation point of view, the differences between Nemerle 
macros and string mixins are mostly syntactic.
The one thing about them that I find really impressive is the IDE 
integration, especially that they got syntax highlighting to work. I 
don't know they've done that.
Dec 14 2010
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-15 08:57, Don wrote:

 Jacob Carlborg wrote:

 On 2010-12-14 13:05, Don wrote:

 Graham St Jack wrote:

 On 14/12/10 20:33, Vladimir Panteleev wrote:

 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:


 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code
 everywhere
 instead of just deep in the bowels of framework libraries.


 TBH, I'm more excited by AST macros which I understood are planned for
 D3, as mentioned here:
 http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf
 They seem to promise the power of mixins, but without the mess.


 I took a look at the pdf, but couldn't see how the AST macros could
 come
 close to the kinds of things that are possible (but difficult) with
 mixins.


 That fact was recognized at the conference, on the following day. As a
 result, AST macros were dropped from D2.


 Do you have an example that would work with string mixins but not with
 AST macros?


 Well, it's a bit hard without a formal definition of AST macros.
 But the stuff I talked about at the conference, I have no idea how to do
 with AST macros.

 There's code like this, which generates an asm instruction.

 ------
 mixin( opToSSE[operations[done+1]] ~ suffix ~ " " ~ XMM(numOnStack-1) ~
 ", " ~ indexedSSEVector(ranklist, operations[done], vectorsize));
 -------

 using the functions:
 ============
 const char [][5] vectorRegister = ["ECX", "EDX", "EBX", "ESI", "EDI"];

 char [] indexedSSEVector(char [] ranklist, char var, char [] vecsize)
 {
 return "[" ~ vectorRegister[vectorNum(ranklist, var)] ~ " + " ~ vecsize
 ~"*EAX]";
 }

 char [] XMM(int k) { return "XMM"~ itoa(k); }

 char [][char] opToSSE() {
 return ['*':"mulp"[], '+': "addp", '-': "subp", '/': "divp"]; }

 int vectorNum(char [] ranklist, char var)
 {
 int numVecs=0;
 for (int i=0; i<var-'A'; ++i) {
 if (ranklist[i]=='1') ++numVecs;
 }
 return numVecs;
 }
 ============


I can't quite visualize how the final code will look like and as you say 
it's hard without a formal definition of AST macros. But I think somehow 
it would be possible, I mean instead of combining strings one combine 
expressions/syntax. But I don't know if it would be possible to combine 
incomplete expressions.


 They need to roughly match string mixins in power. At this stage, there
 is no proposal for how they should work.


 I think someone, Nick Sabalausky perhaps, suggested to have something
 like the hygiene macros in Nemerle:
 http://nemerle.org/wiki/index.php?title=Macros


  From an implementation point of view, the differences between Nemerle
 macros and string mixins are mostly syntactic.
 The one thing about them that I find really impressive is the IDE
 integration, especially that they got syntax highlighting to work. I
 don't know they've done that.


As far as I can see the content of a macro in Nemerle is just code. But 
if you're referring to the syntax expression/statement which adds new 
syntax to the language then I agree.

-- 
/Jacob Carlborg
Dec 16 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:iednio$2vj5$1 digitalmars.com...

 I can't quite visualize how the final code will look like and as you say 
 it's hard without a formal definition of AST macros. But I think somehow 
 it would be possible, I mean instead of combining strings one combine 
 expressions/syntax. But I don't know if it would be possible to combine 
 incomplete expressions.


One parallel that may or may not be applicable, but might be worth 
considering, is dynamically building XHTML: Using a string-template system 
is generally found to work very well, but building it via a DOM (essentially 
an AST) is often considered a bit of a pain. I guess the simplest take-away 
would be that string-based approaches may be easier get working well. FWIW.
Dec 16 2010
next sibling parent Jacob Carlborg <doob me.com> writes:
On 2010-12-16 21:13, Nick Sabalausky wrote:

 "Jacob Carlborg"<doob me.com>  wrote in message
 news:iednio$2vj5$1 digitalmars.com...

 I can't quite visualize how the final code will look like and as you say
 it's hard without a formal definition of AST macros. But I think somehow
 it would be possible, I mean instead of combining strings one combine
 expressions/syntax. But I don't know if it would be possible to combine
 incomplete expressions.


 One parallel that may or may not be applicable, but might be worth
 considering, is dynamically building XHTML: Using a string-template system
 is generally found to work very well, but building it via a DOM (essentially
 an AST) is often considered a bit of a pain. I guess the simplest take-away
 would be that string-based approaches may be easier get working well. FWIW.


I the case of XML I think it can be quite easy if you use the right 
libraries/tools. I think the easiest library I've used for building XML 
files is the Ruby library Bulilder, a code example using Builder can 
look like this:

xml = Builder::XmlMarkup.new

xml.person do
     xml.first_name "John"
     xml.last_name "Doe"
     xml.phone "5484654", :type => "mobile"
end

Which will generate this:

<person>
     <first_name>John</first_name>
     <last_name>Doe</last_name>
     <phone type="mobile">5484654</phone>
</person

Now I don't think that this library is a DOM library, which allows you 
to manipulate a DOM tree, it's something simpler that just generates XML.

Link: http://builder.rubyforge.org/

-- 
/Jacob Carlborg
Dec 16 2010
prev sibling parent Lutger Blijdestijn <lutger.blijdestijn gmail.com> writes:
Nick Sabalausky wrote:


 "Jacob Carlborg" <doob me.com> wrote in message
 news:iednio$2vj5$1 digitalmars.com...

 I can't quite visualize how the final code will look like and as you say
 it's hard without a formal definition of AST macros. But I think somehow
 it would be possible, I mean instead of combining strings one combine
 expressions/syntax. But I don't know if it would be possible to combine
 incomplete expressions.


 
 One parallel that may or may not be applicable, but might be worth
 considering, is dynamically building XHTML: Using a string-template system
 is generally found to work very well, but building it via a DOM
 (essentially an AST) is often considered a bit of a pain. I guess the
 simplest take-away would be that string-based approaches may be easier get
 working well. FWIW.


The newest vb.net has this feature, if I understand you correctly, via xml 
literals and linq expressions. It is quite convenient.

http://msdn.microsoft.com/en-us/library/bb384460.aspx
Dec 17 2010
prev sibling parent reply VladD2 <vc rsdn.ru> writes:
Hi,

I'm one of Nemerle developers.


 From an implementation point of view, the differences between Nemerle
 macros and string mixins are mostly syntactic.


You are deeply mistaken!
I'd advise to study the Nemerle macro-system.

You missed: pattern matching, quasi-quotation, running of fully functional
Nemerle
code in compile time, access to compiler API in macros and IntelliSense support.


The one thing about them that I find really impressive is the IDE
integration, especially that they got syntax highlighting to work.
I don't know they've done that.


"syntax highlighting" ? :)
We have full IntelliSense support: syntax highlighting, code completion, macro
expansion (in tooltips).
We simply use the Nemerle compiler to achive that.
Dec 17 2010
parent reply Don <nospam nospam.com> writes:
VladD2 wrote:

 Hi,
 
 I'm one of Nemerle developers.


Cool!


 From an implementation point of view, the differences between Nemerle
 macros and string mixins are mostly syntactic.


 
 You are deeply mistaken!
 I'd advise to study the Nemerle macro-system.
 
 You missed: pattern matching, 


Yes, you've got me there. I've assumed that pattern matching, while a 
major feature, is not fundamental to the Nemerle macro-system, but I may 
be mistaken.

One obvious difference which *is* fundamental is that Nemerle macros 
allow new syntax.

Am I correct in thinking that Nemerle always requires complete ASTs?
That is, given a name "x", can you access variables "x1", "x2", "x3" ?


 quasi-quotation,


No, that's present in D. It's the primary reason I say that the 
differences are mostly syntactic, since I see this as THE fundamental 
feature. You need to be able to reenter the macro system after you have 
left it. Once you can do that, you can do pretty much anything.

BTW if you argued that D's current syntax is quite horrible, I couldn't 
disagree with you.


 running of fully functional Nemerle
 code in compile time, 


Yes, but that's a different issue. In D, running code at compile time is 
  regarded as an aspect of constant-folding, and is not restricted to 
macros.


 access to compiler API in macros


In practice, has to be a library, right? Otherwise the compiler 
internals would be exposed? (This is an issue we're struggling with).

Another one of the big differences is that D doesn't allow compile-time 
code to call external functions. Although it could certainly be done, it 
raises some big issues. Eg, we cannot assume that the target CPU is the 
same as the one we're running on. With a JIT compiler, you don't have 
that problem.


 and IntelliSense support.


I did mention that...


 The one thing about them that I find really impressive is the IDE
 integration, especially that they got syntax highlighting to work.
 I don't know they've done that.


 
 "syntax highlighting" ? :)
 We have full IntelliSense support: syntax highlighting, code completion, macro
 expansion (in tooltips).


I meant Intellisense, not just syntax highlighting. Though note that you 
can't do syntax highlighting *perfectly* unless the IDE understands the 
code.


 We simply use the Nemerle compiler to achive that.


Doesn't leave me any less impressed. <g>
Dec 17 2010
parent reply foobar <foo bar.com> writes:
Don Wrote:


 VladD2 wrote:

 Hi,
 
 I'm one of Nemerle developers.


 
 Cool!
 

 From an implementation point of view, the differences between Nemerle
 macros and string mixins are mostly syntactic.


 
 You are deeply mistaken!
 I'd advise to study the Nemerle macro-system.
 
 You missed: pattern matching, 


 
 Yes, you've got me there. I've assumed that pattern matching, while a 
 major feature, is not fundamental to the Nemerle macro-system, but I may 
 be mistaken.
 
 One obvious difference which *is* fundamental is that Nemerle macros 
 allow new syntax.
 
 Am I correct in thinking that Nemerle always requires complete ASTs?
 That is, given a name "x", can you access variables "x1", "x2", "x3" ?
 
  > quasi-quotation,
 
 No, that's present in D. It's the primary reason I say that the 
 differences are mostly syntactic, since I see this as THE fundamental 
 feature. You need to be able to reenter the macro system after you have 
 left it. Once you can do that, you can do pretty much anything.
 
 BTW if you argued that D's current syntax is quite horrible, I couldn't 
 disagree with you.
 

 running of fully functional Nemerle
 code in compile time, 


 
 Yes, but that's a different issue. In D, running code at compile time is 
   regarded as an aspect of constant-folding, and is not restricted to 
 macros.
 

 access to compiler API in macros


 
 In practice, has to be a library, right? Otherwise the compiler 
 internals would be exposed? (This is an issue we're struggling with).
 
 Another one of the big differences is that D doesn't allow compile-time 
 code to call external functions. Although it could certainly be done, it 
 raises some big issues. Eg, we cannot assume that the target CPU is the 
 same as the one we're running on. With a JIT compiler, you don't have 
 that problem.


Don, can you please elaborate on this point? 

Here's my understanding:
The D compiler is run once to both 'execute' compile time code which you refer
above as constant folding AND to generate the binary to execute at run-time. 

Nemerle separates this into two distinct steps:
1. you compile regular code inside macro definitions into a compiler plugin. 
2. when compiling the intended run-time code you need to load the compiled
macros from step 1 above on the command line of the compiler.

Since you're talking above about cross-compilation and let's say we run the
compiler on X and compile for Y, I see no problem to load precompiled macros
for X in order to compile the code for Y. The only limit as far as I can see is
that it won't be possible to load macros compiled by the such a cross compiler
on X unless as you say JIT is employed. Perhaps it makes sense to disable macro
definition in a cross-compilation scenario and only allow usage.

Is there something else that I'm missing here? 


 

 and IntelliSense support.


 
 I did mention that...
 

 The one thing about them that I find really impressive is the IDE
 integration, especially that they got syntax highlighting to work.
 I don't know they've done that.


 
 "syntax highlighting" ? :)
 We have full IntelliSense support: syntax highlighting, code completion, macro
 expansion (in tooltips).


 
 I meant Intellisense, not just syntax highlighting. Though note that you 
 can't do syntax highlighting *perfectly* unless the IDE understands the 
 code.
 

 We simply use the Nemerle compiler to achive that.


 
 Doesn't leave me any less impressed. <g>
Dec 17 2010
parent reply Don <nospam nospam.com> writes:
foobar wrote:

 Don Wrote:



 Another one of the big differences is that D doesn't allow compile-time 
 code to call external functions. Although it could certainly be done, it 
 raises some big issues. Eg, we cannot assume that the target CPU is the 
 same as the one we're running on. With a JIT compiler, you don't have 
 that problem.


 
 Don, can you please elaborate on this point? 
 
 Here's my understanding:
 The D compiler is run once to both 'execute' compile time code which you refer
above as constant folding AND to generate the binary to execute at run-time. 



 
 Nemerle separates this into two distinct steps:
 1. you compile regular code inside macro definitions into a compiler plugin. 
 2. when compiling the intended run-time code you need to load the compiled
macros from step 1 above on the command line of the compiler.
 
 Since you're talking above about cross-compilation and let's say we run the
compiler on X and compile for Y, I see no problem to load precompiled macros
for X in order to compile the code for Y. The only limit as far as I can see is
that it won't be possible to load macros compiled by the such a cross compiler
on X unless as you say JIT is employed. Perhaps it makes sense to disable macro
definition in a cross-compilation scenario and only allow usage.


Suppose the pre-compiled code, when run, asks what CPU it's on. What's 
the answer? Is it X or Y?
Dec 17 2010
parent reply VladD2 <vc rsdn.ru> writes:
Don Wrote:

 Suppose the pre-compiled code, when run, asks what CPU it's on. What's 
 the answer? Is it X or Y?


Current: X
Target: Y

Macro - a plugin to the compiler. It works on the same platform as the
compiler, but generates code through the API which abstracts the macro from the
target platform. If you need generate platform specific code, you should worry
about it in macro logic.
In any case macro is a meta-programm wich generate or/and transform code.
Dec 17 2010
parent reply Don <nospam nospam.com> writes:
VladD2 wrote:

 Don Wrote:

 Suppose the pre-compiled code, when run, asks what CPU it's on. What's 
 the answer? Is it X or Y?


 
 Current: X
 Target: Y
 
 Macro - a plugin to the compiler. It works on the same platform as the
compiler, but generates code through the API which abstracts the macro from the
target platform. If you need generate platform specific code, you should worry
about it in macro logic.
 In any case macro is a meta-programm wich generate or/and transform code. 


Yes. But in D there's no distinction between code which is destined for 
a macro, versus any other function. You can call a function once at 
compile time, and the same function at compile time. My understanding of 
Nemerle (which is quite likely to be wrong!) is that at least some 
functions are callable only at compile-time.

I'm also scared of the implications of allowing arbitrary code execution 
during compilation. Make a typo in your program, and then compilation 
may wipe files from your hard disk, or corrupt an external database, 
etc... On some platforms you may be able to sandbox it, but since it's 
running as part of the compilation process, rather than with the 
permissions it will eventually have, it just seems like a security 
nightmare.

All these problems would be reduced somewhat if it were only permissible 
to call pure functions at compile time. Although even that has issues, 
since not all compiler platforms may allow calling of shared libraries. 
If we build that into the language, we'd be cutting ourselves off from 
those platforms.

I'm not certain that there are any unsurmountable problems, but these 
issues make me really cautious.
Dec 18 2010
parent reply foobar <foo bar.com> writes:
Don Wrote:


 VladD2 wrote:

 Don Wrote:

 Suppose the pre-compiled code, when run, asks what CPU it's on. What's 
 the answer? Is it X or Y?


 
 Current: X
 Target: Y
 
 Macro - a plugin to the compiler. It works on the same platform as the
compiler, but generates code through the API which abstracts the macro from the
target platform. If you need generate platform specific code, you should worry
about it in macro logic.
 In any case macro is a meta-programm wich generate or/and transform code. 


 
 Yes. But in D there's no distinction between code which is destined for 
 a macro, versus any other function. You can call a function once at 
 compile time, and the same function at compile time. My understanding of 
 Nemerle (which is quite likely to be wrong!) is that at least some 
 functions are callable only at compile-time.
 


I don't see how there needs to be different code to accomplish the above use
case. You have a function that tests the hardware it's being run on. When this
function is called in the compiler context it would return X, when it's called
from the target executable it returns Y. 


 I'm also scared of the implications of allowing arbitrary code execution 
 during compilation. Make a typo in your program, and then compilation 
 may wipe files from your hard disk, or corrupt an external database, 
 etc... On some platforms you may be able to sandbox it, but since it's 
 running as part of the compilation process, rather than with the 
 permissions it will eventually have, it just seems like a security 
 nightmare.
 


This is a void argument since templates are Turing complete.
It's *already* possible to do all of the above in D at compile time, it's just
a matter of how much code is required to accomplish this. 


 All these problems would be reduced somewhat if it were only permissible 
 to call pure functions at compile time. Although even that has issues, 
 since not all compiler platforms may allow calling of shared libraries. 
 If we build that into the language, we'd be cutting ourselves off from 
 those platforms.
 
 I'm not certain that there are any unsurmountable problems, but these 
 issues make me really cautious.
Dec 18 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"foobar" <foo bar.com> wrote in message 
news:ieijt6$21mh$1 digitalmars.com...

 Don Wrote:


 VladD2 wrote:

 Don Wrote:

 Suppose the pre-compiled code, when run, asks what CPU it's on. What's
 the answer? Is it X or Y?


 Current: X
 Target: Y

 Macro - a plugin to the compiler. It works on the same platform as the 
 compiler, but generates code through the API which abstracts the macro 
 from the target platform. If you need generate platform specific code, 
 you should worry about it in macro logic.
 In any case macro is a meta-programm wich generate or/and transform 
 code.


 Yes. But in D there's no distinction between code which is destined for
 a macro, versus any other function. You can call a function once at
 compile time, and the same function at compile time.




I think you mean "You can call a function once at compile time, and the same 
function at **runtime**."



 My understanding of
 Nemerle (which is quite likely to be wrong!) is that at least some
 functions are callable only at compile-time.




I'd be surprised. I would think that all you would have to do to use the 
same Nemerle function at both runtime and compile-time would be to include 
its module in both the "compile the compiler-plugin step" and in the "load 
compiler-plugins and compile the app" step.



 I don't see how there needs to be different code to accomplish the above 
 use case. You have a function that tests the hardware it's being run on. 
 When this function is called in the compiler context it would return X, 
 when it's called from the target executable it returns Y.


The problem with that is, what if you're generating target-platform-specific 
code at compile-time? You'd be generating code for the wrong platform. I 
think VladD2 is right: You need to keep track of both "current" system and 
"target" system. Unfortunately, there is some information about the "target" 
system the compile-time code wouldn't be able discern without giving it the 
ability to run code (RPC? Virtualization? Really, really good emulator?) on 
the target system, but then again, that's a limitation with any 
cross-compiling scenario.



 I'm also scared of the implications of allowing arbitrary code execution
 during compilation. Make a typo in your program, and then compilation
 may wipe files from your hard disk, or corrupt an external database,
 etc... On some platforms you may be able to sandbox it, but since it's
 running as part of the compilation process, rather than with the
 permissions it will eventually have, it just seems like a security
 nightmare.




That's an interesting point. OTOH, it's common for software to come with its 
own build script or makefile, and those can certainly do literally anything 
like you describe above - but I haven't seen that as being a real problem.



 This is a void argument since templates are Turing complete.
 It's *already* possible to do all of the above in D at compile time, it's 
 just a matter of how much code is required to accomplish this.


Not true. This is a frequent misconception about Turning-completeness. Just 
because something is Turing complete does *not* mean it can do anything that 
any other Turing complete system can do (contrary to how many college profs 
explain it). It *does* mean that it can *calculate* anything that any other 
Turing complete system can calculate. But it doesn't necessarily have 
*access* to everything that any other Turing complete system has access to. 
And D's compile-time system, whether CTFE or templates, does not currently 
provide any way to access any I/O, launch any processes, or do any direct 
memory access.
Dec 18 2010
parent reply Don <nospam nospam.com> writes:
Nick Sabalausky wrote:

 "foobar" <foo bar.com> wrote in message 
 news:ieijt6$21mh$1 digitalmars.com...

 Don Wrote:


 VladD2 wrote:

 Don Wrote:

 Suppose the pre-compiled code, when run, asks what CPU it's on. What's
 the answer? Is it X or Y?


 Current: X
 Target: Y

 Macro - a plugin to the compiler. It works on the same platform as the 
 compiler, but generates code through the API which abstracts the macro 
 from the target platform. If you need generate platform specific code, 
 you should worry about it in macro logic.
 In any case macro is a meta-programm wich generate or/and transform 
 code.


 Yes. But in D there's no distinction between code which is destined for
 a macro, versus any other function. You can call a function once at
 compile time, and the same function at compile time.




 
 I think you mean "You can call a function once at compile time, and the same 
 function at **runtime**."
 
 

 My understanding of
 Nemerle (which is quite likely to be wrong!) is that at least some
 functions are callable only at compile-time.




 
 I'd be surprised. I would think that all you would have to do to use the 
 same Nemerle function at both runtime and compile-time would be to include 
 its module in both the "compile the compiler-plugin step" and in the "load 
 compiler-plugins and compile the app" step.
 
 

 I don't see how there needs to be different code to accomplish the above 
 use case. You have a function that tests the hardware it's being run on. 
 When this function is called in the compiler context it would return X, 
 when it's called from the target executable it returns Y.


 
 The problem with that is, what if you're generating target-platform-specific 
 code at compile-time? You'd be generating code for the wrong platform. I 
 think VladD2 is right: You need to keep track of both "current" system and 
 "target" system. Unfortunately, there is some information about the "target" 
 system the compile-time code wouldn't be able discern without giving it the 
 ability to run code (RPC? Virtualization? Really, really good emulator?) on 
 the target system, but then again, that's a limitation with any 
 cross-compiling scenario.


Note that for this to work at all, the compiler needs to be able to 
generate exectuable code for platform X as well as for Y -- that is, it 
needs to include two back-ends.


 I'm also scared of the implications of allowing arbitrary code execution
 during compilation. Make a typo in your program, and then compilation
 may wipe files from your hard disk, or corrupt an external database,
 etc... On some platforms you may be able to sandbox it, but since it's
 running as part of the compilation process, rather than with the
 permissions it will eventually have, it just seems like a security
 nightmare.




 
 That's an interesting point. OTOH, it's common for software to come with its 
 own build script or makefile, and those can certainly do literally anything 
 like you describe above - but I haven't seen that as being a real problem.


I don't think it's quite the same. In a makefile, every executable is 
listed, and so you can have some degree of control over it. But in this 
scenario, the compiler is making calls to arbitrary shared libraries 
with arbitrary parameters.
It means the compiler cannot be trusted *at all*.
Dec 18 2010
next sibling parent "Nick Sabalausky" <a a.a> writes:
"Don" <nospam nospam.com> wrote in message 
news:iej7eu$b1l$1 digitalmars.com...

 Nick Sabalausky wrote:

 "foobar" <foo bar.com> wrote in message 
 news:ieijt6$21mh$1 digitalmars.com...

 I don't see how there needs to be different code to accomplish the above 
 use case. You have a function that tests the hardware it's being run on. 
 When this function is called in the compiler context it would return X, 
 when it's called from the target executable it returns Y.


 The problem with that is, what if you're generating 
 target-platform-specific code at compile-time? You'd be generating code 
 for the wrong platform. I think VladD2 is right: You need to keep track 
 of both "current" system and "target" system. Unfortunately, there is 
 some information about the "target" system the compile-time code wouldn't 
 be able discern without giving it the ability to run code (RPC? 
 Virtualization? Really, really good emulator?) on the target system, but 
 then again, that's a limitation with any cross-compiling scenario.


 Note that for this to work at all, the compiler needs to be able to 
 generate exectuable code for platform X as well as for Y -- that is, it 
 needs to include two back-ends.


But if the compiler doesn't have both backends then the whole question of 
"how does the user's compile-time code get handled if it's being 
cross-compiled?" becomes irrelevent - you'd have to compile it on the target 
plaform anyway, so X == Y. Or am I missing your point?


 I'm also scared of the implications of allowing arbitrary code 
 execution
 during compilation. Make a typo in your program, and then compilation
 may wipe files from your hard disk, or corrupt an external database,
 etc... On some platforms you may be able to sandbox it, but since it's
 running as part of the compilation process, rather than with the
 permissions it will eventually have, it just seems like a security
 nightmare.




 That's an interesting point. OTOH, it's common for software to come with 
 its own build script or makefile, and those can certainly do literally 
 anything like you describe above - but I haven't seen that as being a 
 real problem.


 I don't think it's quite the same. In a makefile, every executable is 
 listed, and so you can have some degree of control over it. But in this 
 scenario, the compiler is making calls to arbitrary shared libraries with 
 arbitrary parameters.
 It means the compiler cannot be trusted *at all*.


I suppose that's a reasonable point.
Dec 18 2010
prev sibling next sibling parent reply VladD2 <vc rsdn.ru> writes:
Don Wrote:


 I 
 think VladD2 is right: You need to keep track of both "current" system and 
 "target" system. Unfortunately, there is some information about the "target" 
 system the compile-time code wouldn't be able discern without giving it the 
 ability to run code (RPC? Virtualization? Really, really good emulator?) on 
 the target system, but then again, that's a limitation with any 
 cross-compiling scenario.


 
 Note that for this to work at all, the compiler needs to be able to 
 generate exectuable code for platform X as well as for Y -- that is, it 
 needs to include two back-ends.


If the macros have been compiled and are in binary (executable) form, the
compiler must only be able to generate code for platform X, and run macros
(execute code from DLL). This is exactly what makes Nemerle compiler.

In this case, compiling of the same macros looks like any other compilation
process (on the platform X for the platform Y).



 I don't think it's quite the same. In a makefile, every executable is 
 listed, and so you can have some degree of control over it. 


Trust to rmdir ... lol!
And what about NAnt or MSBuild which can have binary extensions?

I think, you are completely wrong.


 But in this 
 scenario, the compiler is making calls to arbitrary shared libraries 
 with arbitrary parameters.
 It means the compiler cannot be trusted *at all*.


The experience of Lisp (50 years!) and Nemerel (about 6 years) shows that the
ability to access any library - is not a problem. This is a huge advantage.

And limit the possibility of a macro, you can simply forbidding them to use
some libraries.
Dec 19 2010
parent Don <nospam nospam.com> writes:
VladD2 wrote:

 Don Wrote:
 

 I 
 think VladD2 is right: You need to keep track of both "current" system and 
 "target" system. Unfortunately, there is some information about the "target" 
 system the compile-time code wouldn't be able discern without giving it the 
 ability to run code (RPC? Virtualization? Really, really good emulator?) on 
 the target system, but then again, that's a limitation with any 
 cross-compiling scenario.


 Note that for this to work at all, the compiler needs to be able to 
 generate exectuable code for platform X as well as for Y -- that is, it 
 needs to include two back-ends.


 
 If the macros have been compiled and are in binary (executable) form, the
compiler must only be able to generate code for platform X,


Yes, but it's not a compiler for platform X! It's only a compiler for 
platform Y.


 and run macros (execute code from DLL). This is exactly what makes Nemerle
compiler.


The .NET system always has a compiler for the platform it's running on. 
That's not necessarily true for D compilers.


 In this case, compiling of the same macros looks like any other compilation
process (on the platform X for the platform Y).
 
 

 I don't think it's quite the same. In a makefile, every executable is 
 listed, and so you can have some degree of control over it. 


 
 Trust to rmdir ... lol!
 And what about NAnt or MSBuild which can have binary extensions?
 
 I think, you are completely wrong.
 

 But in this 
 scenario, the compiler is making calls to arbitrary shared libraries 
 with arbitrary parameters.
 It means the compiler cannot be trusted *at all*.


 
 The experience of Lisp (50 years!) and Nemerel (about 6 years) shows that the
ability to access any library - is not a problem. 


I don't think Nemerle has been sufficiently widely used, to be able to 
draw strong conclusions from it. The Lisp argument is strong though.


 This is a huge advantage.
 
 And limit the possibility of a macro, you can simply forbidding them to use
some libraries. 


I hope you're right, because it's indeed a powerful feature. But I'd 
want to hear the opinion of a security expert.
In particular, if it can be shown that it's exactly the same as Lisp, I 
would be convinced.
Dec 20 2010
prev sibling parent reply "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"Don" <nospam nospam.com> wrote:

 I don't think it's quite the same. In a makefile, every executable is 
 listed, and so you can have some degree of control over it. But in this 
 scenario, the compiler is making calls to arbitrary shared libraries with 
 arbitrary parameters.
 It means the compiler cannot be trusted *at all*.


You are right only partially - it's unsafe for browser language where code 
is taken from untrusted source. But this feature gives so much power to the 
macro sysrem  - that I think is worth considering it. IMO, usually compiled 
code is run just after compilation (with the same prermissions as 
compiler) - so compiled code can make dangerous things and can't be trusted 
at all, but no one is worry about that. Yes compiler can't be *trusted* with 
this features, but if one knows what he is doing, why to prevent him - add 
option --enable-ctfe-DANGEROUS-features to allow potentially dangerous 
features then it wouldn't be so unexpected. Are those features hard to add 
to the current implementation?
Dec 19 2010
parent reply Don <nospam nospam.com> writes:
Alex_Dovhal wrote:

 "Don" <nospam nospam.com> wrote:

 I don't think it's quite the same. In a makefile, every executable is 
 listed, and so you can have some degree of control over it. But in this 
 scenario, the compiler is making calls to arbitrary shared libraries with 
 arbitrary parameters.
 It means the compiler cannot be trusted *at all*.


 
 You are right only partially - it's unsafe for browser language where code 
 is taken from untrusted source. But this feature gives so much power to the 
 macro sysrem  - that I think is worth considering it. IMO, usually compiled 
 code is run just after compilation (with the same prermissions as 
 compiler) - so compiled code can make dangerous things and can't be trusted 
 at all, but no one is worry about that. Yes compiler can't be *trusted* with 
 this features, but if one knows what he is doing, why to prevent him - add 
 option --enable-ctfe-DANGEROUS-features to allow potentially dangerous 
 features then it wouldn't be so unexpected. Are those features hard to add 
 to the current implementation? 


In order for CTFE code to call pre-compiled code, three things are required:
(1) the compiler needs to be able to find the file (.obj/.lib/shared 
library) containing the compiled code;
(2) the compiler needs to be able to load the module and call it. This 
requires some form of dynamic linking.
(3) We need a marshalling step, to convert from compiler literal to 
compiled data, and back.


Step (3) is straightforward. The challenge is step(2), although note 
that it's a general "allow the compiler to load a plugin" problem, and 
doesn't have much to do with CTFE.
Dec 20 2010
next sibling parent "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"Don" <nospam nospam.com> wrote:

 In order for CTFE code to call pre-compiled code, three things are 
 required:
 (1) the compiler needs to be able to find the file (.obj/.lib/shared 
 library) containing the compiled code;
 (2) the compiler needs to be able to load the module and call it. This 
 requires some form of dynamic linking.
 (3) We need a marshalling step, to convert from compiler literal to 
 compiled data, and back.


 Step (3) is straightforward. The challenge is step(2), although note that 
 it's a general "allow the compiler to load a plugin" problem, and doesn't 
 have much to do with CTFE.


Understand. So, it should be dynamic loaded, compiler should know which D 
library to load for used function and this function's name mangling, also 
then phobos should be dynamic library to call it's functions in macro. This 
is non trivial stuff, and compiler itselt is written in C++ so this plugin 
architecture should be working in C++ too. Also when cross-compile it's 
neeaded compiler for both X and Y architectures or two compilers, 
communicating among them. So that compiler for Y when finds macro should 
call compiler X and dynamically load to itself produced function. OK, IMO 
it's too complex and experimental to be of any priority in nearest future.
Dec 20 2010
prev sibling parent reply "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"Don" <nospam nospam.com> wrote:

 In order for CTFE code to call pre-compiled code, three things are 
 required:
 (1) the compiler needs to be able to find the file (.obj/.lib/shared 
 library) containing the compiled code;
 (2) the compiler needs to be able to load the module and call it. This 
 requires some form of dynamic linking.
 (3) We need a marshalling step, to convert from compiler literal to 
 compiled data, and back.


 Step (3) is straightforward. The challenge is step(2), although note that 
 it's a general "allow the compiler to load a plugin" problem, and doesn't 
 have much to do with CTFE.


I thought it over, and got:
(1) can be solved by adding compiler option: --macro-libs=<list_of_libs>, so 
the compiler knows in which dynamic libraries to search for the plugins ;
(2) plugin library functions should be *stdcall*, so C++ compiler can load 
them. That library should implement function like that:

PluginInfo* getFunctionsInfo () ;
where:
typedef struct _PluginInfo{
 struct _PluginInfo * nextPlugin ;
 char* fcnName ; // short name of a function, e.g. "fcn"
 char* params ; // param list, e.g. "(int, float, char*)"
 char* mangledName;//name of funct. in the library, e.g. "_fcn 12"
 char* returnType ; // e.g. "int"
 bool  isNothrow ; // if useful?
 bool  isPure ;    // ditto
 //... etc ...
} PluginInfo ;

And also should implement all the functions, info about which is returned by 
getFunctionsInfo().
D compiler calls getFunctionsInfo from each plugin library - and then loads 
all the implemented functions with full info about them.
If one plugin function name found in two or more libraries - D throws 
compiler error.
Not a perfect solution, but at least straightforward one and solves given 
problem.

One more note: this approach makes CTFE functions calling such plugins 
uncallable at runtime, which IMO is OK, as one should not call it in any 
case.

Is that suitable?
Dec 21 2010
parent reply Don <nospam nospam.com> writes:
Alex_Dovhal wrote:

 "Don" <nospam nospam.com> wrote:

 In order for CTFE code to call pre-compiled code, three things are 
 required:
 (1) the compiler needs to be able to find the file (.obj/.lib/shared 
 library) containing the compiled code;
 (2) the compiler needs to be able to load the module and call it. This 
 requires some form of dynamic linking.
 (3) We need a marshalling step, to convert from compiler literal to 
 compiled data, and back.


 Step (3) is straightforward. The challenge is step(2), although note that 
 it's a general "allow the compiler to load a plugin" problem, and doesn't 
 have much to do with CTFE.


 
 I thought it over, and got:
 (1) can be solved by adding compiler option: --macro-libs=<list_of_libs>, so 
 the compiler knows in which dynamic libraries to search for the plugins ;
 (2) plugin library functions should be *stdcall*, so C++ compiler can load 
 them. That library should implement function like that:
 
 PluginInfo* getFunctionsInfo () ;
 where:
 typedef struct _PluginInfo{
  struct _PluginInfo * nextPlugin ;
  char* fcnName ; // short name of a function, e.g. "fcn"
  char* params ; // param list, e.g. "(int, float, char*)"
  char* mangledName;//name of funct. in the library, e.g. "_fcn 12"
  char* returnType ; // e.g. "int"
  bool  isNothrow ; // if useful?
  bool  isPure ;    // ditto
  //... etc ...
 } PluginInfo ;
 
 And also should implement all the functions, info about which is returned by 
 getFunctionsInfo().
 D compiler calls getFunctionsInfo from each plugin library - and then loads 
 all the implemented functions with full info about them.
 If one plugin function name found in two or more libraries - D throws 
 compiler error.
 Not a perfect solution, but at least straightforward one and solves given 
 problem.
 
 One more note: this approach makes CTFE functions calling such plugins 
 uncallable at runtime, which IMO is OK, as one should not call it in any 
 case.
 
 Is that suitable? 


It's not that complicated. Once you can load the library and call *one* 
function in it, the problem is solved.
But at a deeper level, I'm not sure what you'd hope to achieve with this.
I mean (excluding some not-yet implemented features), CTFE allows you to 
execute any pure + safe function which you have source code for.

So, if you had such a plugin, it could do things like make database 
queries. Everything else you can do already.
Dec 21 2010
parent "Alex_Dovhal" <alex_dovhal yahoo.com> writes:
"Don" <nospam nospam.com> wrote:

 It's not that complicated. Once you can load the library and call *one* 
 function in it, the problem is solved.
 But at a deeper level, I'm not sure what you'd hope to achieve with this.
 I mean (excluding some not-yet implemented features), CTFE allows you to 
 execute any pure + safe function which you have source code for.

 So, if you had such a plugin, it could do things like make database 
 queries. Everything else you can do already.


May be you are right, I don't know. So yes - adding plugins is questionable 
feature now. I thought on it in general, and found several things they are 
suitable for, not sure if so useful:

(1) To call external tools, e.g. SQL-syntax validator, DSL code parser ;
(2) To load and handle resources at CT, e.g.
    (*) load XML, JSON, etc. generated by IDE GUI and convert it to D code;
    (*) load resources as handled data in inner program format, like: auto 
imgData = mixin (loadJPEG ("./img/photo1.jpeg")) ; (that seems useful)
(3) make impure CTFE functions, e.g. one macro configures the state, and 
some/all other depend on it (like optimisation options, hardware resources, 
etc.) or cache result from call to call like fibonacci numbers.
(4) Workaround CTFE limitations if really needed (questionable, as you 
pointed above).
Dec 22 2010
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-14 12:42, Graham St Jack wrote:

 On 14/12/10 20:33, Vladimir Panteleev wrote:

 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:


 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.


 TBH, I'm more excited by AST macros which I understood are planned for
 D3, as mentioned here:
 http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf
 They seem to promise the power of mixins, but without the mess.


 I took a look at the pdf, but couldn't see how the AST macros could come
 close to the kinds of things that are possible (but difficult) with
 mixins. Is there more information somewhere?

 Jacob, I can see how your proposed syntax would make simple mixins
 easier and clearer, but how would it do something more complex?


I don't know, do you have an example ?


 For example taking a classname and a bunch of field types and names, and
 turning it into a class definition complete with constructor from field
 values, constructor from an input stream, a method to write to an output
 stream, and const getters. Or maybe std.typecons.AutoImplement.


Could you post an example of how that mixin would be used and the code 
it would generate then I can see if I can translate it to my syntax. 
AutoImplement seems to just contain template mixins which is something else.

-- 
/Jacob Carlborg
Dec 14 2010
parent reply Graham St Jack <Graham.StJack internode.on.net> writes:
 I don't know, do you have an example ?


 For example taking a classname and a bunch of field types and names, and
 turning it into a class definition complete with constructor from field
 values, constructor from an input stream, a method to write to an output
 stream, and const getters. Or maybe std.typecons.AutoImplement.


 Could you post an example of how that mixin would be used and the code 
 it would generate then I can see if I can translate it to my syntax. 
 AutoImplement seems to just contain template mixins which is something 
 else.


I have attached my concurrency framework, which relies heavily on 
mixins, plus its unit test to show how it is used. I haven't included 
the various dependencies because I assume you just want the example 
code. Let me know if you want something buildable, or perhaps something 
more cut-down.

What the code-generating template does is to create from something like 
this (you can have any number of Messages in a Protocol):


alias Protocol!("Requests",    Message!("job",    string, "name")).code 
jobCode;
mixin(jobCode);


this code:


class Requests {
     struct jobMsg {
         string name;
         this(string name) {
              this.name = name;
         }
         void read(InStream stream) {
              name = stream.get!string;
         }
         void write(OutStream stream) {
             stream(name);
         }
     }
     struct Message {
         uint kind;
         union {
         jobMsg job;
         }
         this(ref jobMsg msg) {
             kind = 0;
             job = msg;
         }
         this(InStream stream) {
             kind = stream.get!uint;
             switch(kind) {
             case 0: job.read(stream); break;
                 default: assert(0, "Cannot read unsupported message kind");
             }
         }
         void write(OutStream stream) {
             stream(kind);
             switch(kind) {
             case 0: job.write(stream); break;
                 default: assert(0, "Cannot write unsupported message 
kind");
             }
         }
     }
     private alias Channel!(Message) _Chan;
     private alias shared _Chan Chan;

     private Chan channel;
     this() { channel = new Chan(); }
     ChannelSelectable newSelectable() { return channel.newSelectable(); }
     void finalize() { channel.finalize; }

     interface IHandler {
         void job(string name);
     }
     void job(string name) {
         channel.add(Message(jobMsg(name)));
     }
     void receive(IHandler handler) {
         auto message = channel.remove;
         switch (message.kind) {
             case 0: handler.job(message.job.name); break;
             default: assert(0, "Cannot dispatch unsupported message kind");
         }
     }
}



I use this for inter-thread communications, and I use the discriminated 
union to pass messages between processes. The manual mixin after the 
alias is a debugging aid - the compiler errors when doing mixins aren't 
helpful at all. I case you are wondering why I did all this, it was 
partly a learning experience, but mostly an attempt to do something 
properly thread-safe (!hasAliasing), using shared, const and immutable 
properly.

-- 
Graham St Jack
Dec 14 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-15 00:19, Graham St Jack wrote:

 I don't know, do you have an example ?


 For example taking a classname and a bunch of field types and names, and
 turning it into a class definition complete with constructor from field
 values, constructor from an input stream, a method to write to an output
 stream, and const getters. Or maybe std.typecons.AutoImplement.


 Could you post an example of how that mixin would be used and the code
 it would generate then I can see if I can translate it to my syntax.
 AutoImplement seems to just contain template mixins which is something
 else.


 I have attached my concurrency framework, which relies heavily on
 mixins, plus its unit test to show how it is used. I haven't included
 the various dependencies because I assume you just want the example
 code. Let me know if you want something buildable, or perhaps something
 more cut-down.

 What the code-generating template does is to create from something like
 this (you can have any number of Messages in a Protocol):


 alias Protocol!("Requests", Message!("job", string, "name")).code jobCode;
 mixin(jobCode);


 this code:


 class Requests {
 struct jobMsg {
 string name;
 this(string name) {
 this.name = name;
 }
 void read(InStream stream) {
 name = stream.get!string;
 }
 void write(OutStream stream) {
 stream(name);
 }
 }
 struct Message {
 uint kind;
 union {
 jobMsg job;
 }
 this(ref jobMsg msg) {
 kind = 0;
 job = msg;
 }
 this(InStream stream) {
 kind = stream.get!uint;
 switch(kind) {
 case 0: job.read(stream); break;
 default: assert(0, "Cannot read unsupported message kind");
 }
 }
 void write(OutStream stream) {
 stream(kind);
 switch(kind) {
 case 0: job.write(stream); break;
 default: assert(0, "Cannot write unsupported message kind");
 }
 }
 }
 private alias Channel!(Message) _Chan;
 private alias shared _Chan Chan;

 private Chan channel;
 this() { channel = new Chan(); }
 ChannelSelectable newSelectable() { return channel.newSelectable(); }
 void finalize() { channel.finalize; }

 interface IHandler {
 void job(string name);
 }
 void job(string name) {
 channel.add(Message(jobMsg(name)));
 }
 void receive(IHandler handler) {
 auto message = channel.remove;
 switch (message.kind) {
 case 0: handler.job(message.job.name); break;
 default: assert(0, "Cannot dispatch unsupported message kind");
 }
 }
 }



 I use this for inter-thread communications, and I use the discriminated
 union to pass messages between processes. The manual mixin after the
 alias is a debugging aid - the compiler errors when doing mixins aren't
 helpful at all. I case you are wondering why I did all this, it was
 partly a learning experience, but mostly an attempt to do something
 properly thread-safe (!hasAliasing), using shared, const and immutable
 properly.


I've attached a part of how concurrency.d could look like translated to 
my suggested syntax. It probably contains a lot of errors because did a 
quick translation and I had some trouble understanding the mixins.

-- 
/Jacob Carlborg
Dec 16 2010
parent reply Graham St Jack <Graham.StJack internode.on.net> writes:
 I've attached a part of how concurrency.d could look like translated 
 to my suggested syntax. It probably contains a lot of errors because 
 did a quick translation and I had some trouble understanding the mixins.


Yes, even I couldn't understand them even a week later. Maintenance is a 
real problem.

My initial reaction is that the proposed syntax helps a bit, but isn't 
really a game-changer. I will let you know if I change my mind on closer 
inspection.

-- 
Graham St Jack
Dec 16 2010
parent Jacob Carlborg <doob me.com> writes:
On 2010-12-17 00:34, Graham St Jack wrote:

 I've attached a part of how concurrency.d could look like translated
 to my suggested syntax. It probably contains a lot of errors because
 did a quick translation and I had some trouble understanding the mixins.


 Yes, even I couldn't understand them even a week later. Maintenance is a
 real problem.

 My initial reaction is that the proposed syntax helps a bit, but isn't
 really a game-changer. I will let you know if I change my mind on closer
 inspection.


No it's basically just syntactic sugar.

-- 
/Jacob Carlborg
Dec 18 2010
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2010-12-14 11:03, Vladimir Panteleev wrote:

 On Tue, 14 Dec 2010 09:30:46 +0200, Graham St Jack
 <Graham.StJack internode.on.net> wrote:


 There is of course the worry that it could get so easy that everyone
 starts doing it, and we have (relatively) impenetrable code everywhere
 instead of just deep in the bowels of framework libraries.


 TBH, I'm more excited by AST macros which I understood are planned for
 D3, as mentioned here:
 http://s3.amazonaws.com/dconf2007/WalterAndrei.pdf
 They seem to promise the power of mixins, but without the mess.


I would like to have AST macros too but I was think my suggestion was 
perhaps easier to implement, just some syntactic sugar.

-- 
/Jacob Carlborg
Dec 14 2010
prev sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Graham St Jack" <Graham.StJack internode.on.net> wrote in message 
news:ie76ig$b2v$1 digitalmars.com...

 What you are suggesting here seems to be a way to dramatically reduce the 
 complexity of code that generates source-code and mixes it in. I think 
 something like that is needed before this mind-bogglingly powerful feature 
 of D can realise its potential.


I think a decent string-template library could probably come very close to 
the proposal without needing any language changes at all:

string get_set(T, string name)()
{
     return
     q{
          type _ name;

          type  name ()
         {
             return _ name;
         }

          type  name ( type  name)
         {
             return _ name =  name;
         }
     }.replace( [" type": T.stringof, " name": name] );
}

class Foo
{
     mixin(get_set!(int, "bar")());
}

There are definitely some things about the proposal that are better than 
with this, but I just wanted to point out that the value of the proposal 
should probably be evaluated against something roughly like the above rather 
than something that does a bunch of procedural string splicing.
Dec 14 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-14 19:13, Nick Sabalausky wrote:

 "Graham St Jack"<Graham.StJack internode.on.net>  wrote in message
 news:ie76ig$b2v$1 digitalmars.com...

 What you are suggesting here seems to be a way to dramatically reduce the
 complexity of code that generates source-code and mixes it in. I think
 something like that is needed before this mind-bogglingly powerful feature
 of D can realise its potential.


 I think a decent string-template library could probably come very close to
 the proposal without needing any language changes at all:

 string get_set(T, string name)()
 {
       return
       q{
            type _ name;

            type  name ()
           {
               return _ name;
           }

            type  name ( type  name)
           {
               return _ name =  name;
           }
       }.replace( [" type": T.stringof, " name": name] );
 }

 class Foo
 {
       mixin(get_set!(int, "bar")());
 }

 There are definitely some things about the proposal that are better than
 with this, but I just wanted to point out that the value of the proposal
 should probably be evaluated against something roughly like the above rather
 than something that does a bunch of procedural string splicing.


The whole point of the idea was to get rid of the strings and the mixin 
expression (as it looks like to day).

-- 
/Jacob Carlborg
Dec 14 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:ie8i8c$15f0$1 digitalmars.com...

 On 2010-12-14 19:13, Nick Sabalausky wrote:

 "Graham St Jack"<Graham.StJack internode.on.net>  wrote in message
 news:ie76ig$b2v$1 digitalmars.com...

 What you are suggesting here seems to be a way to dramatically reduce 
 the
 complexity of code that generates source-code and mixes it in. I think
 something like that is needed before this mind-bogglingly powerful 
 feature
 of D can realise its potential.


 I think a decent string-template library could probably come very close 
 to
 the proposal without needing any language changes at all:

 string get_set(T, string name)()
 {
       return
       q{
            type _ name;

            type  name ()
           {
               return _ name;
           }

            type  name ( type  name)
           {
               return _ name =  name;
           }
       }.replace( [" type": T.stringof, " name": name] );
 }

 class Foo
 {
       mixin(get_set!(int, "bar")());
 }

 There are definitely some things about the proposal that are better than
 with this, but I just wanted to point out that the value of the proposal
 should probably be evaluated against something roughly like the above 
 rather
 than something that does a bunch of procedural string splicing.


 The whole point of the idea was to get rid of the strings and the mixin 
 expression (as it looks like to day).


While I'm not necessarily opposed to the idea of getting rid of the strings, 
I guess I don't really see what improvement your proposal provides other 
than not having to manually specify the mapping of "replace what identifier 
with what data".

Getting rid of the "mixin" I see as a separate issue. We could just as 
easily say that given the function "string get_set(...)":

     get_set("int", "bar");
    or
     get_set int bar;

...Is shorthand for, or is even the new syntax for:

    mixin(get_set("int", "bar"));
Dec 14 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-14 21:44, Nick Sabalausky wrote:

 "Jacob Carlborg"<doob me.com>  wrote in message
 news:ie8i8c$15f0$1 digitalmars.com...

 On 2010-12-14 19:13, Nick Sabalausky wrote:

 "Graham St Jack"<Graham.StJack internode.on.net>   wrote in message
 news:ie76ig$b2v$1 digitalmars.com...

 What you are suggesting here seems to be a way to dramatically reduce
 the
 complexity of code that generates source-code and mixes it in. I think
 something like that is needed before this mind-bogglingly powerful
 feature
 of D can realise its potential.


 I think a decent string-template library could probably come very close
 to
 the proposal without needing any language changes at all:

 string get_set(T, string name)()
 {
        return
        q{
             type _ name;

             type  name ()
            {
                return _ name;
            }

             type  name ( type  name)
            {
                return _ name =  name;
            }
        }.replace( [" type": T.stringof, " name": name] );
 }

 class Foo
 {
        mixin(get_set!(int, "bar")());
 }

 There are definitely some things about the proposal that are better than
 with this, but I just wanted to point out that the value of the proposal
 should probably be evaluated against something roughly like the above
 rather
 than something that does a bunch of procedural string splicing.


 The whole point of the idea was to get rid of the strings and the mixin
 expression (as it looks like to day).


 While I'm not necessarily opposed to the idea of getting rid of the strings,
 I guess I don't really see what improvement your proposal provides other
 than not having to manually specify the mapping of "replace what identifier
 with what data".

 Getting rid of the "mixin" I see as a separate issue. We could just as
 easily say that given the function "string get_set(...)":

       get_set("int", "bar");
      or
       get_set int bar;

 ...Is shorthand for, or is even the new syntax for:

      mixin(get_set("int", "bar"));


That was my idea as well, that

 get_set("int", "bar");

could be translated into

mixin(get_set("int", "bar")); just like

just like scope statements are translated into try/catch/finally.

-- 
/Jacob Carlborg
Dec 15 2010
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:

 On 2010-12-14 21:44, Nick Sabalausky wrote:

 "Jacob Carlborg"<doob me.com>  wrote in message
 news:ie8i8c$15f0$1 digitalmars.com...
 

 On 2010-12-14 19:13, Nick Sabalausky wrote:

 "Graham St Jack"<Graham.StJack internode.on.net>   wrote in message
 news:ie76ig$b2v$1 digitalmars.com...
 

 What you are suggesting here seems to be a way to dramatically reduce
 the
 complexity of code that generates source-code and mixes it in. I think
 something like that is needed before this mind-bogglingly powerful
 feature
 of D can realise its potential.


 
 I think a decent string-template library could probably come very close
 to
 the proposal without needing any language changes at all:
 
 string get_set(T, string name)()
 {
 
        return
        q{
        
             type _ name;
            
             type  name ()
            {
            
                return _ name;
            
            }
            
             type  name ( type  name)
            {
            
                return _ name =  name;
            
            }
        
        }.replace( [" type": T.stringof, " name": name] );
 
 }
 
 class Foo
 {
 
        mixin(get_set!(int, "bar")());
 
 }
 
 There are definitely some things about the proposal that are better
 than with this, but I just wanted to point out that the value of the
 proposal should probably be evaluated against something roughly like
 the above rather
 than something that does a bunch of procedural string splicing.


 
 The whole point of the idea was to get rid of the strings and the mixin
 expression (as it looks like to day).


 
 While I'm not necessarily opposed to the idea of getting rid of the
 strings, I guess I don't really see what improvement your proposal
 provides other than not having to manually specify the mapping of
 "replace what identifier with what data".
 
 Getting rid of the "mixin" I see as a separate issue. We could just as
 
 easily say that given the function "string get_set(...)":
       get_set("int", "bar");
      or
       get_set int bar;
 
 ...Is shorthand for, or is even the new syntax for:
      mixin(get_set("int", "bar"));


 
 That was my idea as well, that
 
  get_set("int", "bar");
 
 could be translated into
 
 mixin(get_set("int", "bar")); just like
 
 just like scope statements are translated into try/catch/finally.


Honestly, I don't see much gain in using   rather than mixin(). It's a little 
less typing, but that's it. And it precludes stuff like mixin("lhs " ~ op ~ " 
rhs") like happens all the time in overloaded operator functions.

- Jonathan M Davis
Dec 15 2010
parent reply "Nick Sabalausky" <a a.a> writes:
"Jonathan M Davis" <jmdavisProg gmx.com> wrote in message 
news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...

 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:

 That was my idea as well, that

  get_set("int", "bar");

 could be translated into

 mixin(get_set("int", "bar")); just like

 just like scope statements are translated into try/catch/finally.


 Honestly, I don't see much gain in using   rather than mixin(). It's a 
 little
 less typing, but that's it.


It does seem like a small difference, just replacing "mixin" with " " and 
removing one layer of parens. But I think that extra layer of parens, minor 
as it seems, makes a big difference in the readability (and "typeability") 
of mixin invocations. Those extra parens do get to be a real bother, major 
visual noise at least to my eyes.


 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.


I don't see why these shouldn't work:

 "int foo;";
return  ("lhs " ~ op ~ " rhs");

At least with just the " " part of the proposal. Maybe the delegate thing 
might make it tricker, I dunno.
Dec 15 2010
next sibling parent reply =?ISO-8859-1?Q?Pelle_M=E5nsson?= <pelle.mansson gmail.com> writes:
On 12/15/2010 11:00 PM, Nick Sabalausky wrote:

 "Jonathan M Davis"<jmdavisProg gmx.com>  wrote in message
 news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...

 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:

 That was my idea as well, that

  get_set("int", "bar");

 could be translated into

 mixin(get_set("int", "bar")); just like

 just like scope statements are translated into try/catch/finally.


 Honestly, I don't see much gain in using   rather than mixin(). It's a
 little
 less typing, but that's it.


 It does seem like a small difference, just replacing "mixin" with " " and
 removing one layer of parens. But I think that extra layer of parens, minor
 as it seems, makes a big difference in the readability (and "typeability")
 of mixin invocations. Those extra parens do get to be a real bother, major
 visual noise at least to my eyes.


I agree with this. Actually, just removing the parenthesis would be a 
huge gain for me.


 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.


 I don't see why these shouldn't work:

  "int foo;";
 return  ("lhs " ~ op ~ " rhs");

 At least with just the " " part of the proposal. Maybe the delegate thing
 might make it tricker, I dunno.


This could work, but I don't think anyone is suggesting completely 
replacing the mixin. I think   could be a function/template thing, and 
have strings in a more explicit mixin"";

Then again, inconsistency sucks.
Dec 16 2010
parent Jacob Carlborg <doob me.com> writes:
On 2010-12-16 11:18, Pelle Månsson wrote:

 On 12/15/2010 11:00 PM, Nick Sabalausky wrote:

 "Jonathan M Davis"<jmdavisProg gmx.com> wrote in message
 news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...

 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:

 That was my idea as well, that

  get_set("int", "bar");

 could be translated into

 mixin(get_set("int", "bar")); just like

 just like scope statements are translated into try/catch/finally.


 Honestly, I don't see much gain in using   rather than mixin(). It's a
 little
 less typing, but that's it.


 It does seem like a small difference, just replacing "mixin" with " " and
 removing one layer of parens. But I think that extra layer of parens,
 minor
 as it seems, makes a big difference in the readability (and
 "typeability")
 of mixin invocations. Those extra parens do get to be a real bother,
 major
 visual noise at least to my eyes.


 I agree with this. Actually, just removing the parenthesis would be a
 huge gain for me.


 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.


 I don't see why these shouldn't work:

  "int foo;";
 return  ("lhs " ~ op ~ " rhs");

 At least with just the " " part of the proposal. Maybe the delegate thing
 might make it tricker, I dunno.


 This could work, but I don't think anyone is suggesting completely
 replacing the mixin. I think   could be a function/template thing, and
 have strings in a more explicit mixin"";

 Then again, inconsistency sucks.


The whole point of the idea/suggestion was to get rid of the strings. 
Then if the syntax I've suggested is translated into the string mixin 
syntax we have now then I would be fine with that.

-- 
/Jacob Carlborg
Dec 16 2010
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-15 23:00, Nick Sabalausky wrote:

 "Jonathan M Davis"<jmdavisProg gmx.com>  wrote in message
 news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...

 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:

 That was my idea as well, that

  get_set("int", "bar");

 could be translated into

 mixin(get_set("int", "bar")); just like

 just like scope statements are translated into try/catch/finally.


 Honestly, I don't see much gain in using   rather than mixin(). It's a
 little
 less typing, but that's it.


 It does seem like a small difference, just replacing "mixin" with " " and
 removing one layer of parens. But I think that extra layer of parens, minor
 as it seems, makes a big difference in the readability (and "typeability")
 of mixin invocations. Those extra parens do get to be a real bother, major
 visual noise at least to my eyes.


^^ I completely agree.


 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.


 I don't see why these shouldn't work:

  "int foo;";
 return  ("lhs " ~ op ~ " rhs");

 At least with just the " " part of the proposal. Maybe the delegate thing
 might make it tricker, I dunno.


My idea was actually to get rid of the strings where the code to be 
mixed in is defined and to have a better syntax where it's used.

The delegates are just a way of passing a block of code around. If you 
just use it in place then maybe one could do like this:

 (int foo;);
return  (lhs (op)rhs);

-- 
/Jacob Carlborg
Dec 16 2010
next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:iedpbg$3i0$1 digitalmars.com...

 On 2010-12-15 23:00, Nick Sabalausky wrote:

 I don't see why these shouldn't work:

  "int foo;";
 return  ("lhs " ~ op ~ " rhs");

 At least with just the " " part of the proposal. Maybe the delegate thing
 might make it tricker, I dunno.


 My idea was actually to get rid of the strings where the code to be mixed 
 in is defined and to have a better syntax where it's used.

 The delegates are just a way of passing a block of code around. If you 
 just use it in place then maybe one could do like this:

  (int foo;);
 return  (lhs (op)rhs);


Yea, my point was just that the " ..." stuff could work either way, with the 
string-based system or with your delegate-based one.

I don't mean to come across like I'm ignoring or against the idea of the 
whole delegate aspect, and I understand that the main point of the OP is to 
replace the strings with delegates, but with the q{...} syntax and 
string-templating, I'm still struggling to see a big enough benefit compared 
to the status quo. I see that using delegates instead of strings could 
probably be made to work, but my questions are "For what benefit(s)?" and 
"Would those benefits be sufficient to warrant the change?" I'm not 
necessarily saying the answer is "no", but I'm unconvinced so far.

And here's another thing: Suppose we got a Ruby/PHP-like syntax for 
embedding code substitutions directly into a string (which would have other 
useful applications besides mixins):

auto name = "Joe";


Joe++;

Would that eliminate much (or all) of the benefit of the delegate approach?
Dec 16 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-16 21:35, Nick Sabalausky wrote:

 "Jacob Carlborg"<doob me.com>  wrote in message
 news:iedpbg$3i0$1 digitalmars.com...

 On 2010-12-15 23:00, Nick Sabalausky wrote:

 I don't see why these shouldn't work:

  "int foo;";
 return  ("lhs " ~ op ~ " rhs");

 At least with just the " " part of the proposal. Maybe the delegate thing
 might make it tricker, I dunno.


 My idea was actually to get rid of the strings where the code to be mixed
 in is defined and to have a better syntax where it's used.

 The delegates are just a way of passing a block of code around. If you
 just use it in place then maybe one could do like this:

  (int foo;);
 return  (lhs (op)rhs);


 Yea, my point was just that the " ..." stuff could work either way, with the
 string-based system or with your delegate-based one.

 I don't mean to come across like I'm ignoring or against the idea of the
 whole delegate aspect, and I understand that the main point of the OP is to
 replace the strings with delegates, but with the q{...} syntax and
 string-templating, I'm still struggling to see a big enough benefit compared
 to the status quo. I see that using delegates instead of strings could
 probably be made to work, but my questions are "For what benefit(s)?" and
 "Would those benefits be sufficient to warrant the change?" I'm not
 necessarily saying the answer is "no", but I'm unconvinced so far.

 And here's another thing: Suppose we got a Ruby/PHP-like syntax for
 embedding code substitutions directly into a string (which would have other
 useful applications besides mixins):

 auto name = "Joe";


 Joe++;

 Would that eliminate much (or all) of the benefit of the delegate approach?


I guess using q{...} with string interpolation is very similar to the 
delegate approach. It just feels wrong passing around strings to 
represent code. I haven't though much about it but with delegates one 
could at lest hope for better help from the compiler validating the 
code. I don't know how IDEs will treat q{...} but with delegates you 
would get the full benefit of the IDE like autocompletion and similar 
features.

-- 
/Jacob Carlborg
Dec 16 2010
parent "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:iee4en$ttb$1 digitalmars.com...

 On 2010-12-16 21:35, Nick Sabalausky wrote:

 "Jacob Carlborg"<doob me.com>  wrote in message
 news:iedpbg$3i0$1 digitalmars.com...

 On 2010-12-15 23:00, Nick Sabalausky wrote:

 I don't see why these shouldn't work:

  "int foo;";
 return  ("lhs " ~ op ~ " rhs");

 At least with just the " " part of the proposal. Maybe the delegate 
 thing
 might make it tricker, I dunno.


 My idea was actually to get rid of the strings where the code to be 
 mixed
 in is defined and to have a better syntax where it's used.

 The delegates are just a way of passing a block of code around. If you
 just use it in place then maybe one could do like this:

  (int foo;);
 return  (lhs (op)rhs);


 Yea, my point was just that the " ..." stuff could work either way, with 
 the
 string-based system or with your delegate-based one.

 I don't mean to come across like I'm ignoring or against the idea of the
 whole delegate aspect, and I understand that the main point of the OP is 
 to
 replace the strings with delegates, but with the q{...} syntax and
 string-templating, I'm still struggling to see a big enough benefit 
 compared
 to the status quo. I see that using delegates instead of strings could
 probably be made to work, but my questions are "For what benefit(s)?" and
 "Would those benefits be sufficient to warrant the change?" I'm not
 necessarily saying the answer is "no", but I'm unconvinced so far.

 And here's another thing: Suppose we got a Ruby/PHP-like syntax for
 embedding code substitutions directly into a string (which would have 
 other
 useful applications besides mixins):

 auto name = "Joe";


 Joe++;

 Would that eliminate much (or all) of the benefit of the delegate 
 approach?


 I guess using q{...} with string interpolation is very similar to the 
 delegate approach. It just feels wrong passing around strings to represent 
 code.


Well, code *is* text after all. But I know what you mean - after all, it 
does have more semantic structure than just ordinary generic strings.


 I haven't though much about it but with delegates one could at lest hope 
 for better help from the compiler validating the code. I don't know how 
 IDEs will treat q{...} but with delegates you would get the full benefit 
 of the IDE like autocompletion and similar features.


My editor (Programmer's Notepad 2, based off Scintilla) handles that fine. 
It doesn't know anything about q{}, so it assumes it's an identifier ("q") 
followed by a normal code block. And since it doesn't try to do any 
grammatical/semantic validation (only lexical, and only for the purpose of 
highlighting) it doesn't complain about "identifier { ... }" being invalid 
or any of the indentifiers-to-be-replaced inside of it being undeclared.

But for fancier IDE's, like Eclipse with Descent or DDT, I don't know - 
that's a good question.

OTOH, even with the delegate approach, I'm assuming that delegate would 
still get evaluated in a different context from where it's defined (which 
you'd probably want). So that might still cause some trouble with the more 
intelligent IDEs trying to tell you that identifierXYZ isn't accessable from 
within what it thinks is the delegate's scope.
Dec 16 2010
prev sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, December 16, 2010 11:28:03 Jacob Carlborg wrote:

 On 2010-12-15 23:00, Nick Sabalausky wrote:

 "Jonathan M Davis"<jmdavisProg gmx.com>  wrote in message
 news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...
 

 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:

 That was my idea as well, that
 
  get_set("int", "bar");
 
 could be translated into
 
 mixin(get_set("int", "bar")); just like
 
 just like scope statements are translated into try/catch/finally.


 
 Honestly, I don't see much gain in using   rather than mixin(). It's a
 little
 less typing, but that's it.


 
 It does seem like a small difference, just replacing "mixin" with " " and
 removing one layer of parens. But I think that extra layer of parens,
 minor as it seems, makes a big difference in the readability (and
 "typeability") of mixin invocations. Those extra parens do get to be a
 real bother, major visual noise at least to my eyes.


 
 ^^ I completely agree.
 

 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.


 
 I don't see why these shouldn't work:
 
  "int foo;";
 return  ("lhs " ~ op ~ " rhs");
 
 At least with just the " " part of the proposal. Maybe the delegate thing
 might make it tricker, I dunno.


 
 My idea was actually to get rid of the strings where the code to be
 mixed in is defined and to have a better syntax where it's used.
 
 The delegates are just a way of passing a block of code around. If you
 just use it in place then maybe one could do like this:
 
  (int foo;);
 return  (lhs (op)rhs);


I would have thought that template mixins would be the thing to use when you 
didn't want to deal with strings. string mixins are extremely powerful and 
flexible, and I'd really hate to lose them. And IIRC, Kenji Hara was working on
a 
module to really help make dealing with complicated string mixins easier and 
less painful. Anything that you propose is going to have to have major benefits 
over the current string mixin situation for it to stand any chance of being 
accepted.

- Jonathan M Davis
Dec 16 2010
parent reply Jacob Carlborg <doob me.com> writes:
On 2010-12-16 23:05, Jonathan M Davis wrote:

 On Thursday, December 16, 2010 11:28:03 Jacob Carlborg wrote:

 On 2010-12-15 23:00, Nick Sabalausky wrote:

 "Jonathan M Davis"<jmdavisProg gmx.com>   wrote in message
 news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...


 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:

 That was my idea as well, that

  get_set("int", "bar");

 could be translated into

 mixin(get_set("int", "bar")); just like

 just like scope statements are translated into try/catch/finally.


 Honestly, I don't see much gain in using   rather than mixin(). It's a
 little
 less typing, but that's it.


 It does seem like a small difference, just replacing "mixin" with " " and
 removing one layer of parens. But I think that extra layer of parens,
 minor as it seems, makes a big difference in the readability (and
 "typeability") of mixin invocations. Those extra parens do get to be a
 real bother, major visual noise at least to my eyes.


 ^^ I completely agree.


 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.


 I don't see why these shouldn't work:

  "int foo;";
 return  ("lhs " ~ op ~ " rhs");

 At least with just the " " part of the proposal. Maybe the delegate thing
 might make it tricker, I dunno.


 My idea was actually to get rid of the strings where the code to be
 mixed in is defined and to have a better syntax where it's used.

 The delegates are just a way of passing a block of code around. If you
 just use it in place then maybe one could do like this:

  (int foo;);
 return  (lhs (op)rhs);


 I would have thought that template mixins would be the thing to use when you
 didn't want to deal with strings. string mixins are extremely powerful and
 flexible, and I'd really hate to lose them. And IIRC, Kenji Hara was working
on a
 module to really help make dealing with complicated string mixins easier and
 less painful. Anything that you propose is going to have to have major benefits
 over the current string mixin situation for it to stand any chance of being
 accepted.

 - Jonathan M Davis


Template mixins and string mixins are used for different things. There's 
a lot of things that string mixins can do that template mixins can't. I 
have no intention what so ever to suggest something that isn't as 
powerful as string mixins, just a new syntax. If it turns out that a 
having a powerful syntax without strings isn't possibles than I'll guess 
we have to live with the strings.

Don't know if you read my first post put there I wrote that it wasn't a 
real suggestion (at least not yet) I just wanted the community's 
thoughts on the idea and see if we could turn it into something useful 
that could become a real suggestion, if people where interested.

The ideas I wrote in my extended suggestion, "Taking it one step 
further", I think that those can have benefits over string mixins. 
Basically allowing you to pass the whole body of a class declaration to 
a function, as a delegate, with a syntax looking like Java annotations.

-- 
/Jacob Carlborg
Dec 16 2010
parent "Nick Sabalausky" <a a.a> writes:
"Jacob Carlborg" <doob me.com> wrote in message 
news:iee561$v5s$1 digitalmars.com...

 On 2010-12-16 23:05, Jonathan M Davis wrote:

 On Thursday, December 16, 2010 11:28:03 Jacob Carlborg wrote:

 On 2010-12-15 23:00, Nick Sabalausky wrote:

 "Jonathan M Davis"<jmdavisProg gmx.com>   wrote in message
 news:mailman.1035.1292441722.21107.digitalmars-d puremagic.com...


 On Wednesday, December 15, 2010 11:27:47 Jacob Carlborg wrote:

 That was my idea as well, that

  get_set("int", "bar");

 could be translated into

 mixin(get_set("int", "bar")); just like

 just like scope statements are translated into try/catch/finally.


 Honestly, I don't see much gain in using   rather than mixin(). It's a
 little
 less typing, but that's it.


 It does seem like a small difference, just replacing "mixin" with " " 
 and
 removing one layer of parens. But I think that extra layer of parens,
 minor as it seems, makes a big difference in the readability (and
 "typeability") of mixin invocations. Those extra parens do get to be a
 real bother, major visual noise at least to my eyes.


 ^^ I completely agree.


 And it precludes stuff like mixin("lhs " ~ op ~ "
 rhs") like happens all the time in overloaded operator functions.


 I don't see why these shouldn't work:

  "int foo;";
 return  ("lhs " ~ op ~ " rhs");

 At least with just the " " part of the proposal. Maybe the delegate 
 thing
 might make it tricker, I dunno.


 My idea was actually to get rid of the strings where the code to be
 mixed in is defined and to have a better syntax where it's used.

 The delegates are just a way of passing a block of code around. If you
 just use it in place then maybe one could do like this:

  (int foo;);
 return  (lhs (op)rhs);


 I would have thought that template mixins would be the thing to use when 
 you
 didn't want to deal with strings. string mixins are extremely powerful 
 and
 flexible, and I'd really hate to lose them. And IIRC, Kenji Hara was 
 working on a
 module to really help make dealing with complicated string mixins easier 
 and
 less painful. Anything that you propose is going to have to have major 
 benefits
 over the current string mixin situation for it to stand any chance of 
 being
 accepted.

 - Jonathan M Davis


 Template mixins and string mixins are used for different things. There's a 
 lot of things that string mixins can do that template mixins can't. I have 
 no intention what so ever to suggest something that isn't as powerful as 
 string mixins, just a new syntax. If it turns out that a having a powerful 
 syntax without strings isn't possibles than I'll guess we have to live 
 with the strings.

 Don't know if you read my first post put there I wrote that it wasn't a 
 real suggestion (at least not yet) I just wanted the community's thoughts 
 on the idea and see if we could turn it into something useful that could 
 become a real suggestion, if people where interested.

 The ideas I wrote in my extended suggestion, "Taking it one step further", 
 I think that those can have benefits over string mixins. Basically 
 allowing you to pass the whole body of a class declaration to a function, 
 as a delegate, with a syntax looking like Java annotations.


It would seem to make sense to treat "chucks of code" the same way 
regardless of whether you're passing them around, mixing them in or 
instantiating them as a template.
Dec 16 2010
prev sibling next sibling parent VladD2 <vc rsdn.ru> writes:
Don Wrote:


 You missed: pattern matching, 


 
 Yes, you've got me there. I've assumed that pattern matching, while a 
 major feature, is not fundamental to the Nemerle macro-system, but I may 
 be mistaken.
 
 One obvious difference which *is* fundamental is that Nemerle macros 
 allow new syntax.
 
 Am I correct in thinking that Nemerle always requires complete ASTs?
 That is, given a name "x", can you access variables "x1", "x2", "x3" ?
 
  > quasi-quotation,
 
 No, that's present in D. It's the primary reason I say that the 
 differences are mostly syntactic, since I see this as THE fundamental 
 feature. You need to be able to reenter the macro system after you have 
 left it. Once you can do that, you can do pretty much anything.


You not understand... pattern matching and quasi-quotation tightly bound. The
quasi-quotation produce instances of Algebraic Data Type (AlgTD) which can be
decomposed (and matched) by pattern matching (PM). We use PM to "read" code of
a project.


http://code.google.com/p/nemerle/source/browse/nemerle/trunk/snippets/csharp-parser/CSharpParser/Parser.n?r=9436
which made by PegGrammar macro:
http://code.google.com/p/nemerle/source/browse/nemerle/trunk/snippets/peg-parser/Nemerle.Peg.Macros?r=9436

Or see our foreach macro:



 running of fully functional Nemerle
 code in compile time, 


 
 Yes, but that's a different issue. In D, running code at compile time is 
   regarded as an aspect of constant-folding, and is not restricted to 
 macros.


The constant-folding is too limited solution. 

The power of of macros, including the ability to use are any libraries and any
data sources.

And ... you use enterpretation... slow way.


 access to compiler API in macros


 
 In practice, has to be a library, right? Otherwise the compiler 
 internals would be exposed? (This is an issue we're struggling with).


Yes. The Nemerle compiler is library. :)
Unfortunately Nemerle API is not pretty clear. But I believe it's a right way.

P.S.

Sorry for my (Russian) English. :)
Dec 17 2010
prev sibling parent foobar <foo bar.com> writes:
Nick Sabalausky Wrote:


 
 The problem with that is, what if you're generating target-platform-specific 
 code at compile-time? You'd be generating code for the wrong platform. I 
 think VladD2 is right: You need to keep track of both "current" system and 
 "target" system. Unfortunately, there is some information about the "target" 
 system the compile-time code wouldn't be able discern without giving it the 
 ability to run code (RPC? Virtualization? Really, really good emulator?) on 
 the target system, but then again, that's a limitation with any 
 cross-compiling scenario.
 
 


I don't understand this. If you're talking about version blocks, obviously all
the current D compile time features are incompatible with the macro system.
They're also redundant cause you can (and should) simply write instead:

macro (version) {
  if (version == Version.Linux) { ...}
}

Also, the macro dll is compiled for some platform Z and will only be loadable
by a compiler that also runs on platform Z.  I don't think that macros need to
be cross-compiled for our (X,Y) scenario, they need to evaluate a
TargetPlatform parameter at their run-time which the cross-compiler will
provide (much like current D have predefined version identifiers defined by the
compiler)
Dec 19 2010