www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Why does this mixin fail to compile?

reply ryuukk_ <ryuukk.dev gmail.com> writes:
This simple mixin fails to compile, anyone know why?

```D
mixin implement;

mixin template implement() {
     mixin("struct _gen(T) {");
     mixin("}");
}

void main(){}
```

```
onlineapp.d-mixin-5(5): Error: `}` expected following members in 
`struct` declaration
onlineapp.d-mixin-5(5):        struct `_gen` starts here
onlineapp.d(6): Error: mixin `__anonymous` incomplete mixin 
declaration `}`
onlineapp.d(2): Error: mixin `onlineapp.implement!()` error 
instantiating
```
Jul 01
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Monday, 1 July 2024 at 09:25:39 UTC, ryuukk_ wrote:

 This simple mixin fails to compile, anyone know why?

 ```D
 mixin implement;

 mixin template implement() {
     mixin("struct _gen(T) {");
     mixin("}");
 }


A string mixin must form a complete declaration / statement / 
expression / type, so you can't end on an open brace.
Jul 01
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Monday, 1 July 2024 at 09:29:50 UTC, Dennis wrote:

 On Monday, 1 July 2024 at 09:25:39 UTC, ryuukk_ wrote:

 This simple mixin fails to compile, anyone know why?

 ```D
 mixin implement;

 mixin template implement() {
     mixin("struct _gen(T) {");
     mixin("}");
 }


 A string mixin must form a complete declaration / statement / 
 expression / type, so you can't end on an open brace.


this is sad, how i am supposed to do a foreach inbetween?

```D
mixin implement!TEST;

mixin template implement(stuff) {
     mixin("struct _gen(T) {");

     // pseudo code
     static foreach(args)
         mixin("stuff");

     mixin("}");
}

void main(){}
```
Jul 01
parent reply drug007 <drug2004 bk.ru> writes:
On 01.07.2024 12:39, ryuukk_ wrote:

 On Monday, 1 July 2024 at 09:29:50 UTC, Dennis wrote:

 On Monday, 1 July 2024 at 09:25:39 UTC, ryuukk_ wrote:

 This simple mixin fails to compile, anyone know why?

 ```D
 mixin implement;

 mixin template implement() {
     mixin("struct _gen(T) {");
     mixin("}");
 }


 A string mixin must form a complete declaration / statement / 
 expression / type, so you can't end on an open brace.


 
 this is sad, how i am supposed to do a foreach inbetween?
 
 ```D
 mixin implement!TEST;
 
 mixin template implement(stuff) {
      mixin("struct _gen(T) {");
 
      // pseudo code
      static foreach(args)
          mixin("stuff");
 
      mixin("}");
 }
 
 void main(){}
 ```


No problem, make a full string then mixin it.
Jul 01
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Monday, 1 July 2024 at 10:20:25 UTC, drug007 wrote:

 On 01.07.2024 12:39, ryuukk_ wrote:

 On Monday, 1 July 2024 at 09:29:50 UTC, Dennis wrote:

 On Monday, 1 July 2024 at 09:25:39 UTC, ryuukk_ wrote:

 This simple mixin fails to compile, anyone know why?

 ```D
 mixin implement;

 mixin template implement() {
     mixin("struct _gen(T) {");
     mixin("}");
 }


 A string mixin must form a complete declaration / statement / 
 expression / type, so you can't end on an open brace.


 
 this is sad, how i am supposed to do a foreach inbetween?
 
 ```D
 mixin implement!TEST;
 
 mixin template implement(stuff) {
      mixin("struct _gen(T) {");
 
      // pseudo code
      static foreach(args)
          mixin("stuff");
 
      mixin("}");
 }
 
 void main(){}
 ```


 No problem, make a full string then mixin it.


Your "no problem" = lot of string concatenation therefore slower 
and memory hungry, i have no desire to do that
Jul 01
parent reply drug007 <drug2004 bk.ru> writes:
On 01.07.2024 13:31, ryuukk_ wrote:

 On Monday, 1 July 2024 at 10:20:25 UTC, drug007 wrote:

 No problem, make a full string then mixin it.


 
 Your "no problem" = lot of string concatenation therefore slower and 
 memory hungry, i have no desire to do that


Do you think that string concatenation is the most heavy operation on 
using string mixin?

When you pass the mixin string to the compiler it allocates memory too. 
If you want to use incomplete mixin strings the compiler will accumulate 
it (allocating additional memory) until it builds a complete AST node to 
mix in at once. At best, it will cost the same as building a full string 
as I suggested but in general it may cost even more. Also, compilation 
would take longer because the compiler would need to have additional 
logic to detect the end of your string mixin etc. Your profit from 
avoiding string concatenation would almost negative.
Jul 01
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
Ok, i'll just do it and benchmark at the end

Another question:


Why doesn't this work?:


```D
mixin implement;

mixin template implement()
{
     char[4096] buffer = 0;
     int pos = 0;

     void append(string str)
     {
         buffer[pos .. pos + str.length] = str[];
         pos += str.length;
     }

     append("void* ctx;");
     mixin(cast(string) buffer[0 .. pos]);
}
```


```
onlineapp.d(13): Error: unexpected `(` in declarator
onlineapp.d(13): Error: basic type expected, not `"void* ctx;"`
onlineapp.d(13): Error: found `"void* ctx;"` when expecting `)`
onlineapp.d(13): Error: no identifier for declarator 
`append(_error_)`
onlineapp.d(13): Error: semicolon expected following function 
declaration, not `)`
onlineapp.d(13): Error: declaration expected, not `)`
```
Jul 01
parent reply drug007 <drug2004 bk.ru> writes:
On 01.07.2024 15:26, ryuukk_ wrote:

 Ok, i'll just do it and benchmark at the end
 
 Another question:
 
 
 Why doesn't this work?:
 


I'm pretty sure your code does not work the way you think it does. 
String Mixins and Template Mixins are two different things. If you want 
to use String Mixins just make a function that creates a complete string 
with the source code and then mix it in. No need for Template Mixins in 
this case.

```D
string generateSourceCode(T)()
{
      return T.stringof ~ "* ctx;";
}

void main()
{
     mixin(generateSourceCode!int);

     static assert(is(typeof(ctx) == int*));

     assert(ctx is null);
}
```
Jul 01
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Monday, 1 July 2024 at 12:43:01 UTC, drug007 wrote:

 On 01.07.2024 15:26, ryuukk_ wrote:

 Ok, i'll just do it and benchmark at the end
 
 Another question:
 
 
 Why doesn't this work?:
 


 I'm pretty sure your code does not work the way you think it 
 does. String Mixins and Template Mixins are two different 
 things. If you want to use String Mixins just make a function 
 that creates a complete string with the source code and then 
 mix it in. No need for Template Mixins in this case.

 ```D
 string generateSourceCode(T)()
 {
      return T.stringof ~ "* ctx;";
 }

 void main()
 {
     mixin(generateSourceCode!int);

     static assert(is(typeof(ctx) == int*));

     assert(ctx is null);
 }
 ```


please stick to what i wrote, i don't want string concatenation, 
i provide a reduced example from my project, everything should be 
a single template block, no extra functions other than the 
append() one
Jul 01
next sibling parent monkyyy <crazymonkyyy gmail.com> writes:
On Monday, 1 July 2024 at 13:00:55 UTC, ryuukk_ wrote:

 i don't want string concatenation


This limitation is very intentional, add it to the pile like file 
io in ctfe of stuff that the core devs think "you shouldnt even 
want that" for "safety"
Jul 01
prev sibling parent reply Dennis <dkorpel gmail.com> writes:
On Monday, 1 July 2024 at 13:00:55 UTC, ryuukk_ wrote:

 please stick to what i wrote, i don't want string 
 concatenation, i provide a reduced example from my project, 
 everything should be a single template block, no extra 
 functions other than the append() one


Mixin templates are a declaration scope, not a function scope, so 
they aren't necessarily analyzed top to bottom. This is why 
allowing partial string mixins would be complicated, consider 
this example:

```D
mixin template implement()
{
     mixin("struct " ~ str);

     mixin("{");

     immutable string str = "T";

     mixin("}");
}
```

It's not obvious how this should be compiled, and this is before 
throwing `static if` and `static foreach` in the mix! If you want 
to procedurally build a type in sequential steps, you'll have to 
do that in a function scope.

If your concern is that such a function would add needless code 
generation, you can use an immediately invoked anonymous function 
like so:

```D
mixin template implement(string typeName, string[] members)
{
     mixin(() {
         string result = "struct " ~ typeName ~ " {";
         foreach (name; members)
         {
             result ~= "int " ~ name ~ ";";
         }
         result ~= "}";
         return result;
     } ());
}

mixin implement!("S", ["x", "y", "z"]);

immutable s = S(x: 3, y: 5, z: 7);
```

You can use your fixed size array append function to try and 
improve CTFE performance, but I'd start with straightforward 
concatenation, and see if it's actually too slow. In that case, 
maybe see if you can reduce it to a self-contained example and 
post it on bugzilla as a performance bug.
Jul 01
parent reply ryuukk_ <ryuukk.dev gmail.com> writes:
On Monday, 1 July 2024 at 21:43:02 UTC, Dennis wrote:

 On Monday, 1 July 2024 at 13:00:55 UTC, ryuukk_ wrote:

 please stick to what i wrote, i don't want string 
 concatenation, i provide a reduced example from my project, 
 everything should be a single template block, no extra 
 functions other than the append() one


 Mixin templates are a declaration scope, not a function scope, 
 so they aren't necessarily analyzed top to bottom. This is why 
 allowing partial string mixins would be complicated, consider 
 this example:

 ```D
 mixin template implement()
 {
     mixin("struct " ~ str);

     mixin("{");

     immutable string str = "T";

     mixin("}");
 }
 ```

 It's not obvious how this should be compiled, and this is 
 before throwing `static if` and `static foreach` in the mix! If 
 you want to procedurally build a type in sequential steps, 
 you'll have to do that in a function scope.

 If your concern is that such a function would add needless code 
 generation, you can use an immediately invoked anonymous 
 function like so:

 ```D
 mixin template implement(string typeName, string[] members)
 {
     mixin(() {
         string result = "struct " ~ typeName ~ " {";
         foreach (name; members)
         {
             result ~= "int " ~ name ~ ";";
         }
         result ~= "}";
         return result;
     } ());
 }

 mixin implement!("S", ["x", "y", "z"]);

 immutable s = S(x: 3, y: 5, z: 7);
 ```

 You can use your fixed size array append function to try and 
 improve CTFE performance, but I'd start with straightforward 
 concatenation, and see if it's actually too slow. In that case, 
 maybe see if you can reduce it to a self-contained example and 
 post it on bugzilla as a performance bug.


I said it 2 times already, i don't want string concatenation, 
i'll benchmark later, but not right now, right now i'm looking 
for a functioning code without string concatenation
Jul 02
next sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, July 2, 2024 1:23:42 AM MDT ryuukk_ via Digitalmars-d-learn wrote:

 I said it 2 times already, i don't want string concatenation,
 i'll benchmark later, but not right now, right now i'm looking
 for a functioning code without string concatenation


D has two options for mixins, and both of them require complete statements.

Template mixins mix in the contents of a template rather than strings, but
they only allow you to compile in declarations, because that's all that
templates can contain:

https://dlang.org/spec/template-mixin.html

You can control what is compiled into a template mixin via conditional
compilation based on the template arguments (e.g. with static if or static
foreach), but the declarations do need to be complete just like they would
for any template, so it will really only work for you if you just need a set
of declarations and not if you're looking to build a list of statements to
run.

For anything more complex, you'll need string mixins:

https://dlang.org/spec/statement.html#mixin-statement

A string mixin takes any string that's known at compile time and compiles it
as code, but the string must contain a complete statement (or a sequence of
statements). You can provide the mixin a string literal, an enum, a list of
strings which are concatenated together, a function call, or any other
expression so long as it evaluates to a string at compile time.

You can of course choose to not use string concatenation to create such a
string, but unless you can provide a string literal with the exact piece of
code that you need, you're going to need to actually build a string from
pieces one way or another, and that normally means either appending to a
string and/or concatenating strings together.

The only way that I can think of at the moment which would involve building
a string at compile time but which would not involve explicit allocations
would be to use a static array and set the unused characters to whitespace,
since the compiler will let you mix in a static array of characters as a
string, e.g.

mixin(()
      {
          char[1024] buffer = ' ';
          ... build the string by setting the characters in buffer ...
          return buffer;
      }());

However, given that this is CTFE we're talking about here, I'm pretty sure
that this particular solution not only allocates, but it potentially
allocates more than simply appending to a string would. IIRC, CTFE's current
implementation doesn't actually use mutatation. Rather, it creates a new
value every time that you mutate a variable. And if that is indeed the case,
then each time you set a character in the static array, you'd basically be
duping the array. CTFE does not work in a manner that pretty much anyone who
hasn't actively worked on it is going to expect, and the generated code
isn't necessarily anything like what would be generated for runtime code.
So, not many D programmers are going to be very good at guessing what's
going to be most efficient with CTFE.

Regardless, because a string mixin must be given an expression that
evaluates to a string, your options are basically

1. Use string concatenation in an expression that you pass to mixin.

2. Call a function that returns a string (with the internals of that
function doing whatever is required to generate that string and return it)
and pass the return value to mixin.



So, if you want to build a statement from pieces instead of providing a
known statement or known list of statements, and you don't want to use
string concatenation, you basically have to write a function which builds
and returns a string in whatever the most efficient fashion is that you can
come up with.

And if you want to measure the performance of a particular function that
builds a string during CTFE, you _will_ need to measure build times rather
than testing it at runtime because of how differently CTFE behaves from
running code at runtime. And honestly, I expect that you would have to be
generating a lot of string mixins to be able to get a consistent, measurable
difference in build times between different implementations, but if you care
that much about build times, you'll need to experiment to see what the most
efficient way to build a string is for your code. It is certainly true
though that CTFE tends to be pretty inefficient, unfortunately. It really is
a shame that "newCTFE" was never completed.

In general, best practice would be to just build the string in whatever the
simplest, most maintainable fashion would be, particularly since it will
have no impact on the runtime performance of the program, and you could
easily spend more time trying to optimize your build times than you would
ever save by trying generate the string more efficiently, but it's your
code.

- Jonathan M Davis
Jul 02
next sibling parent IchorDev <zxinsworld gmail.com> writes:
On Tuesday, 2 July 2024 at 16:46:11 UTC, Jonathan M Davis wrote:

 On Tuesday, July 2, 2024 1:23:42 AM MDT ryuukk_ via 
 Digitalmars-d-learn wrote:

 I said it 2 times already, i don't want string concatenation, 
 i'll benchmark later, but not right now, right now i'm looking 
 for a functioning code without string concatenation


 D has two options for mixins, and both of them require complete 
 statements.


Not a huge deal, but you forgot about the third option—mixin 
types—which does not require a complete statement: 
https://dlang.org/spec/type.html#mixin_types
Aug 12
prev sibling parent reply IchorDev <zxinsworld gmail.com> writes:
On Tuesday, 2 July 2024 at 16:46:11 UTC, Jonathan M Davis wrote:

 It really is a shame that "newCTFE" was never completed.


I didn’t know about this. I thought a new CTFE engine was 
actually implemented a few years ago. Is ‘newCTFE’ too old to be 
resurrected? Would it be feasible to make something similar from 
the ground up?
Aug 12
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 12 August 2024 at 11:26:50 UTC, IchorDev wrote:

 On Tuesday, 2 July 2024 at 16:46:11 UTC, Jonathan M Davis wrote:

 It really is a shame that "newCTFE" was never completed.


 I didn’t know about this. I thought a new CTFE engine was 
 actually implemented a few years ago. Is ‘newCTFE’ too old to 
 be resurrected? Would it be feasible to make something similar 
 from the ground up?


It's about 12.000 lines of code.
I am not sure how much has rotted away, I last touched it in 2021.

     The relevant files would be:
       the bytecode interpreter:
         
https://github.com/UplinkCoder/dmd/blob/newCTFE_2093/src/dmd/ctfe/bc.d
       the visitor that builds the bytodecode:
         
https://github.com/UplinkCoder/dmd/blob/newCTFE_2093/src/dmd/ctfe/ctfe_bc.d

as you can see by the branch name it was rebased on dmd 2.093.
so that's a few versions old now.
As far as I know Max Haughton has tried to rebase it on master 
and improve it but I have not heard from him about that.
Aug 12
parent reply IchorDev <zxinsworld gmail.com> writes:
On Monday, 12 August 2024 at 15:23:31 UTC, Stefan Koch wrote:

 It's about 12.000 lines of code.
 I am not sure how much has rotted away, I last touched it in 
 2021.

 The relevant files would be:
 the bytecode interpreter:
 https://github.com/UplinkCoder/dmd/blob/newCTFE_2093/src/dmd/ctfe/bc.d
 the visitor that builds the bytodecode:
 https://github.com/UplinkCoder/dmd/blob/newCTFE_2093/src/dmd/ctfe/ctfe_bc.d

 as you can see by the branch name it was rebased on dmd 2.093.
 so that's a few versions old now.


My word! I wonder how difficult it would be to maintain that if 
it actually got merged into dmd. Do you remember what parts of it 
were left unfinished?


 As far as I know Max Haughton has tried to rebase it on master 
 and improve it but I have not heard from him about that.


I won’t get my hopes up for one person to rebase 12k lines 
through 14.5K commits. It’s simply not feasible. I think a larger 
community effort would be required to successfully resurrect this 
implementation; although I am perhaps foolhardy enough to try it 
myself…
Aug 13
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 13 August 2024 at 19:40:30 UTC, IchorDev wrote:

 On Monday, 12 August 2024 at 15:23:31 UTC, Stefan Koch wrote:

 It's about 12.000 lines of code.
 I am not sure how much has rotted away, I last touched it in 
 2021.

 The relevant files would be:
 the bytecode interpreter:
 https://github.com/UplinkCoder/dmd/blob/newCTFE_2093/src/dmd/ctfe/bc.d
 the visitor that builds the bytodecode:
 https://github.com/UplinkCoder/dmd/blob/newCTFE_2093/src/dmd/ctfe/ctfe_bc.d

 as you can see by the branch name it was rebased on dmd 2.093.
 so that's a few versions old now.


 My word! I wonder how difficult it would be to maintain that if 
 it actually got merged into dmd. Do you remember what parts of 
 it were left unfinished?


I do not know on the top of my head.
I remember some math was not interpreted correctly.
Though after I stopped working with D I never went back to look.


 As far as I know Max Haughton has tried to rebase it on master 
 and improve it but I have not heard from him about that.


 I won’t get my hopes up for one person to rebase 12k lines 
 through 14.5K commits. It’s simply not feasible. I think a 
 larger community effort would be required to successfully 
 resurrect this implementation; although I am perhaps foolhardy 
 enough to try it myself…


The code is rather self contained.
I usually don't rebase but I simply copy the files from the ctfe 
directory into the master checkout do what I can to make it 
compile and then commit them as new files.
The history has been lost a long time ago already.

The only change to existing files should be in dinterpret.d where 
I hook myself into the existing evaluator.
Aug 15
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On Tuesday, 2 July 2024 at 07:23:42 UTC, ryuukk_ wrote:


 I said it 2 times already, i don't want string concatenation, 
 i'll benchmark later, but not right now, right now i'm looking 
 for a functioning code without string concatenation


Your buffer solution works, but you need to put it inside a 
*function*, not at declaration scope. What you wrote declares 
*runtime* variables, which wouldn't be usable at compile time (if 
you got it past the parser, which is where it was failing).

So for instance:

```d
mixin template implement()
{
     mixin(() {
         // ctfe new array is basically the same as static array
         char[] buffer = new char[4096];
         int pos = 0;

         void append(string str)
         {
             buffer[pos .. pos + str.length] = str[];
             pos += str.length;
         }

         append("void* ctx;");
         return buffer[0 .. pos];
     }());
}
```

And yes, it is faster to do this than appending. But you have to 
do a *lot* of it to make a huge difference.

-Steve
Jul 02
next sibling parent reply Lance Bachmeier <no spam.net> writes:
On Wednesday, 3 July 2024 at 03:52:41 UTC, Steven Schveighoffer 
wrote:


 ```d
 mixin template implement()
 {
     mixin(() {
         // ctfe new array is basically the same as static array
         char[] buffer = new char[4096];
         int pos = 0;

         void append(string str)
         {
             buffer[pos .. pos + str.length] = str[];
             pos += str.length;
         }

         append("void* ctx;");
         return buffer[0 .. pos];
     }());
 }
 ```

 And yes, it is faster to do this than appending. But you have 
 to do a *lot* of it to make a huge difference.


What about creating an array of strings and then using join? One 
obvious problem with this solution is the requirement to specify 
the size of `buffer`.
Jul 03
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On Wednesday, 3 July 2024 at 11:57:58 UTC, Lance Bachmeier wrote:

 On Wednesday, 3 July 2024 at 03:52:41 UTC, Steven Schveighoffer 
 wrote:



 And yes, it is faster to do this than appending. But you have 
 to do a *lot* of it to make a huge difference.


 What about creating an array of strings and then using join? 
 One obvious problem with this solution is the requirement to 
 specify the size of `buffer`.


It's important to remember that what happens at CTFE is not the 
same as what happens at runtime. CTFE is a tree-walking 
interpreter, and does weird things like allocate storage when you 
change a value. And there is no optimization.

So the best thing to do is try it and see what happens. You can't 
use your knowledge of how code compiles to judge CTFE performance.

My general feel is that the more code you give to the CTFE 
interpreter, the worse it gets. Like, try doing `format` in CTFE 
vs. simple appending it will be vastly different.

-Steve
Jul 03
prev sibling parent ryuukk_ <ryuukk.dev gmail.com> writes:
On Wednesday, 3 July 2024 at 03:52:41 UTC, Steven Schveighoffer 
wrote:

 On Tuesday, 2 July 2024 at 07:23:42 UTC, ryuukk_ wrote:


 I said it 2 times already, i don't want string concatenation, 
 i'll benchmark later, but not right now, right now i'm looking 
 for a functioning code without string concatenation


 Your buffer solution works, but you need to put it inside a 
 *function*, not at declaration scope. What you wrote declares 
 *runtime* variables, which wouldn't be usable at compile time 
 (if you got it past the parser, which is where it was failing).

 So for instance:

 ```d
 mixin template implement()
 {
     mixin(() {
         // ctfe new array is basically the same as static array
         char[] buffer = new char[4096];
         int pos = 0;

         void append(string str)
         {
             buffer[pos .. pos + str.length] = str[];
             pos += str.length;
         }

         append("void* ctx;");
         return buffer[0 .. pos];
     }());
 }
 ```

 And yes, it is faster to do this than appending. But you have 
 to do a *lot* of it to make a huge difference.

 -Steve


That's annoying that it couldn't have been more straight 
forward.. but that'll do it for now, it works, thanks
Jul 06
prev sibling parent cc <cc nevernet.com> writes:
I've been using patterns such as this with no difficulty.  
ctfe+mixins == it just works.
I don't know if there's some character limit where eventually the 
ctfe string return will give up, but for moderate programs it 
seems fine.

```d
// Copy fields from another struct, wrapping non-primary key 
elements in Nullable!T
private static string UpdaterFields(STRUCT)() {
	string[] s;
	static foreach (idx, field; STRUCT.tupleof) {{
		enum NAME = field.stringof;
		enum bool isPrimary = hasUDA!(field, PRIMARY);
		static if (isPrimary) {
			enum typestr = format("typeof(STRUCT.%s)", NAME);
		} else {
			enum typestr = format("Nullable!(typeof(STRUCT.%s))", NAME);
		}
		string[] udas;
		static foreach (uda; __traits(getAttributes, field)) {
			udas ~= " ("~uda.stringof~")";
		}
		auto udastr = udas.length ? udas.join(" ")~" " : "";
		s ~= format("%s%s %s;", udastr, typestr, NAME);
	}}
	return s.join("\n");
}
pragma(msg, UpdaterFields!STRUCT);
mixin(UpdaterFields!STRUCT);
```
Aug 14