digitalmars.D.internals - Detect CTFE in AssignExp:semantic
- Lucia Cojocaru (102/102) Jan 10 2017 Hello,
- Walter Bright (6/6) Jan 10 2017 The way to attack these sorts of problems is to not attempt the complete...
- Andrei Alexandrescu (4/8) Jan 10 2017 [snip]
- Stefan Koch (6/13) Jan 10 2017 I cannot say much from this snippet alone. Please discribe what
- Lucia Cojocaru (10/10) Jan 11 2017 The dmd code (e2ir.d and expression.d are of interest):
- Martin Nowak (14/22) Jan 11 2017 Calls for the old dmd<->C-API are very different from template
- Andrei Alexandrescu (12/32) Jan 11 2017 Cool. That looks different from
- Martin Nowak (20/64) Jan 11 2017 Don't really understand your question. What are the 2 problems
- Andrei Alexandrescu (37/58) Jan 11 2017 The current implementation:
- Stefan Koch (4/5) Jan 11 2017 I would guess foreach(i; auto ref e; from) to[i] = e;
- Andrei Alexandrescu (2/6) Jan 11 2017 Why? Aren't operands at this point built-in slices? -- Andrei
- Stefan Koch (3/10) Jan 11 2017 it could still be that you are assigning from a const slice.
- Martin Nowak (2/3) Jan 12 2017 In which case ref for the foreach parameter still works.
- Lucia Cojocaru (8/12) Jan 16 2017 I updated the code based on your comments. Let me know if this is
- Martin Nowak (21/25) Jan 16 2017 Looks like you're running into some issues with the "magic" temporary
- Martin Nowak (8/14) Jan 12 2017 Sorry, it's not the best example, b/c that also generates a
- Stefan Koch (10/20) Jan 11 2017 You should not need to special case ctfe inside the compiler for
- Andrei Alexandrescu (4/8) Jan 11 2017 That's what confuses me, it's the read of the temporary not the code
- Stefan Koch (7/18) Jan 11 2017 Depending on how you choose optimize the certain cases.
- Walter Bright (15/17) Jan 11 2017 The example is about 40 lines of code. The way to figure it out is to si...
- Walter Bright (4/4) Jan 11 2017 Take a look at the D test suite, such as:
- Lucia Cojocaru (28/33) Jan 16 2017 Here is a smaller example:
- Walter Bright (6/21) Jan 16 2017 I just compiled it with:
- Martin Nowak (5/6) Jan 17 2017 The problem comes from also doing the lowering for
- Walter Bright (2/5) Jan 17 2017 Thanks!
Hello,
Brief context description:
I'm trying to replace calls to druntime functions such as
_d_arraycopy [0] with calls to templates. These calls are
generated by the compiler here [1] for array assignments like
below:
int[2] a = [1, 2];
int[2] b;
b[] = a[]; // _d_array_copy call
There are several such druntime functions and the purpose of this
is to avoid pulling druntime as a dependency unless necessary.
The templates don't "exist" if not used and many of them can be
implemented without pulling any dependency.
--------------------------------------------------------------
I implemented a lowering in AssignExp:semantic to a template call
for the use case described above. However compilation fails for
this code (I extracted this bit from druntime code):
1
2 char[] mangle(T)(const(char)[] fqn, char[] dst = null) safe
pure nothrow
3 {
4 import core.internal.string : numDigits,
unsignedToTempString;
5
6 static struct DotSplitter
7 {
8 safe pure nothrow:
9 const(char)[] s;
10
11 property bool empty() const { return !s.length; }
12
13 property const(char)[] front() const
14 {
15 immutable i = indexOfDot();
16 return i == -1 ? s[0 .. $] : s[0 .. i];
17 }
18
19 void popFront()
20 {
21 }
22
23 private ptrdiff_t indexOfDot() const
24 {
25 foreach (i, c; s) if (c == '.') return i;
26 return -1;
27 }
28 }
29
30 size_t len = "_D".length;
31 foreach (comp; DotSplitter(fqn))
32 len += numDigits(comp.length) + comp.length;
33
34 return ['a', 'b', 'c'];
35 }
36
37 char[] mangleFunc(T:FT*, FT)(const(char)[] fqn, char[] dst =
null) safe pure nothrow if (is(FT == function))
38 {
39 return mangle!FT(fqn, dst);
40 }
41
42 pragma(mangle, mangleFunc!(int function(int))("a.b"));
43
44 int main() {
45 return 0;
46 }
The error is:
ctfe.d(28): Error: 2 variable __r57 cannot be read at compile time
ctfe.d(28): called from here: _d_arraycopyT(this.s[],
__r57, 1u)
ctfe.d(18): called from here: this.indexOfDot()
ctfe.d(34): called from here: __r58.front()
ctfe.d(42): called from here: mangle(fqn, dst)
ctfe.d(45): called from here: mangleFunc("a.b", null)
This is probably because of an issue with CTFE and the solution
may be to avoid generating calls to the template in a CTFE
context. The code in AssignExp:semantic [2] would look like this:
if (__ctfe)
{
// add template call to AST
}
else
{
// old code
}
However I was not able to detect this situation from
AssignExp:semantic.
1. __ctfe doesn't seem to be set
2. performing similar checks with the ones in dinterpret which
throw the error [3] don't distinguish this situation
3. checking the scope for ctfe doesn't work either: sc.flags &
SCOPEctfe
Is there a way to achieve this? Maybe I am looking at this
problem from a wrong angle.
Thank you,
Lucia
[0]https://github.com/dlang/druntime/blob/2db828bd4f21807254b770b3ec304f14596a9805/src/rt/arraycat.d#L22
[1] https://github.com/dlang/dmd/blob/master/src/e2ir.d#L2702
[2]
https://github.com/dlang/dmd/blob/master/src/expression.d#L13545
[3]
https://github.com/dlang/dmd/blob/master/src/dinterpret.d#L2381
Jan 10 2017
Walter Bright <newshound2 digitalmars.com> The way to attack these sorts of problems is to not attempt the complete solution as the first step. Start with a massive simplification - for example, just having mangle() return "hello". If that fails in a confusing manner, go even simpler. Once the very simple solution works, then start adding in the complex solution bit by bit, ensuring each piece works before adding the next.
Jan 10 2017
On 01/10/2017 07:13 AM, Lucia Cojocaru wrote:Hello, Brief context description: I'm trying to replace calls to druntime functions such as _d_arraycopy [0] with calls to templates.[snip] You may also want to push your WIP into your dmd branch on github so people can take a look. Thanks! -- Andrei
Jan 10 2017
On Tuesday, 10 January 2017 at 12:13:18 UTC, Lucia Cojocaru wrote:Hello, Brief context description: I'm trying to replace calls to druntime functions such as _d_arraycopy [0] with calls to templates. These calls are generated by the compiler here [1] for array assignments like below: [...]I cannot say much from this snippet alone. Please discribe what you want to archive concretely. As previously mentioned it would also help to push your code to a visible branch. I will take a look shortly.
Jan 10 2017
The dmd code (e2ir.d and expression.d are of interest): https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090 The druntime template: https://github.com/somzzz/druntime/commit/6cf9cbc6650697d8a038be7076e588601aefe954 The example which doesn't compile is a standalone code snippet which reproduces the error encountered. I started from the code in druntime and simplified it to that point. As of your suggestions, I will simplify it further and come back with another example. Thanks!
Jan 11 2017
On Wednesday, 11 January 2017 at 09:05:44 UTC, Lucia Cojocaru wrote:The dmd code (e2ir.d and expression.d are of interest): https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090Calls for the old dmd<->C-API are very different from template functions calls, e.g. take a look at how _xOpEquals is called. https://github.com/dlang/dmd/blob/538a895157acdbbfc5869791f9504f7e86b4fdd0/src/clone.d#L496The druntime template:https://github.com/somzzz/druntime/commit/6cf9cbc6650697d8a038be7076e588601aefe954The example which doesn't compile is a standalone code snippet which reproduces the error encountered. I started from the code in druntime and simplified it to that point. As of your suggestions, I will simplify it further and come back with another example.You cannot distinguish between ctfe/non-ctfe during semantic. Only the backend/glue layer differs between CTFE (interpret.d) and IR/codegen (e2ir). If you want to convert a C-API intrinsic to a template function call, you'd usually deal with __ctfe in druntime not in the compiler. DMD will always call the druntime function and if that happens during CTFE, it'll get interpreted.
Jan 11 2017
On 1/11/17 3:16 PM, Martin Nowak wrote:On Wednesday, 11 January 2017 at 09:05:44 UTC, Lucia Cojocaru wrote:Cool. That looks different from https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090, is the latter appropriate as well? Or perhaps that's why the error with reading the variable during compilation?The dmd code (e2ir.d and expression.d are of interest): https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090Calls for the old dmd<->C-API are very different from template functions calls, e.g. take a look at how _xOpEquals is called. https://github.com/dlang/dmd/blob/538a895157acdbbfc5869791f9504f7e86b4fdd0/src/clone.d#L496OK, but the problem here is it indicates a problem at the call site of _d_arraycopyT, not inside the implementation. Is there an issue with the way the code is generated? Also, as an aside: the _d_arraycopyT should probably go like this: D _d_arraycopyT(S, D)(S from, D to) { ... } You don't need size because it's from[0].sizeof. Correct? AndreiThe druntime template:https://github.com/somzzz/druntime/commit/6cf9cbc6650697d8a038be7076e588601aefe954The example which doesn't compile is a standalone code snippet which reproduces the error encountered. I started from the code in druntime and simplified it to that point. As of your suggestions, I will simplify it further and come back with another example.You cannot distinguish between ctfe/non-ctfe during semantic. Only the backend/glue layer differs between CTFE (interpret.d) and IR/codegen (e2ir). If you want to convert a C-API intrinsic to a template function call, you'd usually deal with __ctfe in druntime not in the compiler. DMD will always call the druntime function and if that happens during CTFE, it'll get interpreted.
Jan 11 2017
On Wednesday, 11 January 2017 at 17:10:15 UTC, Andrei Alexandrescu wrote:On 1/11/17 3:16 PM, Martin Nowak wrote:Don't really understand your question. What are the 2 problems you refer to? The difference is fairly simple but huge: - C intrinsics - AssignExp.semantic - e2ir => call RTLSYM_SYM - interpret => special handling - D lowering - AssignExp.semantic lowered to CallExp of object._arrayCopy - normal function call and no special CTFE handlingOn Wednesday, 11 January 2017 at 09:05:44 UTC, Lucia Cojocaru wrote:Cool. That looks different from https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090, is the latter appropriate as well? Or perhaps that's why the error with reading the variable during compilation?The dmd code (e2ir.d and expression.d are of interest): https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090Calls for the old dmd<->C-API are very different from template functions calls, e.g. take a look at how _xOpEquals is called. https://github.com/dlang/dmd/blob/538a895157acdbbfc5869791f9504f7e86b4fdd0/src/clone.d#L496OK, but the problem here is it indicates a problem at the call site of _d_arraycopyT, not inside the implementation. Is there an issue with the way the code is generated?The druntime template:https://github.com/somzzz/druntime/commit/6cf9cbc6650697d8a038be7076e588601aefe954The example which doesn't compile is a standalone code snippet which reproduces the error encountered. I started from the code in druntime and simplified it to that point. As of your suggestions, I will simplify it further and come back with another example.You cannot distinguish between ctfe/non-ctfe during semantic. Only the backend/glue layer differs between CTFE (interpret.d) and IR/codegen (e2ir). If you want to convert a C-API intrinsic to a template function call, you'd usually deal with __ctfe in druntime not in the compiler. DMD will always call the druntime function and if that happens during CTFE, it'll get interpreted.Also, as an aside: the _d_arraycopyT should probably go like this: D _d_arraycopyT(S, D)(S from, D to) { ... } You don't need size because it's from[0].sizeof. Correct?Just convert the assignment to a function call, the backend deals with optimizations et.al. Also this seems to be used not only for static arrays. NB: - leave aways the _d prefix it's only needed to namespace extern(C) functions with flat mangling - prolly should be _arrayCopy(T)(in T[] from, T[] to) as AssignExp.semantic already takes care of conversions
Jan 11 2017
On 1/11/17 7:02 PM, Martin Nowak wrote:Don't really understand your question. What are the 2 problems you refer to?The current implementation: D _d_arraycopyT(S, D)(S from, D to, uint size) { import core.stdc.string; (() trusted => memcpy(cast(void*)to.ptr, from.ptr, to.length * size))(); return to; } The current error: ctfe.d(28): Error: 2 variable __r57 cannot be read at compile time ctfe.d(28): called from here: _d_arraycopyT(this.s[], __r57, 1u) i.e. NOT in the use of cast, ptr, or memcpy. The error points to the call site. The recommended implementation: D _d_arraycopyT(S, D)(S from, D to, uint size) { if (__ctfe) import core.stdc.string; (() trusted => memcpy(cast(void*)to.ptr, from.ptr, to.length * size))(); } else { foreach (i, ref e; from) to[i] = e; } return to; } It doesn't seem like the recommended implementation will make the error go away.The difference is fairly simple but huge: - C intrinsics - AssignExp.semantic - e2ir => call RTLSYM_SYM - interpret => special handling - D lowering - AssignExp.semantic lowered to CallExp of object._arrayCopy - normal function call and no special CTFE handlingI'm not familiar with the code, so although yes these are different I don't know how they translate into what needs to be done to make this work.I'm saying the third parameter is entirely redundant in all cases. It's the sizeof the element.Also, as an aside: the _d_arraycopyT should probably go like this: D _d_arraycopyT(S, D)(S from, D to) { ... } You don't need size because it's from[0].sizeof. Correct?Just convert the assignment to a function call, the backend deals with optimizations et.al. Also this seems to be used not only for static arrays.NB: - leave aways the _d prefix it's only needed to namespace extern(C) functions with flat mangling - prolly should be _arrayCopy(T)(in T[] from, T[] to) as AssignExp.semantic already takes care of conversionsYah, once we're past the hurdle of getting this to basically work these are good touches. Andrei
Jan 11 2017
On Wednesday, 11 January 2017 at 18:59:29 UTC, Andrei
Alexandrescu wrote:
foreach (i, ref e; from) to[i] = e;
I would guess foreach(i; auto ref e; from) to[i] = e;
is a little more compatible.
Jan 11 2017
On 1/11/17 8:34 PM, Stefan Koch wrote:On Wednesday, 11 January 2017 at 18:59:29 UTC, Andrei Alexandrescu wrote:Why? Aren't operands at this point built-in slices? -- Andreiforeach (i, ref e; from) to[i] = e;I would guess foreach(i; auto ref e; from) to[i] = e; is a little more compatible.
Jan 11 2017
On Wednesday, 11 January 2017 at 22:14:18 UTC, Andrei Alexandrescu wrote:On 1/11/17 8:34 PM, Stefan Koch wrote:it could still be that you are assigning from a const slice.On Wednesday, 11 January 2017 at 18:59:29 UTC, Andrei Alexandrescu wrote:Why? Aren't operands at this point built-in slices? -- Andreiforeach (i, ref e; from) to[i] = e;I would guess foreach(i; auto ref e; from) to[i] = e; is a little more compatible.
Jan 11 2017
On Wednesday, 11 January 2017 at 23:44:29 UTC, Stefan Koch wrote:it could still be that you are assigning from a const slice.In which case ref for the foreach parameter still works.
Jan 12 2017
On Thursday, 12 January 2017 at 11:09:09 UTC, Martin Nowak wrote:On Wednesday, 11 January 2017 at 23:44:29 UTC, Stefan Koch wrote:I updated the code based on your comments. Let me know if this is better. The issue still persists. druntime https://github.com/somzzz/druntime/commit/fb51f34cb0bdd96daaa92ab22773bf93778d4d11 dmd https://github.com/somzzz/dmd/commit/b3cf4625ce733e4b3f219bcd5069d9042430b594it could still be that you are assigning from a const slice.In which case ref for the foreach parameter still works.
Jan 16 2017
On 01/16/2017 11:22 PM, Lucia Cojocaru wrote:druntime https://github.com/somzzz/druntime/commit/fb51f34cb0bdd96daaa92ab22773bf93778d4d11https://github.com/somzzz/druntime/commit/fb51f34cb0bdd96daaa92ab22773bf93778d4d11#commitcomment-20497420dmd https://github.com/somzzz/dmd/commit/b3cf4625ce733e4b3f219bcd5069d9042430b594Looks like you're running into some issues with the "magic" temporary variables generated by the compiler. https://github.com/dlang/dmd/blob/a5f823a59d11bb02a56384891eb80a55af467e00/src/statementsem.d#L886-L907 The error is generated here https://github.com/dlang/dmd/blob/c1d138cc3860ecf8cbe06090cc321f4d5502b6ee/src/dinterpret.d#L2441 with hasValue(v) == false meaning that the variable was not interpreted (put on the stack) before usage. I guess this happens because you're actually converting the initialization of the temporary into a function call. T __r154 = slice[]; lowered into: _d_arraycopyT(__r154, slice[]); So the function call gets interpreted before the variable was initialized by CTFE, not exactly sure how to best resolve this. Maybe simply skip lowering ConstructExp (subclass of AssignExp used for construction) for now. You can recognize construction inside of AssignExp by looking at the op, `op == TOKconstruct`. -Martin
Jan 16 2017
On Wednesday, 11 January 2017 at 17:10:15 UTC, Andrei Alexandrescu wrote:Sorry, it's not the best example, b/c that also generates a function. Added inline comments on https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090 which isn't far off.Calls for the old dmd<->C-API are very different from template functions calls, e.g. take a look at how _xOpEquals is called. https://github.com/dlang/dmd/blob/538a895157acdbbfc5869791f9504f7e86b4fdd0/src/clone.d#L496Or perhaps that's why the error with reading the variable during compilation?Not sure where that comes from, there are unrelated changes as well.
Jan 12 2017
On Wednesday, 11 January 2017 at 09:05:44 UTC, Lucia Cojocaru wrote:The dmd code (e2ir.d and expression.d are of interest): https://github.com/somzzz/dmd/commit/8bccc49ba661567c523545650aad30c01fd25090 The druntime template: https://github.com/somzzz/druntime/commit/6cf9cbc6650697d8a038be7076e588601aefe954 The example which doesn't compile is a standalone code snippet which reproduces the error encountered. I started from the code in druntime and simplified it to that point. As of your suggestions, I will simplify it further and come back with another example. Thanks!You should not need to special case ctfe inside the compiler for this. Rather the template should have if (__ctfe) inside it if those are needed. However I would advise against splitting code-paths, if it is not _strictly_ necessary. I will be able to provide more help should you need it. Just contact me if you encounter further problems.
Jan 11 2017
On 1/11/17 4:25 PM, Stefan Koch wrote:You should not need to special case ctfe inside the compiler for this. Rather the template should have if (__ctfe) inside it if those are needed. However I would advise against splitting code-paths, if it is not _strictly_ necessary.That's what confuses me, it's the read of the temporary not the code inside the function. Would branching inside the function help with that? -- Andrei
Jan 11 2017
On Wednesday, 11 January 2017 at 18:51:08 UTC, Andrei Alexandrescu wrote:On 1/11/17 4:25 PM, Stefan Koch wrote:Depending on how you choose optimize the certain cases. You e.g. using sse2 instructions for copies of 128bit values, It may become non-ctfeable. Then you need a ctfeable branch in order for the function to still work at ctfe.You should not need to special case ctfe inside the compiler for this. Rather the template should have if (__ctfe) inside it if those are needed. However I would advise against splitting code-paths, if it is not _strictly_ necessary.That's what confuses me, it's the read of the temporary not the code inside the function. Would branching inside the function help with that? -- Andrei
Jan 11 2017
On 1/11/2017 10:51 AM, Andrei Alexandrescu wrote:That's what confuses me, it's the read of the temporary not the code inside the function. Would branching inside the function help with that? -- AndreiThe example is about 40 lines of code. The way to figure it out is to simplify the example. For starters, replace: char[] mangle(T)(const(char)[] fqn, char[] dst = null) with: char[] mangle(const(char)[] fqn) Note that the template doesn't even use T or dst. Nor does it use unsignedToTempString. Replace numDigits() with 1. Rewrite the foreach loop to use primitives, then reduce the primitives (like removing popFront() which does nothing). Never going to figure out what is wrong if the example is larded up with obfuscation and distraction. I bet it will shrink down to 3 lines that exhibit the same issue, at which point the problem will be obvious.
Jan 11 2017
Take a look at the D test suite, such as:
https://github.com/dlang/dmd/blob/master/test/runnable/test42.d
Each of these tests cases started out as some long complicated thing that got
reduced to a very small self-contained test case.
Jan 11 2017
On Thursday, 12 January 2017 at 01:29:12 UTC, Walter Bright wrote:
Take a look at the D test suite, such as:
https://github.com/dlang/dmd/blob/master/test/runnable/test42.d
Each of these tests cases started out as some long complicated
thing that got reduced to a very small self-contained test case.
Here is a smaller example:
ptrdiff_t f(char[] s) {
// foreach(c; s) if (c == '.') return 0; // (0)fails with the
same error message
foreach(char c; s) if (c == '.') return 0; // (1)
// foreach(dchar c; s) if (c == '.') return 0; // (2)
different compile path, doesn't fail
/*
for (size_t i = 0; i < s.length; i++)
{
if (s[i] == '.') return i;
}
*/
return -1;
}
pragma(msg, f(['z', 'b', 'c', '.', 'd']));
-------------------------------------------------------
ctfe.d(5): Error: variable __r54 cannot be read at compile time
ctfe.d(5): called from here: _d_arraycopyT(__r54, s[])
ctfe.d(15): called from here: f(['z', 'b', 'c', '.', 'd'])
ctfe.d(15): while evaluating pragma(msg, f(['z', 'b', 'c',
'.', 'd']))
-------------------------------------------------------
It looks like the problem is with the foreach loop. In the
example above, the (0) and (1) loops result in the compilation
error mentioned. Loop (2) with dchar is fine.
I also tried using a regular for loop and this one compiled.
Jan 16 2017
On 1/16/2017 2:09 PM, Lucia Cojocaru wrote:
ptrdiff_t f(char[] s) {
// foreach(c; s) if (c == '.') return 0; // (0)fails with the same error
message
foreach(char c; s) if (c == '.') return 0; // (1)
// foreach(dchar c; s) if (c == '.') return 0; // (2) different compile
path, doesn't fail
/*
for (size_t i = 0; i < s.length; i++)
{
if (s[i] == '.') return i;
}
*/
return -1;
}
pragma(msg, f(['z', 'b', 'c', '.', 'd']));
I just compiled it with:
dmd -c bug8
and it prints:
0
with no error message.
Jan 16 2017
On Tuesday, 17 January 2017 at 03:32:41 UTC, Walter Bright wrote:I just compiled it with:The problem comes from also doing the lowering for initialization, hence the CTFE interpreter has never seen the variable value before it is used. http://forum.dlang.org/post/o5jrti$n0t$1 digitalmars.com
Jan 17 2017
On 1/17/2017 4:39 AM, Martin Nowak wrote:The problem comes from also doing the lowering for initialization, hence the CTFE interpreter has never seen the variable value before it is used. http://forum.dlang.org/post/o5jrti$n0t$1 digitalmars.comThanks!
Jan 17 2017