www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - newCTFE Status August 2019

reply Stefan Koch <uplink.coder googlemail.com> writes:
Hi Guys,

I had 2 weeks of vacation recently, and I used them to work on 
newCTFE again.

The main addition was enriched internal debug data.
Now newCTFE can emit proper dwarf source_locations which used 
with the gccjit backend.
This means given you use the gccjit-backend for newCTFE you can 
(step through!) ctfe-code.
At least in theory ... in reality some of the time there will not 
be source code for you to step through, (think mixins).

Another rather important bugfix was done the handling of for 
loops, it turned out that for loops without increment statement 
would not properly behave when `continue` was encountered (they'd 
jump to the end of the loop instead of the beginning).

There are currently 2 main bugs still to be fixed:
-  1. floating point corner cases
-  2. wrong code on indirections inside closures.

It should be noted that I just spent the better part of a week 
working on the aforementioned 2 issues, and it does not look like 
they'll be fixed soon.

That said, I am hopeful to be able to release newCTFE in 2020!

My Thanks to everyone in this forum who encouraged me.

Also a special thanks to Johnathan Blow, who's occasional streams 
on Jai's progress led me to pick up the newCTFE torch again!

Cheers,

Stefan

P.S. the current newCTFE patch has about 17000 additional lines.
10700 of which are actual interpreter and test-code, and the rest 
are internal testcases, alternative back-ends 
(LLVM/GCCJIT/C/IR-Printer).

Re-basing to newer versions of the compiler will be done once the 
dmd/druntime/phobos test-suites and my internal tests are passing.
Aug 15 2019
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Aug 15, 2019 at 09:42:47PM +0000, Stefan Koch via Digitalmars-d wrote:

 Hi Guys,
 
 I had 2 weeks of vacation recently, and I used them to work on newCTFE
 again.


Yes!!!


[...]

 That said, I am hopeful to be able to release newCTFE in 2020!


*celebratory dance*



 My Thanks to everyone in this forum who encouraged me.
 
 Also a special thanks to Johnathan Blow, who's occasional streams on
 Jai's progress led me to pick up the newCTFE torch again!


[...]

Go, Stefan, go!  Bring newCTFE into a reality!  D badly needs this.
Non-trivial CTFE is what D can do that no other language I know can.
Right now it's seriously hampered by lackluster speed and memory usage;
newCTFE can turn this into a huge game-changer.


T

-- 
Do not reason with the unreasonable; you lose by definition.
Aug 15 2019
prev sibling next sibling parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Thursday, 15 August 2019 at 21:42:47 UTC, Stefan Koch wrote:

 Hi Guys,

 I had 2 weeks of vacation recently, and I used them to work on 
 newCTFE again.


Nice to see that you keep the ball rolling!

[…]

 It should be noted that I just spent the better part of a week 
 working on the aforementioned 2 issues, and it does not look 
 like they'll be fixed soon.


Last time you told me about that other difficult issue, you were 
able to fix it shortly after. So who knows, maybe you will get a 
similar breakthrough soon.


 That said, I am hopeful to be able to release newCTFE in 2020!


That would be great!


 My Thanks to everyone in this forum who encouraged me.

 Also a special thanks to Johnathan Blow, who's occasional 
 streams on Jai's progress led me to pick up the newCTFE torch 
 again!


Interesting…

Keep having fun,
Bastiaan.
Aug 17 2019
prev sibling next sibling parent reply Gregor =?UTF-8?B?TcO8Y2ts?= <gregormueckl gmx.de> writes:
On Thursday, 15 August 2019 at 21:42:47 UTC, Stefan Koch wrote:

 P.S. the current newCTFE patch has about 17000 additional lines.
 10700 of which are actual interpreter and test-code, and the 
 rest are internal testcases, alternative back-ends 
 (LLVM/GCCJIT/C/IR-Printer).

 Re-basing to newer versions of the compiler will be done once 
 the dmd/druntime/phobos test-suites and my internal tests are 
 passing.


Are you developing this out in the open or are you keeping the 
code closer to your chest until you are done?

Both are fine with me. I'm mostly curious.

Keep up the good work! You have taken on a beast of a spare time 
project!
Aug 18 2019
parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Sunday, 18 August 2019 at 22:18:29 UTC, Gregor Mückl wrote:

 Are you developing this out in the open or are you keeping the 
 code closer to your chest until you are done?


These branches have been recently updated:

https://github.com/UplinkCoder/dmd/tree/newCTFE_reboot_20741
https://github.com/UplinkCoder/dmd/tree/newCTFE_reboot_20741_gccjit
Aug 19 2019
prev sibling next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Thursday, 15 August 2019 at 21:42:47 UTC, Stefan Koch wrote:

 That said, I am hopeful to be able to release newCTFE in 2020!


Can't wait! We'll be able to make a lot more parsing at 
compile-time :)
Aug 19 2019
parent reply Ethan <gooberman gmail.com> writes:
On Monday, 19 August 2019 at 13:37:18 UTC, Guillaume Piolat wrote:

 Can't wait! We'll be able to make a lot more parsing at 
 compile-time :)


My compiles are already at 8GB memory usage thanks to CTFE.

Even just a patch to fix array memory handling behind the scenes 
would be great. But newCTFE would certainly be the preferred 
option.
Aug 19 2019
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/19/2019 7:02 AM, Ethan wrote:

 Even just a patch to fix array memory handling behind the scenes would be
great.


The memory handling could be fixed if the CTFE's allocated AST's were all 
allocated from a separate pool, and marked as such. Then, the returned AST from 
CTFE can be scanned and every marked node replaced. Then, the entire pool can
be 
discarded.

Should be fairly simple to implement.
Aug 19 2019
next sibling parent reply Paolo Invernizzi <paolo.invernizzi gmail.com> writes:
On Monday, 19 August 2019 at 23:04:23 UTC, Walter Bright wrote:

 On 8/19/2019 7:02 AM, Ethan wrote:

 Even just a patch to fix array memory handling behind the 
 scenes would be great.


 The memory handling could be fixed if the CTFE's allocated 
 AST's were all allocated from a separate pool, and marked as 
 such. Then, the returned AST from CTFE can be scanned and every 
 marked node replaced. Then, the entire pool can be discarded.

 Should be fairly simple to implement.


Replying to this to keep the topic on top!
That's an amazing outcome!

What would be also, IYUO, the expected impact on speed 
performance?

/Paolo
Aug 20 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/20/2019 6:43 AM, Paolo Invernizzi wrote:

 What would be also, IYUO, the expected impact on speed performance?


Probably none, excluding the effects of reduced swapping.
Aug 20 2019
prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 19 August 2019 at 23:04:23 UTC, Walter Bright wrote:

 On 8/19/2019 7:02 AM, Ethan wrote:

 Even just a patch to fix array memory handling behind the 
 scenes would be great.


 The memory handling could be fixed if the CTFE's allocated 
 AST's were all allocated from a separate pool, and marked as 
 such. Then, the returned AST from CTFE can be scanned and every 
 marked node replaced. Then, the entire pool can be discarded.

 Should be fairly simple to implement.


At least last time I tried that, (which was 2.8 years ago), I ran 
into problems with deciding which nodes to keep. As CTFE can 
trigger semantic() on FunctionDeclarations, and all of the nodes 
created there have to be kept, even though they might not be 
referenced from the ctfe result.
Aug 20 2019
parent Walter Bright <newshound2 digitalmars.com> writes:
On 8/20/2019 5:38 PM, Stefan Koch wrote:

 At least last time I tried that, (which was 2.8 years ago), I ran into
problems 
 with deciding which nodes to keep. As CTFE can trigger semantic() on 
 FunctionDeclarations, and all of the nodes created there have to be kept, even 
 though they might not be referenced from the ctfe result.


Doing a reference count on the Region allocator should do the trick.
Aug 24 2019
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/19/2019 4:04 PM, Walter Bright wrote:

 Should be fairly simple to implement.


https://github.com/dlang/dmd/pull/10343
Aug 24 2019
next sibling parent Ethan <gooberman gmail.com> writes:
On Saturday, 24 August 2019 at 08:23:52 UTC, Walter Bright wrote:

 https://github.com/dlang/dmd/pull/10343


I will eagerly test this out in the very near future.
Aug 24 2019
prev sibling parent reply RazvanN <razvan.nitu1305 gmail.com> writes:
On Saturday, 24 August 2019 at 08:23:52 UTC, Walter Bright wrote:

 On 8/19/2019 4:04 PM, Walter Bright wrote:

 Should be fairly simple to implement.


 https://github.com/dlang/dmd/pull/10343


Does this fix: https://issues.dlang.org/show_bug.cgi?id=1382 ? I 
would test it myself, but last time I tried, my computer hanged.
Aug 27 2019
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 8/27/2019 7:12 AM, RazvanN wrote:

 Does this fix: https://issues.dlang.org/show_bug.cgi?id=1382 ? I would test it 
 myself, but last time I tried, my computer hanged.


Not yet, it's just a start.
Aug 28 2019
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 28 August 2019 at 21:31:45 UTC, Walter Bright wrote:

 On 8/27/2019 7:12 AM, RazvanN wrote:

 Does this fix: https://issues.dlang.org/show_bug.cgi?id=1382 ? 
 I would test it myself, but last time I tried, my computer 
 hanged.


 Not yet, it's just a start.


The region allocater can be used in newCTFE as well.
So in this case our work is synergistic.

Any more work poured into the interpret.d is likely to be 
superseeded by newCTFE in a few months.
Aug 29 2019
prev sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 27 August 2019 at 14:12:50 UTC, RazvanN wrote:

 On Saturday, 24 August 2019 at 08:23:52 UTC, Walter Bright 
 wrote:

 On 8/19/2019 4:04 PM, Walter Bright wrote:

 Should be fairly simple to implement.


 https://github.com/dlang/dmd/pull/10343


 Does this fix: https://issues.dlang.org/show_bug.cgi?id=1382 ? 
 I would test it myself, but last time I tried, my computer 
 hanged.


newCTFE will fix this.
Infact for the example posted in the bug, it already does.
Aug 29 2019
prev sibling parent reply Jon Degenhardt <jond noreply.com> writes:
On Thursday, 15 August 2019 at 21:42:47 UTC, Stefan Koch wrote:

 Hi Guys,

 I had 2 weeks of vacation recently, and I used them to work on 
 newCTFE again.
 [...]

 That said, I am hopeful to be able to release newCTFE in 2020!


Hi Stefan,

Great that you're still getting some time to work on this!

By coincidence, I just ran into a case. I'm curious, will newCTFE 
enable the usage below?

------------------
$ cat ctfe_case.d
struct C
{
     import std.conv : to;
     string s = double.init.to!string;   // Fails to compile.
}

void main()
{
     import std.stdio;
     C c;
     writeln(c.s);
}

$ dmd ctfe_case.d
/Library/D/dmd/src/phobos/std/exception.d(515): Error: uncaught 
CTFE exception std.format.FormatException("Cannot format floating 
point types at compile-time")
ctfe_case.d(4):        called from here: to(nan)
---------------

Using a similar code construct in a function works fine:

----------------
$ cat ctfe_case2.d
void main()
{
     import std.stdio;
     import std.conv : to;

     string s = double.init.to!string;
     writeln(s);
}
$ dmd ctfe_case2.d
$ ./ctfe_case2
nan
-----------------

--Jon
Aug 31 2019
parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Saturday, 31 August 2019 at 21:44:38 UTC, Jon Degenhardt wrote:

 By coincidence, I just ran into a case. I'm curious, will 
 newCTFE enable the usage below?


This isn't actually a CTFE engine thing, just the library 
implementation.

The current CTFE engine can do it if you use a different library 
function to do the conversion (Stefan I believe has ported one 
from C to D before and it is ... somewhere...).

The reason it won't do it now is that it calls out to a C 
function to format floating point, and CTFE - new and old - can 
only evaluate D functions.


 Using a similar code construct in a function works fine:

 ----------------
 $ cat ctfe_case2.d


this one is not ctfe btw.
Aug 31 2019
parent reply Jon Degenhardt <jond noreply.com> writes:
On Saturday, 31 August 2019 at 21:49:18 UTC, Adam D. Ruppe wrote:

 On Saturday, 31 August 2019 at 21:44:38 UTC, Jon Degenhardt 
 wrote:

 By coincidence, I just ran into a case. I'm curious, will 
 newCTFE enable the usage below?


 This isn't actually a CTFE engine thing, just the library 
 implementation.


Interesting. Thanks Adam. Suggests that there's more to having 
consistent behavior than just the engine itself.
Aug 31 2019
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 31 August 2019 at 22:00:47 UTC, Jon Degenhardt wrote:

 On Saturday, 31 August 2019 at 21:49:18 UTC, Adam D. Ruppe 
 wrote:

 On Saturday, 31 August 2019 at 21:44:38 UTC, Jon Degenhardt 
 wrote:

 By coincidence, I just ran into a case. I'm curious, will 
 newCTFE enable the usage below?


 This isn't actually a CTFE engine thing, just the library 
 implementation.


 Interesting. Thanks Adam. Suggests that there's more to having 
 consistent behavior than just the engine itself.


https://github.com/night-shift/fpconv/blob/master/src/fpconv_ctfe.d

You can use the link a quoted above.
Aug 31 2019
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Saturday, August 31, 2019 4:00:47 PM MDT Jon Degenhardt via Digitalmars-d 
wrote:

 On Saturday, 31 August 2019 at 21:49:18 UTC, Adam D. Ruppe wrote:

 On Saturday, 31 August 2019 at 21:44:38 UTC, Jon Degenhardt

 wrote:

 By coincidence, I just ran into a case. I'm curious, will
 newCTFE enable the usage below?


 This isn't actually a CTFE engine thing, just the library
 implementation.


 Interesting. Thanks Adam. Suggests that there's more to having
 consistent behavior than just the engine itself.


Well, the second example didn't even use CTFE. CTFE gets used when the
result needs to be known at compile time. The init value of a struct needs
to be known at compile time, so anything that directly initializes a
struct's members has to be known at compile time and will trigger CTFE. On
the other hand, a non-static, local variable doesn't need to have its value
known at compile time, so CTFE isn't used.

The only question of consistent behavior here is whether the expression
being evaluated can be done at both compile time and at runtime, and due to
how to!string uses a C function to format floating point values, it only
works at runtime. The same will happen any time that a piece of code calls a
C function or does anything else that can't be done at compile time. That's
why some code uses

if(__ctfe)

to provide an alternate implementation that works at compile time (e.g. if
pointer arithmetic were being used for performance in the non-CTFE branch of
the code). However, in this case, rather than having a small change to the
code for a CTFE branch, having a CTFE branch of the code would require
providing an implementation for formatting floating point values which was
entirely written in D - which IIRC Stefan does have floating around
somewhere (he ported to D from somewhere else) - but as long as std.conv.to
always calls into C functions when formatting floating point values, then it
can't work with floating points and CTFE any more than writeln can.

Unfortunately, it's typically the case that you can't know whether something
will work at compile time until you try it, because it depends on how the
functions you're calling were implemented, and that's not likely to ever
change. Somehow, everything would have to work at compile time for that not
to be the case.

As for newCTFE, it's my understanding that it doesn't actually support
anything that the current CTFE engine doesn't support; rather it's just
implementing it all in a far more efficient manner so that it's much faster
and doesn't consume ludicrous amounts of memory (which will be a huge
enabler but doesn't necessarily allow any particular code to compile which
can't currently compile unless compilation time or memory usage is what's
currently preventing it). In fact, IIRC, Stefan set it up so that it will
fall back to the old engine in some cases, so it's not even going to fully
replace the old one. However, I haven't been following everything he's been
doing closely, so maybe some of what's he's doing or planning as changed.
Having a much saner CTFE engine will likely make it easier to improve
though. From what I understand about it, the current way that CTFE works is
kind of insane, which is a lot of why it's so inefficient.

- Jonathan M Davis
Sep 01 2019