www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Battle-plan for CTFE

reply Stefan Koch <uplink.coder googlemail.com> writes:
Hi Guys,

I have been looking into the DMD now to see what I can do about 
CTFE.
Unfortunately It is a pretty big mess to untangle.
Code responsible for CTFE is in at least 3 files.
[dinterpret.d, ctfeexpr.d, constfold.d]
I was shocked to discover that the PowExpression actually depends 
on phobos! (depending on the exact codePath it may or may not 
compile...)
which let to me prematurely stating that it worked at ctfe 
[http://forum.dlang.org/thread/ukcoibejffinknrbzktv forum.dlang.org]

My Plan is as follows.

Add a new file for my ctfe-interpreter and update it gradually to 
take more and more of the cases the code in the files mentioned 
above was used for.

Do Dataflow analysis on the code that is to be ctfe'd so we can 
tell beforehand if we need to store state in the ctfe stack or 
not.

Or baring proper data-flow analysis: RefCouting the variables on 
the ctfe-stack could also be a solution.

I will post more details as soon as I dive deeper into the code.
May 09 2016
next sibling parent Rory McGuire via Digitalmars-d-announce writes:
On 09 May 2016 19:01, "Stefan Koch via Digitalmars-d-announce" <
digitalmars-d-announce puremagic.com> wrote:

 Hi Guys,

 I have been looking into the DMD now to see what I can do about CTFE.
 Unfortunately It is a pretty big mess to untangle.
 Code responsible for CTFE is in at least 3 files.
 [dinterpret.d, ctfeexpr.d, constfold.d]
 I was shocked to discover that the PowExpression actually depends on


phobos! (depending on the exact codePath it may or may not compile...)

 which let to me prematurely stating that it worked at ctfe [


http://forum.dlang.org/thread/ukcoibejffinknrbzktv forum.dlang.org]

 My Plan is as follows.

 Add a new file for my ctfe-interpreter and update it gradually to take


more and more of the cases the code in the files mentioned above was used
for.

 Do Dataflow analysis on the code that is to be ctfe'd so we can tell


beforehand if we need to store state in the ctfe stack or not.

 Or baring proper data-flow analysis: RefCouting the variables on the


ctfe-stack could also be a solution.

 I will post more details as soon as I dive deeper into the code.


Will be awesome. Particularly if you document the workings of ctfe, might
make a great set of articles for a blog.
May 09 2016
prev sibling next sibling parent reply David Nadlinger <code klickverbot.at> writes:
Hi Stefan,

On Monday, 9 May 2016 at 16:57:39 UTC, Stefan Koch wrote:

 My Plan is as follows.


I think you guys talked about it at the conference, but be sure 
to coordinate with Timon Gehr. You'll want to steal all the best 
ideas from the various implementations anyway. ;)


 Do Dataflow analysis on the code that is to be ctfe'd so we can 
 tell beforehand if we need to store state in the ctfe stack or 
 not.

 Or baring proper data-flow analysis: RefCouting the variables 
 on the ctfe-stack could also be a solution.


I presume by "store state" you mean persisting objects beyond the 
bounds of a single CTFE invocation? My first inclination here 
would simply be to make all allocations from a new arena each 
time CTFE is entered (can also re-use memory from prior runs for 
that), do a deep-copy of the result (converting it to full AST 
nodes, etc.), and then drop the entire arena. But probably you 
have thought of (and discarded) this already.

  — David
May 09 2016
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 9 May 2016 at 18:19:53 UTC, David Nadlinger wrote:

 Hi Stefan,

 On Monday, 9 May 2016 at 16:57:39 UTC, Stefan Koch wrote:

 My Plan is as follows.


 I think you guys talked about it at the conference, but be sure 
 to coordinate with Timon Gehr. You'll want to steal all the 
 best ideas from the various implementations anyway. ;)


 Do Dataflow analysis on the code that is to be ctfe'd so we 
 can tell beforehand if we need to store state in the ctfe 
 stack or not.

 Or baring proper data-flow analysis: RefCouting the variables 
 on the ctfe-stack could also be a solution.


 I presume by "store state" you mean persisting objects beyond 
 the bounds of a single CTFE invocation? My first inclination 
 here would simply be to make all allocations from a new arena 
 each time CTFE is entered (can also re-use memory from prior 
 runs for that), do a deep-copy of the result (converting it to 
 full AST nodes, etc.), and then drop the entire arena. But 
 probably you have thought of (and discarded) this already.

  — David


The current implementation stores persistent state for every ctfe 
incovation.
While caching nothing. Not even the compiled for of a function 
body.
Because it cannot relax purity.
Which is why a simple while-loop from 0 to 100_000_00 can crash 
dmd if executed at ctfe.

Your advice is a good one. I am happy to discuss with others!
There is no need to duplicate mental workload.
May 09 2016
prev sibling next sibling parent reply Robert burner Schadek <rburners gmail.com> writes:
awesome news :-) thanks you
May 09 2016
parent =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Monday, 9 May 2016 at 18:20:46 UTC, Robert burner Schadek 
wrote:

 awesome news :-) thanks you


I very much agree.
May 09 2016
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/9/2016 9:57 AM, Stefan Koch wrote:

[...]


The memory consumption problem, at least, can be resolved by using stack 
temporaries instead of allocating new nodes. This was already done in 
constfold.d, but not in the rest of the interpreter.

Doing that will (I predict) also double its speed right there.
May 09 2016
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 9 May 2016 at 20:24:56 UTC, Walter Bright wrote:

 On 5/9/2016 9:57 AM, Stefan Koch wrote:

[...]


 The memory consumption problem, at least, can be resolved by 
 using stack temporaries instead of allocating new nodes. This 
 was already done in constfold.d, but not in the rest of the 
 interpreter.

 Doing that will (I predict) also double its speed right there.


Thanks, your advice is most helpful and a good first stop-gap.

Still the current state of CTFE is almost not maintainable
  and I would really prefer a clean-slate approach.

SDC benefits extremely from the extra level of indirection, 
however I do understand that SDC and DMD have diffrent goals 
regarding compile-speed.

Also I feel that good code has found it's most beautiful shape 
when it's simplicity makes it inevitable, at least the 
Ctfe-Mechanism has not reached this point yet, imo.
May 09 2016
prev sibling parent Jonathan M Davis via Digitalmars-d-announce writes:
On Monday, May 09, 2016 13:24:56 Walter Bright via Digitalmars-d-announce 
wrote:

 On 5/9/2016 9:57 AM, Stefan Koch wrote:

[...]


 The memory consumption problem, at least, can be resolved by using stack
 temporaries instead of allocating new nodes. This was already done in
 constfold.d, but not in the rest of the interpreter.

 Doing that will (I predict) also double its speed right there.


Previously, Don stated that he thought that simply making it so that CTFE
didn't allocate a new integer every time it mutated it would be a _huge_
speed-up by itself (which presumably is what you're talking about with
regards to allocating new nodes). Unfortunately, he got too busy to actually
do that work and no one else stepped in to do. But if Stefan is going to
step up and improve CTFE, that's fantastic. It's one of D's best features,
but it's also one of its most problematic. Fixng that would be huge.

- Jonathan M Davis
May 09 2016
prev sibling next sibling parent reply Andrej Mitrovic via Digitalmars-d-announce writes:
On 5/9/16, Stefan Koch via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:

 I was shocked to discover that the PowExpression actually depends
 on phobos!


I seem to remember having to implement diagnostics in DMD asking for
the user to import std.math. I'm fairly confident it had something to
do with power expressions.

Yah, it's a mess. :)
May 09 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/9/2016 2:32 PM, Andrej Mitrovic via Digitalmars-d-announce wrote:

 On 5/9/16, Stefan Koch via Digitalmars-d-announce
 <digitalmars-d-announce puremagic.com> wrote:

 I was shocked to discover that the PowExpression actually depends
 on phobos!


 I seem to remember having to implement diagnostics in DMD asking for
 the user to import std.math. I'm fairly confident it had something to
 do with power expressions.

 Yah, it's a mess. :)


The pow stuff should just be done in dmd without reference to the library.
May 09 2016
next sibling parent "H. S. Teoh via Digitalmars-d-announce" writes:
On Mon, May 09, 2016 at 05:36:21PM -0700, Walter Bright via
Digitalmars-d-announce wrote:

 On 5/9/2016 2:32 PM, Andrej Mitrovic via Digitalmars-d-announce wrote:

On 5/9/16, Stefan Koch via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:

I was shocked to discover that the PowExpression actually depends on
phobos!


I seem to remember having to implement diagnostics in DMD asking for
the user to import std.math. I'm fairly confident it had something to
do with power expressions.

Yah, it's a mess. :)


 
 The pow stuff should just be done in dmd without reference to the
 library.


+1.  It makes little sense for a language built-in operator to require
std.math, when the whole point behind the druntime/phobos split is to
make it possible for implementations *not* to implement Phobos.


T

-- 
Some days you win; most days you lose.
May 10 2016
prev sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 10 May 2016 at 00:36:21 UTC, Walter Bright wrote:

 On 5/9/2016 2:32 PM, Andrej Mitrovic via Digitalmars-d-announce 
 wrote:

 On 5/9/16, Stefan Koch via Digitalmars-d-announce
 <digitalmars-d-announce puremagic.com> wrote:

 I was shocked to discover that the PowExpression actually 
 depends
 on phobos!


 I seem to remember having to implement diagnostics in DMD 
 asking for
 the user to import std.math. I'm fairly confident it had 
 something to
 do with power expressions.

 Yah, it's a mess. :)


 The pow stuff should just be done in dmd without reference to 
 the library.


I just made a PR to fix it for ctfe.
It's a hack but then again ...
The whole handling of PowExp is a hack.
Jul 07 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 7 July 2016 at 13:55:44 UTC, Stefan Koch wrote:

 I just made a PR to fix it for ctfe.
 It's a hack but then again ...
 The whole handling of PowExp is a hack.


Clarification now it works on Literals.
It is still not available at ctfe and the full non-hackish fix 
will take a while.
Jul 07 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 7 July 2016 at 17:47:28 UTC, Stefan Koch wrote:

 On Thursday, 7 July 2016 at 13:55:44 UTC, Stefan Koch wrote:

 I just made a PR to fix it for ctfe.
 It's a hack but then again ...
 The whole handling of PowExp is a hack.


 Clarification now it works on Literals.
 It is still not available at ctfe and the full non-hackish fix 
 will take a while.


I forgot to mention I posted a short article about the CTFE 
design on my blog.
https://codesoldier.blogspot.com

Feel free to comment or give suggestions.
Jul 08 2016
parent reply Rene Zwanenburg <renezwanenburg gmail.com> writes:
On Friday, 8 July 2016 at 11:32:10 UTC, Stefan Koch wrote:

 I forgot to mention I posted a short article about the CTFE 
 design on my blog.
 https://codesoldier.blogspot.com

 Feel free to comment or give suggestions.


Thanks! Posts like these are always interesting to read. I 
noticed a few mistakes:

I have been working a rewrite -> I have been working on a rewrite
complicated corer-case -> complicated corner-case
that have to handled correctly -> that have to be handled 
correctly
a cononical from -> a canonical from
Jul 08 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 8 July 2016 at 12:17:29 UTC, Rene Zwanenburg wrote:

 On Friday, 8 July 2016 at 11:32:10 UTC, Stefan Koch wrote:

 I forgot to mention I posted a short article about the CTFE 
 design on my blog.
 https://codesoldier.blogspot.com

 Feel free to comment or give suggestions.


 Thanks! Posts like these are always interesting to read. I 
 noticed a few mistakes:

 I have been working a rewrite -> I have been working on a 
 rewrite
 complicated corer-case -> complicated corner-case
 that have to handled correctly -> that have to be handled 
 correctly
 a cononical from -> a canonical from


I addressed the spelling mistakes.
Thanks for pointed them out.
Also more complex SwitchStatements work now.
Jul 08 2016
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Friday, 8 July 2016 at 23:31:38 UTC, Stefan Koch wrote:
and mine is segfaulting in some bizarre ways (i failed my basic 
++ and -- math, and so the stack ;-).

still, it is almost working, with support for both compiled and 
interpreted function calls, almost full range of integer math and 
some string ops. druntime still compiles and passes unittests, 
but phobos segfaulted somewhere deep in my CTFE engine. i really 
need to re-check all added opcodes and codegen.

writing that was surprisingly easy. and sometimes simply 
surprisingly: "how did you came to me with this, DMD?! eh? aha, i 
see now, thank you, grep." ;-)

some stats: virtual machine is <30KB of code now. compiler is 
~70KB, but there is alot of copypasta in there (you know, it 
happens when the code grows organically).

if someone is interested in interop between frontend and backend, 
writing engine like this is a nice way to make yourself familiar, 
as CTFE interpreter effectively gets already semanticed AST, with 
most things correctly lowered and so on. so you only have to copy 
dummy CtfeCompiler from dinterpret.d (it only count vars there, 
otherwise doing nothing), and start extending it.

even when we'll get new shiny engine from Stefan, i still 
recommend to anyone who wish to understand how compiled and 
processed code looks inside the frontend at least try to 
implement some simple bytecode compiler. with some care seeing 
that you are effectively replacing compiler parts, and it is 
still able to compile complex unittests is... yeah, go find the 
word. ;-)
Jul 08 2016
parent ketmar <ketmar ketmar.no-ip.org> writes:
p.s. it is, btw, completely possible to add at least something 
like "ctfe tracer" (something like old basic "trace on" command), 
or even some kind of gdb-like debugger to ctfe engine.

it won't, of course, find it's way into mainline, but may still 
be a fun project to do.
Jul 08 2016
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/9/16 7:57 PM, Stefan Koch wrote:

 Hi Guys,

 I have been looking into the DMD now to see what I can do about CTFE.
 Unfortunately It is a pretty big mess to untangle.
 Code responsible for CTFE is in at least 3 files.
 [dinterpret.d, ctfeexpr.d, constfold.d]
 I was shocked to discover that the PowExpression actually depends on
 phobos! (depending on the exact codePath it may or may not compile...)
 which let to me prematurely stating that it worked at ctfe
 [http://forum.dlang.org/thread/ukcoibejffinknrbzktv forum.dlang.org]

 My Plan is as follows.

 Add a new file for my ctfe-interpreter and update it gradually to take
 more and more of the cases the code in the files mentioned above was
 used for.

 Do Dataflow analysis on the code that is to be ctfe'd so we can tell
 beforehand if we need to store state in the ctfe stack or not.

 Or baring proper data-flow analysis: RefCouting the variables on the
 ctfe-stack could also be a solution.

 I will post more details as soon as I dive deeper into the code.


Thanks Stefan, that's a good start! (This is probably better for the 
general forum.) -- Andrei
May 09 2016
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-05-09 18:57, Stefan Koch wrote:

 Hi Guys,

 I have been looking into the DMD now to see what I can do about CTFE.
 Unfortunately It is a pretty big mess to untangle.
 Code responsible for CTFE is in at least 3 files.
 [dinterpret.d, ctfeexpr.d, constfold.d]
 I was shocked to discover that the PowExpression actually depends on
 phobos! (depending on the exact codePath it may or may not compile...)
 which let to me prematurely stating that it worked at ctfe
 [http://forum.dlang.org/thread/ukcoibejffinknrbzktv forum.dlang.org]

 My Plan is as follows.

 Add a new file for my ctfe-interpreter and update it gradually to take
 more and more of the cases the code in the files mentioned above was
 used for.

 Do Dataflow analysis on the code that is to be ctfe'd so we can tell
 beforehand if we need to store state in the ctfe stack or not.

 Or baring proper data-flow analysis: RefCouting the variables on the
 ctfe-stack could also be a solution.

 I will post more details as soon as I dive deeper into the code.


I was listening to a discussion Don and Daniel had about the current 
implementation of CTFE. They talked about using a byte code interpreter. 
Even implementing a really crappy byte code interpreter would be a huge 
improvement.

-- 
/Jacob Carlborg
May 09 2016
next sibling parent reply Rory McGuire via Digitalmars-d-announce writes:
On Tue, May 10, 2016 at 8:45 AM, Jacob Carlborg via
Digitalmars-d-announce <digitalmars-d-announce puremagic.com> wrote:

 I was listening to a discussion Don and Daniel had about the current
 implementation of CTFE. They talked about using a byte code interpreter.
 Even implementing a really crappy byte code interpreter would be a huge
 improvement.

 --
 /Jacob Carlborg


Does anyone know whether it would be worth while keeping the LLVM JIT
in mind when writing this new CTFE engine?
May 10 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 10 May 2016 at 07:44:54 UTC, Rory McGuire wrote:

 On Tue, May 10, 2016 at 8:45 AM, Jacob Carlborg via 
 Digitalmars-d-announce <digitalmars-d-announce puremagic.com> 
 wrote:

 I was listening to a discussion Don and Daniel had about the 
 current implementation of CTFE. They talked about using a byte 
 code interpreter. Even implementing a really crappy byte code 
 interpreter would be a huge improvement.

 --
 /Jacob Carlborg


 Does anyone know whether it would be worth while keeping the 
 LLVM JIT in mind when writing this new CTFE engine?


Yes I do know the llvm jit, it is slow as a three legged dog.

But I do plan for a way of plugging it in. This is not a main 
priority however.
May 10 2016
parent jmh530 <john.michael.hall gmail.com> writes:
On Tuesday, 10 May 2016 at 11:31:33 UTC, Stefan Koch wrote:

 Yes I do know the llvm jit, it is slow as a three legged dog.

 But I do plan for a way of plugging it in. This is not a main 
 priority however.


What about libjit?
May 16 2016
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/10/2016 08:45 AM, Jacob Carlborg wrote:

 
 I was listening to a discussion Don and Daniel had about the current
 implementation of CTFE. They talked about using a byte code interpreter.
 Even implementing a really crappy byte code interpreter would be a huge
 improvement.


No need for a byte-code interpreter, it mostly just adds overhead and
complexity over an AST interpreter. If you want to go really fast you
need some sort of JIT anyhow, but a proper interpreter will be orders of
mangnitude faster than the current implementation.

I might refer you to
http://dconf.org/2013/talks/chevalier_boisvert.pdf
page 59 ff.
May 15 2016
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Sunday, 15 May 2016 at 10:29:21 UTC, Martin Nowak wrote:

 On 05/10/2016 08:45 AM, Jacob Carlborg wrote:
 overhead and complexity over an AST interpreter. If you want to 
 go really fast you need some sort of JIT anyhow, but a proper 
 interpreter will be orders of mangnitude faster than the 
 current implementation.


If you are going to have fast evaluation of loops/recursion then 
you need to use a solver. And well, doing worse than O(log N) at 
compile time is a very bad idea.

Why not start with the most difficult case first? Then the simple 
cases will resolve themselves for free, most likely.
May 15 2016
parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/15/2016 01:55 PM, Ola Fosheim Grøstad wrote:

 If you are going to have fast evaluation of loops/recursion then you
 need to use a solver. And well, doing worse than O(log N) at compile
 time is a very bad idea.
 
 Why not start with the most difficult case first? Then the simple cases
 will resolve themselves for free, most likely.


Why not do something that takes about a month and is much more likely to
succeed?
If someone has more time and needs an even faster interpreter she can
write a new one, or add optimizations or JIT to the simple interpreter.
May 15 2016
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Sunday, 15 May 2016 at 12:00:43 UTC, Martin Nowak wrote:

 On 05/15/2016 01:55 PM, Ola Fosheim Grøstad wrote:

 If you are going to have fast evaluation of loops/recursion 
 then you need to use a solver. And well, doing worse than 
 O(log N) at compile time is a very bad idea.
 
 Why not start with the most difficult case first? Then the 
 simple cases will resolve themselves for free, most likely.


 Why not do something that takes about a month and is much more 
 likely to succeed?


Well, you can, but it won't bring improvements to the language 
down the line. If typical CTFE can be rewritten as simple tail 
recursion then you probably can find existing libraries that will 
do it for you.
May 15 2016
parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/15/2016 02:13 PM, Ola Fosheim Grøstad wrote:

 
 Well, you can, but it won't bring improvements to the language down the
 line.


Maybe you don't know the actual problem of the current interpreter?
I leaks memory like hell b/c it allocates new AST nodes for almost every
expression evaluation.
Even an interpreter that is as slow as ruby will fly during compile time
in comparision to the current one.

Let me illustrate the point.

---
import std.algorithm, std.array, std.random, std.range;

enum count = 2 ^^ 10;
enum sorted = {
    auto gen = Random(123);
    return generate!(() => uniform(byte.min, byte.max,
gen)).take(count).array.sort().release;
}();
pragma(msg, sorted.length);
---
count = 2 ** 10
nums = Enumerator.new do |yielder|
  prng = Random.new(123)
  loop do
    yielder.yield prng.rand(-128 .. 127)
  end
end.take(count).sort
print nums.length
---

   N   |     CTFE     |     Ruby
       | Time  | Mem  | Time  | Mem
-------|-------|------|-------|------
 2^^10 | 0.16s | 82M  | 0.11s | 9.3M
 2^^11 | 0.22s | 110M | 0.12s | 9.3M
 2^^12 | 0.4s  | 190M | 0.12s | 9.4M
 2^^13 | 0.7s  | 450M | 0.12s | 9.5M
 2^^14 | 1.5s  | 1.4G | 0.12s | 9.7M
 2^^15 | 3.7s  | 4.8G | 0.13s | 10.0M
 2^^16 | 5:30m | 15G  | 0.13s | 10.8M

D's CTFE grows O(N^2) b/c it leaks for almost every operation.
We don't currently need a superfast interpreter, even the simplest
possible interpreter will allow so much more that we're more likely
limited by the lack of I/O before we need a faster interpreter.
May 15 2016
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Sunday, 15 May 2016 at 13:44:45 UTC, Martin Nowak wrote:

  2^^15 | 3.7s  | 4.8G | 0.13s | 10.0M
  2^^16 | 5:30m | 15G  | 0.13s | 10.8M

 D's CTFE grows O(N^2) b/c it leaks for almost every operation.
 We don't currently need a superfast interpreter, even the 
 simplest
 possible interpreter will allow so much more that we're more 
 likely
 limited by the lack of I/O before we need a faster interpreter.


Well, this looks really bad.  But a solver would get you much 
more than an interpreter. E.g. proving that asserts always hold 
etc.
May 15 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 15 May 2016 at 14:56:02 UTC, Ola Fosheim Grøstad wrote:

 Well, this looks really bad.  But a solver would get you much 
 more than an interpreter. E.g. proving that asserts always hold 
 etc.


You want it ?
Write it.
May 15 2016
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Sunday, 15 May 2016 at 15:09:17 UTC, Stefan Koch wrote:

 You want it ?
 Write it.


I don't «want» anything.
May 15 2016
prev sibling next sibling parent reply Daniel Murphy <yebbliesnospam gmail.com> writes:
On 15/05/2016 8:29 PM, Martin Nowak wrote:

 No need for a byte-code interpreter, it mostly just adds overhead and
 complexity over an AST interpreter. If you want to go really fast you
 need some sort of JIT anyhow, but a proper interpreter will be orders of
 mangnitude faster than the current implementation.


The biggest advantage of bytecode is not the interpreter speed, it's 
that by lowering you can substitute VarExps etc with actual references 
to memory without modifying the AST.

By working with something lower level than the AST, you should end up 
with something much less complex and with fewer special cases.

The current goal is not a full JIT, just something that manages memory 
in a less insane way.
May 15 2016
parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/15/2016 01:58 PM, Daniel Murphy wrote:

 The biggest advantage of bytecode is not the interpreter speed, it's
 that by lowering you can substitute VarExps etc with actual references
 to memory without modifying the AST.
 
 By working with something lower level than the AST, you should end up
 with something much less complex and with fewer special cases.


Which is a bad assessment, you can stick variable indexes into
VarDeclaration (we already do that) and thereby access them in O(1).
Converting control flow and references into byte code is far from
trivial, we're talking about another s2ir and e2ir here.

-Martin
May 15 2016
next sibling parent reply Daniel Murphy <yebbliesnospam gmail.com> writes:
On 15/05/2016 9:57 PM, Martin Nowak wrote:

 On 05/15/2016 01:58 PM, Daniel Murphy wrote:

 The biggest advantage of bytecode is not the interpreter speed, it's
 that by lowering you can substitute VarExps etc with actual references
 to memory without modifying the AST.

 By working with something lower level than the AST, you should end up
 with something much less complex and with fewer special cases.


 Which is a bad assessment, you can stick variable indexes into
 VarDeclaration (we already do that) and thereby access them in O(1).
 Converting control flow and references into byte code is far from
 trivial, we're talking about another s2ir and e2ir here.

 -Martin


For simple types that's true.  For more complicated reference types...

Variable indexes are not enough, you also need heap memory, but slices 
and pointers (and references) can refer to values either on the heap or 
the stack, and you can have a slice of a member static array of a class 
on the stack, etc.  Then there are closures...

Neither e2ir or s2ir are actually that complex.  A lot of the mess there 
comes from the backend IR interface being rather difficult to work with. 
  We can already save a big chunk of complexity by not having to 
translate the frontend types.  E.g.  implementing the logic in the 
interpreter to correctly unwind through destructors is unlikely to be 
simpler than lowering to an IR.
May 15 2016
next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/15/2016 02:17 PM, Daniel Murphy wrote:

 
 For simple types that's true.  For more complicated reference types...
 
 Variable indexes are not enough, you also need heap memory, but slices
 and pointers (and references) can refer to values either on the heap or
 the stack, and you can have a slice of a member static array of a class
 on the stack, etc.  Then there are closures...


So we do need a GC or RC for arrays, structs, classes (anything
heapish). Values for those could be allocated by a simple bump/region
allocator or a dedicated allocator that support individual freeing (via
RC or GC).

In any case struct Pointer { int index; /* 2B positive values for stack,
2B negative for heap*/ } wouldn't be much more complicated than a raw
pointer (and a bit simpler to garbage collect).


 Neither e2ir or s2ir are actually that complex.  A lot of the mess there
 comes from the backend IR interface being rather difficult to work with.


Think of a simple switch statement where you even need to introduce
relocations (or keep a list of fixup addresses) b/c you don't know the
jump addresses in advance.
In a visitor you simply test the cases and execute the first case body.

Not to mention that we can reuse existing solutions from the current
interpreter (e.g. for gotos see
https://github.com/dlang/dmd/blob/0757504342e48e272372b7ac52cda5a333b2a2bc/src/dinterpret.d#L1014
and
https://github.com/dlang/dmd/blob/0757504342e48e272372b7ac52cda5a333b2a2bc/src/dinterpret.d#L1094).
May 15 2016
next sibling parent reply Daniel Murphy <yebbliesnospam gmail.com> writes:
On 15/05/2016 11:25 PM, Martin Nowak wrote:

 On 05/15/2016 02:17 PM, Daniel Murphy wrote:

 For simple types that's true.  For more complicated reference types...

 Variable indexes are not enough, you also need heap memory, but slices
 and pointers (and references) can refer to values either on the heap or
 the stack, and you can have a slice of a member static array of a class
 on the stack, etc.  Then there are closures...


 So we do need a GC or RC for arrays, structs, classes (anything
 heapish). Values for those could be allocated by a simple bump/region
 allocator or a dedicated allocator that support individual freeing (via
 RC or GC).

 In any case struct Pointer { int index; /* 2B positive values for stack,
 2B negative for heap*/ } wouldn't be much more complicated than a raw
 pointer (and a bit simpler to garbage collect).


The problem is, if index refers to a single variable on the stack, then 
it's insufficient to refer to a variable inside an aggregate on the 
stack.  Then you need to start building constructs for member of struct 
in array of struct pointers and it gets fairly messy...  It's all 
solvable, I'm not sure the simplicity would survive.


 Neither e2ir or s2ir are actually that complex.  A lot of the mess there
 comes from the backend IR interface being rather difficult to work with.


 Think of a simple switch statement where you even need to introduce
 relocations (or keep a list of fixup addresses) b/c you don't know the
 jump addresses in advance.


This is not exactly difficult to do.


 In a visitor you simply test the cases and execute the first case body.

 Not to mention that we can reuse existing solutions from the current
 interpreter (e.g. for gotos see
 https://github.com/dlang/dmd/blob/0757504342e48e272372b7ac52cda5a333b2a2bc/src/dinterpret.d#L1014
 and
 https://github.com/dlang/dmd/blob/0757504342e48e272372b7ac52cda5a333b2a2bc/src/dinterpret.d#L1094).


Flow control is really not where the complexity lies IMO.  The weird 
ways in which different types of reference types can combine leads to 
either very complicated or very low level descriptions of memory.
May 15 2016
parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/15/2016 04:00 PM, Daniel Murphy wrote:

 The problem is, if index refers to a single variable on the stack, then
 it's insufficient to refer to a variable inside an aggregate on the
 stack.  Then you need to start building constructs for member of struct
 in array of struct pointers and it gets fairly messy...  It's all
 solvable, I'm not sure the simplicity would survive.


Taking what I said further down below there would be one union Value
type (untagged b/c the compiler knows what it is).

Then an Array could be implementd as HeapValueRef[], a hash table as
HeapValueRef[ValueRef], a struct/class as HeapValueRef[] (with
HeapValueRef being a pointer or int to a heap allocated Value and
ValueRef being a pointer or int to either a stack value or a heap value).
A D Pointer/Ref would just be a ValueRef in the interpreter and the
aforementioned int (2B + for stack, 2B - for heap) encoding should still
work for that.
Depending on how much pointer arithmetic we support, Pointer must
contain a ValueRef to the outermost aggregate and needs to store the
type of that an additional byte offset. The type and offset could then
be used to compute the actual pointee. While that sounds a bit complex,
it's merely just https://en.wikipedia.org/wiki/Offsetof.


 Flow control is really not where the complexity lies IMO.  The weird
 ways in which different types of reference types can combine leads to
 either very complicated or very low level descriptions of memory.


Maybe you're right, but it'll be hard to figure out w/o an actual
implementation. And the AST still looks like a lot less to code and execute.
May 15 2016
prev sibling parent reply Kagamin <spam here.lot> writes:
On Sunday, 15 May 2016 at 13:25:42 UTC, Martin Nowak wrote:

 So we do need a GC or RC for arrays, structs, classes (anything 
 heapish). Values for those could be allocated by a simple 
 bump/region allocator or a dedicated allocator that support 
 individual freeing (via RC or GC).


Wasn't it possible to enable GC for entire compiler? There can be 
hybrid approach: 1) first allocate from bump heap 2) when it 
reaches, say, 200MB, switch to GC.
May 16 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Monday, 16 May 2016 at 10:01:47 UTC, Kagamin wrote:

 Wasn't it possible to enable GC for entire compiler? There can 
 be hybrid approach: 1) first allocate from bump heap 2) when it 
 reaches, say, 200MB, switch to GC.


Well, I wouldn't use D's GC for that dedicated heap.
Allocation of CTFE values are completely independent and call be 
freed once the evaluation is finished.
May 16 2016
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/16/16 7:20 AM, Martin Nowak wrote:

 On Monday, 16 May 2016 at 10:01:47 UTC, Kagamin wrote:

 Wasn't it possible to enable GC for entire compiler? There can be
 hybrid approach: 1) first allocate from bump heap 2) when it reaches,
 say, 200MB, switch to GC.


 Well, I wouldn't use D's GC for that dedicated heap.
 Allocation of CTFE values are completely independent and call be freed
 once the evaluation is finished.


A reap would be great there! std.experimental.allocator offers that and 
a variety of others. -- Andrei
May 16 2016
parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/16/2016 01:36 PM, Andrei Alexandrescu wrote:

 
 A reap would be great there! std.experimental.allocator offers that and
 a variety of others. -- Andrei


Yes indeed, a malloc backed Region Allocator w/ a FreeList or a
BitmappedBlock would be a good starting point.

That might finally be a compelling enough case to start using phobos in
dmd. Last time people forced me to spend several hours on reimplementing
and debugging a BitArray implementation [¹].

[¹]: https://github.com/dlang/dmd/pull/5426#discussion_r52833955
May 16 2016
parent safety0ff <safety0ff.dev gmail.com> writes:
On Monday, 16 May 2016 at 12:13:14 UTC, Martin Nowak wrote:

 Last time people forced me to spend several hours on 
 reimplementing and debugging a BitArray implementation


Ouch.
src/tk/vec.(h|c) already contained an implementation.
May 16 2016
prev sibling parent Daniel Murphy <yebbliesnospam gmail.com> writes:
On 16/05/2016 9:20 PM, Martin Nowak wrote:

 On Monday, 16 May 2016 at 10:01:47 UTC, Kagamin wrote:

 Wasn't it possible to enable GC for entire compiler? There can be
 hybrid approach: 1) first allocate from bump heap 2) when it reaches,
 say, 200MB, switch to GC.


 Well, I wouldn't use D's GC for that dedicated heap.
 Allocation of CTFE values are completely independent and call be freed
 once the evaluation is finished.


Maybe you wouldn't, but you certainly could...
May 16 2016
prev sibling parent reply Don Clugston <sendmespam turnyourkidsintocash.com> writes:
On Sunday, 15 May 2016 at 12:17:30 UTC, Daniel Murphy wrote:

 On 15/05/2016 9:57 PM, Martin Nowak wrote:

 On 05/15/2016 01:58 PM, Daniel Murphy wrote:

 The biggest advantage of bytecode is not the interpreter 
 speed, it's
 that by lowering you can substitute VarExps etc with actual 
 references
 to memory without modifying the AST.

 By working with something lower level than the AST, you 
 should end up
 with something much less complex and with fewer special cases.


 Which is a bad assessment, you can stick variable indexes into
 VarDeclaration (we already do that) and thereby access them in 
 O(1).
 Converting control flow and references into byte code is far 
 from
 trivial, we're talking about another s2ir and e2ir here.

 -Martin


 For simple types that's true.  For more complicated reference 
 types...

 Variable indexes are not enough, you also need heap memory, but 
 slices and pointers (and references) can refer to values either 
 on the heap or the stack, and you can have a slice of a member 
 static array of a class on the stack, etc.  Then there are 
 closures...

 Neither e2ir or s2ir are actually that complex.  A lot of the 
 mess there comes from the backend IR interface being rather 
 difficult to work with.
  We can already save a big chunk of complexity by not having to 
 translate the frontend types.  E.g.  implementing the logic in 
 the interpreter to correctly unwind through destructors is 
 unlikely to be simpler than lowering to an IR.


Exactly. I think the whole idea of trying to avoid a glue layer 
is a mistake.
CTFE is a backend. It really is. And it should be treated as one. 
A very simple one, of course.
Once you do this, you'll find all sorts of commonalities with the 
existing glue layers.
We should end up with at least 4 backends: DMD, GCD, LDC, and 
CTFE.

Many people here are acting like this is something complicated, 
and making dangerous suggestions like using Phobos inside the 
compiler. (I think everyone who has fixed a compiler bug that was 
discovered in Phobos, will know what a nightmare that would be. 
The last thing compiler development needs is another level of 
complexity in the compiler).

As I've tried to explain, the problems with CTFE historically 
were never with the CTFE engine itself. They were always with the 
interface between CTFE and the remainder of the compiler -- 
finding every case where CTFE can be called, finding all the 
bizarre cases (tuple variables, variables without a stack because 
they are local variables declared in comma expressions in global 
scope, local 'ref' variables, etc), finding all the cases where 
the syntax trees were invalid...

There's no need for grandiose plans, as if there is some 
almost-insurmountable problem to be solved. THIS IS NOT 
DIFFICULT. With the interface cleaned up, it is the well-studied 
problem of creating an interpreter. Everyone knows how to do 
this, it's been done thousands of times. The complete test suite 
is there for you. Someone just needs to do it.

I think I took the approach of using syntax trees about as far as 
it can go. It's possible, but it's really vile. Look at the code 
for doing assignments. Bleagh. The only thing in its favour is 
that originally it was the only implementation that was possible 
at all. Even the first, minimal step towards creating a ctfe 
backend -- introducing a syntax-tree-validation step -- 
simplified parts of the code immensely.

You might imagine that it's easier to work with syntax trees than 
to start from scratch but I'm certain that's not true. I'm pretty 
sure that the simplest approach is to use the simplest possible 
machine-independent bytecode that you can come up with. I had got 
to the point of starting that, but I just couldn't face doing it 
in C++.

TL;DR:  CTFE is actually a backend, so don't be afraid of 
creating a glue layer for it.
May 17 2016
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Tuesday, 17 May 2016 at 10:42:30 UTC, Don Clugston wrote:

 TL;DR:  CTFE is actually a backend, so don't be afraid of 
 creating a glue layer for it.


My point exactly.
The AST is not something I want to handle while inside the 
interpreter.
It introduces too much complexity.
There needs to be some kind of further lowering.
May 17 2016
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/17/2016 12:42 PM, Don Clugston wrote:

 There's no need for grandiose plans, as if there is some
 almost-insurmountable problem to be solved. THIS IS NOT DIFFICULT. With
 the interface cleaned up, it is the well-studied problem of creating an
 interpreter. Everyone knows how to do this, it's been done thousands of
 times. The complete test suite is there for you. Someone just needs to
 do it.


Yes, exactly my words.


 I think I took the approach of using syntax trees about as far as it can
 go. It's possible, but it's really vile. Look at the code for doing
 assignments. Bleagh. The only thing in its favour is that originally it
 was the only implementation that was possible at all. Even the first,
 minimal step towards creating a ctfe backend -- introducing a
 syntax-tree-validation step -- simplified parts of the code immensely.


Yes, this
https://github.com/dlang/dmd/blob/7d00095301c4780b41addcfeb50f4743a9a6c5d4/src/dinterpret.d#L3418
is really ugly and complex, but you won't get rid of this inherent
complexity. The e2ir code for AssingExp looks almost the same
https://github.com/dlang/dmd/blob/7d00095301c4780b41addcfeb50f4743a9a6c5d4/src/e2ir.c#L2466.


 You might imagine that it's easier to work with syntax trees than to
 start from scratch but I'm certain that's not true. I'm pretty sure that
 the simplest approach is to use the simplest possible
 machine-independent bytecode that you can come up with. I had got to the
 point of starting that, but I just couldn't face doing it in C++.


All I'm saying is that interpreting the AST to generate bytecode is
going to be as complex as interpreting the AST directly, but then you
still a bytecode interpreter and later might still want a JIT.

Using dedicated value types during interpretation instead of recycling
the AST for that will also make the transitions clearer and get rid of
some of the lifetime complexities in the current code.

-Martin
May 17 2016
parent reply Daniel Murphy <yebbliesnospam gmail.com> writes:
On 18/05/2016 9:01 AM, Martin Nowak wrote:

 Yes, this
 https://github.com/dlang/dmd/blob/7d00095301c4780b41addcfeb50f4743a9a6c5d4/src/dinterpret.d#L3418
 is really ugly and complex, but you won't get rid of this inherent
 complexity. The e2ir code for AssingExp looks almost the same
 https://github.com/dlang/dmd/blob/7d00095301c4780b41addcfeb50f4743a9a6c5d4/src/e2ir.c#L2466.



IMO this is a different problem, that AssignExp is stupidly complex and 
needs to be split up.


 You might imagine that it's easier to work with syntax trees than to
 start from scratch but I'm certain that's not true. I'm pretty sure that
 the simplest approach is to use the simplest possible
 machine-independent bytecode that you can come up with. I had got to the
 point of starting that, but I just couldn't face doing it in C++.


 All I'm saying is that interpreting the AST to generate bytecode is
 going to be as complex as interpreting the AST directly, but then you
 still a bytecode interpreter and later might still want a JIT.


The bytecode generator and bytecode interpreter can be debugged (and 
tested!) independently.  So the total amount of code will increase but 
the components themselves will be better isolated and easier to work with.

I don't think a possible future need for a JIT is a good reason to avoid 
an bytecode interpreter.  A large part of the work of adding a new (JIT) 
backend is pinning down the semantics, and adding a bytecode interpreter 
will certainly help to do that.


 Using dedicated value types during interpretation instead of recycling
 the AST for that will also make the transitions clearer and get rid of
 some of the lifetime complexities in the current code.


Meh, sure.  But this feels just as difficult as switching to a simple 
bytecode, without all the benefits.
May 18 2016
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Wednesday, 18 May 2016 at 14:59:09 UTC, Daniel Murphy wrote:


 Meh, sure.  But this feels just as difficult as switching to a 
 simple bytecode, without all the benefits.


Indeed.

I am currently designing an IR to feed into the CTFE Evaluator.
I am aware that this could potentially make it harder to get 
things merged since DMD already has the glue-layer.

However I do think that the benefits outweigh the drawbacks by 
far.
Especially when one looks at the possibility to eventually plug 
llvm or the gcc-jit in.

My CTFE needs are rather heavy weight. So I will go for a 
solution that can support this.
I believe the pressure on CTFE performance will increase as soon 
as the preformance increases. Since this will enable much more 
things.

I.E. running a query optimizer at compile-time.
May 18 2016
next sibling parent Daniel Murphy <yebbliesnospam gmail.com> writes:
On 19/05/2016 3:50 AM, Stefan Koch wrote:

 I am currently designing an IR to feed into the CTFE Evaluator.
 I am aware that this could potentially make it harder to get things
 merged since DMD already has the glue-layer.


It's always more difficult to justify merging more complexity.  But if 
you can present a working and superior solution the specific 
implementation is less important.  It is still important that it matches 
the existing style of the compiler, especially with respect to adding 
dependencies.

Also be aware that even with agreement on the eventual goal, it is still 
very slow to get big changes approved.  eg ddmd took more than twice as 
long as it should have.  This is why I suggested looking for incremental 
improvements, so you can overlap getting earlier things merged and 
developing later parts.  I would be on the lookout for things that can 
be justified on their own (untangling AssignExp :) ) and try to push 
those through first.
May 19 2016
prev sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/18/2016 07:50 PM, Stefan Koch wrote:

 Indeed.
 
 I am currently designing an IR to feed into the CTFE Evaluator.
 I am aware that this could potentially make it harder to get things
 merged since DMD already has the glue-layer.


As a compat layer between different interpreters or as a compat layer
between all backends? Adding another translation might not be
acceptable, at least for real backends.


 However I do think that the benefits outweigh the drawbacks by far.
 Especially when one looks at the possibility to eventually plug llvm or
 the gcc-jit in.


Indeed, but it heavily increases the chance that your project lands on
the unfinished D project pile.


 My CTFE needs are rather heavy weight. So I will go for a solution that
 can support this.
 I believe the pressure on CTFE performance will increase as soon as the
 preformance increases. Since this will enable much more things.
 I.E. running a query optimizer at compile-time.


That might be true, but scripting languages are still fast enough to be
used everywhere. You won't need native CTFE performance for it to be an
enabling technique.

-Martin
May 21 2016
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/18/2016 04:59 PM, Daniel Murphy wrote:

 The bytecode generator and bytecode interpreter can be debugged (and
 tested!) independently.  So the total amount of code will increase but
 the components themselves will be better isolated and easier to work with.


It's simpler to debug an AST interpreter working with a stack of
high-level values, than it is to debug a bytecode interpreter where lots
of context has been converted to jumping goto code.

Just to illustrate my point, here are an AST and a BC interpreter for
very simplistic functional language I wrote recently. The later one
still missing the actual interpretation.
https://github.com/MartinNowak/CC1/commit/ed28b8966de86e7449f93ce4e4cf7aed3082180b
https://github.com/MartinNowak/CC1/commit/899e67cf7038050b86eed533c9165bd2ba06e609

There is nothing simpler about a BC interpreter. Instead you have to
deal with converting control flow and computing addressing.

The debugging metaphor would be comparing a program that only uses
pointer arithmetic against one that is memory safe, the former can
randomly write everywhere from anywhere, the latter could use the wrong
reference.


 I don't think a possible future need for a JIT is a good reason to avoid
 an bytecode interpreter.


It's a very good reason, b/c once you work on JIT, there is no benefit
for BC left, e.g. all the extra work for nothing.
That said, I doubt we'll need a JIT anytime soon.


 A large part of the work of adding a new (JIT) backend is pinning down the
semantics,
 and adding a bytecode interpreter will certainly help to do that.


The semantic problem is already solved, in a file called interpret.d by
sth. that's an AST interpreter, that just happens to use the wrong value
structures and leaks memory. Converting that to BC will be quite
difficult, cleaning it up and changing it to use a better stack and
deterministic memory management is rather straightforward.

Last but not least, don't forget that we have the same situation since
over 3 years already. It has always been similarly easy to write a
better interpreter, it just never happened b/c the ambitions never
matched the available resources.

-Martin
May 21 2016
parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/21/2016 11:18 PM, Martin Nowak wrote:

 The debugging metaphor would be comparing a program that only uses
 pointer arithmetic against one that is memory safe, the former can
 randomly write everywhere from anywhere, the latter could use the wrong
 reference.


It's also similar to comparing assembly code to C.
May 21 2016
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Saturday, 21 May 2016 at 21:20:54 UTC, Martin Nowak wrote:

 On 05/21/2016 11:18 PM, Martin Nowak wrote:

 The debugging metaphor would be comparing a program that only 
 uses pointer arithmetic against one that is memory safe, the 
 former can randomly write everywhere from anywhere, the latter 
 could use the wrong reference.


 It's also similar to comparing assembly code to C.


I am just working on a change that introduces a temporary-Stack 
to the memory management of dmd.
So far it crashes :)
May 27 2016
prev sibling parent reply Daniel Murphy <yebbliesnospam gmail.com> writes:
On 15/05/2016 9:57 PM, Martin Nowak wrote:

 On 05/15/2016 01:58 PM, Daniel Murphy wrote:

 The biggest advantage of bytecode is not the interpreter speed, it's
 that by lowering you can substitute VarExps etc with actual references
 to memory without modifying the AST.

 By working with something lower level than the AST, you should end up
 with something much less complex and with fewer special cases.


 Which is a bad assessment, you can stick variable indexes into
 VarDeclaration (we already do that) and thereby access them in O(1).
 Converting control flow and references into byte code is far from
 trivial, we're talking about another s2ir and e2ir here.

 -Martin


We really should have discussed this last week!
May 15 2016
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 15 May 2016 at 12:54:33 UTC, Daniel Murphy wrote:

 We really should have discussed this last week!


I agree.
Maybe we can have a skype conference or something in the next 
days.

About the whole to BC or not to BC discussion.
As Daniel already outlined, the main purpose it not speed, but 
having a simple  lowered representation to interpret.
Many AST interpreters switched to a byte-code because it's much 
easier to maintain.
May 15 2016
parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/15/2016 03:55 PM, Stefan Koch wrote:

 
 About the whole to BC or not to BC discussion.
 As Daniel already outlined, the main purpose it not speed, but having a
 simple  lowered representation to interpret.
 Many AST interpreters switched to a byte-code because it's much easier
 to maintain.
 


I just don't buy the argument for using BC.

In order to generate linearized bytecode you'll have to fully walk
(a.k.a. interpret) your AST tree, collect and fixup variable references
and jump addresses, precompute frame sizes, and still maintain a
dedicated stack and heap for variables (b/c you don't want to use D's GC
to maintain the lifetime, and using raw pointers will easily result in
time-consuming memory corruptions).

This effort might be worthwhile for hot loops if you wanted to generate
simple asm code so that the CPU can do the interpretation (2nd step
after AST interpreter in [¹]). But at this point this would be a
premature optimization.
Clearly a BC interpreter is both more complex than an AST interpreter
and less optimal than a simple JIT.

Sure supporting D's pointer arithmetic in an interpreter will be
challenging, but in fact you'll have to solve the same problem for a
bytecode interpreter (how to maintain ownership when pointing to a part
of a struct).
The simple but bad solution, using raw pointers and relying on D's GC,
would work for both interpreters.

Another simple but RC friendly solution is to compose everything from
reference Values, e.g. a struct or array being modeled as RC!Value[],
and a hash as RC!Value[RC!Value].

One could even do a hybrid between value and reference type by lazily
moving values onto the heap when needed.

struct Value
{
  static struct Impl
  {
    mixin(bitfields!(
      uint, "refCount", 31,
      bool, "onHeap", 1));

    union
    {
      dinteger_t int_;
      uinteger_t uint_;
      real_t real_;
      String str_;
      Value[] array;
      Impl* heap;
    }
  }
  Impl impl;

  ref Impl get() { return onHeap ? *impl.heap : impl; }
  alias get this;

  void moveToHeap()
  {
    if (impl.onHeap)
      return;
    auto p = heapAllocator.alloc!Impl;
    *p = impl;
    p.refCount = 1;
    impl.heap = p;
    impl.onHeap = true;
  }

  ~this()
  {
    if (impl.onHeap && --impl.heap.refCount == 0)
      heapAllocator.free(impl.heap);
  }
}

auto heapAllocator = FreeList!(AllocatorList!(
    (size_t n) => Region!Mallocator(max(n, 1024 * 1024))
))();

//...
class Interpreter
{
  //...
  void visit(PtrExp e)
  {
    stack.push(interpret(e.e1, istate, ctfeNeedLvalue));
  }

  void visit(IndexExp e)
  {
    accept(e1);
    auto v1 = stack.pop();
    accept(e2);
    auto v2 = stack.pop();

    if (ctfeGoal == ctfeNeedLvalue)
        v1.array[v2.uint_].moveToHeap();
    stack.push(v1.array[v2.uint_]);
  }
//...
}

That way you could benefit from memory efficient value types while still
being able to take references from any element.
Should be possible to do the same with references/pointers to stack
variables, but you want check that refCount == 1 when a stack frame gets
cleaned up.

[¹]: http://dconf.org/2013/talks/chevalier_boisvert.pdf p. 59 ff
May 16 2016
parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/16/2016 03:03 PM, Martin Nowak wrote:

   ~this()
   {
     if (impl.onHeap && --impl.heap.refCount == 0)
       heapAllocator.free(impl.heap);
   }


Of course missing the increment for copies.

    this(this)
    {
      if (impl.onHeap)
        ++impl.heap.refCount;
    }
May 16 2016
prev sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/15/2016 02:54 PM, Daniel Murphy wrote:

 
 We really should have discussed this last week!


I talked about it w/ Stefan, and asked him to volunteer for an
implementation, that's why we have this thread ;).
In any case I'm convinced that the simple-first strategy has a much
higher chance to be implemented this year, whereas the bug report [¹]
for slow CTFE is already 5 years old.

[¹]: https://issues.dlang.org/show_bug.cgi?id=6498
May 15 2016
prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Sunday, 15 May 2016 at 10:29:21 UTC, Martin Nowak wrote:

 On 05/10/2016 08:45 AM, Jacob Carlborg wrote:

 
 I was listening to a discussion Don and Daniel had about the 
 current implementation of CTFE. They talked about using a byte 
 code interpreter. Even implementing a really crappy byte code 
 interpreter would be a huge improvement.


 No need for a byte-code interpreter, it mostly just adds 
 overhead and complexity over an AST interpreter. If you want to 
 go really fast you need some sort of JIT anyhow, but a proper 
 interpreter will be orders of mangnitude faster than the 
 current implementation.

 I might refer you to
 http://dconf.org/2013/talks/chevalier_boisvert.pdf
 page 59 ff.


Correct. A ByteCode Interpreter will add even more implementation 
overhead, and the benefit is only realizable if the ByteCode  is 
a standard format that can be read other backends such as a jit.
May 15 2016
parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/15/2016 02:02 PM, Stefan Koch wrote:

 
 Correct. A ByteCode Interpreter will add even more implementation
 overhead, and the benefit is only realizable if the ByteCode  is a
 standard format that can be read other backends such as a jit.


This indeed would be an interesting proposal, interpretable IR that is
portable between the backends. But I guess we won't find such IR or at
least need to lower it into RealIR for dmd/gcc/llvm.
Sounds like an interesting candidate for the next GSoC.
May 15 2016
prev sibling parent deadalnix <deadalnix gmail.com> writes:
On Sunday, 15 May 2016 at 10:29:21 UTC, Martin Nowak wrote:

 On 05/10/2016 08:45 AM, Jacob Carlborg wrote:

 
 I was listening to a discussion Don and Daniel had about the 
 current implementation of CTFE. They talked about using a byte 
 code interpreter. Even implementing a really crappy byte code 
 interpreter would be a huge improvement.


 No need for a byte-code interpreter, it mostly just adds 
 overhead and complexity over an AST interpreter. If you want to 
 go really fast you need some sort of JIT anyhow, but a proper 
 interpreter will be orders of mangnitude faster than the 
 current implementation.

 I might refer you to
 http://dconf.org/2013/talks/chevalier_boisvert.pdf
 page 59 ff.


+1 . One need to walk the tree anyway to generate bytecode, which 
makes it impossible to make it faster for a one time execution.

For frequent executions, then a JIT is preferable, which let the 
bytecode the favorite choice for more than one, but not too many 
executions.
May 17 2016
prev sibling next sibling parent reply maik klein <maikklein googlemail.com> writes:
On Monday, 9 May 2016 at 16:57:39 UTC, Stefan Koch wrote:

 Hi Guys,

 I have been looking into the DMD now to see what I can do about 
 CTFE.
 Unfortunately It is a pretty big mess to untangle.
 Code responsible for CTFE is in at least 3 files.
 [dinterpret.d, ctfeexpr.d, constfold.d]
 I was shocked to discover that the PowExpression actually 
 depends on phobos! (depending on the exact codePath it may or 
 may not compile...)
 which let to me prematurely stating that it worked at ctfe 
 [http://forum.dlang.org/thread/ukcoibejffinknrbzktv forum.dlang.org]

 My Plan is as follows.

 Add a new file for my ctfe-interpreter and update it gradually 
 to take more and more of the cases the code in the files 
 mentioned above was used for.

 Do Dataflow analysis on the code that is to be ctfe'd so we can 
 tell beforehand if we need to store state in the ctfe stack or 
 not.

 Or baring proper data-flow analysis: RefCouting the variables 
 on the ctfe-stack could also be a solution.

 I will post more details as soon as I dive deeper into the code.


What is the current problem with ctfe?

Before I switched from C++ to D a few months ago I was heavily 
using boost hana in C++. I tried to emulate hana in D which 
worked quite well but the compile time performance was absolutely 
horrific

https://maikklein.github.io/2016/03/01/metaprogramming-typeobject/

After that I tried a few other things and I compared the compile 
times with

https://github.com/boostorg/hana/tree/master/benchmark

which I could never beat. The fastest thing, if I remember 
correctly, was string mixins but they used up too much memory.

But I have to say that I don't know much about the D internals 
and therefore don't know how I would optimize ct code execution.
May 11 2016
parent Jonathan M Davis via Digitalmars-d-announce writes:
On Wednesday, May 11, 2016 07:06:59 maik klein via Digitalmars-d-announce 
wrote:

 What is the current problem with ctfe?


The biggest problem is that it uses up a _lot_ of memory and is generally
slow. For instance, as I understand it, every time it mutates a variable, it
actually allocates a new one to hold the new state. This combined with the
fact that the compiler doesn't actually ever free memory (since it's
normally more efficient for it to work that way), and you risk running out
of memory while compiling. CTFE is a fantastic feature, but it evolved over
time rather than being designed up front, and it's suffered a lot because of
that. Don did a _lot_ of work to improve it, but he wasn't able to continue
working on it, and until now, no one has ever really stepped up to finish
the job. Don's post gives a good background on why CTFE is the way it is
and some of what he did to make it as solid as it is now:

http://forum.dlang.org/post/jmvsbhdpsjgeykpukoxf forum.dlang.org

But having someone like Stefan reimplement will be _huge_, and the D
community will be _very_ grateful.

- Jonathan M Davis
May 13 2016
prev sibling next sibling parent reply Don Clugston <sendmespam turnyourkidsintocash.com> writes:
On Monday, 9 May 2016 at 16:57:39 UTC, Stefan Koch wrote:

 Hi Guys,

 I have been looking into the DMD now to see what I can do about 
 CTFE.
 Unfortunately It is a pretty big mess to untangle.
 Code responsible for CTFE is in at least 3 files.
 [dinterpret.d, ctfeexpr.d, constfold.d]
 I was shocked to discover that the PowExpression actually 
 depends on phobos! (depending on the exact codePath it may or 
 may not compile...)


Yes. This is because of lowering. Walter said in his DConf talk 
that lowering was a success; actually, it's a quick-and-dirty 
hack that inevitably leads to a disaster.
Lowering always needs to be reverted.


 which let to me prematurely stating that it worked at ctfe 
 [http://forum.dlang.org/thread/ukcoibejffinknrbzktv forum.dlang.org]

 My Plan is as follows.

 Add a new file for my ctfe-interpreter and update it gradually 
 to take more and more of the cases the code in the files 
 mentioned above was used for.

 Do Dataflow analysis on the code that is to be ctfe'd so we can 
 tell beforehand if we need to store state in the ctfe stack or 
 not.


You don't need dataflow analysis. The CtfeCompile pass over the 
semantic tree was intended to determine how many variables are 
required by each function.


 Or baring proper data-flow analysis: RefCouting the variables 
 on the ctfe-stack could also be a solution.

 I will post more details as soon as I dive deeper into the code.



 The current implementation stores persistent state for every 
 ctfe incovation.
 While caching nothing. Not even the compiled for of a function 
 body.
 Because it cannot relax purity.


No. Purity is not why it doesn't save the state. It's because of 
history.

I think I need to explain the history of CTFE.
Originally, we had constant-folding. Then constant-folding was 
extended to do things like slicing a string at compile time. 
Constant folding leaks memory like the Exxon Valdez leaks oil, 
but that's OK because it only ever happens once.
Then, the constant folding was extended to include function 
calls, for loops, etc. All using the existing constant-folding 
code. Now the crappy memory usage is a problem. But it's OK 
because the CTFE code was kind of proof-of-concept thing anyway.

Now, everyone asks, why doesn't it use some kind of byte-code 
interpreter or something?
Well, the reason is, it just wasn't possible. There was actually 
no single CTFE entry point. Instead, it was a complete mess. For 
example, with template arguments, the compiler would first try to 
run CTFE on the argument, with error messages suppressed. If that 
succeeded, it was a template value argument. If it generated 
errors, it would then see if was a type. If that failed as well, 
it assumed it was a template alias argument.
The other big problem was that CTFE was also often called on a 
function which had semantic errors.

So, here is what I did with CTFE:
(1) Implement all the functionality, so that CTFE code can be 
developed. The valuable legacy of this, which I am immensely 
proud of, is the file "interpret3.d" in the test suite. It is 
very comprehensive. If an CTFE implementation passes the test 
suite, it's good to go.
The CTFE implementation itself is practically worthless. It's 
value was to get the test suite developed.

(2) Created a single entry point for CTFE. This involved working 
out rules for every place that CTFE is actually required, 
removing the horrid speculative execution of CTFE.
It made sure that functions had actually been semantically 
analyzed before they were executed (there were really horrific 
cases where the function had its semantic tree modified while it 
was being executed!!)
Getting to this point involved about 60 pull requests and six 
months of nearly full-time work. Finally it was possible to 
consider a byte-code interpreter or JITer.

We reached this point around Nov 2012.

(3) Added a 'ctfeCompile' step which runs over the semantic tree 
the first time the function is executed at compile time. Right 
now it does nothing much except that check that the semantic tree 
is valid. This detected many errors in the rest of the compiler.

We reached this point around March 2013.

My intention was to extend the cfteCompile step to a byte-code 
generator. But then I had to stop working on it and concentrate 
on looking after my family.

Looking at the code without knowing the history, you'll think, 
the obvious way to do this would be with a byte-code generator or 
JITer, and wonder why the implementation is so horrible. But for 
most of the history, that kind of implementation was just not 
possible.
People come up with all these elaborate schemes to speed up CTFE. 
It's totally not necessary. All that's needed is a very simple 
bytecode interpreter.
May 13 2016
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 13.05.2016 15:59, Don Clugston wrote:

 All that's needed is a very simple bytecode interpreter.


Here is the one I have hacked together:
https://github.com/tgehr/d-compiler/blob/master/interpret.d

This file does both constant folding and byte-code interpretation for 
most of the language. I still need to implement exception handling.

I'll let you know when it passes interpret3.d. :)
May 13 2016
prev sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 13 May 2016 at 13:59:57 UTC, Don Clugston wrote:

 I think I need to explain the history of CTFE.
 Originally, we had constant-folding. Then constant-folding was 
 extended to do things like slicing a string at compile time. 
 Constant folding leaks memory like the Exxon Valdez leaks oil, 
 but that's OK because it only ever happens once.
 Then, the constant folding was extended to include function 
 calls, for loops, etc. All using the existing constant-folding 
 code. Now the crappy memory usage is a problem. But it's OK 
 because the CTFE code was kind of proof-of-concept thing anyway.

 [...]


Thanks for the explanation, and for doing so much work on CTFE.


I would like to work on a solution that does scale.
The Problem is not making a byteCode-interpreter.
That part is relatively easy. Currently I am trying to get a 
detailed understanding of dmd and it's data-structures. (mainly 
it's AST.)

Generating the byte-code seems to be non-trivial.

I wonder in how far the glue layer can be of help...
May 13 2016
parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/13/2016 06:32 PM, Stefan Koch wrote:

 I would like to work on a solution that does scale. The Problem is 
 not making a byteCode-interpreter. That part is relatively easy. 
 Currently I am trying to get a detailed understanding of dmd and 
 it's data-structures. (mainly it's AST.)
 
 Generating the byte-code seems to be non-trivial.
 
 I wonder in how far the glue layer can be of help...


Seems like I've to repeat this once more, b/c everyone including me
didn't got it in the first place. We don't need a bytecode
interpreter, it mostly adds overhead and a complicated second layer
between walking the AST and interpreting it (think of transforming a
for loop with goto into linear bytecode, almost as complicated as in
the glue layer).

What we basically need is a stack of values, a stack of frames (for
function calls and variables in scopes), and an AST visitor that does
the interpretation.
It would be most helpful for the success of this to follow common CS
examples like [¹], [²], or [³].
With realistic expectations we might have a working implementation in
a month or so. With more requirements like bytecode, using dmd's
backend, or JIT we end up with a long newsgroup discussion ;).

Tricky things for a CTFE interpreter include:

- enumerating VarDeclarations (they already have a ctfeAdrOnStack
field) in each scope, and referring to outer variables from nested scopes

At best just use a continuous stack and just set the stack pointer to
the last frame pointer when leaving a scope.

- getting function calls right

Push arguments, on return shift top of stack under arguments and pop
them (caller cleanup). If possible detect and support tail recursion.

- converting AST values to and from Interpreter Values.

Literals and constant VarExp from the AST need to be converted to an
interpreter Value before being pushed on the stack. The result of
interpretation (top of stack) needs to be converted back to an AST
literal.
Using separate data types (instead of creating AST values in the
interpreter) will be a major performance improvement over using AST
values (e.g. 16 vs. ~100 bytes). It also creates a clear boundary
between Interpreter and AST values. Currently quite some complexity is
thrown at cleaning interpreter generated AST values, and
distinguishing interpreter owned from normal AST values (which allows
some optimizations) [⁴].

We don't need a tagged union b/c the AST already contains the type
information, but a tag could be helpful for debugging [⁵].

Values can be deleted when popped from stack (no ref counting needed I
think).

- Implementing more complex data structures (arrays, strings, hash
tables, aggregates)

Use Value[], Value[Value], and a dedicated String
(char[]/wchar[]/dchar[]). For structs/classes field indexes are known
=> use fix-sized Value[]. Value.class_ needs to hold a reference to
the actual class instance for vtable calls.

Last time I was working on this (also on a bytecode interpreter) the
entry point was fairly clear [⁶] (thanks to Don).

-Martin

[¹]: [The Interpreter In An Undergraduate Compilers Course
](http://arxiv.org/pdf/1412.0426.pdf)
[²]: [L8: Interpreters &
Visitors](http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-005-elements-of-software-construction-fall-2011/lecture-notes/MIT6_005F11_lec08.pdf)
[³]: [PA 2:
Interpreter](https://sites.google.com/a/bodik.org/cs164/projects/pa2)
[⁴]: https://github.com/dlang/dmd/search?q=ownedByCtfe
[⁵]:
https://github.com/MartinNowak/dmd/blob/28ffb0ab4fa6950f60c085f33f8a2ce23df7c0cd/src/interpret.c#L73
[⁶]:
https://github.com/MartinNowak/dmd/blob/28ffb0ab4fa6950f60c085f33f8a2ce23df7c0cd/src/interpret.c#L693
May 15 2016
prev sibling next sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 05/09/2016 06:57 PM, Stefan Koch wrote:

 I was shocked to discover that the PowExpression actually depends on
 phobos! (depending on the exact codePath it may or may not compile...)
 which let to me prematurely stating that it worked at ctfe
 [http://forum.dlang.org/thread/ukcoibejffinknrbzktv forum.dlang.org]


There is a really old bug report for that [Issue 3749 – cannot evaluate
yl2x (log) and exp functions at compile
time](https://issues.dlang.org/show_bug.cgi?id=3749). The lack of exp is
really limiting for many nice table precomputation use-cases in
scientific contexts.

-Martin
May 15 2016
prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 9 May 2016 at 16:57:39 UTC, Stefan Koch wrote:

 Hi Guys,

 I have been looking into the DMD now to see what I can do about 
 CTFE.



 I will post more details as soon as I dive deeper into the code.


Update :
int bug6498(int x)
{
     int n = 0;
     while (n < x)
         ++n;
     return n;
}
static assert(bug6498(10_000_000)==10_000_000);
This compiles now for me. Although it still takes 10 seconds ;)
May 27 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 27 May 2016 at 23:31:24 UTC, Stefan Koch wrote:

 On Monday, 9 May 2016 at 16:57:39 UTC, Stefan Koch wrote:

 Hi Guys,

 I have been looking into the DMD now to see what I can do 
 about CTFE.



 I will post more details as soon as I dive deeper into the 
 code.


 Update :
 int bug6498(int x)
 {
     int n = 0;
     while (n < x)
         ++n;
     return n;
 }
 static assert(bug6498(10_000_000)==10_000_000);
 This compiles now for me. Although it still takes 10 seconds ;)


Big Whoop!
Just make the number slightly lager and it fails again.
May 28 2016
parent Taylor Hillegeist <taylorh140 gmail.com> writes:
On Saturday, 28 May 2016 at 12:27:26 UTC, Stefan Koch wrote:

 On Friday, 27 May 2016 at 23:31:24 UTC, Stefan Koch wrote:

 On Monday, 9 May 2016 at 16:57:39 UTC, Stefan Koch wrote:

 Hi Guys,

 I have been looking into the DMD now to see what I can do 
 about CTFE.



 I will post more details as soon as I dive deeper into the 
 code.


 Update :
 int bug6498(int x)
 {
     int n = 0;
     while (n < x)
         ++n;
     return n;
 }
 static assert(bug6498(10_000_000)==10_000_000);
 This compiles now for me. Although it still takes 10 seconds ;)


 Big Whoop!
 Just make the number slightly lager and it fails again.


but smaller than 2_147_483_647 right :)
May 28 2016
prev sibling next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 9 May 2016 at 16:57:39 UTC, Stefan Koch wrote:


 I will post more details as soon as I dive deeper into the code.


Okay I briefly evaluated the current IR dmd uses for backend 
communication, and it seems usable for the purposes of a 
CTFE-Interpreter/JIT.

The Memory-Management issue is mostly orthogonal to CTFE.
But nonetheless very important.
Jun 03 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Friday, 3 June 2016 at 15:46:27 UTC, Stefan Koch wrote:

 On Monday, 9 May 2016 at 16:57:39 UTC, Stefan Koch wrote:


 I will post more details as soon as I dive deeper into the 
 code.


 Okay I briefly evaluated the current IR dmd uses for backend 
 communication, and it seems usable for the purposes of a 
 CTFE-Interpreter/JIT.

 The Memory-Management issue is mostly orthogonal to CTFE.
 But nonetheless very important.


I have to admit currently I am a bit stuck.

 Timon Gehr cooperation would be highly appreciated.
Jun 07 2016
next sibling parent reply Stefan Koch <uplink.coder googlemail.com> writes:
First small code example compiles!

int bug6498(int x) {
  int n = 0;

  while (n < x) {
   n++;
  }

  return n;
}

evaluation of bug6498(100_000_00) took 226 msecs.
evaluation of bug6498(100_000_000) took 2228 msecs.

The memory allocated by the Evaluator is exactly 12 bytes.
Jun 29 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Thursday, 30 June 2016 at 01:20:08 UTC, Stefan Koch wrote:

 First small code example compiles!

 int bug6498(int x) {
  int n = 0;

  while (n < x) {
   n++;
  }

  return n;
 }

 evaluation of bug6498(100_000_00) took 226 msecs.
 evaluation of bug6498(100_000_000) took 2228 msecs.

 The memory allocated by the Evaluator is exactly 12 bytes.


The speedup comes from interpreting the IR or fixing the memory 
leaking?
Jun 29 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 30 June 2016 at 01:32:47 UTC, Martin Nowak wrote:

 On Thursday, 30 June 2016 at 01:20:08 UTC, Stefan Koch wrote:

 First small code example compiles!

 int bug6498(int x) {
  int n = 0;

  while (n < x) {
   n++;
  }

  return n;
 }

 evaluation of bug6498(100_000_00) took 226 msecs.
 evaluation of bug6498(100_000_000) took 2228 msecs.

 The memory allocated by the Evaluator is exactly 12 bytes.


 The speedup comes from interpreting the IR or fixing the memory 
 leaking?


Both. Actually I could not imagine fixing the memory problem 
without doing IR interpretation.
I will tackle compiling more difficult code later today.
As soon as I can run my compiletime brainfuck I will open a PR.

Until then you can see my progress at 
https://github.com/UplinkCoder/dmd/tree/newCTFE
I will try to always keep the branch in a healthy state.
Jun 29 2016
next sibling parent reply =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Thursday, 30 June 2016 at 03:17:38 UTC, Stefan Koch wrote:

 Until then you can see my progress at 
 https://github.com/UplinkCoder/dmd/tree/newCTFE
 I will try to always keep the branch in a healthy state.


I can't wait to see the benchmarks.

Keep up the good work!
Jun 30 2016
parent Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 30 June 2016 at 07:15:30 UTC, Nordlöw wrote:

 On Thursday, 30 June 2016 at 03:17:38 UTC, Stefan Koch wrote:

 Until then you can see my progress at 
 https://github.com/UplinkCoder/dmd/tree/newCTFE
 I will try to always keep the branch in a healthy state.


 I can't wait to see the benchmarks.

 Keep up the good work!


Currently the interpreter is about 10x-15x slower then native 
execution.
and about 2x-4000x faster then the old Interpreter :)

code I can currently compile :

uint test2(int x) {
	uint r = 0;
	const uint end = 9;
	for(int i = 0;i < end;++i) {
		for(int j = 0;j < 9;++j) {
			r += j;
			r += i;
		}
	}

	return r;
}

dmd's interpreter and mine return the same value :)
Jun 30 2016
prev sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 06/30/2016 05:17 AM, Stefan Koch wrote:

 Both. Actually I could not imagine fixing the memory problem without
 doing IR interpretation.
 I will tackle compiling more difficult code later today.
 As soon as I can run my compiletime brainfuck I will open a PR.
 
 Until then you can see my progress at
 https://github.com/UplinkCoder/dmd/tree/newCTFE
 I will try to always keep the branch in a healthy state.


Looks good and will definitely lead to a proper CTFE interpreter.

You might want to go back and look at
https://github.com/MartinNowak/dmd/blob/28ffb0ab4fa6950f60c085f33f8a2ce23df7c0cd/src/interpret.c,
it could already do a few more things than your interpreter atm.

[¹]: https://github.com/MartinNowak/dmd/commits/ctfe
Jun 30 2016
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 08.06.2016 03:20, Stefan Koch wrote:

 On Friday, 3 June 2016 at 15:46:27 UTC, Stefan Koch wrote:

 On Monday, 9 May 2016 at 16:57:39 UTC, Stefan Koch wrote:


 I will post more details as soon as I dive deeper into the code.


 Okay I briefly evaluated the current IR dmd uses for backend
 communication, and it seems usable for the purposes of a
 CTFE-Interpreter/JIT.

 The Memory-Management issue is mostly orthogonal to CTFE.
 But nonetheless very important.


 I have to admit currently I am a bit stuck.

  Timon Gehr cooperation would be highly appreciated.


Sorry, I had missed this. I see you were able to make progress.
Jun 30 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 30 June 2016 at 14:17:30 UTC, Timon Gehr wrote:

 Sorry, I had missed this. I see you were able to make progress.


It's fine.
Honestly the hardest thing was to actually get started.

Once I see something working the excitement carries me forward :)

I would appreciate a critical opinion, anytime!
Jun 30 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 30 June 2016 at 14:26:29 UTC, Stefan Koch wrote:

 On Thursday, 30 June 2016 at 14:17:30 UTC, Timon Gehr wrote:

 Sorry, I had missed this. I see you were able to make progress.


 It's fine.
 Honestly the hardest thing was to actually get started.

 Once I see something working the excitement carries me forward 
 :)

 I would appreciate a critical opinion, anytime!


Another update.
_very_ Basic SwtichStatements work now :)

I expect a simple project like a compile-time bf interpreter to 
work by the end of next week.
Jul 03 2016
parent reply ZombineDev <petar.p.kirov gmail.com> writes:
On Monday, 4 July 2016 at 01:54:29 UTC, Stefan Koch wrote:

 On Thursday, 30 June 2016 at 14:26:29 UTC, Stefan Koch wrote:

 On Thursday, 30 June 2016 at 14:17:30 UTC, Timon Gehr wrote:

 Sorry, I had missed this. I see you were able to make 
 progress.


 It's fine.
 Honestly the hardest thing was to actually get started.

 Once I see something working the excitement carries me forward 
 :)

 I would appreciate a critical opinion, anytime!


 Another update.
 _very_ Basic SwtichStatements work now :)

 I expect a simple project like a compile-time bf interpreter to 
 work by the end of next week.


Nice work! Any chance that you could also improve AliasSeq 
algorithms, like those in std.meta to compile faster and use less 
memory during compilation? Or is that too different from CTFE?
Jul 04 2016
next sibling parent reply ZombineDev <petar.p.kirov gmail.com> writes:
On Monday, 4 July 2016 at 07:29:49 UTC, ZombineDev wrote:

 On Monday, 4 July 2016 at 01:54:29 UTC, Stefan Koch wrote:

 On Thursday, 30 June 2016 at 14:26:29 UTC, Stefan Koch wrote:

 On Thursday, 30 June 2016 at 14:17:30 UTC, Timon Gehr wrote:

 Sorry, I had missed this. I see you were able to make 
 progress.


 It's fine.
 Honestly the hardest thing was to actually get started.

 Once I see something working the excitement carries me 
 forward :)

 I would appreciate a critical opinion, anytime!


 Another update.
 _very_ Basic SwtichStatements work now :)

 I expect a simple project like a compile-time bf interpreter 
 to work by the end of next week.


 Nice work! Any chance that you could also improve AliasSeq 
 algorithms, like those in std.meta to compile faster and use 
 less memory during compilation? Or is that too different from 
 CTFE?


BTW, do you plan to handle stuff like exceptions, virtual calls 
and associative arrays and if so how? Sounds quite challenging.
Jul 04 2016
next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 4 July 2016 at 07:33:48 UTC, ZombineDev wrote:

 On Monday, 4 July 2016 at 07:29:49 UTC, ZombineDev wrote:

 Nice work! Any chance that you could also improve AliasSeq 
 algorithms, like those in std.meta to compile faster and use 
 less memory during compilation? Or is that too different from 
 CTFE?


 BTW, do you plan to handle stuff like exceptions, virtual calls 
 and associative arrays and if so how? Sounds quite challenging.


Templates do interact with CTFE.
They will improve as well as a consequence of CTFE being less 
wasteful.

I will do it bit by bit.
For now getting a simple parser to work with the new engine is 
the goal.
Everything else comes later.
Jul 04 2016
prev sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Monday, 4 July 2016 at 07:33:48 UTC, ZombineDev wrote:

 BTW, do you plan to handle stuff like exceptions, virtual calls 
 and associative arrays and if so how? Sounds quite challenging.


on the lower level, it's all just arrays and jumps. ;-)
Jul 05 2016
prev sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Monday, 4 July 2016 at 07:29:49 UTC, ZombineDev wrote:

 Nice work! Any chance that you could also improve AliasSeq 
 algorithms, like those in std.meta to compile faster and use 
 less memory during compilation? Or is that too different from 
 CTFE?


Not that I opposes this, but please keep it focused. Doing a 
bytecode interpreter is a huge task on its own and this seems 
largely orthogonal.
Jul 05 2016
parent ZombineDev <petar.p.kirov gmail.com> writes:
On Tuesday, 5 July 2016 at 21:11:39 UTC, deadalnix wrote:

 On Monday, 4 July 2016 at 07:29:49 UTC, ZombineDev wrote:

 Nice work! Any chance that you could also improve AliasSeq 
 algorithms, like those in std.meta to compile faster and use 
 less memory during compilation? Or is that too different from 
 CTFE?


 Not that I opposes this, but please keep it focused. Doing a 
 bytecode interpreter is a huge task on its own and this seems 
 largely orthogonal.


I was actually wondering if they're orthogonal or some of the 
CTFE improvements could be reused. I was not saying the he should 
work on both if they're not related.
Jul 05 2016
prev sibling next sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
so, we played a little game with Stefan: i wrote a simple 
stack-based VM implementation for our beloved bug6498, and got 
270 msecs with debug build, 140 msecs with release build. can be 
made even faster with some simple peephole optimizer (it has two 
hacks to get fair times on 6498, but this is not a good peephole 
opt ;-). of course, it scales linearly too.

don't expect me to join the race, though: i was just curious how 
well stack-based VM can do.
Jul 05 2016
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= writes:
On Tuesday, 5 July 2016 at 16:40:05 UTC, ketmar wrote:

 so, we played a little game with Stefan: i wrote a simple 
 stack-based VM implementation for our beloved bug6498, and got


What is «bug6498»?
Jul 05 2016
parent "H. S. Teoh via Digitalmars-d-announce" writes:
On Tue, Jul 05, 2016 at 05:44:14PM +0000, Ola Fosheim Grstad via
Digitalmars-d-announce wrote:

 On Tuesday, 5 July 2016 at 16:40:05 UTC, ketmar wrote:

 so, we played a little game with Stefan: i wrote a simple
 stack-based VM implementation for our beloved bug6498, and got


 
 What is bug6498?


https://issues.dlang.org/show_bug.cgi?id=6498


T

-- 
Give me some fresh salted fish, please.
Jul 05 2016
prev sibling next sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
and just to make sure that my approach is working: bytecode 
compiler now compiles simple free functions without args and 
returning int (yep, there are some in phobos! ;-), while passing 
everything other to old interpreter. it works, and all phobos 
unittests are passed (and some functions were actually executed 
with VM).

that's where i should stop, i think, until it comes too far. ;-)
Jul 05 2016
parent reply Rory McGuire via Digitalmars-d-announce writes:
On Tue, Jul 5, 2016 at 11:54 PM, ketmar via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 and just to make sure that my approach is working: bytecode compiler now
 compiles simple free functions without args and returning int (yep, there
 are some in phobos! ;-), while passing everything other to old interpreter.
 it works, and all phobos unittests are passed (and some functions were
 actually executed with VM).

 that's where i should stop, i think, until it comes too far. ;-)


Are you sharing this code / how you did it with the GSOC intern?
Jul 05 2016
next sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Wednesday, 6 July 2016 at 05:54:48 UTC, Rory McGuire wrote:

 Are you sharing this code


yes. we are chatting with Stefan in IRC, and the repository is 
public (i mean, the link was there ;-). yet it won't compile with 
"vanilla" dmd anyway -- i require my dmd fork ("aliced"). and i 
don't really want to make it more public, as i have no intentions 
to turn that into full-blown engine.

also, not only our codebases are completely different, but our 
designs and approaches are drastically different. we just can't 
share/reuse the code, those projects are as compatible as WLIV 
and Z80. ;-)


 how you did it with the GSOC intern?


Stefan in not GSOC intern. ;-)
Jul 05 2016
parent reply Rory McGuire via Digitalmars-d-announce writes:
On Wed, Jul 6, 2016 at 8:09 AM, ketmar via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On Wednesday, 6 July 2016 at 05:54:48 UTC, Rory McGuire wrote:


 Are you sharing this code


 yes. we are chatting with Stefan in IRC, and the repository is public (i
 mean, the link was there ;-). yet it won't compile with "vanilla" dmd
 anyway -- i require my dmd fork ("aliced"). and i don't really want to make
 it more public, as i have no intentions to turn that into full-blown engine.


Nice! Sounds like fun.



 also, not only our codebases are completely different, but our designs and
 approaches are drastically different. we just can't share/reuse the code,
 those projects are as compatible as WLIV and Z80. ;-)


Thought as much, wondered if the ideas about how they work were being
shared; often one idea sparks another.



 how you did it with the GSOC intern?


 Stefan in not GSOC intern. ;-)


doh! my bad.



 Jul 06 2016



  parent  ketmar <ketmar ketmar.no-ip.org>  writes:


On Wednesday, 6 July 2016 at 08:16:32 UTC, Rory McGuire wrote:

 Thought as much, wondered if the ideas about how they work were 
 being shared; often one idea sparks another.


sure, some ideas flows; mostly very high-level. for example, 
proper fallback to the original interpreter was invented by 
Stefan, but i was faster to make it work. ;-)

after all, we aren't competitors. he is still working hard on 
full implementation, and i am just having some fun, investigating 
creation of DMD backends by the way.



 Jul 06 2016



prev sibling  parent  ketmar <ketmar ketmar.no-ip.org>  writes:


On Wednesday, 6 July 2016 at 05:54:48 UTC, Rory McGuire wrote:

 Are you sharing this code


i mean "my code is publicly available, but we aren't sharing any 
code between our projects".



 Jul 05 2016



prev sibling  parent reply ketmar <ketmar ketmar.no-ip.org>  writes:


just... can't... stop...

some current statistics:

druntime
total interpreted calls: 27,146
total complied calls   : 1,330

phobos
total interpreted calls: 6,492,826
total complied calls   : 19,975

i can't do function calls from compiled code yet, so number of 
compiled calls is low, and no speedup can be seen. also, i can do 
only integer math and string concatenation.

yes, phobos has ALOT of CTFE! ;-)

this is statistics from "make -f posix.mak unittest" (yes, i can 
pass all unittests too!)

still, not that bad for a toy implementation! ;-)



 Jul 06 2016



  parent  ketmar <ketmar ketmar.no-ip.org>  writes:


yay. ackemann(3, 7) in 70 milliseconds! and it does TCO.

default interpreter is unable to execute that -- recursion is too 
deep.



 Jul 07 2016



prev sibling  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Monday, 9 May 2016 at 16:57:39 UTC, Stefan Koch wrote:

 Hi Guys,

 I have been looking into the DMD now to see what I can do about 
 CTFE.
 [ .... ]
 I will post more details as soon as I dive deeper into the code.


I decided to keep a gist updated to represent the current state 
the new engine can handle.

https://gist.github.com/UplinkCoder/89faa06311e417aa93ea99bc92934d3e

This is the currentState after approx. 50 hours of work



 Jul 09 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Saturday, 9 July 2016 at 20:09:14 UTC, Stefan Koch wrote:

 I decided to keep a gist updated to represent the current state 
 the new engine can handle.

 https://gist.github.com/UplinkCoder/89faa06311e417aa93ea99bc92934d3e

 This is the currentState after approx. 50 hours of work


First StringIndexing works now.
The next step is getting structs right,



 Jul 13 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 14 July 2016 at 00:45:53 UTC, Stefan Koch wrote:

 On Saturday, 9 July 2016 at 20:09:14 UTC, Stefan Koch wrote:

 I decided to keep a gist updated to represent the current 
 state the new engine can handle.

 https://gist.github.com/UplinkCoder/89faa06311e417aa93ea99bc92934d3e

 This is the currentState after approx. 50 hours of work


 First StringIndexing works now.
 The next step is getting structs right,


Support for _basic_ support for structs has just landed in my 
branch.
only structs with int or uint members are supported for now.
And no arrays.
But this is coming soon :)



 Jul 15 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Friday, 15 July 2016 at 20:36:46 UTC, Stefan Koch wrote:

 I decided to keep a gist updated to represent the current state 
 the new engine can handle.
 https://gist.github.com/UplinkCoder/89faa06311e417aa93ea99bc92934d3e


Internal changes to the bytecode engine make the manipulation of 
values on the ctfe stack possible.

this allows the following code to properly compile now.

struct V3 {int x; int y; int z;}
int fn() {

         auto b = V3(1,2,3);
         b.y += 19;
         return b.y;
}

static assert(fn() == 21);



 Jul 17 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


Nice! how are you keeping track of changes? Are you tagging / branching
each time you get something new working?

On Sun, Jul 17, 2016 at 12:12 PM, Stefan Koch via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On Friday, 15 July 2016 at 20:36:46 UTC, Stefan Koch wrote:


 I decided to keep a gist updated to represent the current state the new
 engine can handle.
 https://gist.github.com/UplinkCoder/89faa06311e417aa93ea99bc92934d3e


 Internal changes to the bytecode engine make the manipulation of values on
 the ctfe stack possible.

 this allows the following code to properly compile now.

 struct V3 {int x; int y; int z;}
 int fn() {

         auto b = V3(1,2,3);
         b.y += 19;
         return b.y;
 }

 static assert(fn() == 21);





 Jul 17 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Sunday, 17 July 2016 at 12:12:17 UTC, Rory McGuire wrote:

 Nice! how are you keeping track of changes? Are you tagging / 
 branching each time you get something new working?


I just push a new commit.
And add code to the gist.

Why would I branch ?
It all goes towards the same goal.



 Jul 17 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Sun, Jul 17, 2016 at 2:41 PM, Stefan Koch via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On Sunday, 17 July 2016 at 12:12:17 UTC, Rory McGuire wrote:


 Nice! how are you keeping track of changes? Are you tagging / branching
 each time you get something new working?


 I just push a new commit.
 And add code to the gist.

 Why would I branch ?
 It all goes towards the same goal.


A commit does pretty much the same thing. Tags/Branches just make UIs show
a "menu" for selecting a change, not important really. I had just wondered
if you were marking these milestones.
Someone could always branch off of any of your commits, it they wanted to
try an alternative approach (or learn from you).

Thanks for all the work you are doing. I love CTFE. I have a lot of big
vibe (diet) templates, and my own Jade ctfe implementation, really looking
forward to faster compile times.

R



 Jul 17 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Sunday, 17 July 2016 at 13:04:33 UTC, Rory McGuire wrote:

 On Sun, Jul 17, 2016 at 2:41 PM, Stefan Koch via 
 Digitalmars-d-announce < digitalmars-d-announce puremagic.com> 
 wrote:


[...]


 A commit does pretty much the same thing. Tags/Branches just 
 make UIs show
 a "menu" for selecting a change, not important really. I had 
 just wondered
 if you were marking these milestones.
 Someone could always branch off of any of your commits, it they 
 wanted to
 try an alternative approach (or learn from you).

 Thanks for all the work you are doing. I love CTFE. I have a 
 lot of big vibe (diet) templates, and my own Jade ctfe 
 implementation, really looking forward to faster compile times.

 R


If you let me have a look at your jade-impl. I can probably work 
on speeding up it up. Vibed is still going to take a while. I 
need to figure all this UTF stuff out to make string handling 
robust.



 Jul 17 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Sun, Jul 17, 2016 at 3:28 PM, Stefan Koch via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On Sunday, 17 July 2016 at 13:04:33 UTC, Rory McGuire wrote:


 On Sun, Jul 17, 2016 at 2:41 PM, Stefan Koch via Digitalmars-d-announce <
 digitalmars-d-announce puremagic.com> wrote:

 [...]


 A commit does pretty much the same thing. Tags/Branches just make UIs show
 a "menu" for selecting a change, not important really. I had just wondered
 if you were marking these milestones.
 Someone could always branch off of any of your commits, it they wanted to
 try an alternative approach (or learn from you).

 Thanks for all the work you are doing. I love CTFE. I have a lot of big
 vibe (diet) templates, and my own Jade ctfe implementation, really looking
 forward to faster compile times.

 R


 If you let me have a look at your jade-impl. I can probably work on
 speeding up it up. Vibed is still going to take a while. I need to figure
 all this UTF stuff out to make string handling robust.


Thanks that would be great, however I think its a while off before it can
work on your ctfe implementation. It uses Pegged, so getting the Pegged
JSON parser or pegged.peg working would be a first step.

If you were just meaning to show me any improvements I could make, the
source code is at:
https://github.com/rjmcguire/jaded


Cheers,
R



 Jul 17 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Sunday, 17 July 2016 at 15:57:28 UTC, Rory McGuire wrote:

 Thanks that would be great, however I think its a while off 
 before it can work on your ctfe implementation. It uses Pegged, 
 so getting the Pegged JSON parser or pegged.peg working would 
 be a first step.


pegged uses templates.
There is an intersection on which it can profit from CTFE 
improvments. But for the most part, it's a different sub-system 
in the compiler.

That said, templates are next on my TODO list.



 Jul 17 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Sun, Jul 17, 2016 at 6:08 PM, Stefan Koch via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On Sunday, 17 July 2016 at 15:57:28 UTC, Rory McGuire wrote:


 Thanks that would be great, however I think its a while off before it can
 work on your ctfe implementation. It uses Pegged, so getting the Pegged
 JSON parser or pegged.peg working would be a first step.


 pegged uses templates.
 There is an intersection on which it can profit from CTFE improvments. But
 for the most part, it's a different sub-system in the compiler.

 That said, templates are next on my TODO list.


Nice, so I'll be able to see separate improvements in my CTFE stuff vs the
pegged template stuff once structs,  classes, etc.. are handled in your new
ctfe.



 Jul 17 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Sunday, 17 July 2016 at 16:43:18 UTC, Rory McGuire wrote:

 Nice, so I'll be able to see separate improvements in my CTFE 
 stuff vs the pegged template stuff once structs,  classes, 
 etc.. are handled in your new ctfe.


Yes.
Btw while working on printing template-instantiations I 
discovered that ctfe improvements will most likely have a much 
higher impact in template heavy code then I fist expected.



 Jul 19 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


Hi,

I have fresh performance statistics:
The test[1] involved running an empty while loop.

Machine 1 Linux :
DMD release : Interpreted : 3400 ms || Pseudo-Jited 230 || 50ms 
Native
DMD Debug : Interpreted 4560 || Pseudo-Jited 260ms || Native 230 
ms

LDC release : Interpreted 2400 ms || Pseudo-Jited 0 ms || Native 
0 ms
LDC debug : Interpreted 8700 ms || Pseudo-Jited 330 ms || Native 
250 ms

Machine 2 Windows :

DMD release : Interpreted : 5500 ms || Pseudo-Jited 180ms || 40ms 
Native
DMD Debug : Interpreted 5500 || Pseudo-Jited 320ms || Native 180 
ms


Pay attention to the ratio (~4.5) between (pseudo)jit[2] and 
native.
This slowdown inherent in the VM design I choose.

please share your thoughts.

[1] https://github.com/UplinkCoder/dmd/blob/newCTFE/bcPerf.d
[2] 
https://github.com/UplinkCoder/dmd/blob/newCTFE/src/bc_c_backend.d



 Jul 29 2016



 next sibling parent reply Robert burner Schadek <rburners gmail.com>  writes:


On Friday, 29 July 2016 at 11:30:20 UTC, Stefan Koch wrote:

 please share your thoughts.


When is this moving into dmd master?



 Jul 29 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Friday, 29 July 2016 at 12:47:55 UTC, Robert burner Schadek 
wrote:

 On Friday, 29 July 2016 at 11:30:20 UTC, Stefan Koch wrote:

 please share your thoughts.


 When is this moving into dmd master?


As soon as it passes the test-suite.
And can execute diet-ng on the fast-path.

Currently I have issues with chained binary expressions.
such as

if ((a && b) || (a && c)) {//bla}



 Jul 29 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Friday, 29 July 2016 at 14:28:03 UTC, Stefan Koch wrote:

 if ((a && b) || (a && c)) {//bla}


This is now solved although quite naively at the cost of 
inserting twice the number of instructions for thoose cases.
Then agian we are still much faster then the old interpreter.
And I can still optimize this if it should turn out to be a 
bottleneck.

Tomorrow I will fix the regressed string handling.



 Jul 29 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


I am very happy to announce that calls are almost working now.
And the code-gen-interface[1] is near finalization.

It should be very easy to provide different back-ends such as 
LLVM or libJIT now.
The missing pieces well be added during next week.

I invite everyone to comment on the interface and code.

Don't be shy and feel free to comment

[1] 
https://github.com/UplinkCoder/dmd/blob/newCTFE/src/bc_common.d#L313

So far the strong decoupling between dmd and the bytecode has 
worked out pretty well.

Being able to debug and test the bytecode component independently 
from the AST-processing makes for a faster development process 
and allows insight into the code-gen behavior.



 Jul 31 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


Time for an update.
(ASCII)-Strings work reasonably well.

I am now working on supporting general Sliceing and Appending.
The effort on function calls is also still ongoing.

I added a switch to my version of dmd which allows to toggle the 
ctfe engine.
So now I can compare apples to apples when posting perf data.

A nice weekend to all of you.



 Aug 06 2016



 next sibling parent reply Rory McGuire via Digitalmars-d-announce  writes:


On 06 Aug 2016 16:30, "Stefan Koch via Digitalmars-d-announce" <
digitalmars-d-announce puremagic.com> wrote:

 Time for an update.
 (ASCII)-Strings work reasonably well.

 I am now working on supporting general Sliceing and Appending.
 The effort on function calls is also still ongoing.

 I added a switch to my version of dmd which allows to toggle the ctfe


engine.

 So now I can compare apples to apples when posting perf data.

 A nice weekend to all of you.


Hi Stefan,

Are you saying we can play around with ascii string slicing/appending
already?

The gist with the current state of ctfe support seems to have last been
changed 21 days ago? Am I reading that right? If so where could I find your
current tests, if I wanted to run the tests on my machine.

Cheers,
Rory



 Aug 06 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Saturday, 6 August 2016 at 19:07:10 UTC, Rory McGuire wrote:


 Hi Stefan,

 Are you saying we can play around with ascii string 
 slicing/appending already?


No, not now, but very soon. I want to have _basic_ utf8 support 
before I am comfortable with enabling string operations.


 The gist with the current state of ctfe support seems to have 
 last been changed 21 days ago? Am I reading that right? If so 
 where could I find your current tests, if I wanted to run the 
 tests on my machine.


That could be right.
Many features are half-implemented at the moment and will fail in 
random and exciting ways.
Therefore it is too soon to publish code testing those.

If you want particular features the best way is contacting me.



 Aug 06 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Saturday, 6 August 2016 at 23:04:48 UTC, Stefan Koch wrote:

 On Saturday, 6 August 2016 at 19:07:10 UTC, Rory McGuire wrote:


 Hi Stefan,

 Are you saying we can play around with ascii string 
 slicing/appending already?


 No, not now, but very soon. I want to have _basic_ utf8 support 
 before I am comfortable with enabling string operations.


Tomorrow this is going to work.



 Aug 07 2016



  parent reply Johan Engelen <j j.nl>  writes:


On Sunday, 7 August 2016 at 16:52:28 UTC, Stefan Koch wrote:

 On Saturday, 6 August 2016 at 23:04:48 UTC, Stefan Koch wrote:

 No, not now, but very soon. I want to have _basic_ utf8 
 support before I am comfortable with enabling string 
 operations.


 Tomorrow this is going to work.


Hi Stefan,

How much effort will it be to integrate your work into LDC? I'd 
like to test it on some Weka code.

Definitely let us know once you can CTFE something like 
std.traits.ParameterStorageClassTuple.

cheers,
   Johan



 Aug 08 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Monday, 8 August 2016 at 09:51:03 UTC, Johan Engelen wrote:

 On Sunday, 7 August 2016 at 16:52:28 UTC, Stefan Koch wrote:

 On Saturday, 6 August 2016 at 23:04:48 UTC, Stefan Koch wrote:

 No, not now, but very soon. I want to have _basic_ utf8 
 support before I am comfortable with enabling string 
 operations.


 Tomorrow this is going to work.


 Hi Stefan,

 How much effort will it be to integrate your work into LDC? I'd 
 like to test it on some Weka code.

 Definitely let us know once you can CTFE something like 
 std.traits.ParameterStorageClassTuple.

 cheers,
   Johan


Hi Johan,

It won't be too much effort. Since it was designed with that in 
mind.



 Aug 08 2016



  parent reply Kagamin <spam here.lot>  writes:


1. You said CTFE engine can be ctfeable itself? But it uses 
unions in BCValue - it's not going to work in CTFE, is it? Just 
wondering myself what's the way to have polymorphism at compile 
time.
2. The byte code generator interface has no mean to declare 
functions? In case of LLVM jit one would want to declare 
functions so that jit can use registers to pass parameters for a 
bit more extra speed, though it's probably not worth it.



 Aug 09 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Tuesday, 9 August 2016 at 12:30:18 UTC, Kagamin wrote:

 1. You said CTFE engine can be ctfeable itself? But it uses 
 unions in BCValue - it's not going to work in CTFE, is it? Just 
 wondering myself what's the way to have polymorphism at compile 
 time.
 2. The byte code generator interface has no mean to declare 
 functions? In case of LLVM jit one would want to declare 
 functions so that jit can use registers to pass parameters for 
 a bit more extra speed, though it's probably not worth it.


1. unions work fine when you use them as discriminated unions.
2. The BCGen does not have to declare functions. Since every-time 
it is created it will implicitly define a function.
It will all make more sense once the function-call system works.



 Aug 09 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


I have just committed the changes necessary for (ascii) string 
indexing and (ascii) string-foreach.

Currently working on UTF8 => UTF32 auto-re-encoding and 
StringConcat and Slice-Support.

Completion of Slice support will also fix the interplay between 
structs and arrays.

Cheers,
Stefan



 Aug 11 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


Hi,

I took a break from work on string operations and focused instead 
of improving the robustness of the engine.
I.E. for it not to halt the compiler on unsupported expressions.

right now,
I can compile druntime without failures.
Phobos should be working by the end of next week.

Have a nice Sunday,

Stefan

PS. String foreach regressed.
And Struct-handling regressed.
Due to a change of memory-layout.

I will fix this as soon as the engine does no longer abort.



 Aug 13 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Sun, Aug 14, 2016 at 5:21 AM, Stefan Koch via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 Hi,

 I took a break from work on string operations and focused instead of
 improving the robustness of the engine.
 I.E. for it not to halt the compiler on unsupported expressions.

 right now,
 I can compile druntime without failures.
 Phobos should be working by the end of next week.

 Have a nice Sunday,

 Stefan

 PS. String foreach regressed.
 And Struct-handling regressed.
 Due to a change of memory-layout.

 I will fix this as soon as the engine does no longer abort.


Nice! thanks! Will be nice to be able to test newCTFE with our "normal"
code every now and then.



 Aug 14 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


Just a small update today.
if(__ctfe) and if(!__ctfe)
now get special treatment.
Also working on getting compiletime-parsers to run.



 Aug 17 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On 17 Aug 2016 18:50, "Stefan Koch via Digitalmars-d-announce" <
digitalmars-d-announce puremagic.com> wrote:

 Just a small update today.
 if(__ctfe) and if(!__ctfe)
 now get special treatment.
 Also working on getting compiletime-parsers to run.


Nice tease with the "compile time parsers to run" aside.

We're salivating here. What exactly did you mean by that?

Which compile time parser are you using for testing?

PS: thanks for the updates!



 Aug 17 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Wednesday, 17 August 2016 at 18:24:37 UTC, Rory McGuire wrote:

 On 17 Aug 2016 18:50, "Stefan Koch via Digitalmars-d-announce" 
 < digitalmars-d-announce puremagic.com> wrote:

 Just a small update today.
 if(__ctfe) and if(!__ctfe)
 now get special treatment.
 Also working on getting compiletime-parsers to run.


 Nice tease with the "compile time parsers to run" aside.

 We're salivating here. What exactly did you mean by that?

 Which compile time parser are you using for testing?

 PS: thanks for the updates!


I am using ctfe-bf for a start and others of my own design.
Of course pegged is also on the menu.



 Aug 17 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


Hi Guys,
First of all,
parsers will not make it before September.
I am sorry about that.

Currently I am fixing issues with the design, that for example 
prevent slices of slices to work.

Also I am writing analysis and debugging code to (such as 
generating a call-graph and primitive DFA) that will hopefully 
make function-calls work reliably.
The DMD-AST does not make it easy to track things like variables 
originating from a parent-scope.


I feel that this can have a positive impact on the whole of dmd, 
since that will allow better frontend-optimisations.

I am happy for all comments or suggestions.

Cheers,
Stefan



 Aug 28 2016



 next sibling parent reply Dominikus Dittes Scherkl <Dominikus.Scherkl continental-corporation.com>  writes:


On Monday, 29 August 2016 at 00:24:01 UTC, Stefan Koch wrote:

 I feel that this can have a positive impact on the whole of 
 dmd, since that will allow better frontend-optimisations.

 I am happy for all comments or suggestions.


The work you are doing is just awesome!
Many thanks.



 Aug 29 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Mon, Aug 29, 2016 at 9:51 AM, Dominikus Dittes Scherkl via
Digitalmars-d-announce <digitalmars-d-announce puremagic.com> wrote:


 On Monday, 29 August 2016 at 00:24:01 UTC, Stefan Koch wrote:


 I feel that this can have a positive impact on the whole of dmd, since
 that will allow better frontend-optimisations.

 I am happy for all comments or suggestions.


 The work you are doing is just awesome!
 Many thanks.


+1 your work is key for our success as a community.

R



 Aug 29 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Monday, 29 August 2016 at 08:05:10 UTC, Rory McGuire wrote:

 On Mon, Aug 29, 2016 at 9:51 AM, Dominikus Dittes Scherkl via

 The work you are doing is just awesome!
 Many thanks.


 +1 your work is key for our success as a community.

 R


Thanks guys.

I just came up with a nifty little patch that makes it possible 
to ensure that a function is _only_ used at ctfe.
Or the opposite.

static assert(__ctfe, "This function is not supposed to be called 
outside of ctfe");
and static assert(!__ctfe, "This function is not supposed to be 
called during ctfe");

similarly you can use static if (__ctfe).

Is it worth trying to get it into master ?



 Aug 29 2016



 next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2016-08-29 10:39, Stefan Koch wrote:


 Thanks guys.

 I just came up with a nifty little patch that makes it possible to
 ensure that a function is _only_ used at ctfe.
 Or the opposite.

 static assert(__ctfe, "This function is not supposed to be called
 outside of ctfe");
 and static assert(!__ctfe, "This function is not supposed to be called
 during ctfe");

 similarly you can use static if (__ctfe).

 Is it worth trying to get it into master ?


Wow, absolutely :)

-- 
/Jacob Carlborg



 Aug 29 2016



  parent reply Johannes Pfau <nospam example.com>  writes:


Am Tue, 30 Aug 2016 08:57:25 +0200
schrieb Jacob Carlborg <doob me.com>:


 On 2016-08-29 10:39, Stefan Koch wrote:
 

 Thanks guys.

 I just came up with a nifty little patch that makes it possible to
 ensure that a function is _only_ used at ctfe.
 Or the opposite.

 static assert(__ctfe, "This function is not supposed to be called
 outside of ctfe");
 and static assert(!__ctfe, "This function is not supposed to be
 called during ctfe");

 similarly you can use static if (__ctfe).

 Is it worth trying to get it into master ?  


 
 Wow, absolutely :)
 


There are some nice use cases for this:
* Do not enforce  nogc for CTFE only functions, so you can mark a
  complete module nogc: and CTFE only functions will get ignored
* Do not emit code for CTFE only functions. This is useful for bare
  metal programming: The CTFE only function can depend on druntime
  functions not available in the bare metal runtime.

These are common problems when using string mixins: The functions
to generate the mixed in code use string manipulation which is usually
not  nogc and might require runtime functions. But the code generation
function is exclusively used in CTFE.



 Aug 30 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Tuesday, 30 August 2016 at 08:18:47 UTC, Johannes Pfau wrote:

 There are some nice use cases for this:
 * Do not enforce  nogc for CTFE only functions, so you can mark 
 a
   complete module nogc: and CTFE only functions will get ignored
 * Do not emit code for CTFE only functions. This is useful for 
 bare
   metal programming: The CTFE only function can depend on 
 druntime
   functions not available in the bare metal runtime.

 These are common problems when using string mixins: The 
 functions
 to generate the mixed in code use string manipulation which is 
 usually
 not  nogc and might require runtime functions. But the code 
 generation
 function is exclusively used in CTFE.


I do not see how this could affect  nogc.
But yes you can circumvent the code-generation and pulling in the 
druntime dependency using a static if.



 Aug 30 2016



 next sibling parent  Rory McGuire via Digitalmars-d-announce  writes:


On Tue, Aug 30, 2016 at 10:35 AM, Stefan Koch via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 [snip]
 But yes you can circumvent the code-generation and pulling in the druntime
 dependency using a static if.


That will be awesome!



 Aug 30 2016



prev sibling  parent reply Johannes Pfau <nospam example.com>  writes:


Am Tue, 30 Aug 2016 08:35:20 +0000
schrieb Stefan Koch <uplink.coder googlemail.com>:


 I do not see how this could affect  nogc.


 nogc is only meaningful at runtime. So this could work:

module foo;
 nogc:
[...]

string ctfeOnly(string name)
{
    static assert(!__ctfe);
    // error: cannot use operator ~ in  nogc function
    return "int " ~ name ~ ";";
}
mixin(ctfeOnly("foo"));



 Aug 30 2016



  parent  Johannes Pfau <nospam example.com>  writes:


Am Tue, 30 Aug 2016 12:42:35 +0200
schrieb Johannes Pfau <nospam example.com>:


  nogc is only meaningful at runtime. So this could work:
 
 module foo;
  nogc:
 [...]
 
 string ctfeOnly(string name)
 {
     static assert(!__ctfe);


This should be
    static assert(__ctfe);
of course.



 Aug 30 2016



prev sibling next sibling parent reply deadalnix <deadalnix gmail.com>  writes:


On Monday, 29 August 2016 at 08:39:56 UTC, Stefan Koch wrote:

 On Monday, 29 August 2016 at 08:05:10 UTC, Rory McGuire wrote:

 On Mon, Aug 29, 2016 at 9:51 AM, Dominikus Dittes Scherkl via

 The work you are doing is just awesome!
 Many thanks.


 +1 your work is key for our success as a community.

 R


 Thanks guys.

 I just came up with a nifty little patch that makes it possible 
 to ensure that a function is _only_ used at ctfe.
 Or the opposite.

 static assert(__ctfe, "This function is not supposed to be 
 called outside of ctfe");
 and static assert(!__ctfe, "This function is not supposed to be 
 called during ctfe");

 similarly you can use static if (__ctfe).

 Is it worth trying to get it into master ?


I would say maybe, but let's keep separate things separate. This 
is a language change. I would not include it in the same series 
of patch that change CTFE behavior.



 Aug 30 2016



  parent  Stefan Koch <uplink.coder googlemail.com>  writes:


On Tuesday, 30 August 2016 at 17:29:19 UTC, deadalnix wrote:
worth trying to get it into master ?

 I would say maybe, but let's keep separate things separate. 
 This is a language change. I would not include it in the same 
 series of patch that change CTFE behavior.


Yes. It would be confusing. And it is not done with the patch I 
mentioned.



 Aug 30 2016



prev sibling  parent reply tsbockman <thomas.bockman gmail.com>  writes:


On Monday, 29 August 2016 at 08:39:56 UTC, Stefan Koch wrote:

 I just came up with a nifty little patch that makes it possible 
 to ensure that a function is _only_ used at ctfe.
 Or the opposite.

 static assert(__ctfe, "This function is not supposed to be 
 called outside of ctfe");
 and static assert(!__ctfe, "This function is not supposed to be 
 called during ctfe");

 similarly you can use static if (__ctfe).

 Is it worth trying to get it into master ?


Yes, please. I've often wished I could use `__ctfe` in a `static 
if`.



 Aug 30 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Tuesday, 30 August 2016 at 22:01:45 UTC, tsbockman wrote:

 On Monday, 29 August 2016 at 08:39:56 UTC, Stefan Koch wrote:

 I just came up with a nifty little patch that makes it 
 possible to ensure that a function is _only_ used at ctfe.
 Or the opposite.

 static assert(__ctfe, "This function is not supposed to be 
 called outside of ctfe");
 and static assert(!__ctfe, "This function is not supposed to 
 be called during ctfe");

 similarly you can use static if (__ctfe).

 Is it worth trying to get it into master ?


 Yes, please. I've often wished I could use `__ctfe` in a 
 `static if`.


Sorry. It I overlooked a something rather important. static 
__ctfe is currently not possible and it's rather expensive to 
make it possible.



 Aug 31 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Wed, Aug 31, 2016 at 10:43 PM, Stefan Koch via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On Tuesday, 30 August 2016 at 22:01:45 UTC, tsbockman wrote:


 On Monday, 29 August 2016 at 08:39:56 UTC, Stefan Koch wrote:


 I just came up with a nifty little patch that makes it possible to
 ensure that a function is _only_ used at ctfe.
 Or the opposite.

 static assert(__ctfe, "This function is not supposed to be called
 outside of ctfe");
 and static assert(!__ctfe, "This function is not supposed to be called
 during ctfe");

 similarly you can use static if (__ctfe).

 Is it worth trying to get it into master ?


 Yes, please. I've often wished I could use `__ctfe` in a `static if`.


 Sorry. It I overlooked a something rather important. static __ctfe is
 currently not possible and it's rather expensive to make it possible.



Surely changing the current implementation slightly could still work if we
made a library implementation like:

========
// ? module std.exception.enforce_ctfe;

void main() {
ctfefunc();
}


string ctfefunc() {
static if (assertCTFE!true) {
throw new Exception("asdf");
}
return `import std.stdio; writeln("ctfe generated this");`;
}

template assertCTFE(bool b) {
enum assertCTFE = __traits(compiles, _checkCTFE());
}
void _checkCTFE() {
import std.uuid;
enum id = new UUID("8AB3060E-2cba-4f23-b74c-b52db3bdfb46");
}


the _checkCTFE() function is just a function that does something we're not
allowed to do at CTFE, but current implementation does not respect
__traits(compiles, ....);



As far as I can tell that is a bug. Thoughts?



 Sep 01 2016



 next sibling parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 1 September 2016 at 13:18:18 UTC, Rory McGuire wrote:

 the _checkCTFE() function is just a function that does 
 something we're not
 allowed to do at CTFE, but current implementation does not 
 respect
 __traits(compiles, ....);



 As far as I can tell that is a bug. Thoughts?


It is not a bug, because there is no way to mark something as 
CTFE-only.
static ifs are not visible at the time where ctfe sees the 
function, they have already been resolved.

getting a static if (__ctfe) to work would require significant 
changes to the semantic-analysis path for functions.



 Sep 01 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Thu, Sep 1, 2016 at 6:29 PM, Stefan Koch via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On Thursday, 1 September 2016 at 13:18:18 UTC, Rory McGuire wrote:


 the _checkCTFE() function is just a function that does something we're not
 allowed to do at CTFE, but current implementation does not respect
 __traits(compiles, ....);



 As far as I can tell that is a bug. Thoughts?


 It is not a bug, because there is no way to mark something as CTFE-only.
 static ifs are not visible at the time where ctfe sees the function, they
 have already been resolved.

 getting a static if (__ctfe) to work would require significant changes to
 the semantic-analysis path for functions.



Ah, right, understood.


is that in one of the "semantic" passes the compiler has?



 Sep 01 2016



  parent reply David Nadlinger <code klickverbot.at>  writes:


On Thursday, 1 September 2016 at 16:43:53 UTC, Rory McGuire wrote:

 is that in one of the "semantic" passes the compiler has?


For reference, I've laid out the reasons why this proposal 
couldn't work to Stefan here: 
https://github.com/dlang/dmd/pull/6098#issuecomment-243375543

The real reason is not so much in the pass structure of the 
compiler as in the fact that CTFE by definition executes the same 
function body that is also emitted to the runtime binary.

  — David



 Sep 01 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Thu, Sep 1, 2016 at 9:09 PM, David Nadlinger via Digitalmars-d-announce =
<
digitalmars-d-announce puremagic.com> wrote:


 On Thursday, 1 September 2016 at 16:43:53 UTC, Rory McGuire wrote:


 is that in one of the "semantic" passes the compiler has?


 For reference, I've laid out the reasons why this proposal couldn't work
 to Stefan here: https://github.com/dlang/dmd/p
 ull/6098#issuecomment-243375543

 The real reason is not so much in the pass structure of the compiler as i=


n

 the fact that CTFE by definition executes the same function body that is
 also emitted to the runtime binary.

  =E2=80=94 David


okay thanks, I'll take a look at the link.

Question: if the function runs the same, why can't I catch the exception?

From what Stefan said even if I could catch the exception static if still


wouldn't compile different code.

bool _checkCTFE() {
try {
import std.uuid;
enum id =3D new UUID("8AB3060E-2cba-4f23-b74c-b52db3bdfb46");
return false;
} catch (Throwable t) {
return true;
}
}

The above still fails during CTFE, compiler just exists with:
enforceCTFE.d(20): Error: variable enforceCTFE._checkCTFE.id : Unable to
initialize enum with class or pointer to struct. Use static const variable
instead.


So why can't we even catch the Error during CTFE, that would at least help
somewhat.

At the moment I just have a verbose logging mode with pragma(msg) for my
CTFE stuff.



 Sep 01 2016



 next sibling parent reply David Nadlinger <code klickverbot.at>  writes:


On Thursday, 1 September 2016 at 19:27:17 UTC, Rory McGuire wrote:

 So why can't we even catch the Error during CTFE, that would at 
 least help somewhat.


You are mixing up runtime exceptions ("throw …") with compiler 
errors (missing a semicolon). dm.D.learn should be able to help 
clear that up.

  — David



 Sep 01 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On 01 Sep 2016 21:36, "David Nadlinger via Digitalmars-d-announce" <
digitalmars-d-announce puremagic.com> wrote:

 On Thursday, 1 September 2016 at 19:27:17 UTC, Rory McGuire wrote:

 So why can't we even catch the Error during CTFE, that would at least




help somewhat.

 You are mixing up runtime exceptions ("throw =E2=80=A6") with compiler er=


rors
(missing a semicolon). dm.D.learn should be able to help clear that up.

  =E2=80=94 David


I'm actually asking why we can't catch the ctfe error. It's quite obvious
that it is a temporary limitation of ctfe, there have been many limitations
removed over the years.

Surely the ctfe engine could be changed to catch unsupported code errors.
(Not invalid, just unsupported at CT).

The ctfe engine would have to "go past" the try anyway, right? This is the
part I don't understand. Is it because ctfe actually hasn't started yet and
some other analysis is freaking out?



 Sep 01 2016



 next sibling parent reply David Nadlinger <code klickverbot.at>  writes:


On Thursday, 1 September 2016 at 21:01:46 UTC, Rory McGuire wrote:

 I'm actually asking why we can't catch the ctfe error.


You can, by using __traits(compiles, …).


 Surely the ctfe engine could be changed to catch unsupported 
 code errors. (Not invalid, just unsupported at CT).


It already does, see above. How is this related to try/catch?

  — David



 Sep 01 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Thu, Sep 1, 2016 at 11:05 PM, David Nadlinger via Digitalmars-d-announce
<digitalmars-d-announce puremagic.com> wrote:


 On Thursday, 1 September 2016 at 21:01:46 UTC, Rory McGuire wrote:


 I'm actually asking why we can't catch the ctfe error.


 You can, by using __traits(compiles, =E2=80=A6).

 Surely the ctfe engine could be changed to catch unsupported code errors.

 (Not invalid, just unsupported at CT).


 It already does, see above. How is this related to try/catch?

  =E2=80=94 David


I was hoping that the error was coming from the CTFE engine as it ran the
code, but it comes up before ctfe execution I guess.

__traits(compiles, _some_function_that_calls_), thinks that the invalid
code compiles even when it doesn't compile, e.g.:

template assertCTFE(bool b) {
enum assertCTFE =3D __traits(compiles, _checkCTFE1());
}
void _checkCTFE1() {
static if (__ctfe) { // this still stops compilation (probably because this
is checked before __traits(compiled ...) is evaluated

}
}



 Sep 01 2016



  parent  David Nadlinger <code klickverbot.at>  writes:


On Thursday, 1 September 2016 at 21:48:41 UTC, Rory McGuire wrote:

 I was hoping that the error was coming from the CTFE engine as 
 it ran the code, but it comes up before ctfe execution I guess.


As a general comment, there is no such thing as a CTFE phase. It 
is performed in-line with other semantic analysis as required.


 __traits(compiles, _some_function_that_calls_), thinks that the 
 invalid code compiles even when it doesn't compile […]


No, you are missing the point. This:

---
void foo() {

}
---

will always cause a compiler error, regardless of whether there 
is a `__traits(compiles, foo())` somewhere else in the module. 
__traits(compiles, …) only applies to its argument and doesn't 
magically cause errors in code somewhere else to be ignored.

  — David



 Sep 01 2016



prev sibling  parent reply ag0aep6g <anonymous example.com>  writes:


On 09/01/2016 11:01 PM, Rory McGuire via Digitalmars-d-announce wrote:

 I'm actually asking why we can't catch the ctfe error.


There is no CTFE error in your example. It doesn't compile in the first 
place, even without attempting any CTFE.

[...]

 Surely the ctfe engine could be changed to catch unsupported code
 errors. (Not invalid, just unsupported at CT).


Maybe. It isn't obvious to me that it would be a good idea, but it's not 
obviously terrible either.


 The ctfe engine would have to "go past" the try anyway, right? This is
 the part I don't understand. Is it because ctfe actually hasn't started
 yet and some other analysis is freaking out?


It's just a compiler error. The snippet on its own fails compilation and 
there is no CTFE in there. Something like `static assert(_checkCTFE());` 
would involve CTFE. But that's not needed to trigger the error.



 Sep 01 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Thu, Sep 1, 2016 at 11:26 PM, ag0aep6g via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On 09/01/2016 11:01 PM, Rory McGuire via Digitalmars-d-announce wrote:


 I'm actually asking why we can't catch the ctfe error.


 There is no CTFE error in your example. It doesn't compile in the first
 place, even without attempting any CTFE.

 [...]


 Surely the ctfe engine could be changed to catch unsupported code
 errors. (Not invalid, just unsupported at CT).


 Maybe. It isn't obvious to me that it would be a good idea, but it's not
 obviously terrible either.

 The ctfe engine would have to "go past" the try anyway, right? This is

 the part I don't understand. Is it because ctfe actually hasn't started
 yet and some other analysis is freaking out?


 It's just a compiler error. The snippet on its own fails compilation and
 there is no CTFE in there. Something like `static assert(_checkCTFE());`
 would involve CTFE. But that's not needed to trigger the error.




Yeah, I'm using an enum to force the evaluation during CT, and its dying
before CTFE from what I can tell. Here is another example:

void main() {
enum ret = ctfefunc();
mixin(ret);
}


string ctfefunc() {
static if (!assertCTFE!true) {
throw new Exception("Only during ctfe please...");
}
return `import std.stdio; writeln("ctfe generated this");`;
}

template assertCTFE(bool b) {
enum assertCTFE = __traits(compiles, _checkCTFE1());
}
void _checkCTFE1() {
static if (__ctfe) {

}
}



 Sep 01 2016



  parent reply ag0aep6g <anonymous example.com>  writes:


On 09/01/2016 11:51 PM, Rory McGuire via Digitalmars-d-announce wrote:

 Yeah, I'm using an enum to force the evaluation during CT,


I had missed that detail. Still, the error seems to be independent of 
any CTFE that's going on.

[...]

 Here is another example:

 void main() {
 enum ret = ctfefunc();
 mixin(ret);
 }


 string ctfefunc() {
 static if (!assertCTFE!true) {
 throw new Exception("Only during ctfe please...");
 }
 return `import std.stdio; writeln("ctfe generated this");`;
 }

 template assertCTFE(bool b) {
 enum assertCTFE = __traits(compiles, _checkCTFE1());
 }
 void _checkCTFE1() {
 static if (__ctfe) {

 }
 }


It's not clear to me what this example is supposed to show. In 
particular, how does it relate to catching CTFE errors?

Obviously, `static if (__ctfe)` doesn't compile right now, so 
_checkCTFE1 must be rejected regardless of anything else. And with 
_checkCTFE1, the whole program goes down.

Your code looks like it's supposed to throw an exception when ctfefunc 
is called during normal run-time instead of CTFE. That can be done:

----
string ctfefunc()
{
     if (!__ctfe) throw new Exception("Only during ctfe please...");
     return `import std.stdio; writeln("ctfe generated this");`;
}
----

But I feel like that's missing the point of the example.



 Sep 01 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


FunctionCall support is done. (with a lot of room for improvement)
you can already return strings.
now I just have to finish string-comparison and string-concat 
support to make it usable.

And of course the correct translation of logical-expressions...



 Sep 05 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Mon, Sep 5, 2016 at 1:39 PM, Stefan Koch via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 FunctionCall support is done. (with a lot of room for improvement)
 you can already return strings.
 now I just have to finish string-comparison and string-concat support to
 make it usable.

 And of course the correct translation of logical-expressions...



Great news! Any chance you post the commit link in your announcements?
Would be really interested in being able to do a quick scan of what changed.

For that latest commit I just see the below. Is this all the changes? :)
what does it mean? (you replaced the old ctfe for functions with your new
one?)

% git show 4ba35de160cdb82bb289ad2860cf5163e1636ab5
commit 4ba35de160cdb82bb289ad2860cf5163e1636ab5
Author: Stefan Koch <Uplink.Coder googlemail.com>
Date:   Mon Sep 5 13:35:26 2016 +0200

    hack in ability to do function calls

diff --git a/src/ctfe_bc.d b/src/ctfe_bc.d
index 2e9d0fa..a96d37c 100644
--- a/src/ctfe_bc.d
+++ b/src/ctfe_bc.d
   -143,6 +143,7    else static if (UsePrinterBackend)
 else
 {
     import ddmd.ctfe.bc;
+import ddmd.dinterpret;

     alias BCGenT = BCGen;

   -2252,6 +2253,10    public:
             IGaveUp = true;
             return;
         }
+        import ddmd.dinterpret;
+        ctfeInterpret(ce).accept(this);
+        return ;
+        /*
         auto fn = _sharedCtfeState.getFunctionIndex(ce.f);
         if (!fn)
         {
   -2296,6 +2301,8    public:
                 assert(0, "Could not gen Function: " ~ ce.f.toString);
             IGaveUp = true;
         }
+        */
+
     }

     override void visit(ReturnStatement rs)




BTW: thanks for all the work!



 Sep 05 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Monday, 5 September 2016 at 13:13:03 UTC, Rory McGuire wrote:

 Great news! Any chance you post the commit link in your 
 announcements? Would be really interested in being able to do a 
 quick scan of what changed.


At the moment writing a change-log would be tedious busywork 
since everything changes all the time.

 For that latest commit I just see the below. Is this all the 
 changes? :)
 what does it mean? (you replaced the old ctfe for functions 
 with your new
 one?)


Yes and no, I did not replace the old ctfe I merely placed myself 
in front of it.
Therefore I can leverage the existing mechanism for calls.
And replace a function call by the expression it yields.

For certain kind of function-calls (Those that change visible 
state) This is going to fail or produce incorrect code.

But I am working on it :)



 Sep 05 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


compiling parts of phobos does no longer crash the new engine.
However it still produces incorrect byte-code :)



 Sep 08 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 8 September 2016 at 10:44:55 UTC, Stefan Koch wrote:

 compiling parts of phobos does no longer crash the new engine.
 However it still produces incorrect byte-code :)


I think I have taken care of the incorrect bytecode.
It was not an issue with the engine itself but rather with the 
way I plugged it in.
The new engine is not supposed being called if the old one has 
already started to interpret the function, because the two do not 
and should not share state.



 Sep 08 2016



 next sibling parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Thu, Sep 8, 2016 at 3:11 PM, Stefan Koch via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On Thursday, 8 September 2016 at 10:44:55 UTC, Stefan Koch wrote:


 compiling parts of phobos does no longer crash the new engine.
 However it still produces incorrect byte-code :)


 I think I have taken care of the incorrect bytecode.
 It was not an issue with the engine itself but rather with the way I
 plugged it in.
 The new engine is not supposed being called if the old one has already
 started to interpret the function, because the two do not and should not
 share state.


!! Does this mean I can start testing new ctfe and only some of my CT will
be faster?



 Sep 08 2016



  parent  Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 8 September 2016 at 14:48:25 UTC, Rory McGuire wrote:

 On Thu, Sep 8, 2016 at 3:11 PM, Stefan Koch via 
 Digitalmars-d-announce < digitalmars-d-announce puremagic.com> 
 wrote:


 On Thursday, 8 September 2016 at 10:44:55 UTC, Stefan Koch 
 wrote:


 compiling parts of phobos does no longer crash the new 
 engine. However it still produces incorrect byte-code :)


 I think I have taken care of the incorrect bytecode.
 It was not an issue with the engine itself but rather with the 
 way I
 plugged it in.
 The new engine is not supposed being called if the old one has 
 already
 started to interpret the function, because the two do not and 
 should not
 share state.


 !! Does this mean I can start testing new ctfe and only some of 
 my CT will be faster?


You can start yes.
However it is very limited right now I would expect it to slow 
you down by a little bit.



 Sep 08 2016



prev sibling  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 8 September 2016 at 13:11:23 UTC, Stefan Koch wrote:

 On Thursday, 8 September 2016 at 10:44:55 UTC, Stefan Koch 
 wrote:

 compiling parts of phobos does no longer crash the new engine.
 However it still produces incorrect byte-code :)


 I think I have taken care of the incorrect bytecode.
 It was not an issue with the engine itself but rather with the 
 way I plugged it in.
 The new engine is not supposed being called if the old one has 
 already started to interpret the function, because the two do 
 not and should not share state.


I found more incorrect code.
this time it's located more deeply in the engine.
I am investigating the cause.
It seems to be related closures somehow.



 Sep 08 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 8 September 2016 at 18:54:52 UTC, Stefan Koch wrote:

 On Thursday, 8 September 2016 at 13:11:23 UTC, Stefan Koch 
 wrote:

 On Thursday, 8 September 2016 at 10:44:55 UTC, Stefan Koch 
 wrote:

 compiling parts of phobos does no longer crash the new engine.
 However it still produces incorrect byte-code :)


 I think I have taken care of the incorrect bytecode.
 It was not an issue with the engine itself but rather with the 
 way I plugged it in.
 The new engine is not supposed being called if the old one has 
 already started to interpret the function, because the two do 
 not and should not share state.


 I found more incorrect code.
 this time it's located more deeply in the engine.
 I am investigating the cause.
 It seems to be related closures somehow.


Compiling all of phobos does not crash my engine anymore!
However running the unittests does show incorrect results :(
This is concerning since I am mostly bailing out.

I think this too seems to be related to closures produced in 
templates.
Really nasty stuff.



 Sep 19 2016



 next sibling parent reply Lurker <lurker gmail.com>  writes:


On Monday, 19 September 2016 at 10:07:06 UTC, Stefan Koch wrote:

 Compiling all of phobos does not crash my engine anymore!
 However running the unittests does show incorrect results :(
 This is concerning since I am mostly bailing out.

 I think this too seems to be related to closures produced in 
 templates.
 Really nasty stuff.


Good news anyway! Do you have any preliminary results or goals 
and expectations that you are going to achieve with your rework? 
Is it mostly perf/memory improvements, are there any new features 
that this rework will unlock?

Thanks for your hard work!



 Sep 19 2016



  parent  Stefan Koch <uplink.coder googlemail.com>  writes:


On Monday, 19 September 2016 at 18:05:34 UTC, Lurker wrote:

 Good news anyway! Do you have any preliminary results or goals 
 and expectations that you are going to achieve with your 
 rework? Is it mostly perf/memory improvements, are there any 
 new features that this rework will unlock?

 Thanks for your hard work!


I stated my expectations earlier I think there will be a 2-6x 
performance increase in the non-jited version and around 10-14x 
for the jit.
My preliminary results do support that claim.

As for new features I do not plan on doing language-changes as I 
do not have the ability to anticipate the outcome of doing so.

My work will probably have side-effects on the rest of dmd, for 
example the AST is hard to work with in some cases. I would to 
add like some support for frontend-optimisations as well.

I will also improve documentation in some areas and hope to lower 
the entry-barrier for dmd-development.



 Sep 19 2016



prev sibling  parent reply =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com>  writes:


On Monday, 19 September 2016 at 10:07:06 UTC, Stefan Koch wrote:

 Compiling all of phobos does not crash my engine anymore!


Great work! Keep up still!



 Sep 19 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Tuesday, 20 September 2016 at 05:06:57 UTC, Nordlöw wrote:

 On Monday, 19 September 2016 at 10:07:06 UTC, Stefan Koch wrote:

 Compiling all of phobos does not crash my engine anymore!


 Great work! Keep up still!


I am proud to announce,
(and slightly embarssed because it took to long)

that the following code can now be executed using the new CTFE 
engine :

string fn(bool b)
{
     return b ? "true" : "false";
}
static assert(fn(true) == "true");
static assert(fn(false) == "false");

although this seems trivial it took me about 3 months to get to 
this point.
I believe from here on the road will be less steep.



 Sep 25 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Sun, Sep 25, 2016 at 4:47 PM, Stefan Koch via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On Tuesday, 20 September 2016 at 05:06:57 UTC, Nordl=C3=B6w wrote:


 On Monday, 19 September 2016 at 10:07:06 UTC, Stefan Koch wrote:


 Compiling all of phobos does not crash my engine anymore!


 Great work! Keep up still!


 I am proud to announce,
 (and slightly embarssed because it took to long)

 that the following code can now be executed using the new CTFE engine :

 string fn(bool b)
 {
     return b ? "true" : "false";
 }
 static assert(fn(true) =3D=3D "true");
 static assert(fn(false) =3D=3D "false");

 although this seems trivial it took me about 3 months to get to this poin=


t.

 I believe from here on the road will be less steep.


:D congrats!



 Sep 25 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Sunday, 25 September 2016 at 18:21:27 UTC, Rory McGuire wrote:

 :D congrats!


I appreciate it.
If all goes well there will be a separate nightly release build 
from the newCTFE branch,  sometime in October.

I hope to get alpha bug reports that way.

Also I am now starting experimentation with actual JIT to make 
sure my API and ABI definitions are not bogus.
(For the record an earlier design made slices impossible) Luckily 
I spotted this before I went to far with it.

Unfortunately many basic features are still very brittle or 
completely dysfunctional.
(Such as function calls or structs.)
Again I apologize for the delay.
My adventures in template-land, were quite time consuming.
Although arguably with interesting results.

If you have any questions regarding my work with the DMD, please 
ask away.

Greetings,
Stefan



 Sep 25 2016



 next sibling parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Sun, Sep 25, 2016 at 10:47 PM, Stefan Koch via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On Sunday, 25 September 2016 at 18:21:27 UTC, Rory McGuire wrote:


 :D congrats!


 I appreciate it.
 If all goes well there will be a separate nightly release build from the
 newCTFE branch,  sometime in October.

 I hope to get alpha bug reports that way.

 Also I am now starting experimentation with actual JIT to make sure my API
 and ABI definitions are not bogus.
 (For the record an earlier design made slices impossible) Luckily I
 spotted this before I went to far with it.

 Unfortunately many basic features are still very brittle or completely
 dysfunctional.
 (Such as function calls or structs.)
 Again I apologize for the delay.
 My adventures in template-land, were quite time consuming.
 Although arguably with interesting results.

 If you have any questions regarding my work with the DMD, please ask away.

 Greetings,
 Stefan


No worries, I've been watching this space for over a decade. I really
believe you are working on one of the most important parts of IT for the
next decade. I am planning/making a library that uses CTFE extensively and
feel much more confident about it purely because of your work on getting
CTFE performance to be a non-issue.

R



 Oct 05 2016



  parent reply Uplink_Coder <uplink.coder googlemail.com>  writes:


On Wednesday, 5 October 2016 at 08:34:06 UTC, Rory McGuire wrote:

 No worries, I've been watching this space for over a decade. I 
 really believe you are working on one of the most important 
 parts of IT for the next decade. I am planning/making a library 
 that uses CTFE extensively and feel much more confident about 
 it purely because of your work on getting CTFE performance to 
 be a non-issue.

 R


Little update here:
The LLVM backend is almost on feature parity.
Meaning that that soon the new CTFE engine is a real jit.
In the process I discoverd quite a few horrible bugs and 
inconsistency in the API.
I am quite astonished that it ever ran before :)



 Oct 15 2016



 next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com>  writes:


On 10/16/16 2:27 AM, Uplink_Coder wrote:

 On Wednesday, 5 October 2016 at 08:34:06 UTC, Rory McGuire wrote:

 No worries, I've been watching this space for over a decade. I really
 believe you are working on one of the most important parts of IT for
 the next decade. I am planning/making a library that uses CTFE
 extensively and feel much more confident about it purely because of
 your work on getting CTFE performance to be a non-issue.

 R


 Little update here:
 The LLVM backend is almost on feature parity.
 Meaning that that soon the new CTFE engine is a real jit.
 In the process I discoverd quite a few horrible bugs and inconsistency
 in the API.
 I am quite astonished that it ever ran before :)


That LLVM thing is surely nice to have but I highly doubt it will be 
allowed as dependency for DMD.

---
Dmitry Olshansky



 Oct 16 2016



  parent reply Uplink_Coder <uplink.coder googlemail.com>  writes:


On Sunday, 16 October 2016 at 10:58:57 UTC, Dmitry Olshansky 
wrote:

 That LLVM thing is surely nice to have but I highly doubt it 
 will be allowed as dependency for DMD.

 ---
 Dmitry Olshansky


LLVM is purely optional.
A pure D interpreter exists.

LLVM optimises most ctfe btw and returns constants.



 Oct 16 2016



  parent reply Uplink_Coder <uplink.coder googlemail.com>  writes:


On Sunday, 16 October 2016 at 13:51:55 UTC, Uplink_Coder wrote:

 On Sunday, 16 October 2016 at 10:58:57 UTC, Dmitry Olshansky 
 wrote:

 That LLVM thing is surely nice to have but I highly doubt it 
 will be allowed as dependency for DMD.

 ---
 Dmitry Olshansky


 LLVM is purely optional.
 A pure D interpreter exists.

 LLVM optimises most ctfe btw and returns constants.


If anyone want to take a look the lastest llvm_backend 
development is happening here :
https://github.com/UplinkCoder/dmd/blob/_ctfe/src/bc_llvm_backend.d



 Oct 16 2016



  parent reply Uplink_Coder <uplink.coder googlemail.com>  writes:


On Monday, 17 October 2016 at 00:56:01 UTC, Uplink_Coder wrote:

 If anyone want to take a look the lastest llvm_backend 
 development is happening here :
 https://github.com/UplinkCoder/dmd/blob/_ctfe/src/bc_llvm_backend.d


The time llvm takes to build it's IR and execute the JITed code 
is absolutely too much! multiple milliseconds!

I will write a very simple x86 codegenerator tomorrow.



 Oct 17 2016



  parent  =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com>  writes:


On Monday, 17 October 2016 at 15:50:37 UTC, Uplink_Coder wrote:

 The time llvm takes to build it's IR and execute the JITed code 
 is absolutely too much! multiple milliseconds!

 I will write a very simple x86 codegenerator tomorrow.


Great work.



 Oct 17 2016



prev sibling  parent reply Stefam Koch <uplink.coder googlemail.com>  writes:


On Sunday, 16 October 2016 at 00:27:50 UTC, Uplink_Coder wrote:

 Little update here:
 The LLVM backend is almost on feature parity.
 Meaning that that soon the new CTFE engine is a real jit.
 In the process I discoverd quite a few horrible bugs and 
 inconsistency in the API.
 I am quite astonished that it ever ran before :)


Hey Guys,

I am still dealing with the many bugs that have surfaced.
It is really crazy how something so broken could have worked that 
well.

Originally I planned adding a ton of features, but that can only 
happen If the fundamental issues are fixed.



 Oct 23 2016



  parent reply Stefam Koch <uplink.coder googlemail.com>  writes:


On Monday, 24 October 2016 at 05:57:42 UTC, Stefam Koch wrote:

 On Sunday, 16 October 2016 at 00:27:50 UTC, Uplink_Coder wrote:

 Little update here:
 The LLVM backend is almost on feature parity.
 Meaning that that soon the new CTFE engine is a real jit.
 In the process I discoverd quite a few horrible bugs and 
 inconsistency in the API.
 I am quite astonished that it ever ran before :)


 Hey Guys,

 I am still dealing with the many bugs that have surfaced.
 It is really crazy how something so broken could have worked 
 that well.

 Originally I planned adding a ton of features, but that can 
 only happen If the fundamental issues are fixed.


However at the D Meetup in Berlin, I have gotten some positive 
feedback concerning my ByteCode-Layer.
Once the bugs are fixed and the edges are smoothed over I can see 
finding it's way into other parts of the compiler.



 Oct 23 2016



  parent reply Rory McGuire via Digitalmars-d-announce  writes:


On Mon, Oct 24, 2016 at 8:17 AM, Stefam Koch via Digitalmars-d-announce <
digitalmars-d-announce puremagic.com> wrote:


 On Monday, 24 October 2016 at 05:57:42 UTC, Stefam Koch wrote:


 On Sunday, 16 October 2016 at 00:27:50 UTC, Uplink_Coder wrote:


 Little update here:
 The LLVM backend is almost on feature parity.
 Meaning that that soon the new CTFE engine is a real jit.
 In the process I discoverd quite a few horrible bugs and inconsistency
 in the API.
 I am quite astonished that it ever ran before :)


 Hey Guys,

 I am still dealing with the many bugs that have surfaced.
 It is really crazy how something so broken could have worked that well.




So true of computer programming. Particularly if the documentation for the
API is awol.



 Originally I planned adding a ton of features, but that can only happen
 If the fundamental issues are fixed.


 However at the D Meetup in Berlin, I have gotten some positive feedback
 concerning my ByteCode-Layer.
 Once the bugs are fixed and the edges are smoothed over I can see finding
 it's way into other parts of the compiler.


Cool, thanks for the feedback.



 Oct 23 2016



  parent reply Stefam Koch <uplink.coder googlemail.com>  writes:


On Monday, 24 October 2016 at 06:37:12 UTC, Rory McGuire wrote:

 Cool, thanks for the feedback.



I have take care of the blocker for now.
I turns out my tests contained wrong code that reused a register 
for multiple purposes.
LLVM does not like that.
So it assumed the wrong things while optimising and went into the 
wrong direction makeing complete bogus out of valid code.



 Oct 25 2016



  parent reply Stefam Koch <uplink.coder googlemail.com>  writes:


On Tuesday, 25 October 2016 at 09:36:12 UTC, Stefam Koch wrote:

 On Monday, 24 October 2016 at 06:37:12 UTC, Rory McGuire wrote:

 Cool, thanks for the feedback.



 I have take care of the blocker for now.
 I turns out my tests contained wrong code that reused a 
 register for multiple purposes.
 LLVM does not like that.
 So it assumed the wrong things while optimising and went into 
 the wrong direction makeing complete bogus out of valid code.


First perf data is in

The is measured with time src/dmd -c -ctfe-bc

old interpreter (without -ctfe-bc) :

real	0m6.839s
user	0m6.423s
sys	0m0.407s

new interpreter (-ctfe-bc)

real	0m0.549s
user	0m0.547s
sys	0m0.000s

LLVM Backend (-ctfe-bc -version=UseLLVMBackend) :

real	0m0.039s
user	0m0.027s
sys	0m0.010s


The compiled code was :
int bug6498(int x)
{
     int n = 0;
     while (n < x)
         ++n;
     return n;
}
static assert(bug6498(10_000_000)==10_000_000);



 Oct 25 2016



 next sibling parent reply Wyatt <wyatt.epp gmail.com>  writes:


On Tuesday, 25 October 2016 at 12:36:56 UTC, Stefam Koch wrote:

 LLVM Backend (-ctfe-bc -version=UseLLVMBackend) :

 real	0m0.039s
 user	0m0.027s
 sys	0m0.010s


I think 20,000% is a pretty good speedup! ;)  Great stuff.

Now that JIT works, are you returning to focusing on feature 
coverage?

-Wyatt



 Oct 25 2016



  parent  Stefam Koch <uplink.coder googlemail.com>  writes:


On Tuesday, 25 October 2016 at 15:57:33 UTC, Wyatt wrote:

 On Tuesday, 25 October 2016 at 12:36:56 UTC, Stefam Koch wrote:

 LLVM Backend (-ctfe-bc -version=UseLLVMBackend) :

 real	0m0.039s
 user	0m0.027s
 sys	0m0.010s


 I think 20,000% is a pretty good speedup! ;)  Great stuff.

 Now that JIT works, are you returning to focusing on feature 
 coverage?

 -Wyatt


Yes I have already started on solidifying struct, string and 
slice support.
Once this is done, most of the ctfe system is in place.
The rest will be costmetics.
Make no mistake structs and slices may seem trivial but there are 
a few complxities hiding involved.
Particularly handling of the this pointer and accsess to 
member-variables.
Slices require a proper memory mangament system to be in place.
All of this needs to be platform agnostic and representable in 
bytecode.



 Oct 25 2016



prev sibling  parent reply Jacob Carlborg <doob me.com>  writes:


On 2016-10-25 14:36, Stefam Koch wrote:


 First perf data is in

 The is measured with time src/dmd -c -ctfe-bc

 old interpreter (without -ctfe-bc) :

 real    0m6.839s
 user    0m6.423s
 sys    0m0.407s

 new interpreter (-ctfe-bc)

 real    0m0.549s
 user    0m0.547s
 sys    0m0.000s

 LLVM Backend (-ctfe-bc -version=UseLLVMBackend) :

 real    0m0.039s
 user    0m0.027s
 sys    0m0.010s


 The compiled code was :
 int bug6498(int x)
 {
     int n = 0;
     while (n < x)
         ++n;
     return n;
 }
 static assert(bug6498(10_000_000)==10_000_000);


Very impressive :)

-- 
/Jacob Carlborg



 Oct 25 2016



  parent reply Stefam Koch <uplink.coder googlemail.com>  writes:


On Tuesday, 25 October 2016 at 17:19:26 UTC, Jacob Carlborg wrote:

 Very impressive :)


Thanks.

I just got the following code to compile and execute correctly.

bool strEq(string a, string b)
{
     if (a.length != b.length)
     {
         return false;
     }
     uint length = cast(uint) a.length;

     while(length--)
     {
         auto c1 = a[length];
         auto c2 = b[length];
         if (c1 != c2)
         {
             return false;
         }
     }

     return true;
}

static assert(!strEq("Hello","World"));
static assert(strEq("World","World"));

I used the generated bytecode to make == for strings work.



 Oct 25 2016



  parent reply Andrea Fontana <nospam example.com>  writes:


On Wednesday, 26 October 2016 at 03:58:05 UTC, Stefam Koch wrote:

 On Tuesday, 25 October 2016 at 17:19:26 UTC, Jacob Carlborg 
 wrote:

 Very impressive :)


 Thanks.

 I just got the following code to compile and execute correctly.

 bool strEq(string a, string b)
 {
     if (a.length != b.length)
     {
         return false;
     }
     uint length = cast(uint) a.length;

     while(length--)
     {
         auto c1 = a[length];
         auto c2 = b[length];
         if (c1 != c2)
         {
             return false;
         }
     }

     return true;
 }

 static assert(!strEq("Hello","World"));
 static assert(strEq("World","World"));

 I used the generated bytecode to make == for strings work.


Why did you cast size_t to uint in this example?



 Oct 26 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Wednesday, 26 October 2016 at 08:19:46 UTC, Andrea Fontana 
wrote:

 On Wednesday, 26 October 2016 at 03:58:05 UTC, Stefam Koch 
 wrote:

 On Tuesday, 25 October 2016 at 17:19:26 UTC, Jacob Carlborg 
 wrote:

 Very impressive :)


 Thanks.

 I just got the following code to compile and execute correctly.

 bool strEq(string a, string b)
 {
     if (a.length != b.length)
     {
         return false;
     }
     uint length = cast(uint) a.length;

     while(length--)
     {
         auto c1 = a[length];
         auto c2 = b[length];
         if (c1 != c2)
         {
             return false;
         }
     }

     return true;
 }

 static assert(!strEq("Hello","World"));
 static assert(strEq("World","World"));

 I used the generated bytecode to make == for strings work.


 Why did you cast size_t to uint in this example?


Currently I am limited to 32bit Arithmetic.
Changing this is on the Table but I have not gotten to it yet.



 Oct 26 2016



  parent reply Stefam Koch <uplink.coder googlemail.com>  writes:


Who can guess what this code does ?


char[] obfuscatedFn(string a, string b, uint aLength = 0, uint 
bLength = 0, uint cLength = 0, uint pos = 0, uint bPos = 0, 
char[] result = [])
{
     aLength = cast(uint)a.length;
     bLength = cast(uint)a.length;
     cLength = aLength + bLength;
     result.length = cLength;
     while(aLength--)
     {
         result[pos] = a[pos];
         ++pos;
     }
     while(bLength--)
     {
         result[pos] = b[bPos++];
         ++pos;
     }

     return result;
}



 Oct 26 2016



 next sibling parent reply Kagamin <spam here.lot>  writes:


On Wednesday, 26 October 2016 at 15:02:28 UTC, Stefam Koch wrote:

     bLength = cast(uint)a.length;


Reads past the end of b if b is shorter than a.



 Oct 26 2016



  parent  Stefam Koch <uplink.coder googlemail.com>  writes:


On Wednesday, 26 October 2016 at 15:38:30 UTC, Kagamin wrote:

 On Wednesday, 26 October 2016 at 15:02:28 UTC, Stefam Koch 
 wrote:

     bLength = cast(uint)a.length;


 Reads past the end of b if b is shorter than a.


you are right.
Thanks for spotting it :)



 Oct 26 2016



prev sibling  parent reply MakersF <lfc morz.com>  writes:


On Wednesday, 26 October 2016 at 15:02:28 UTC, Stefam Koch wrote:

 Who can guess what this code does ?


 char[] obfuscatedFn(string a, string b, uint aLength = 0, uint 
 bLength = 0, uint cLength = 0, uint pos = 0, uint bPos = 0, 
 char[] result = [])
 {
     aLength = cast(uint)a.length;
     bLength = cast(uint)a.length;
     cLength = aLength + bLength;
     result.length = cLength;
     while(aLength--)
     {
         result[pos] = a[pos];
         ++pos;
     }
     while(bLength--)
     {
         result[pos] = b[bPos++];
         ++pos;
     }

     return result;
 }


Shouldn't

 cLength = aLength + bLength;


actually be

 cLength = aLength + bLength + pos;


?
Otherwise when pos is > 0 it writes past the end of result.

Same problem in the first and second loop, where the legths 
should probably be


     aLength = cast(uint)a.length - pos;
     bLength = cast(uint)a.length - bPos;


If you fix the a and b Length then cLength is fine.

Or am I missing something?



 Oct 26 2016



  parent reply Stefam Koch <uplink.coder googlemail.com>  writes:


On Wednesday, 26 October 2016 at 16:24:49 UTC, MakersF wrote:

 On Wednesday, 26 October 2016 at 15:02:28 UTC, Stefam Koch 
 wrote:

 Who can guess what this code does ?


 char[] obfuscatedFn(string a, string b, uint aLength = 0, uint 
 bLength = 0, uint cLength = 0, uint pos = 0, uint bPos = 0, 
 char[] result = [])
 {
     aLength = cast(uint)a.length;
     bLength = cast(uint)a.length;
     cLength = aLength + bLength;
     result.length = cLength;
     while(aLength--)
     {
         result[pos] = a[pos];
         ++pos;
     }
     while(bLength--)
     {
         result[pos] = b[bPos++];
         ++pos;
     }

     return result;
 }


 Shouldn't

 cLength = aLength + bLength;


 actually be

 cLength = aLength + bLength + pos;


 ?
 Otherwise when pos is > 0 it writes past the end of result.

 Same problem in the first and second loop, where the legths 
 should probably be


     aLength = cast(uint)a.length - pos;
     bLength = cast(uint)a.length - bPos;


 If you fix the a and b Length then cLength is fine.

 Or am I missing something?


Ah the default paramters are never touched.
The reason they are there is because of the way the ctfe engine 
lowers parameters.
It's easier to create byte-code macros if all used variables are 
function paramters :)



 Oct 26 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


Another update on CTFE.

I have found a few errors in my handling of switch-statments.
An efficient solution for this is still pending,

Futhermore I have begun to work on ctfe handling refernces.
These are a little bit harder to do in bytecode and do pessimise 
performance if overused.

I hope to make another leap at the end of this month.

We should have string Concat-support fairly soon.

Cheers,
stefan



 Oct 28 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Friday, 28 October 2016 at 16:52:46 UTC, Stefan Koch wrote:

 Another update on CTFE.

 I have found a few errors in my handling of switch-statments.
 An efficient solution for this is still pending,

 Futhermore I have begun to work on ctfe handling refernces.
 These are a little bit harder to do in bytecode and do 
 pessimise performance if overused.

 I hope to make another leap at the end of this month.

 We should have string Concat-support fairly soon.

 Cheers,
 stefan


I just made progress on another fundamental feature,
function call support.

It's does not work yet, but it shows promise.



 Oct 29 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Sunday, 30 October 2016 at 03:07:13 UTC, Stefan Koch wrote:

 I just made progress on another fundamental feature,
 function call support.

 It's does not [fully] work yet, but it shows promise.


The following just compiled :
int fn(int a)
{
     return a + fn2(2);
}


int fn2(int a)
{
     return a + 2;
}

static assert(fn2(4) == 6);
static assert(fn(4) == 8);
static assert(fn(fn(2)) == 10);



 Oct 30 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Sunday, 30 October 2016 at 21:09:19 UTC, Stefan Koch wrote:

 On Sunday, 30 October 2016 at 03:07:13 UTC, Stefan Koch wrote:

 I just made progress on another fundamental feature,
 function call support.

 It's does not [fully] work yet, but it shows promise.


 The following just compiled :
 int fn(int a)
 {
     return a + fn2(2);
 }


 int fn2(int a)
 {
     return a + 2;
 }

 static assert(fn2(4) == 6);
 static assert(fn(4) == 8);
 static assert(fn(fn(2)) == 10);


Oh shoot!
I did not enable the new call system :(
This computed by the old interpreter.

The new engine fails :(



 Oct 30 2016



  parent  Dicebot <public dicebot.lv>  writes:


 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,a,n,n,o,u,n,c,e
Subject: Re: Battle-plan for CTFE
References: <rnzxxfvmeeytnvkmwcxj forum.dlang.org>
 <nqa6e7$1fd7$1 digitalmars.com>
 <mailman.104.1472766720.2965.digitalmars-d-announce puremagic.com>
 <nqaa8a$1kcd$1 digitalmars.com> <qzdiqnxfdaytulaszppy forum.dlang.org>
 <mailman.210.1473081196.2965.digitalmars-d-announce puremagic.com>
 <wjwmowqkdcttntkosept forum.dlang.org> <zolwvouqhqnmmoqpxofo forum.dlang.org>
 <hmvfsrdobpeuqpwiufdx forum.dlang.org> <lcxgcoxtuvbujbqzqias forum.dlang.org>
 <yofxponrmwlanyghkjrq forum.dlang.org> <jttaymputcolwmpbgzcv forum.dlang.org>
 <dfgwdgajygyvjrtzhqcj forum.dlang.org>
 <mailman.406.1474827700.2994.digitalmars-d-announce puremagic.com>
 <acmqjjnecwryyzpklgjx forum.dlang.org>
 <mailman.902.1475656459.2994.digitalmars-d-announce puremagic.com>
 <uprtsdxodwsbdpoxxkpe forum.dlang.org> <qywmrwidasgvvnqpvfgf forum.dlang.org>
 <jmtczpvmxgxuaofxphyv forum.dlang.org>
 <mailman.162.1477291045.3398.digitalmars-d-announce puremagic.com>
 <hprtrdyiowkxconetmyp forum.dlang.org> <rwvuaudnlfwhkeavwbvu forum.dlang.org>
 <nuo46u$25rm$1 digitalmars.com> <vcjwcfmaqrievijciouq forum.dlang.org>
 <gdzaxcoiwzdaxzcrjycl forum.dlang.org> <zgxwctswlhmdbzibwjfr forum.dlang.org>
 <okfrlchbefeqjvesscrc forum.dlang.org> <kkfteaolhhihmkrmozcm forum.dlang.org>
 <eneqlnfnhcpubcbyxrdb forum.dlang.org> <caqoqagchdwiienjrwve forum.dlang.org>
 <ejtexhsgtzcqlijnvxuw forum.dlang.org> <citszaxzagriezhtjgrd forum.dlang.org>
 <sasytwhgedikcnzigvvi forum.dlang.org>
In-Reply-To: <sasytwhgedikcnzigvvi forum.dlang.org>

--omJ5HxNF3lN2qBjXTBoLAxqVL4WIgcBTr
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/30/2016 11:19 PM, Stefan Koch wrote:

 Oh shoot!
 I did not enable the new call system :(
 This computed by the old interpreter.
=20
 The new engine fails :(


Stefan, would you mind creating a dedicated topic in main D newsgroup to
report your development progress? Bumping the thread in announce NG like
that is rather distracting, we should aim for posting only most
important/relevant info there.


--omJ5HxNF3lN2qBjXTBoLAxqVL4WIgcBTr--



 Oct 31 2016



prev sibling  parent reply =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com>  writes:


On Sunday, 25 September 2016 at 20:47:41 UTC, Stefan Koch wrote:

 If all goes well there will be a separate nightly release build 
 from the newCTFE branch,  sometime in October.

 I hope to get alpha bug reports that way.


Have you benchmarked CTFE-heavy projects like Pegged?



 Oct 19 2016



  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Wednesday, 19 October 2016 at 07:11:24 UTC, Nordlöw wrote:

 On Sunday, 25 September 2016 at 20:47:41 UTC, Stefan Koch wrote:

 If all goes well there will be a separate nightly release 
 build from the newCTFE branch,  sometime in October.

 I hope to get alpha bug reports that way.


 Have you benchmarked CTFE-heavy projects like Pegged?


It is not yet able to handle pegged.
And I suspect alot of slowness is caused by templates not by CTFE.



 Oct 19 2016



  parent  Rory McGuire via Digitalmars-d-announce  writes:


On 19 Oct 2016 17:46, "Stefan Koch via Digitalmars-d-announce" <
digitalmars-d-announce puremagic.com> wrote:

 On Wednesday, 19 October 2016 at 07:11:24 UTC, Nordl=C3=B6w wrote:

 On Sunday, 25 September 2016 at 20:47:41 UTC, Stefan Koch wrote:

 If all goes well there will be a separate nightly release build from






the newCTFE branch,  sometime in October.

 I hope to get alpha bug reports that way.



 Have you benchmarked CTFE-heavy projects like Pegged?



 It is not yet able to handle pegged.
 And I suspect alot of slowness is caused by templates not by CTFE.


diet-ng is probably a better "complex" benchmark.



 Oct 19 2016



prev sibling  parent reply Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 1 September 2016 at 19:27:17 UTC, Rory McGuire wrote:


 At the moment I just have a verbose logging mode with 
 pragma(msg) for my
 CTFE stuff.


I have something that will help with that a little bit.
https://github.com/UplinkCoder/dmd/tree/__ctfeWriteln
when you apply this patch __ctfeWriteln() will output every 
compiletime avilable string to the console.
It's a little bit nicer then using pragma(msg);



 Sep 01 2016



  parent reply David Nadlinger <code klickverbot.at>  writes:


On Thursday, 1 September 2016 at 19:38:13 UTC, Stefan Koch wrote:

 I have something that will help with that a little bit.
 https://github.com/UplinkCoder/dmd/tree/__ctfeWriteln
 when you apply this patch __ctfeWriteln() will output every 
 compiletime avilable string to the console.


More crucially, it also allows you to print runtime values during 
CTFE. pragma(msg, …) suffices to print compile-time constants. 
Very few people seem to have the correct mental model for the the 
interaction between compile-time features and CTFE, hence I think 
using precise terminology is an important first step.

See also: https://github.com/dlang/dmd/pull/692 – it's about time 
we finally got __ctfeWrite() merged.

  — David



 Sep 01 2016



 next sibling parent  Stefan Koch <uplink.coder googlemail.com>  writes:


On Thursday, 1 September 2016 at 20:43:16 UTC, David Nadlinger 
wrote:

 See also: https://github.com/dlang/dmd/pull/692 – it's about 
 time we finally got __ctfeWrite() merged.

  — David


Oh yeah.
Let me get this into PR shape.



 Sep 01 2016



prev sibling  parent  "H. S. Teoh via Digitalmars-d-announce"  writes:


On Thu, Sep 01, 2016 at 08:43:16PM +0000, David Nadlinger via
Digitalmars-d-announce wrote:

 On Thursday, 1 September 2016 at 19:38:13 UTC, Stefan Koch wrote:

 I have something that will help with that a little bit.
 https://github.com/UplinkCoder/dmd/tree/__ctfeWriteln when you apply
 this patch __ctfeWriteln() will output every compiletime avilable
 string to the console.


 
 More crucially, it also allows you to print runtime values during
 CTFE.  pragma(msg, …) suffices to print compile-time constants. Very
 few people seem to have the correct mental model for the the
 interaction between compile-time features and CTFE, hence I think
 using precise terminology is an important first step.


[...]

Part of the problem is that the term "compile-time" is ambiguous. In the
compilation of a D program, there are (at least) 2 distinct phases that
may be termed "compile-time": (1) expansion of templates and evaluation
of static-if, and (2) CTFE.

Phase (1) conceptually happens before the AST is finalized, and hence at
this stage it makes no sense to refer to variables and such: variables
don't even exist yet because the syntax tree of the program is still
being manipulated.

Phase (2) happens just as the compiler is ready to emit object code: the
AST has been finalized, symbols have been resolved, statements have been
analysed / lowered, etc.. At this stage, static-if no longer makes any
sense because the AST can no longer be manipulated at this point.

Of course, the above is a simplified picture of what actually happens.
In a complex D program, you *can* have phase (2) precede phase (1),
e.g., using static-if on the result of a CTFE function. The important
thing to note here, though, is that the CTFE function's body must
already be in phase (2), because otherwise CTFE is impossible. So
another part of the code can depend on the result of the CTFE function,
but the function itself is already past phase (1) and you can't change
its AST anymore.  So you can have different parts of the program be in
different phases, and they can have interdependencies, but the same
piece of code can only progress from phase (1) to phase (2), and never
the other way around.

Referring to both phases as "compile-time" is ambiguous and causes
confusion to people who are not familiar with how the compilation
process actually works.


T

-- 
Windows: the ultimate triumph of marketing over technology. -- Adrian von Bidder



 Sep 01 2016



prev sibling  parent  David Nadlinger <code klickverbot.at>  writes:


On Thursday, 1 September 2016 at 13:18:18 UTC, Rory McGuire wrote:

 void main() {
 ctfefunc();
 }


 string ctfefunc() {
 static if (assertCTFE!true) {
 throw new Exception("asdf");
 }
 return `import std.stdio; writeln("ctfe generated this");`;
 }

 template assertCTFE(bool b) {
 enum assertCTFE = __traits(compiles, _checkCTFE());
 }
 void _checkCTFE() {
 import std.uuid;
 enum id = new UUID("8AB3060E-2cba-4f23-b74c-b52db3bdfb46");
 }

 […] current implementation does not respect __traits(compiles, 
 ....);

 As far as I can tell that is a bug. Thoughts?


This doesn't do what you think it does in more than one way. If 
you laid out how you think it should work, I'd be happy to walk 
you through where it goes wrong in particular. However, as it is, 
the whole program would certainly fail to compile if 
__traits(compiles, …) were false (which it indeed is) because the 
module-level function _checkCTFE() couldn't then be compiled 
either. It's just a top-level function. There is no 
__traits(compiles, ....) involved in that.

  — David



 Sep 01 2016



prev sibling  parent  =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com>  writes:


On Monday, 29 August 2016 at 00:24:01 UTC, Stefan Koch wrote:
I feel that this can have a positive

 I am happy for all comments or suggestions.


Incredible work! Keep up!



 Aug 29 2016



prev sibling  parent  Martin Nowak <code dawg.eu>  writes:


On Saturday, 6 August 2016 at 14:26:00 UTC, Stefan Koch wrote:

 I added a switch to my version of dmd which allows to toggle 
 the ctfe engine.
 So now I can compare apples to apples when posting perf data.


That's indeed very useful, also for testing purposes.



 Aug 07 2016



prev sibling  parent reply Edwin van Leeuwen <edder tkwsping.nl>  writes:


On Friday, 29 July 2016 at 11:30:20 UTC, Stefan Koch wrote:

 I have fresh performance statistics:


Is there any improvement in memory usage?



 Jul 29 2016



  parent  Stefan Koch <uplink.coder googlemail.com>  writes:


On Friday, 29 July 2016 at 13:07:12 UTC, Edwin van Leeuwen wrote:

 On Friday, 29 July 2016 at 11:30:20 UTC, Stefan Koch wrote:

 I have fresh performance statistics:


 Is there any improvement in memory usage?


Yes!
There memory usage is the same as run-time execution.
plus about 16k for the interpreter state.



 Jul 29 2016