www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - CTFE slower than expected

reply Manu <turkeyman gmail.com> writes:
I've been trying to work out why my compile times have gone to hell
recently.

I have a lib, it takes 3.5 seconds to compile.
I add one CTFE heavy module, it's not huge, certainly much smaller than the
rest of the app, and it blows out to 18 seconds. I've done some experiments
removing bits and pieces of code, I can isolate the bits that add seconds
to the compile time, but the big offenders are one-line mixins which use
CTFE fairly aggressively to generate the strings they mix in.

Can anyone comment on CTFE as implemented? Why is it so slow? It's
certainly not executing a lot of code. I can imagine executing the same
routine in an interpreted language like lua would take milliseconds or
less, not multiple seconds.
What are the bottlenecks? Is there any way to improve it?
May 29 2012
next sibling parent reply Michel Fortin <michel.fortin michelf.com> writes:
On 2012-05-29 10:25:54 +0000, Manu <turkeyman gmail.com> said:


 What are the bottlenecks? Is there any way to improve it?


The answer to those questions is usually found by profiling. Asking 
people for what they think is slow is almost certain to give you wrong 
answers.

-- 
Michel Fortin
michel.fortin michelf.com
http://michelf.com/
May 29 2012
parent Manu <turkeyman gmail.com> writes:
On 29 May 2012 14:28, Michel Fortin <michel.fortin michelf.com> wrote:


 On 2012-05-29 10:25:54 +0000, Manu <turkeyman gmail.com> said:

  What are the bottlenecks? Is there any way to improve it?


 The answer to those questions is usually found by profiling. Asking people
 for what they think is slow is almost certain to give you wrong answers.



I'm not in a hurry. I'm mainly asking out of curiosity, and wondering if
others are thinking the same thing, or if there are motions to improve it.



 May 29 2012



prev sibling next sibling parent reply Jacob Carlborg <doob me.com>  writes:


On 2012-05-29 12:25, Manu wrote:

 I've been trying to work out why my compile times have gone to hell
 recently.

 I have a lib, it takes 3.5 seconds to compile.
 I add one CTFE heavy module, it's not huge, certainly much smaller than
 the rest of the app, and it blows out to 18 seconds. I've done some
 experiments removing bits and pieces of code, I can isolate the bits
 that add seconds to the compile time, but the big offenders are one-line
 mixins which use CTFE fairly aggressively to generate the strings they
 mix in.

 Can anyone comment on CTFE as implemented? Why is it so slow? It's
 certainly not executing a lot of code. I can imagine executing the same
 routine in an interpreted language like lua would take milliseconds or
 less, not multiple seconds.
 What are the bottlenecks? Is there any way to improve it?


Many small string mixins are slow, even if they're string literals and 
not generated. If possible, it's better with one huge string mixin.

-- 
/Jacob Carlborg



 May 29 2012



  parent reply Manu <turkeyman gmail.com>  writes:


On 29 May 2012 15:10, Jacob Carlborg <doob me.com> wrote:


 On 2012-05-29 12:25, Manu wrote:


 I've been trying to work out why my compile times have gone to hell
 recently.

 I have a lib, it takes 3.5 seconds to compile.
 I add one CTFE heavy module, it's not huge, certainly much smaller than
 the rest of the app, and it blows out to 18 seconds. I've done some
 experiments removing bits and pieces of code, I can isolate the bits
 that add seconds to the compile time, but the big offenders are one-line
 mixins which use CTFE fairly aggressively to generate the strings they
 mix in.

 Can anyone comment on CTFE as implemented? Why is it so slow? It's
 certainly not executing a lot of code. I can imagine executing the same
 routine in an interpreted language like lua would take milliseconds or
 less, not multiple seconds.
 What are the bottlenecks? Is there any way to improve it?


 Many small string mixins are slow, even if they're string literals and not
 generated. If possible, it's better with one huge string mixin.



That's interesting. I can probably give that a shot.
So you think that's a bigger cost than the CTFE code that generates the
strings?



 May 29 2012



  parent  Jacob Carlborg <doob me.com>  writes:


On 2012-05-29 14:37, Manu wrote:


 That's interesting. I can probably give that a shot.
 So you think that's a bigger cost than the CTFE code that generates the
 strings?


I don't know. I just did a test with Derelict that needed to be 
compatible with D1 and D2 and therefore used string mixins for things 
like __gshared.

For example:

http://www.dsource.org/projects/derelict/browser/branches/Derelict2/DerelictGL/derelict/opengl/glfuncs.d#L699

Putting all those declarations in their own string mixins make a difference.

-- 
/Jacob Carlborg



 May 29 2012



prev sibling  parent reply Don Clugston <dac nospam.com>  writes:


On 29/05/12 12:25, Manu wrote:

 I've been trying to work out why my compile times have gone to hell
 recently.

 I have a lib, it takes 3.5 seconds to compile.
 I add one CTFE heavy module, it's not huge, certainly much smaller than
 the rest of the app, and it blows out to 18 seconds. I've done some
 experiments removing bits and pieces of code, I can isolate the bits
 that add seconds to the compile time, but the big offenders are one-line
 mixins which use CTFE fairly aggressively to generate the strings they
 mix in.



 Can anyone comment on CTFE as implemented? Why is it so slow?


You really don't want to know. What it's actually doing is horrific. Bug 
6498.

The reason why it's still like that is that CTFE bugs have kept cropping 
up (mostly related to pointers and especially AAs), which have prevented 
me from doing anything on the performance issue.


 It's
 certainly not executing a lot of code. I can imagine executing the same
 routine in an interpreted language like lua would take milliseconds or
 less, not multiple seconds.
 What are the bottlenecks?


It's was originally based on the const-folding code used by the 
optimizer. So most of the code was written with totally goals (that 
didn't include performance).


 Is there any way to improve it?


Oh yeah. Orders of magnitude, easily. The slowness is not in any way 
inherent to CTFE. The experience will be completely different, once I 
have some time to work on it -- I know exactly how to do it.



 May 29 2012



 next sibling parent reply Manu <turkeyman gmail.com>  writes:


On 29 May 2012 15:52, Don Clugston <dac nospam.com> wrote:


 On 29/05/12 12:25, Manu wrote:


 Is there any way to improve it?



 Oh yeah. Orders of magnitude, easily. The slowness is not in any way
 inherent to CTFE. The experience will be completely different, once I have
 some time to work on it -- I know exactly how to do it.


Alright, well I've got a case of beer with your name on it if you can pull
it off! ;)



 May 29 2012



  parent reply Walter Bright <newshound2 digitalmars.com>  writes:


On 5/29/2012 8:06 AM, Manu wrote:

 On 29 May 2012 15:52, Don Clugston <dac nospam.com <mailto:dac nospam.com>>
wrote:

     On 29/05/12 12:25, Manu wrote:

         Is there any way to improve it?


     Oh yeah. Orders of magnitude, easily. The slowness is not in any way
     inherent to CTFE. The experience will be completely different, once I have
     some time to work on it -- I know exactly how to do it.


 Alright, well I've got a case of beer with your name on it if you can pull it
 off! ;)


tl,dr: CTFE started out as a glorified constant folder, not an interpreter. An 
interpreter needs a different design.



 May 29 2012



 next sibling parent  "bearophile" <bearophileHUGS lycos.com>  writes:


Walter Bright:


 tl,dr: CTFE started out as a glorified constant folder, not an 
 interpreter. An interpreter needs a different design.


And I presume a basic JITting interpreter for CTFE (like one in 
LDC2 using existing LLVM JIT tools) needs yet another design.

Bye,
bearophile



 May 29 2012



prev sibling  parent  deadalnix <deadalnix gmail.com>  writes:


Le 29/05/2012 23:53, Walter Bright a écrit :

 On 5/29/2012 8:06 AM, Manu wrote:

 On 29 May 2012 15:52, Don Clugston <dac nospam.com
 <mailto:dac nospam.com>> wrote:

 On 29/05/12 12:25, Manu wrote:

 Is there any way to improve it?


 Oh yeah. Orders of magnitude, easily. The slowness is not in any way
 inherent to CTFE. The experience will be completely different, once I
 have
 some time to work on it -- I know exactly how to do it.


 Alright, well I've got a case of beer with your name on it if you can
 pull it
 off! ;)


 tl,dr: CTFE started out as a glorified constant folder, not an
 interpreter. An interpreter needs a different design.


Can you elaborate on that ?

I would be very interested to have your experience with that, which 
problems you faced, what you'd have done differently if you would have 
known the future.



 May 30 2012



prev sibling next sibling parent  d coder <dlang.coder gmail.com>  writes:


 Alright, well I've got a case of beer with your name on it if you can pull
 it off! ;)


+1. I too am waiting for CTFE improvements. I am working on a DSL and with
the present limitations, it is impractically slow and memory consuming
while compiling.



 May 29 2012



prev sibling next sibling parent reply Philippe Sigaud <philippe.sigaud gmail.com>  writes:


On Tue, May 29, 2012 at 2:52 PM, Don Clugston <dac nospam.com> wrote:


 Is there any way to improve it?



 Oh yeah. Orders of magnitude, easily.


!


The slowness is not in any way
 inherent to CTFE. The experience will be completely different, once I have
 some time to work on it -- I know exactly how to do it.


Did 2.058 or 2.059 see any new code for CTFE? Like the OP, I've the
impression CTFE/mixins suddenly became far slower. I'm not
complaining, I understand it's a difficult part of DMD, but I wondered
if what I see is real or imaginary.



 May 29 2012



  parent  Don Clugston <dac nospam.com>  writes:


On 29/05/12 23:23, Philippe Sigaud wrote:

 On Tue, May 29, 2012 at 2:52 PM, Don Clugston<dac nospam.com>  wrote:


 Is there any way to improve it?



 Oh yeah. Orders of magnitude, easily.


 !


 The slowness is not in any way
 inherent to CTFE. The experience will be completely different, once I have
 some time to work on it -- I know exactly how to do it.


 Did 2.058 or 2.059 see any new code for CTFE? Like the OP, I've the
 impression CTFE/mixins suddenly became far slower. I'm not
 complaining, I understand it's a difficult part of DMD, but I wondered
 if what I see is real or imaginary.


The behaviour of __traits(allMembers) changed (it now returns an array 
of string literals) and I expect that to be a little bit slower.
Compiling Phobos is now *much* slower than it used to be, due to changes 
in Phobos. (eg, import std.random; is unbelievably slow).


As for CTFE, certain cases became faster (eg, repeated use of global 
array literals).

But if you can pinpoint a case where CTFE itself became slower, I'd like 
to know.



 May 30 2012



prev sibling  parent  Andrej Mitrovic <andrej.mitrovich gmail.com>  writes:


On 5/29/12, Don Clugston <dac nospam.com> wrote:

 Oh yeah. Orders of magnitude, easily.


Does this also imply speedups for static foreach loops? I've got quite
a few of those and they seem to slow down compilation a bit. I've had
a funny error message once saying there's an error on line ~50_000 in
a 1000-line module, so I guess static foreach can expand to quite a
bit of code, maybe I should be more careful with those..



 May 29 2012