www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Build time

reply JG <JG somewhere.com> writes:
Hi,

The program I writing is around 3000 loc and recently I noticed a 
large slow down in compile time which after investigation seemed 
to be caused by my computer running out of memory. The compile 
was using more than 15GB memory. I tried using lowmem and that 
did solve the memory problem but the compile still takes around 1 
minute. Any suggestion on how to try and improve the build time. 
I am currently using dub.

Of course one could try to use fewer templates and less meta 
programming but that seems to defeat the purpose of using d.
Jul 23 2021
next sibling parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote:

 The program I writing is around 3000 loc


what's the code?
Jul 23 2021
parent reply JG <JG somewhere.com> writes:
On Friday, 23 July 2021 at 18:57:46 UTC, Adam D Ruppe wrote:

 On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote:

 The program I writing is around 3000 loc


 what's the code?


I am not sure how relevant it is but it is a compiler that I have 
been writing, not something serious (yet - if ever).
Jul 23 2021
parent Adam D Ruppe <destructionator gmail.com> writes:
On Friday, 23 July 2021 at 19:09:02 UTC, JG wrote:

 I am not sure how relevant it is but it is a compiler that I 
 have been writing, not something serious (yet - if ever).


Compile time optimizations are a bit weird compared to runtime 
ones - things that would fast at run time may actually be very 
slow at compile time, so eyeballing the code might help me point 
out something to consider.
Jul 23 2021
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jul 23, 2021 at 06:53:06PM +0000, JG via Digitalmars-d-learn wrote:
[...]

 The program I writing is around 3000 loc and recently I noticed a
 large slow down in compile time which after investigation seemed to be
 caused by my computer running out of memory. The compile was using
 more than 15GB memory.  I tried using lowmem and that did solve the
 memory problem but the compile still takes around 1 minute. Any
 suggestion on how to try and improve the build time. I am currently
 using dub.


3000 loc and 1 minute build time?  Sounds like you're using too many
nested templates / CTFE.



 Of course one could try to use fewer templates and less meta
 programming but that seems to defeat the purpose of using d.


I wouldn't say use fewer templates / less meta-programming.  But I'd say
look into how deeply nested templates are, and whether some templates
parameters may be unnecessary.  If you have recursive templates,
consider refactoring it so that it uses linear expansion instead.
Shallowly-nested templates generally don't run into performance
problems.

(Confession: I wrote the variadic version of cartesianProduct in
std.algorithm with recursive templates. It uses an exponential number of
template expansions, and so quickly brings the compiler to its knees
when you try to take 4 or more cartesian products in a row.  Eventually,
I refactored the most common case (no infinite ranges among its
arguments) to use a linear expansion with a nested loop instead.
Compile times improved by a HUGE margin.)

And avoid doing too much work in CTFE, which is known to be slow. But
not as slow as overly-deeply nested templates.

Another way is to have a separate build step for expanding the most
heavy templates, so that you only incur that heavy expansion once in a
while when you change the relevant code.  I had a Vibe.d project where
Diet templates slowed me down too much (they are super template-heavy),
so I split it into several different build targets with a separate link
step, so that when I'm not changing the Diet templates it doesn't slow
me down so much.  Dub unfortunately won't help you here (unless you use
subpackages -- but I doubt it will win much) -- I recommend using a
better build system like CMake or SCons. Dub's architecture simply does
not play well with staged compilation.

Alternatively, use a separate pre-compilation stage for generating code
(e.g., write a utility that emits D code that then gets compiled in a
subsequent step).  As much as I love D's compile-time capabilities,
there comes a time when it's simply more practical to just `writefln`
some D code snippets into a file and compile that into the main program,
instead of trying to tame the memory-guzzling beast that is the D
compiler.


T

-- 
I am Ohm of Borg. Resistance is voltage over current.
Jul 23 2021
parent Adam D Ruppe <destructionator gmail.com> writes:
On Friday, 23 July 2021 at 19:32:08 UTC, H. S. Teoh wrote:

 And avoid doing too much work in CTFE, which is known to be 
 slow.


Well it very much depends HOW you do it. Like the ~= operation in 
ctfe is awfully slow and wastes a lot of memory depending on the 
size of the string, but if you preallocate and copy memory in 
chunks it isn't too bad at all.

And if you use format!"str"() in ctfe that alone can be a real 
speed killer.

That's why I wanna see the code, it is possible there's a fairly 
simple bottleneck to look at.
Jul 23 2021
prev sibling next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote:

 Any suggestion on how to try and improve the build time. I am 
 currently using dub.


You can try profiling it with LDC 1.25 or later. Add this to 
dub.sdl:

```
dflags "--ftime-trace" platform="ldc"
dflags "--ftime-trace-file=./my-trace.json" platform="ldc"
postBuildCommands "import-chrome ./my-trace.json 
./my-trace.tracy" platform="ldc"
```

You can [get import-chrome and tracy 
here](https://github.com/wolfpld/tracy). There's binaries for 
Windows, and on Ubuntu/Debian Linux you can install as follows:

```

only one I didn't have installed yet)
sudo apt install libcapstone-dev


git clone https://github.com/wolfpld/tracy


cd tracy/import-chrome/build/unix/
make

anywhere
sudo cp import-chrome-release /usr/local/bin/import-chrome


cd ../../../
cd profiler/build/unix/
make
sudo cp Tracy-release /usr/local/bin/tracy
```

Then do:
```
dub build --compiler=ldc2
tracy my-trace.tracy
```

And you can inspect what functions the compiler spends the most 
time on compiling. Sometimes there's one dumb thing taking a lot 
of time, so you can improve build times by working around that. 
E.g. I once replaced `std.conv: text` with `snprintf` to reduce 
my build time from 2.1 to 1.6 seconds.
Jul 23 2021
parent reply JG <JG somewhere.com> writes:
On Friday, 23 July 2021 at 20:03:22 UTC, Dennis wrote:

 On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote:

 [...]


 You can try profiling it with LDC 1.25 or later. Add this to 
 dub.sdl:

 [...]


Thanks for this suggestion. Unfortunately this makes the compile 
use too much memory for my system and so it gets killed before 
the end and no my-trace.tracy file is produced. I will try 
building on parts of the program with this and see if I can see 
what is going on.
Jul 24 2021
parent reply JG <JG somewhere.com> writes:
On Saturday, 24 July 2021 at 08:26:39 UTC, JG wrote:

 On Friday, 23 July 2021 at 20:03:22 UTC, Dennis wrote:

 On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote:

 [...]


 You can try profiling it with LDC 1.25 or later. Add this to 
 dub.sdl:

 [...]


 Thanks for this suggestion. Unfortunately this makes the 
 compile use too much memory for my system and so it gets killed 
 before the end and no my-trace.tracy file is produced. I will 
 try building on parts of the program with this and see if I can 
 see what is going on.


Got this to work after removing part of the program, the slowest 
parts are in library code (sumtype match to be precise). I will 
look into whether my usage can be improved.

I should also mention that what I said about compile time was a 
little inaccurate, some of that time linking (which involves 
llvm).
Jul 24 2021
next sibling parent reply JG <JG somewhere.com> writes:
On Saturday, 24 July 2021 at 09:12:15 UTC, JG wrote:

 On Saturday, 24 July 2021 at 08:26:39 UTC, JG wrote:

 On Friday, 23 July 2021 at 20:03:22 UTC, Dennis wrote:

 [...]


 Thanks for this suggestion. Unfortunately this makes the 
 compile use too much memory for my system and so it gets 
 killed before the end and no my-trace.tracy file is produced. 
 I will try building on parts of the program with this and see 
 if I can see what is going on.


 Got this to work after removing part of the program, the 
 slowest parts are in library code (sumtype match to be 
 precise). I will look into whether my usage can be improved.

 I should also mention that what I said about compile time was a 
 little inaccurate, some of that time linking (which involves 
 llvm).


Thanks very much to everyone for the help. With a few minor 
changes so far I have halved the compile time.
Jul 24 2021
parent russhy <russhy gmail.com> writes:
One thing to add:

With LDC you can add this flag ``-linkonce-templates`` to get 
faster link time when using templates a lot
Jul 24 2021
prev sibling next sibling parent Adam D Ruppe <destructionator gmail.com> writes:
On Saturday, 24 July 2021 at 09:12:15 UTC, JG wrote:

 the slowest parts are in library code


This is really common. A lot of libraries do really weird 
things... if you want to keep quick builds it is best to use the 
language directly.
Jul 24 2021
prev sibling parent russhy <russhy gmail.com> writes:
On Saturday, 24 July 2021 at 09:12:15 UTC, JG wrote:

 On Saturday, 24 July 2021 at 08:26:39 UTC, JG wrote:

 On Friday, 23 July 2021 at 20:03:22 UTC, Dennis wrote:

 On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote:

 [...]


 You can try profiling it with LDC 1.25 or later. Add this to 
 dub.sdl:

 [...]


 Thanks for this suggestion. Unfortunately this makes the 
 compile use too much memory for my system and so it gets 
 killed before the end and no my-trace.tracy file is produced. 
 I will try building on parts of the program with this and see 
 if I can see what is going on.


 Got this to work after removing part of the program, the 
 slowest parts are in library code (sumtype match to be 
 precise). I will look into whether my usage can be improved.

 I should also mention that what I said about compile time was a 
 little inaccurate, some of that time linking (which involves 
 llvm).


LLVM is known to be slow to compile

If you are on windows i suggest  to use DMD for iterating 
quickly, i get much faster compile time with it (probably on 
linux too, but i'm not a linux user)
Jul 24 2021
prev sibling parent Vladimir Panteleev <thecybershadow.lists gmail.com> writes:
On Friday, 23 July 2021 at 18:53:06 UTC, JG wrote:

 Any suggestion on how to try and improve the build time.


You could try some of the tools listed on the wiki for build time 
profiling:

https://wiki.dlang.org/Development_tools#Build_time_profiling

(intentional bump to aid search results, as apparently this list 
is not very well known)
Aug 08 2021