www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Compilation is taking a ton of memory

reply Mario Silva <mario tripaneer.com> writes:
Hello,

Our code base has been growing steadily and it's currently at a 
point where my 16GB machine just freezes when we're compiling our 
code. This happens because DMD just consumes all my memory for a 
while.

Also, it's taking a long time to compile it. Less than an year 
ago our project was taking around 17 seconds to compile - no libs 
requiring compilation - and maybe around 50 seconds for full 
compilation, and it now takes around 50 seconds for an 
incremental compilations and around 1.5 minutes for a full one.

For you guys to have an idea of the size of our project, we have 
21151 lines of code and then 50933 in our libs. This is just our 
code without counting dependencies fetched by dub like vibe.d for 
example.

Are there any plans to work on compiler performance in terms of 
memory consumption and compilation time?

Any tips on how to code in a way that minimizes both compilation 
times and memory consumption when compiling?

Thanks,
Mario
Jun 27 2018
next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jun 27, 2018 at 04:00:37PM +0000, Mario Silva via Digitalmars-d wrote:
 Hello,
 
 Our code base has been growing steadily and it's currently at a point
 where my 16GB machine just freezes when we're compiling our code. This
 happens because DMD just consumes all my memory for a while.
How are you compiling the code? Dub? Makefile? Are you compiling everything all at once? DMD is known to prefer compilation speed over memory efficiency, a choice I personally don't quite agree with, but it is what it is. That means past a certain size, you're going to have to split up your project into multiple separate compilations. The usual convention is to compile packages together, e.g., if you have the source tree structure: main/main.d main/common.d pkg1/package.d pkg1/mod1.d pkg1/mod2.d pkg2/package.d pkg2/mod1.d pkg2/mod2.d then you'd compile main/* first, say into main.a (or main.lib), and compile pkg1/* into pkg1.a, and compile pkg2/* into pkg2.a, then finally link them all together with a final invocation of the compiler. This should reduce the total amount of memory used by any single invocation of dmd, which should keep memory usage sane. It may also reduce compilation time if only one package needs to be recompiled (though template-heavy code may not save that much, depending on what you do with it). [..]
 Are there any plans to work on compiler performance in terms of memory
 consumption and compilation time?
Not that I know of, though it'd be really nice if somebody worked on that. Given Walter's past choice of compilation speed over memory efficiency, I'm not sure how far such an effort would get. But I'd vote for it. There *is* the option of just turning on the GC, now that dmd is bootstrapping, but the last time I tried there were bugs that caused crashes, so it may not (yet) be a viable option.
 Any tips on how to code in a way that minimizes both compilation times
 and memory consumption when compiling?
[...] In general: (1) avoid template bloat, and (2) use separate compilation in strategic ways. (1) implies: - Avoid using (or reduce usage of) libraries / modules that are heavy on templates, except where absolutely necessary. If you're using Phobos, std.format comes to mind (though in a sufficiently large project I'd expect it doesn't play that big of a role, but the principle holds). If you're using vibe.d, Diet templates can be a source of heavy template usage. More generally, avoid recursive templates, or templates instantiation chains that nest too deeply. If not done carefully, you may end up with a quadratic or an exponential number of template instantiations, which will quickly eat up all your memory and CPU cycles (in addition to generating bloated executables). - Avoid doing too much in CTFE, because the current CTFE engine is slow (and also very memory-inefficient). We're all waiting for newCTFE to land, but from what I can tell that's still a ways away. (Note: not to be confused with other compile-time features that are not directly related to CTFE. Using static if's and the like won't add much overhead unless you're using recursive templates, in which case see If template-heavy or CTFE-heavy code is essential to your project, consider splitting them into their own self-contained (sub)modules, and separately compile them, so that recompilations that don't touch those parts of the code will be reasonably fast. In one of my vibe.d projects, I basically separated out the "business logic" of the program into a module outside the code that directly interfaces with vibe.d (Diet templates being the primary target). This has sped up recompilations when only the "business logic" parts of the code needs to change. Furthermore, because Diet templates tend to get very heavy, I also split those up into different compilation groups (e.g., new user / account management pages, error/miscellaneous pages, main pages) so that if I'm working on Diet templates in, say, the new user creation pages, I only have to recompile the Diet templates related to account management, not every other Diet template in the project. This also saves a lot of time waiting for compilation. T -- It is impossible to make anything foolproof because fools are so ingenious. -- Sammy
Jun 27 2018
prev sibling next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/27/18 12:00 PM, Mario Silva wrote:
 Hello,
 
 Our code base has been growing steadily and it's currently at a point 
 where my 16GB machine just freezes when we're compiling our code. This 
 happens because DMD just consumes all my memory for a while.
That is the path of D's compiler toolchain. The front end uses a simple "bump the pointer" memory allocation scheme, and NEVER frees memory. This is reasonable for many things in the compiler (e.g. you never want to forget about symbols you built), but the worst is if you have a lot of CTFE, as that consumes a lot of memory that could easily be reclaimed after it's done running.
 Also, it's taking a long time to compile it. Less than an year ago our 
 project was taking around 17 seconds to compile - no libs requiring 
 compilation - and maybe around 50 seconds for full compilation, and it 
 now takes around 50 seconds for an incremental compilations and around 
 1.5 minutes for a full one.
Yikes, I experience stuff like this, and I'm using vibe.d as well. But I haven't gotten big enough to make it horrible. I'm compiling my vibe.d project on a VM Linux image with only 2GB of RAM. It's definitely a problem for template-heavy or CTFE heavy builds.
 
 For you guys to have an idea of the size of our project, we have 21151 
 lines of code and then 50933 in our libs. This is just our code without 
 counting dependencies fetched by dub like vibe.d for example.
The libraries should build OK. It's really the templates or diet templates that take the longest. And that's not a large project in comparison to other projects.
 Are there any plans to work on compiler performance in terms of memory 
 consumption and compilation time?
This has been a thorn in many sides for a long time. I remember Weka.io's Liran talking about how they required an INSANE amount of time/memory to build their system in dconf 2015 maybe? But things have gotten a bit better since then. I think at some point there will be a reckoning where it has to be fixed. Fast compile-time is cool, but inifinte compile time (i.e. it never finishes because the OOM killer destroys your process) is not acceptable.
 Any tips on how to code in a way that minimizes both compilation times 
 and memory consumption when compiling?
The default for dub is to build "separately", you can try different build modes --build-mode=allAtOnce or --build-mode=singleFile. There is very scarce documentation as to what these actually do. I don't know if these would help at all, but worth a try. Note that in some respects, memory usage and compilation times are related. If you want to keep memory usage low, you may have to forget some things, and then re-figure them out later, which slows you down. But if you consume too much memory, your system thrashes and then you experience a slowdown that way. -Steve
Jun 27 2018
parent Shachar Shemesh <shachar weka.io> writes:
On 28/06/18 01:46, Steven Schveighoffer wrote:
 This has been a thorn in many sides for a long time. I remember 
 Weka.io's Liran talking about how they required an INSANE amount of 
 time/memory to build their system in dconf 2015 maybe? But things have 
 gotten a bit better since then. I think at some point there will be a 
 reckoning where it has to be fixed. Fast compile-time is cool, but 
 inifinte compile time (i.e. it never finishes because the OOM killer 
 destroys your process) is not acceptable.
No, we just use bigger build servers: $ cat /proc/cpuinfo ... processor : 127 vendor_id : GenuineIntel cpu family : 6 model : 63 model name : Intel(R) Xeon(R) CPU E7-8880 v3 2.30GHz stepping : 4 microcode : 0x11 cpu MHz : 2300.072 cache size : 46080 KB physical id : 3 siblings : 32 core id : 15 cpu cores : 16 apicid : 223 initial apicid : 223 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq monitor est ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm ida xsaveopt fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm bogomips : 4662.97 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: ubuntu ip-172-31-14-32:~$ cat /proc/meminfo MemTotal: 2014743200 kB My laptop has 32GB of ram. If I do single threaded compilation and allocate enough swap, I can *barely* squeeze through a debug build compilation of the node. Shachar
Jun 30 2018
prev sibling next sibling parent sarn <sarn theartofmachinery.com> writes:
On Wednesday, 27 June 2018 at 16:00:37 UTC, Mario Silva wrote:
 Any tips on how to code in a way that minimizes both 
 compilation times and memory consumption when compiling?
Here are my tips. I'd love to hear more from others. * Try to reduce imports. E.g., say you use a lot of stuff from std.algorithm in your implementations, but not in your external interface. In that case, instead of doing "import std.algorithm;" at the module top, do things like "import std.algorithm.searching : canFind;" inside function bodies. If you think about this when designing your external interfaces, you can let user code import what it needs from your modules without recursively importing half the rest of the D ecosystem with it. * Templates are great, but not everything has to be one. Sometimes a simple delegate makes a perfectly generic interface. Templated code can be more expensive to compile, while non-templated code can be compiled once in a library and then reused. * CTFE is awesome but currently a memory hog. BTW, vibe.d isn't great with this stuff, TBH. Simple public API functions return types with "auto" that turn out to be several layers of templated wrappers all the way down, each wrapper coming from a different module. That's an example of what to avoid doing.
Jun 27 2018
prev sibling next sibling parent reply crimaniak <crimaniak gmail.com> writes:
On Wednesday, 27 June 2018 at 16:00:37 UTC, Mario Silva wrote:
 Hello,

 Our code base has been growing steadily and it's currently at a 
 point where my 16GB machine just freezes when we're compiling 
 our code. This happens because DMD just consumes all my memory 
 for a while.

 Also, it's taking a long time to compile it. Less than an year 
 ago our project was taking around 17 seconds to compile - no 
 libs requiring compilation - and maybe around 50 seconds for 
 full compilation, and it now takes around 50 seconds for an 
 incremental compilations and around 1.5 minutes for a full one.
The same problem. Freeze because of active swap usage and about 2 minutes to compile after every small change. In fact, when it comes to template code, DMD is neither particularly fast nor memory-efficient. Given that there are no things such as precompiled headers, on a large project, the D experience is much worse than C++. The problem is aggravated by the fact that DUB compiles all the sources in one DMD launch. I tried to solve this problem some time ago, and Sönke advised using reggae (https://code.dlang.org/packages/reggae). I'm still experimenting with different approaches and even wrote my own utility helper (https://code.dlang.org/packages/ifupdated). But at the moment I can state the following: * It is necessary to incorporate into the DUB reggae (== classic make) approach to compile sources. * Like the 'debug' and 'release' modes of the compiler, the build system should also have two modes, for example, 'deploy' and 'development', the first minimizes the time it takes for a full compilation, and the second minimizes the recompilation time after minor changes. * It needs to think seriously about 'compiler as a service' approach and caching intermediate compilation results.
 Any tips on how to code in a way that minimizes both 
 compilation times and memory consumption when compiling?
For the current moment just try reggae. You can find some forgotten imports because of separate compilation, but it is easily fixable.
Jun 28 2018
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Jun 28, 2018 at 04:11:57PM +0000, crimaniak via Digitalmars-d wrote:
[...]
 The problem is aggravated by the fact that DUB compiles all the
 sources in one DMD launch.
Doesn't dub have an option to compile packages (i.e. subdirs) separately? Or does that only apply to dub packages, not to subdirs within a single project?
 I tried to solve this problem some time ago, and Snke advised using
 reggae (https://code.dlang.org/packages/reggae). I'm still
 experimenting with different approaches and even wrote my own utility
 helper (https://code.dlang.org/packages/ifupdated).
[...] This is one of the reasons I was not impressed by dub (sorry, Sonke). I continue to use SCons for my D projects. For dub dependencies, I just create a fake empty dub project with declared dependencies and run that separately for refreshing dependencies, but the actual compiling and linking is handled by SCons. Once a project gets beyond a certain size, you *need* more fine-grained control over exactly how the project should be built. T -- That's not a bug; that's a feature!
Jun 28 2018
next sibling parent reply Eugene Wissner <belka caraus.de> writes:
On Thursday, 28 June 2018 at 16:24:07 UTC, H. S. Teoh wrote:
 On Thu, Jun 28, 2018 at 04:11:57PM +0000, crimaniak via 
 Digitalmars-d wrote: [...]
 The problem is aggravated by the fact that DUB compiles all 
 the sources in one DMD launch.
Doesn't dub have an option to compile packages (i.e. subdirs) separately? Or does that only apply to dub packages, not to subdirs within a single project?
As far as I can see dub builds static libraries from dub dependencies. But it passes all source files from one dub project together to build final static library/executable. You can actually use --verbose to see what commands a called.
 I tried to solve this problem some time ago, and Sönke advised 
 using reggae (https://code.dlang.org/packages/reggae). I'm 
 still experimenting with different approaches and even wrote 
 my own utility helper 
 (https://code.dlang.org/packages/ifupdated).
[...] This is one of the reasons I was not impressed by dub (sorry, Sonke). I continue to use SCons for my D projects. For dub dependencies, I just create a fake empty dub project with declared dependencies and run that separately for refreshing dependencies, but the actual compiling and linking is handled by SCons. Once a project gets beyond a certain size, you *need* more fine-grained control over exactly how the project should be built. T
Some more alternatives: I have good experiance with "rake"-family of build systems. I've seen ruby rake [1] itself being used for D projects. For a smaller project of me I'm using shake [2]. You write your build scripts in Haskell and since Haskell is a compiled language, the build system should be compiled first, but it is worth it, since shake is mature and very fast (I'm pretty sure it is faster than python, ruby or js build systems and probably make too). You can use it to run tests as well. I compile every source file to an object file, put object files from dependencies to its own static libraries and link everything together at the end. But I should also admit, I'm not using dmd at all, but build everything with gdc/gcc. The only problem is handling dependencies and dependencies of dependencies if you have a lot of them. [1] https://ruby.github.io/rake/ [2] https://shakebuild.com/
Jun 28 2018
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Jun 28, 2018 at 06:13:45PM +0000, Eugene Wissner via Digitalmars-d
wrote:
 On Thursday, 28 June 2018 at 16:24:07 UTC, H. S. Teoh wrote:
 On Thu, Jun 28, 2018 at 04:11:57PM +0000, crimaniak via Digitalmars-d
 wrote: [...]
 The problem is aggravated by the fact that DUB compiles all the
 sources in one DMD launch.
Doesn't dub have an option to compile packages (i.e. subdirs) separately? Or does that only apply to dub packages, not to subdirs within a single project?
As far as I can see dub builds static libraries from dub dependencies. But it passes all source files from one dub project together to build final static library/executable. You can actually use --verbose to see what commands a called.
Someone should file an enhancement ticket for this. [...]
 I compile every source file to an object file, put object files from
 dependencies to its own static libraries and link everything together
 at the end.
That's the traditional C/C++ approach, which is stable, if not the most suitable for D. I forget the details now, but I believe there are some savings in compiling template code if you compile multiple source files at once (assuming, of course, that they share some number of template instantiations). So personally I'd group source files by subdir and compile them in batches that way, then pass the resulting static libs to the parent dirs, and so on.
 But I should also admit, I'm not using dmd at all, but build
 everything with gdc/gcc. The only problem is handling dependencies and
 dependencies of dependencies if you have a lot of them.
[...] Unless you're doing imports using mixins, which is a questionable practice to begin with, it should be pretty easy to, say, grep for 'import' lines and build your dependency tree that way. Or use the tup approach of instrumenting the compiler with LD_LIBRARY_* so that any dependencies are automatically caught, and you don't have to worry about it (though that only accounts for dependencies on a specific run, it won't pick up imports across static-if branches, for example, though it's unlikely to be an actual problem in real-life code). T -- You have to expect the unexpected. -- RL
Jun 28 2018
prev sibling next sibling parent crimaniak <crimaniak gmail.com> writes:
On Thursday, 28 June 2018 at 16:24:07 UTC, H. S. Teoh wrote:
 On Thu, Jun 28, 2018 at 04:11:57PM +0000, crimaniak via 
 Digitalmars-d wrote: [...]
 The problem is aggravated by the fact that DUB compiles all 
 the sources in one DMD launch.
Doesn't dub have an option to compile packages (i.e. subdirs) separately? Or does that only apply to dub packages, not to subdirs within a single project?
Yes, it's about dub packages, not about subdirs. And yes, it's possible to split project to subpackages, it gives its benefits and structure code, but I hate the idea of doing this forcibly, just to fix compilation time. In addition, a typical large site is a large number of pages with diet templates, and even if you separate the pages from business logic, but this part itself is still very large and slow in compilation. I had to refuse diet rendering on the server and go to Vue+pug, which allowed to accelerate the development cycle when changes on the frontend from 2 minutes to instantly.
 [...]

 This is one of the reasons I was not impressed by dub (sorry, 
 Sonke).  I continue to use SCons for my D projects...
I believe that this situation is bad for D as a whole. The problem with large projects can be solved by splitting into subpackages, or with Scons, or with reggae, or manually writing the makefile, etc., etc., but the person who just started writing on D does not know all this. He simply does the project and with his growth discovers this problem. I think the modern programmer has the right to expect that the official build system will work well enough with a project of several thousand files.
Jun 28 2018
prev sibling parent sarn <sarn theartofmachinery.com> writes:
On Thursday, 28 June 2018 at 16:24:07 UTC, H. S. Teoh wrote:
 I continue to use SCons for my D projects. For dub 
 dependencies, I just create a fake empty dub project with 
 declared dependencies and run that separately for refreshing 
 dependencies, but the actual compiling and linking is handled 
 by SCons.
Sounds like what I'm doing. I build the dependencies using dub on a dummy project, then use "dub describe" to find all the files I need and install them into my project directory. Then I can use whatever build system I want (currently tup).
Jun 28 2018
prev sibling next sibling parent Nick Treleaven <nick geany.org> writes:
On Wednesday, 27 June 2018 at 16:00:37 UTC, Mario Silva wrote:
 Less than an year ago our project was taking around 17 seconds 
 to compile - no libs requiring compilation - and maybe around 
 50 seconds for full compilation, and it now takes around 50 
 seconds for an incremental compilations and around 1.5 minutes 
 for a full one.
If memory consumption has got worse in recent releases, it's possible it was my fault. I made a pull* to fix it, so it might be worth trying the 2.081.0 beta as it should have the fix merged. * https://github.com/dlang/dmd/pull/8281
Jul 01 2018
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2018-06-27 18:00, Mario Silva wrote:
 Hello,
 
 Our code base has been growing steadily and it's currently at a point 
 where my 16GB machine just freezes when we're compiling our code. This 
 happens because DMD just consumes all my memory for a while.
 
 Also, it's taking a long time to compile it. Less than an year ago our 
 project was taking around 17 seconds to compile - no libs requiring 
 compilation - and maybe around 50 seconds for full compilation, and it 
 now takes around 50 seconds for an incremental compilations and around 
 1.5 minutes for a full one.
 
 For you guys to have an idea of the size of our project, we have 21151 
 lines of code and then 50933 in our libs. This is just our code without 
 counting dependencies fetched by dub like vibe.d for example.
 
 Are there any plans to work on compiler performance in terms of memory 
 consumption and compilation time?
 
 Any tips on how to code in a way that minimizes both compilation times 
 and memory consumption when compiling?
Unless this already has been stated, the issue is usually not the size of the project, it's rather which compile time features which are used. For example the DWT project has over 200k lines of code and compile (to a library) in a couple of seconds. Try to reduce templates and CTFE code, if possible and prefer CTFE over templates. -- /Jacob Carlborg
Jul 04 2018
prev sibling parent reply "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 06/27/2018 12:00 PM, Mario Silva wrote:
 Hello,
 
 Our code base has been growing steadily and it's currently at a point 
 where my 16GB machine just freezes when we're compiling our code. This 
 happens because DMD just consumes all my memory for a while.
 
 Also, it's taking a long time to compile it. Less than an year ago our 
 project was taking around 17 seconds to compile - no libs requiring 
 compilation - and maybe around 50 seconds for full compilation, and it 
 now takes around 50 seconds for an incremental compilations and around 
 1.5 minutes for a full one.
 
Memory consumption is a known issue with the D frontend implementation. As others have said, CTFE never freeing memory is considered to be a big part of the issue here. The CTFE rewrite someone is doing should help with that when it's done. Long compilation times are almost always the fault of Dub. Dub is pretty well-known for taking DMD's near-instant compile times and bloating them out to the ridiculous (by D standards) lengths you're experiencing. I'd say try just writing a basic buildscript that runs DMD directly. Plus, that way you can customize it to your own project's needs: Which files/packages get compiled together, which get compiled separately.
Jul 04 2018
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jul 04, 2018 at 04:17:22PM -0400, Nick Sabalausky (Abscissa) via
Digitalmars-d wrote:
[...]
 Long compilation times are almost always the fault of Dub. Dub is
 pretty well-known for taking DMD's near-instant compile times and
 bloating them out to the ridiculous (by D standards) lengths you're
 experiencing. I'd say try just writing a basic buildscript that runs
 DMD directly. Plus, that way you can customize it to your own
 project's needs: Which files/packages get compiled together, which get
 compiled separately.
Yeah, I'm sorry to say that I have been quite disappointed with dub (as a build system), and currently have taken to avoiding it as much as possible. As a packaging system it's not bad, but it simply doesn't have the options I need to use it as a build tool. But to be fair, it's not always dub's fault. Heavy template / CTFE usage is also known to bloat compilation times, sometimes by a lot. Earlier this year I discovered that the very act of importing std.format or std.regex causes compilation time to almost double. There were some issues related to compiling with -unittest that triggered this behaviour, though we found that the underlying issue was actually more tricky than it might first appear. I don't remember if this issue has been completely resolved yet. T -- Живёшь только однажды.
Jul 05 2018