www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dmd out of memory compiling vibed project

reply Guillaume <guillaume spam.spam> writes:
Hi,

I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d 
project with ~20 views (*.dt). On my main computer, I have no 
problems but my laptop has only 4 gig of RAM and dmd is killed 
before the end of the compilation because it takes more than 4 
gig of RAM.

I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but it 
doesn't change anything. It does work with the "singleFile" 
option with dub but it's really long to compile... And the 
"singleFile" option seems to recompile everything even if I only 
change one ".d" source file.

Are there other solutions?

Thanks,

Guillaume
May 22
next sibling parent reply Suliman <evermind live.ru> writes:
Absolutely same problem. -lowmem did not get any affect on my VPS 
(1GB ram). I was able to compile project only in singleFile mode.
May 22
parent reply krzaq <dlangmailinglist krzaq.cc> writes:
On Wednesday, 22 May 2019 at 13:09:37 UTC, Suliman wrote:
 Absolutely same problem. -lowmem did not get any affect on my 
 VPS (1GB ram). I was able to compile project only in singleFile 
 mode.
In those situations you can go with a swapfile. fallocate -l 4G swapfile chmod 600 swapfile mkswap swapfile then, when you need the file, just sudo swapon swapfile and when you're done sudo swapoff swapfile Even on an SSD the performance will be terrible, but it should get the job done.
May 22
parent reply Anonymouse <asdf asdf.net> writes:
On Wednesday, 22 May 2019 at 13:25:00 UTC, krzaq wrote:
 On Wednesday, 22 May 2019 at 13:09:37 UTC, Suliman wrote:
 Absolutely same problem. -lowmem did not get any affect on my 
 VPS (1GB ram). I was able to compile project only in 
 singleFile mode.
In those situations you can go with a swapfile.
Does it circumvent process memory limits? Regardless it's not an option if you're dealing with CI and you don't have the neccessary permissions. $ dub build Performing "debug" build using /usr/bin/dmd for x86_64. cachetools 0.1.2: building configuration "library"... requests 1.0.8: building configuration "std"... kameloso 1.1.3+commit.66.g4c07959f: building configuration "posix"... Killed /usr/bin/dmd failed with exit code 137. $ fallocate -l 4G /tmp/swapfile $ chmod 600 /tmp/swapfile $ mkswap /tmp/swapfile Setting up swapspace version 1, size = 4 GiB (4294963200 bytes) no label, UUID=7112294b-812a-4765-a3b9-41aa545daa70 $ sudo swapon /tmp/swapfile swapon: /tmp/swapfile: insecure file owner 3434, 0 (root) suggested. swapon: /tmp/swapfile: swapon failed: Operation not permitted
May 22
next sibling parent krzaq <dlangmailinglist krzaq.cc> writes:
On Wednesday, 22 May 2019 at 14:11:07 UTC, Anonymouse wrote:
 On Wednesday, 22 May 2019 at 13:25:00 UTC, krzaq wrote:
 On Wednesday, 22 May 2019 at 13:09:37 UTC, Suliman wrote:
 Absolutely same problem. -lowmem did not get any affect on my 
 VPS (1GB ram). I was able to compile project only in 
 singleFile mode.
In those situations you can go with a swapfile.
Does it circumvent process memory limits? Regardless it's not an option if you're dealing with CI and you don't have the neccessary permissions. $ dub build Performing "debug" build using /usr/bin/dmd for x86_64. cachetools 0.1.2: building configuration "library"... requests 1.0.8: building configuration "std"... kameloso 1.1.3+commit.66.g4c07959f: building configuration "posix"... Killed /usr/bin/dmd failed with exit code 137. $ fallocate -l 4G /tmp/swapfile $ chmod 600 /tmp/swapfile $ mkswap /tmp/swapfile Setting up swapspace version 1, size = 4 GiB (4294963200 bytes) no label, UUID=7112294b-812a-4765-a3b9-41aa545daa70 $ sudo swapon /tmp/swapfile swapon: /tmp/swapfile: insecure file owner 3434, 0 (root) suggested. swapon: /tmp/swapfile: swapon failed: Operation not permitted
Unless it's a 32-bit process, then it should help. That's what I did when I built geordi on a 512MB DigitalOcean droplet, and it allowed me to have a successful compilation. As for CI - I have no suggestions outside of "get a beefier one".
May 22
prev sibling parent Guillaume <guillaume spam.spam> writes:
On Wednesday, 22 May 2019 at 14:11:07 UTC, Anonymouse wrote:
 On Wednesday, 22 May 2019 at 13:25:00 UTC, krzaq wrote:
 On Wednesday, 22 May 2019 at 13:09:37 UTC, Suliman wrote:
 Absolutely same problem. -lowmem did not get any affect on my 
 VPS (1GB ram). I was able to compile project only in 
 singleFile mode.
In those situations you can go with a swapfile.
Does it circumvent process memory limits? Regardless it's not an option if you're dealing with CI and you don't have the neccessary permissions. $ fallocate -l 4G /tmp/swapfile $ chmod 600 /tmp/swapfile $ mkswap /tmp/swapfile Setting up swapspace version 1, size = 4 GiB (4294963200 bytes) no label, UUID=7112294b-812a-4765-a3b9-41aa545daa70 $ sudo swapon /tmp/swapfile swapon: /tmp/swapfile: insecure file owner 3434, 0 (root) suggested. swapon: /tmp/swapfile: swapon failed: Operation not permitted
I think it will work if you use sudo on all the commands (fallocate, chmod, mkswap), not just swapon. But it doesn't work for my case since I cannot have a swapfile on my weird setup. Other ideas?
May 22
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 22, 2019 at 12:49:56PM +0000, Guillaume via Digitalmars-d wrote:
 Hi,
 
 I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d project
 with ~20 views (*.dt). On my main computer, I have no problems but my
 laptop has only 4 gig of RAM and dmd is killed before the end of the
 compilation because it takes more than 4 gig of RAM.
 
 I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but it
 doesn't change anything. It does work with the "singleFile" option
 with dub but it's really long to compile... And the "singleFile"
 option seems to recompile everything even if I only change one ".d"
 source file.
[...] Use separate compilation. (I don't know if dub support this.) Also, in my own vibe.d project, I split up the Diet templates across multiple modules, because they are extremely compile-time intensive. For example, I had one set of pages dealing with account creation, another set of pages with the main functionality, so I split them up into two modules -- since it's unlikely I'll be changing both sets of pages at the same time. That, plus separate compilation, cuts compile time by half. Better yet, I split up front-facing UI code (mainly Diet templates) from backend logic, so that the Diet templates are not recompiled while I'm changing only the backend logic. T -- Customer support: the art of getting your clients to pay for your own incompetence.
May 22
parent reply Guillaume <guillaume spam.spam> writes:
On Wednesday, 22 May 2019 at 16:23:30 UTC, H. S. Teoh wrote:
 On Wed, May 22, 2019 at 12:49:56PM +0000, Guillaume via 
 Digitalmars-d wrote:
 Hi,
 
 I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d 
 project with ~20 views (*.dt). On my main computer, I have no 
 problems but my laptop has only 4 gig of RAM and dmd is killed 
 before the end of the compilation because it takes more than 4 
 gig of RAM.
 
 I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but 
 it doesn't change anything. It does work with the "singleFile" 
 option with dub but it's really long to compile... And the 
 "singleFile" option seems to recompile everything even if I 
 only change one ".d" source file.
[...] Use separate compilation. (I don't know if dub support this.)
You mean compiling each file to create .o and then linking them? dub supports this with the "singleFile" option but recompiles everything each time. What tool do you use to automate compilation and linking? Guillaume
May 22
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 22, 2019 at 05:23:19PM +0000, Guillaume via Digitalmars-d wrote:
 On Wednesday, 22 May 2019 at 16:23:30 UTC, H. S. Teoh wrote:
 On Wed, May 22, 2019 at 12:49:56PM +0000, Guillaume via Digitalmars-d
 wrote:
 Hi,
 
 I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d
 project with ~20 views (*.dt). On my main computer, I have no
 problems but my laptop has only 4 gig of RAM and dmd is killed
 before the end of the compilation because it takes more than 4 gig
 of RAM.
 
 I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but it
 doesn't change anything. It does work with the "singleFile" option
 with dub but it's really long to compile... And the "singleFile"
 option seems to recompile everything even if I only change one
 ".d" source file.
[...] Use separate compilation. (I don't know if dub support this.)
You mean compiling each file to create .o and then linking them? dub supports this with the "singleFile" option but recompiles everything each time. What tool do you use to automate compilation and linking?
[...] IME, dub is useful as a packaging tool but not so much as a build tool. Because of that I have given up on using dub as a build tool, and only use it for pulling in dependencies. For builds, I use SCons, though you really could any general build system that isn't crippled. Even Makefiles can do it, despite Make's many issues. Whichever build system you choose, you don't really have to compile *individual* files; you can compile clusters of them into static libraries (the usual recommendation is one static library per package, but that's only a convention; you can break it up however is most advantageous for you), then link them together into the final executable. For example, for my vibe.d project I split up the source files into one group for backend logic, then a group for base pages (front page and a few other common "utility" pages), another group for creating new accounts, and another group for the main functionality pages. In theory I could also break up the backend logic into multiple groups, but IME Diet templates take long enough to compile that that's not really the bottleneck, so I left it as a single group for the time being. Be warned, however, that you'll have to dig into some dirty details of how dub works if you decide to go to go this route -- e.g., you may have to parse dub output in order to get import paths, the list of object files to link into the final executable, etc.. It's all scriptable once you figure it out, of course, but you do have to get your hands dirty once. T -- Give me some fresh salted fish, please.
May 22
prev sibling parent reply Francesco Mecca <me francescomecca.eu> writes:
On Wednesday, 22 May 2019 at 12:49:56 UTC, Guillaume wrote:
 Hi,

 I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d 
 project with ~20 views (*.dt). On my main computer, I have no 
 problems but my laptop has only 4 gig of RAM and dmd is killed 
 before the end of the compilation because it takes more than 4 
 gig of RAM.

 I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but 
 it doesn't change anything. It does work with the "singleFile" 
 option with dub but it's really long to compile... And the 
 "singleFile" option seems to recompile everything even if I 
 only change one ".d" source file.

 Are there other solutions?

 Thanks,

 Guillaume
Do you have the same problem with ldc or gdc? I have a pc with 4 GB as well and dmd is usually the only one that gets OOM killed when compiling big projects.
May 22
parent reply Guillaume <guillaume spam.spam> writes:
On Wednesday, 22 May 2019 at 22:07:41 UTC, Francesco Mecca wrote:
 On Wednesday, 22 May 2019 at 12:49:56 UTC, Guillaume wrote:
 Hi,

 I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d 
 project with ~20 views (*.dt). On my main computer, I have no 
 problems but my laptop has only 4 gig of RAM and dmd is killed 
 before the end of the compilation because it takes more than 4 
 gig of RAM.

 I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but 
 it doesn't change anything. It does work with the "singleFile" 
 option with dub but it's really long to compile... And the 
 "singleFile" option seems to recompile everything even if I 
 only change one ".d" source file.

 Are there other solutions?

 Thanks,

 Guillaume
Do you have the same problem with ldc or gdc? I have a pc with 4 GB as well and dmd is usually the only one that gets OOM killed when compiling big projects.
Yep. I've tryed the latest ldc and gdc and both cashes... Although I don't know how to add a "-lowmem" switch to those compiler... but I suspect it would not change anything. If dub "singleFile" could be more intelligent and not recompile everything it would solve my problem! Guillaume
May 22
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 23 May 2019 at 00:26:03 UTC, Guillaume wrote:
 I don't know how to add a "-lowmem" switch to those compiler... 
 but I suspect it would not change anything.
If you're using a dub.sdl: dflags "-lowmem" dub.json: { ... "dflags" : ["-lowmem"] } The compiler directly: dmd -lowmem [rest of args...] See dflags under Build settings in https://dub.pm/package-format-sdl https://dub.pm/package-format-json.html
May 22
parent reply Guillaume <guillaume spam.spam> writes:
On Thursday, 23 May 2019 at 01:17:26 UTC, Nicholas Wilson wrote:
 On Thursday, 23 May 2019 at 00:26:03 UTC, Guillaume wrote:
 I don't know how to add a "-lowmem" switch to those 
 compiler... but I suspect it would not change anything.
If you're using a dub.sdl: dflags "-lowmem" dub.json: { ... "dflags" : ["-lowmem"] } The compiler directly: dmd -lowmem [rest of args...] See dflags under Build settings in https://dub.pm/package-format-sdl https://dub.pm/package-format-json.html
Still no luck. Anybody has a decent build system as easy as dub but that doesn't recompile everything? Or everybody rolls their own? Guillaume
May 23
next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Thursday, 23 May 2019 at 11:06:43 UTC, Guillaume wrote:
 On Thursday, 23 May 2019 at 01:17:26 UTC, Nicholas Wilson wrote:
 On Thursday, 23 May 2019 at 00:26:03 UTC, Guillaume wrote:
 I don't know how to add a "-lowmem" switch to those 
 compiler... but I suspect it would not change anything.
If you're using a dub.sdl: dflags "-lowmem" dub.json: { ... "dflags" : ["-lowmem"] } The compiler directly: dmd -lowmem [rest of args...] See dflags under Build settings in https://dub.pm/package-format-sdl https://dub.pm/package-format-json.html
Still no luck. Anybody has a decent build system as easy as dub but that doesn't recompile everything? Or everybody rolls their own? Guillaume
Dub allows you to create sub packages within you dub package. This might solve the recompile issue for you. Kind regards André
May 23
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 23 May 2019 at 11:06:43 UTC, Guillaume wrote:
 On Thursday, 23 May 2019 at 01:17:26 UTC, Nicholas Wilson wrote:
 On Thursday, 23 May 2019 at 00:26:03 UTC, Guillaume wrote:
 I don't know how to add a "-lowmem" switch to those 
 compiler... but I suspect it would not change anything.
If you're using a dub.sdl: dflags "-lowmem" dub.json: { ... "dflags" : ["-lowmem"] } The compiler directly: dmd -lowmem [rest of args...] See dflags under Build settings in https://dub.pm/package-format-sdl https://dub.pm/package-format-json.html
Still no luck. Anybody has a decent build system as easy as dub but that doesn't recompile everything? Or everybody rolls their own?
https://github.com/atilaneves/reggae/
May 23
parent Guillaume <guillaume spam.spam> writes:
On Thursday, 23 May 2019 at 13:02:26 UTC, Atila Neves wrote:
 On Thursday, 23 May 2019 at 11:06:43 UTC, Guillaume wrote:
 On Thursday, 23 May 2019 at 01:17:26 UTC, Nicholas Wilson 
 wrote:
 On Thursday, 23 May 2019 at 00:26:03 UTC, Guillaume wrote:
 I don't know how to add a "-lowmem" switch to those 
 compiler... but I suspect it would not change anything.
See dflags under Build settings in https://dub.pm/package-format-sdl https://dub.pm/package-format-json.html
Still no luck. Anybody has a decent build system as easy as dub but that doesn't recompile everything? Or everybody rolls their own?
https://github.com/atilaneves/reggae/
It just works (tm)! No config, everything is perfect on the first try! Thank you so much!!! Guilaume
May 23