www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - dmd out of memory compiling vibed project

reply Guillaume <guillaume spam.spam> writes:
Hi,

I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d 
project with ~20 views (*.dt). On my main computer, I have no 
problems but my laptop has only 4 gig of RAM and dmd is killed 
before the end of the compilation because it takes more than 4 
gig of RAM.

I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but it 
doesn't change anything. It does work with the "singleFile" 
option with dub but it's really long to compile... And the 
"singleFile" option seems to recompile everything even if I only 
change one ".d" source file.

Are there other solutions?

Thanks,

Guillaume
May 22 2019
next sibling parent reply Suliman <evermind live.ru> writes:
Absolutely same problem. -lowmem did not get any affect on my VPS 
(1GB ram). I was able to compile project only in singleFile mode.
May 22 2019
parent reply krzaq <dlangmailinglist krzaq.cc> writes:
On Wednesday, 22 May 2019 at 13:09:37 UTC, Suliman wrote:

 Absolutely same problem. -lowmem did not get any affect on my 
 VPS (1GB ram). I was able to compile project only in singleFile 
 mode.


In those situations you can go with a swapfile.

fallocate -l 4G swapfile
chmod 600 swapfile
mkswap swapfile

then, when you need the file, just

sudo swapon swapfile

and when you're done

sudo swapoff swapfile

Even on an SSD the performance will be terrible, but it should 
get the job done.
May 22 2019
parent reply Anonymouse <asdf asdf.net> writes:
On Wednesday, 22 May 2019 at 13:25:00 UTC, krzaq wrote:

 On Wednesday, 22 May 2019 at 13:09:37 UTC, Suliman wrote:

 Absolutely same problem. -lowmem did not get any affect on my 
 VPS (1GB ram). I was able to compile project only in 
 singleFile mode.


 In those situations you can go with a swapfile.


Does it circumvent process memory limits? Regardless it's not an 
option if you're dealing with CI and you don't have the 
neccessary permissions.

$ dub build
Performing "debug" build using /usr/bin/dmd for x86_64.
cachetools 0.1.2: building configuration "library"...
requests 1.0.8: building configuration "std"...
kameloso 1.1.3+commit.66.g4c07959f: building configuration 
"posix"...
Killed
/usr/bin/dmd failed with exit code 137.

$ fallocate -l 4G /tmp/swapfile
$ chmod 600 /tmp/swapfile
$ mkswap /tmp/swapfile
Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
no label, UUID=7112294b-812a-4765-a3b9-41aa545daa70
$ sudo swapon /tmp/swapfile
swapon: /tmp/swapfile: insecure file owner 3434, 0 (root) 
suggested.
swapon: /tmp/swapfile: swapon failed: Operation not permitted
May 22 2019
next sibling parent krzaq <dlangmailinglist krzaq.cc> writes:
On Wednesday, 22 May 2019 at 14:11:07 UTC, Anonymouse wrote:

 On Wednesday, 22 May 2019 at 13:25:00 UTC, krzaq wrote:

 On Wednesday, 22 May 2019 at 13:09:37 UTC, Suliman wrote:

 Absolutely same problem. -lowmem did not get any affect on my 
 VPS (1GB ram). I was able to compile project only in 
 singleFile mode.


 In those situations you can go with a swapfile.


 Does it circumvent process memory limits? Regardless it's not 
 an option if you're dealing with CI and you don't have the 
 neccessary permissions.

 $ dub build
 Performing "debug" build using /usr/bin/dmd for x86_64.
 cachetools 0.1.2: building configuration "library"...
 requests 1.0.8: building configuration "std"...
 kameloso 1.1.3+commit.66.g4c07959f: building configuration 
 "posix"...
 Killed
 /usr/bin/dmd failed with exit code 137.

 $ fallocate -l 4G /tmp/swapfile
 $ chmod 600 /tmp/swapfile
 $ mkswap /tmp/swapfile
 Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
 no label, UUID=7112294b-812a-4765-a3b9-41aa545daa70
 $ sudo swapon /tmp/swapfile
 swapon: /tmp/swapfile: insecure file owner 3434, 0 (root) 
 suggested.
 swapon: /tmp/swapfile: swapon failed: Operation not permitted


Unless it's a 32-bit process, then it should help. That's what I 
did when I built geordi on a 512MB DigitalOcean droplet, and it 
allowed me to have a successful compilation.

As for CI - I have no suggestions outside of "get a beefier one".
May 22 2019
prev sibling parent Guillaume <guillaume spam.spam> writes:
On Wednesday, 22 May 2019 at 14:11:07 UTC, Anonymouse wrote:

 On Wednesday, 22 May 2019 at 13:25:00 UTC, krzaq wrote:

 On Wednesday, 22 May 2019 at 13:09:37 UTC, Suliman wrote:

 Absolutely same problem. -lowmem did not get any affect on my 
 VPS (1GB ram). I was able to compile project only in 
 singleFile mode.


 In those situations you can go with a swapfile.


 Does it circumvent process memory limits? Regardless it's not 
 an option if you're dealing with CI and you don't have the 
 neccessary permissions.

 $ fallocate -l 4G /tmp/swapfile
 $ chmod 600 /tmp/swapfile
 $ mkswap /tmp/swapfile
 Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
 no label, UUID=7112294b-812a-4765-a3b9-41aa545daa70
 $ sudo swapon /tmp/swapfile
 swapon: /tmp/swapfile: insecure file owner 3434, 0 (root) 
 suggested.
 swapon: /tmp/swapfile: swapon failed: Operation not permitted


I think it will work if you use sudo on all the commands 
(fallocate, chmod, mkswap), not just swapon.

But it doesn't work for my case since I cannot have a swapfile on 
my weird setup.

Other ideas?
May 22 2019
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 22, 2019 at 12:49:56PM +0000, Guillaume via Digitalmars-d wrote:

 Hi,
 
 I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d project
 with ~20 views (*.dt). On my main computer, I have no problems but my
 laptop has only 4 gig of RAM and dmd is killed before the end of the
 compilation because it takes more than 4 gig of RAM.
 
 I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but it
 doesn't change anything. It does work with the "singleFile" option
 with dub but it's really long to compile... And the "singleFile"
 option seems to recompile everything even if I only change one ".d"
 source file.


[...]

Use separate compilation. (I don't know if dub support this.)

Also, in my own vibe.d project, I split up the Diet templates across
multiple modules, because they are extremely compile-time intensive. For
example, I had one set of pages dealing with account creation, another
set of pages with the main functionality, so I split them up into two
modules -- since it's unlikely I'll be changing both sets of pages at
the same time. That, plus separate compilation, cuts compile time by
half.  Better yet, I split up front-facing UI code (mainly Diet
templates) from backend logic, so that the Diet templates are not
recompiled while I'm changing only the backend logic.


T

-- 
Customer support: the art of getting your clients to pay for your own
incompetence.
May 22 2019
parent reply Guillaume <guillaume spam.spam> writes:
On Wednesday, 22 May 2019 at 16:23:30 UTC, H. S. Teoh wrote:

 On Wed, May 22, 2019 at 12:49:56PM +0000, Guillaume via 
 Digitalmars-d wrote:

 Hi,
 
 I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d 
 project with ~20 views (*.dt). On my main computer, I have no 
 problems but my laptop has only 4 gig of RAM and dmd is killed 
 before the end of the compilation because it takes more than 4 
 gig of RAM.
 
 I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but 
 it doesn't change anything. It does work with the "singleFile" 
 option with dub but it's really long to compile... And the 
 "singleFile" option seems to recompile everything even if I 
 only change one ".d" source file.


 [...]

 Use separate compilation. (I don't know if dub support this.)


You mean compiling each file to create .o and then linking them? 
dub supports this with the "singleFile" option but recompiles 
everything each time. What tool do you use to automate 
compilation and linking?

Guillaume
May 22 2019
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, May 22, 2019 at 05:23:19PM +0000, Guillaume via Digitalmars-d wrote:

 On Wednesday, 22 May 2019 at 16:23:30 UTC, H. S. Teoh wrote:

 On Wed, May 22, 2019 at 12:49:56PM +0000, Guillaume via Digitalmars-d
 wrote:

 Hi,
 
 I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d
 project with ~20 views (*.dt). On my main computer, I have no
 problems but my laptop has only 4 gig of RAM and dmd is killed
 before the end of the compilation because it takes more than 4 gig
 of RAM.
 
 I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but it
 doesn't change anything. It does work with the "singleFile" option
 with dub but it's really long to compile... And the "singleFile"
 option seems to recompile everything even if I only change one
 ".d" source file.


 [...]
 
 Use separate compilation. (I don't know if dub support this.)
 


 
 You mean compiling each file to create .o and then linking them? dub
 supports this with the "singleFile" option but recompiles everything
 each time. What tool do you use to automate compilation and linking?


[...]

IME, dub is useful as a packaging tool but not so much as a build tool.
Because of that I have given up on using dub as a build tool, and only
use it for pulling in dependencies.  For builds, I use SCons, though you
really could any general build system that isn't crippled. Even
Makefiles can do it, despite Make's many issues.

Whichever build system you choose, you don't really have to compile
*individual* files; you can compile clusters of them into static
libraries (the usual recommendation is one static library per package,
but that's only a convention; you can break it up however is most
advantageous for you), then link them together into the final
executable.

For example, for my vibe.d project I split up the source files into one
group for backend logic, then a group for base pages (front page and a
few other common "utility" pages), another group for creating new
accounts, and another group for the main functionality pages. In theory
I could also break up the backend logic into multiple groups, but IME
Diet templates take long enough to compile that that's not really the
bottleneck, so I left it as a single group for the time being.

Be warned, however, that you'll have to dig into some dirty details of
how dub works if you decide to go to go this route -- e.g., you may have
to parse dub output in order to get import paths, the list of object
files to link into the final executable, etc..  It's all scriptable once
you figure it out, of course, but you do have to get your hands dirty
once.


T

-- 
Give me some fresh salted fish, please.
May 22 2019
prev sibling parent reply Francesco Mecca <me francescomecca.eu> writes:
On Wednesday, 22 May 2019 at 12:49:56 UTC, Guillaume wrote:

 Hi,

 I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d 
 project with ~20 views (*.dt). On my main computer, I have no 
 problems but my laptop has only 4 gig of RAM and dmd is killed 
 before the end of the compilation because it takes more than 4 
 gig of RAM.

 I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but 
 it doesn't change anything. It does work with the "singleFile" 
 option with dub but it's really long to compile... And the 
 "singleFile" option seems to recompile everything even if I 
 only change one ".d" source file.

 Are there other solutions?

 Thanks,

 Guillaume


Do you have the same problem with ldc or gdc?
I have a pc with 4 GB as well and dmd is usually the only one 
that gets OOM killed when compiling big projects.
May 22 2019
parent reply Guillaume <guillaume spam.spam> writes:
On Wednesday, 22 May 2019 at 22:07:41 UTC, Francesco Mecca wrote:

 On Wednesday, 22 May 2019 at 12:49:56 UTC, Guillaume wrote:

 Hi,

 I'm using dmd 2.086.0 and vibe.d 0.85.0 to compile a vibe.d 
 project with ~20 views (*.dt). On my main computer, I have no 
 problems but my laptop has only 4 gig of RAM and dmd is killed 
 before the end of the compilation because it takes more than 4 
 gig of RAM.

 I've tried the new -lowmem switch (in DFLAGS in dmd.conf) but 
 it doesn't change anything. It does work with the "singleFile" 
 option with dub but it's really long to compile... And the 
 "singleFile" option seems to recompile everything even if I 
 only change one ".d" source file.

 Are there other solutions?

 Thanks,

 Guillaume


 Do you have the same problem with ldc or gdc?
 I have a pc with 4 GB as well and dmd is usually the only one 
 that gets OOM killed when compiling big projects.


Yep. I've tryed the latest ldc and gdc and both cashes... 
Although I don't know how to add a "-lowmem" switch to those 
compiler... but I suspect it would not change anything.

If dub "singleFile" could be more intelligent and not recompile 
everything it would solve my problem!

Guillaume
May 22 2019
parent reply Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Thursday, 23 May 2019 at 00:26:03 UTC, Guillaume wrote:

 I don't know how to add a "-lowmem" switch to those compiler... 
 but I suspect it would not change anything.


If you're using a dub.sdl:

dflags "-lowmem"

dub.json:

{
    ...
     "dflags" : ["-lowmem"]
}

The compiler directly:

dmd -lowmem [rest of args...]

See dflags under Build settings in 
https://dub.pm/package-format-sdl 
https://dub.pm/package-format-json.html
May 22 2019
parent reply Guillaume <guillaume spam.spam> writes:
On Thursday, 23 May 2019 at 01:17:26 UTC, Nicholas Wilson wrote:

 On Thursday, 23 May 2019 at 00:26:03 UTC, Guillaume wrote:

 I don't know how to add a "-lowmem" switch to those 
 compiler... but I suspect it would not change anything.


 If you're using a dub.sdl:

 dflags "-lowmem"

 dub.json:

 {
    ...
     "dflags" : ["-lowmem"]
 }

 The compiler directly:

 dmd -lowmem [rest of args...]

 See dflags under Build settings in 
 https://dub.pm/package-format-sdl 
 https://dub.pm/package-format-json.html


Still no luck.

Anybody has a decent build system as easy as dub but that doesn't 
recompile everything? Or everybody rolls their own?

Guillaume
May 23 2019
next sibling parent Andre Pany <andre s-e-a-p.de> writes:
On Thursday, 23 May 2019 at 11:06:43 UTC, Guillaume wrote:

 On Thursday, 23 May 2019 at 01:17:26 UTC, Nicholas Wilson wrote:

 On Thursday, 23 May 2019 at 00:26:03 UTC, Guillaume wrote:

 I don't know how to add a "-lowmem" switch to those 
 compiler... but I suspect it would not change anything.


 If you're using a dub.sdl:

 dflags "-lowmem"

 dub.json:

 {
    ...
     "dflags" : ["-lowmem"]
 }

 The compiler directly:

 dmd -lowmem [rest of args...]

 See dflags under Build settings in 
 https://dub.pm/package-format-sdl 
 https://dub.pm/package-format-json.html


 Still no luck.

 Anybody has a decent build system as easy as dub but that 
 doesn't recompile everything? Or everybody rolls their own?

 Guillaume


Dub allows you to create sub packages within you dub package.
This might solve the recompile issue for you.

Kind regards
André
May 23 2019
prev sibling parent reply Atila Neves <atila.neves gmail.com> writes:
On Thursday, 23 May 2019 at 11:06:43 UTC, Guillaume wrote:

 On Thursday, 23 May 2019 at 01:17:26 UTC, Nicholas Wilson wrote:

 On Thursday, 23 May 2019 at 00:26:03 UTC, Guillaume wrote:

 I don't know how to add a "-lowmem" switch to those 
 compiler... but I suspect it would not change anything.


 If you're using a dub.sdl:

 dflags "-lowmem"

 dub.json:

 {
    ...
     "dflags" : ["-lowmem"]
 }

 The compiler directly:

 dmd -lowmem [rest of args...]

 See dflags under Build settings in 
 https://dub.pm/package-format-sdl 
 https://dub.pm/package-format-json.html


 Still no luck.

 Anybody has a decent build system as easy as dub but that 
 doesn't recompile everything? Or everybody rolls their own?



https://github.com/atilaneves/reggae/
May 23 2019
parent Guillaume <guillaume spam.spam> writes:
On Thursday, 23 May 2019 at 13:02:26 UTC, Atila Neves wrote:

 On Thursday, 23 May 2019 at 11:06:43 UTC, Guillaume wrote:

 On Thursday, 23 May 2019 at 01:17:26 UTC, Nicholas Wilson 
 wrote:

 On Thursday, 23 May 2019 at 00:26:03 UTC, Guillaume wrote:

 I don't know how to add a "-lowmem" switch to those 
 compiler... but I suspect it would not change anything.


 See dflags under Build settings in 
 https://dub.pm/package-format-sdl 
 https://dub.pm/package-format-json.html


 Still no luck.

 Anybody has a decent build system as easy as dub but that 
 doesn't recompile everything? Or everybody rolls their own?



 https://github.com/atilaneves/reggae/


It just works (tm)! No config, everything is perfect on the first 
try! Thank you so much!!!

Guilaume
May 23 2019