www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - About compiler memory consumption

reply "Puming" <zhaopuming gmail.com> writes:
Hi, I'm learning D, and trying to develop a website using vibe.d.

I'm using Digital Ocean's VPS server, with 1GB memory. Many a 
time when I do `dub`, the memory consumption soars and forces dmd 
to quit, sometimes even causes the remote machine droplet to 
freeze.

I understand that due to template and static analysis in the 
compiler, the memory usage is huge for template heavy code. But I 
heard that there was effort going on and progress is already made 
in recent releases.

My question is, is there a plan for this effort? What is the 
general prediction of an ideal D compiler, some hundreds of MBs 
or less? I'd like to switch to 512MB VPS droplet, as it's much 
cheaper. When (which release) would the memory cost be in 
reasonable level?

As an alternative, I can build the project locally and rsync the 
binaries to the VPS, but a simple binary in vibe.d yields a ~12MB 
file, it's somewhat slow to upload it. So I wonder can we refer 
vibe.d/phobos as a dynamic library?

Thanks,
Puming.
Dec 04 2013
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Puming:


 My question is, is there a plan for this effort?


There are plans to reduce memory used by CTFE, but template bloat 
is less easy to face.



 What is the  general prediction of an ideal D compiler,
 some hundreds of MBs or less?


Non one knows, I think.



 but a simple binary in vibe.d yields a ~12MB file,
 it's somewhat slow to upload it.


There are programs to compress binaries a lot.



 So I wonder can we refer vibe.d/phobos as a dynamic library?


This is being worked on in the latest D releases.

Bye,
bearophile
Dec 04 2013
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Dec 04, 2013 at 05:12:26PM +0100, bearophile wrote:

 Puming:


[...]

but a simple binary in vibe.d yields a ~12MB file,
it's somewhat slow to upload it.


 
 There are programs to compress binaries a lot.


Take a look at UPX: http://upx.sourceforge.net/

In my own projects, a typical D program is easily 5-10MB, or, if it's
template-heavy, 15-20MB. But if I run 'strip' on the executable and then
run 'upx' on it, I can quite easily get it down to less than 1MB.

You won't be able to debug it, of course, since there would be no
symbols, etc., left, but if you're deploying it to your production
server anyway, presumably you've already debugged it beforehand, so this
shouldn't be a problem.


T

-- 
Claiming that your operating system is the best in the world because
more people use it is like saying McDonalds makes the best food in the
world. -- Carl B. Constantine
Dec 04 2013
parent "Puming" <zhaopuming gmail.com> writes:
On Wednesday, 4 December 2013 at 16:26:43 UTC, H. S. Teoh wrote:

 On Wed, Dec 04, 2013 at 05:12:26PM +0100, bearophile wrote:

 Puming:


 [...]

but a simple binary in vibe.d yields a ~12MB file,
it's somewhat slow to upload it.


 
 There are programs to compress binaries a lot.


 Take a look at UPX: http://upx.sourceforge.net/

 In my own projects, a typical D program is easily 5-10MB, or, 
 if it's
 template-heavy, 15-20MB. But if I run 'strip' on the executable 
 and then
 run 'upx' on it, I can quite easily get it down to less than 
 1MB.

 You won't be able to debug it, of course, since there would be 
 no
 symbols, etc., left, but if you're deploying it to your 
 production
 server anyway, presumably you've already debugged it 
 beforehand, so this
 shouldn't be a problem.


 T


Thanks, I'll try that :)
Dec 04 2013
prev sibling parent reply "Puming" <zhaopuming gmail.com> writes:
On Wednesday, 4 December 2013 at 16:12:28 UTC, bearophile wrote:

 Puming:


 My question is, is there a plan for this effort?


 There are plans to reduce memory used by CTFE, but template 
 bloat is less easy to face.


Vibe's diet templates would benefit from this I guess.


 What is the  general prediction of an ideal D compiler,
 some hundreds of MBs or less?


 Non one knows, I think.


That is a pity.


 but a simple binary in vibe.d yields a ~12MB file,
 it's somewhat slow to upload it.


 There are programs to compress binaries a lot.


Thanks for the tip, now I'll just need a little script to do the 
compress/rsync/decompress automation.

Did you mean tar/zip or some specialized tool? I found that tar 
makes it to ~2.5MB, which is fairly good


 So I wonder can we refer vibe.d/phobos as a dynamic library?


 This is being worked on in the latest D releases.


That is great news. Would it be out in 2.065? ore 2.066?


 Bye,
 bearophile


Thanks bearophile,

So hopefully the situation might be much better earlier next year 
:-)
Dec 04 2013
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Puming:


 Did you mean tar/zip or some specialized tool?


I mean specialized programs that compile executables and keep 
them executables.



 I found that tar makes it to ~2.5MB, which is fairly good


I think that tar is not a compressor, it's a packager.



 That is great news. Would it be out in 2.065? ore 2.066?


It's already partially present. The next D release (2.065) will 
be about bug fixing, especially ICEs.

Bye,
bearophile
Dec 04 2013
parent reply "Puming" <zhaopuming gmail.com> writes:
On Wednesday, 4 December 2013 at 16:38:28 UTC, bearophile wrote:

 Puming:


 Did you mean tar/zip or some specialized tool?


 I mean specialized programs that compile executables and keep 
 them executables.



 I found that tar makes it to ~2.5MB, which is fairly good


 I think that tar is not a compressor, it's a packager.


Sorry, I mean gzip.


 That is great news. Would it be out in 2.065? ore 2.066?


 It's already partially present. The next D release (2.065) will 
 be about bug fixing, especially ICEs.


Thanks, I'll try that :) are there docs about how to use phobos 
as a shared lib?


 Bye,
 bearophile
Dec 04 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 16:44:14 UTC, Puming wrote:

 Thanks, I'll try that :) are there docs about how to use phobos 
 as a shared lib?


$ cat test.d
import std.stdio;

void main()
{
	writeln("Hello");
}

$ dmd test.d -defaultlib=libphobos2.so
$ ls -lah test
-rwxr-xr-x 1 dicebot users 26K Dec  4 17:51 test
$ ldd test
	linux-vdso.so.1 (0x00007fffc82e7000)
	libphobos2.so.0.64 => /usr/lib/libphobos2.so.0.64 
(0x00007ffd253e0000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007ffd251c2000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007ffd24ebf000)
	librt.so.1 => /usr/lib/librt.so.1 (0x00007ffd24cb7000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007ffd2490c000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007ffd24708000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0x00007ffd244a3000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ffd25a4e000)
	libssh2.so.1 => /usr/lib/libssh2.so.1 (0x00007ffd2427a000)
	libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x00007ffd2400d000)
	libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 
(0x00007ffd23c05000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007ffd239ef000)

(from my Arch Linux x64 box)
Dec 04 2013
parent "Puming" <zhaopuming gmail.com> writes:
Thanks, it works :-)

So I can modify package.json to let dub know how to use phobos as 
shared lib.

Can vibe.d also be used as a shared lib?

It would be great if we have a

```
dub --build=shared
```

or something similar.



On Wednesday, 4 December 2013 at 16:53:13 UTC, Dicebot wrote:

 On Wednesday, 4 December 2013 at 16:44:14 UTC, Puming wrote:

 Thanks, I'll try that :) are there docs about how to use 
 phobos as a shared lib?


 $ cat test.d
 import std.stdio;

 void main()
 {
 	writeln("Hello");
 }

 $ dmd test.d -defaultlib=libphobos2.so
 $ ls -lah test
 -rwxr-xr-x 1 dicebot users 26K Dec  4 17:51 test
 $ ldd test
 	linux-vdso.so.1 (0x00007fffc82e7000)
 	libphobos2.so.0.64 => /usr/lib/libphobos2.so.0.64 
 (0x00007ffd253e0000)
 	libpthread.so.0 => /usr/lib/libpthread.so.0 
 (0x00007ffd251c2000)
 	libm.so.6 => /usr/lib/libm.so.6 (0x00007ffd24ebf000)
 	librt.so.1 => /usr/lib/librt.so.1 (0x00007ffd24cb7000)
 	libc.so.6 => /usr/lib/libc.so.6 (0x00007ffd2490c000)
 	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007ffd24708000)
 	libcurl.so.4 => /usr/lib/libcurl.so.4 (0x00007ffd244a3000)
 	/lib64/ld-linux-x86-64.so.2 (0x00007ffd25a4e000)
 	libssh2.so.1 => /usr/lib/libssh2.so.1 (0x00007ffd2427a000)
 	libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 
 (0x00007ffd2400d000)
 	libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 
 (0x00007ffd23c05000)
 	libz.so.1 => /usr/lib/libz.so.1 (0x00007ffd239ef000)

 (from my Arch Linux x64 box)
Dec 04 2013
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
While in general one may expect CTFE/template memory consumption 
to drop considerably, in vibe.d case it is unlikely to make that 
much of an impact. vibe.d does really lot of computations during 
compile-time, for example, all Diet templates get loaded and 
processed during that phase. I can't imagine optimizations that 
will make possible to fit compilation of any vibe.d project of 
notable size in 512Mb memory. To be honest, I'd hate to do any 
DMD + vibe.d development on anything less than 4Gb of RAM.

In that sense I think rsync'ing binary from development box is 
most reasonable approach in the long term. There is already 
working dynamic Phobos library for Linux, using it + stripping 
the binaries shall reduce their sizes quite a lot. Officially 
shared Phobos is still not very tested but I have not had any 
issues with it so far.
Dec 04 2013
next sibling parent reply "Puming" <zhaopuming gmail.com> writes:
On Wednesday, 4 December 2013 at 16:28:07 UTC, Dicebot wrote:

 While in general one may expect CTFE/template memory 
 consumption to drop considerably, in vibe.d case it is unlikely 
 to make that much of an impact. vibe.d does really lot of 
 computations during compile-time, for example, all Diet 
 templates get loaded and processed during that phase. I can't 
 imagine optimizations that will make possible to fit 
 compilation of any vibe.d project of notable size in 512Mb 
 memory. To be honest, I'd hate to do any DMD + vibe.d 
 development on anything less than 4Gb of RAM.


Other than diet, is there many other CTFE/template heavy parts?

I hope there would be a non-CTFE version of diet template, 
similar to regex/CTRegex. If we use the non-CTFE version in dev 
mode, we would have a better memory & compile time turnaround in 
dev mode. And in the release mode the diet templates would be 
optimized with CTFE. Is that a viable approach?


 In that sense I think rsync'ing binary from development box is 
 most reasonable approach in the long term. There is already 
 working dynamic Phobos library for Linux, using it + stripping 
 the binaries shall reduce their sizes quite a lot. Officially 
 shared Phobos is still not very tested but I have not had any 
 issues with it so far.


Yes, after some trying and struggle, I came to the same 
conclusion. Will try the compression tools mentioned by Teoh :)

Thanks,

Puming.
Dec 04 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 16:41:14 UTC, Puming wrote:

 Other than diet, is there many other CTFE/template heavy parts?


Also vibe.http.rest and vibe.http.form create really lot of 
template instances because of heavy usage of reflection + code 
generation. This is one of domains though that can potentially 
improve a lot by better compiler implementation,


 I hope there would be a non-CTFE version of diet template, 
 similar to regex/CTRegex. If we use the non-CTFE version in dev 
 mode, we would have a better memory & compile time turnaround 
 in dev mode. And in the release mode the diet templates would 
 be optimized with CTFE. Is that a viable approach?


That should be possible and relatively simple (because of CTFE 
nature a lot of code can be reused). It may considerably impact 
Diet performance but I can see desire for that in development 
mode as project size increases. As usual, it is mostly matter of 
manpower - vibe.d has only one persistent contributor right now 
and probably 5-8 guys doing occasional pull request, definitely 
not enough for project of that scale.
Dec 04 2013
parent reply "Puming" <zhaopuming gmail.com> writes:
On Wednesday, 4 December 2013 at 16:50:45 UTC, Dicebot wrote:

 On Wednesday, 4 December 2013 at 16:41:14 UTC, Puming wrote:

 Other than diet, is there many other CTFE/template heavy parts?


 Also vibe.http.rest and vibe.http.form create really lot of 
 template instances because of heavy usage of reflection + code 
 generation. This is one of domains though that can potentially 
 improve a lot by better compiler implementation,



 I hope there would be a non-CTFE version of diet template, 
 similar to regex/CTRegex. If we use the non-CTFE version in 
 dev mode, we would have a better memory & compile time 
 turnaround in dev mode. And in the release mode the diet 
 templates would be optimized with CTFE. Is that a viable 
 approach?


 That should be possible and relatively simple (because of CTFE 
 nature a lot of code can be reused). It may considerably impact 
 Diet performance but I can see desire for that in development 
 mode as project size increases. As usual, it is mostly matter 
 of manpower - vibe.d has only one persistent contributor right 
 now and probably 5-8 guys doing occasional pull request, 
 definitely not enough for project of that scale.


There maybe another traditional approach, by converting the .dt 
diet templates directly into D source code BEFORE the actuall 
compilation. That is the way many other HTML template systems 
use, and the performance is fairly good.

The generated D code could be very C like, and does not use much 
CTFE and templates, so that it doesn't incur a lot of compilation 
resources.

As as the matter of manpower, I'm very interested in D/Vibe.d, 
and will learn and try to contribute :-)

I'm currently using Java/Netty&Friends at work. But found 
D/Vibe.d would be a better solution in the future.
Dec 04 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 5 December 2013 at 04:02:58 UTC, Puming wrote:

 There maybe another traditional approach, by converting the .dt 
 diet templates directly into D source code BEFORE the actuall 
 compilation. That is the way many other HTML template systems 
 use, and the performance is fairly good.


It is somewhat complicated by the very tight integration between 
D handler host and Diet - you can expose D symbols for rendering 
context to template. It is still possible to do it that way but 
is likely to require more efforts and effectively will just to 
the same thing as good JIT-ed CTFE should. I don't like it 
because it is too effort-demanding for a quick hack and not 
suitable in the long term by design (one of D goals was to remove 
reliance on external build steps as much as possible)
Dec 05 2013
parent "Puming" <zhaopuming gmail.com> writes:
On Thursday, 5 December 2013 at 13:49:51 UTC, Dicebot wrote:

 On Thursday, 5 December 2013 at 04:02:58 UTC, Puming wrote:

 There maybe another traditional approach, by converting the 
 .dt diet templates directly into D source code BEFORE the 
 actuall compilation. That is the way many other HTML template 
 systems use, and the performance is fairly good.


 It is somewhat complicated by the very tight integration 
 between D handler host and Diet - you can expose D symbols for 
 rendering context to template. It is still possible to do it 
 that way but is likely to require more efforts and effectively 
 will just to the same thing as good JIT-ed CTFE should. I don't 
 like it because it is too effort-demanding for a quick hack and 
 not suitable in the long term by design (one of D goals was to 
 remove reliance on external build steps as much as possible)


Yes, I agree that for dev mode a pure runtime version would fit 
more and its implementation could take advantage of many 
functions in the CTFE version (or is that already possible?)
Dec 05 2013
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Dec 04, 2013 at 05:28:06PM +0100, Dicebot wrote:

 While in general one may expect CTFE/template memory consumption to
 drop considerably, in vibe.d case it is unlikely to make that much
 of an impact. vibe.d does really lot of computations during
 compile-time, for example, all Diet templates get loaded and
 processed during that phase. I can't imagine optimizations that will
 make possible to fit compilation of any vibe.d project of notable
 size in 512Mb memory. To be honest, I'd hate to do any DMD + vibe.d
 development on anything less than 4Gb of RAM.


Even without vibe.d, dmd eats way too much memory to be usable on a
machine with 512MB RAM. I was quite disappointed to discover this when I
tried running dmd on a not-that-old machine with 512MB RAM (aka my
office PC), and every time it would soak up all available RAM, force my
browser (and probably significant parts of X11) to swap out, making the
PC generally unresponsive for several seconds, before the dmd process
exits and everything goes back to normal.

For all of the good points of D and dmd's compile speeds, the one thing
I can't recommend dmd for is efficient memory usage, sad to say.  Of
course, on a machine with more available RAM, dmd runs at lightning
speed, so that at least is a redeeming quality.


T

-- 
Guns don't kill people. Bullets do.
Dec 04 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 17:46:40 UTC, H. S. Teoh wrote:

 Even without vibe.d, dmd eats way too much memory to be usable 
 on a
 machine with 512MB RAM. I was quite disappointed to discover 
 this when I
 tried running dmd on a not-that-old machine with 512MB RAM (aka 
 my
 office PC), and every time it would soak up all available RAM, 
 force my
 browser (and probably significant parts of X11) to swap out, 
 making the
 PC generally unresponsive for several seconds, before the dmd 
 process
 exits and everything goes back to normal.


Heh, that was reason why I was forced to reconsider from setting 
up a CI suite on my 1Gb-worth VPS :) There are some live projects 
that require 2.5-3 Gb for routine build.
Dec 04 2013
parent reply "Dicebot" <public dicebot.lv> writes:
I want to note though that when it comes to template instance 
bloat which is one of huge memory consumers it is quite possible 
to design compiler in a much more efficient way, possibly with 
minor change to language spec.
Dec 04 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Dec 04, 2013 at 06:55:35PM +0100, Dicebot wrote:

 I want to note though that when it comes to template instance bloat
 which is one of huge memory consumers it is quite possible to design
 compiler in a much more efficient way, possibly with minor change to
 language spec.


I'm pretty sure there's a lot of room for optimization in dmd when it
comes to handling templates. If only I had the time to dig into the dmd
source code...


T

-- 
Why are you blatanly misspelling "blatant"? -- Branden Robinson
Dec 04 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 18:20:36 UTC, H. S. Teoh wrote:

 I'm pretty sure there's a lot of room for optimization in dmd 
 when it
 comes to handling templates. If only I had the time to dig into 
 the dmd
 source code...


I have wasted ~3 months on digging this with no success. And when 
Walter has finally paid attention to this issue and changed 
template instance emitting algorithm, I still was hardly able to 
understand why exactly the changes in pull were necessary to get 
that result.

Just be warned ;)
Dec 04 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Dec 04, 2013 at 07:22:45PM +0100, Dicebot wrote:

 On Wednesday, 4 December 2013 at 18:20:36 UTC, H. S. Teoh wrote:

I'm pretty sure there's a lot of room for optimization in dmd when it
comes to handling templates. If only I had the time to dig into the
dmd source code...


 
 I have wasted ~3 months on digging this with no success. And when
 Walter has finally paid attention to this issue and changed template
 instance emitting algorithm, I still was hardly able to understand
 why exactly the changes in pull were necessary to get that result.
 
 Just be warned ;)


Well, that's why I said I needed more time. :) I'm not expecting it to
be a breeze to read through dmd code. I've actually looked into it
briefly before when tracking down a compiler bug, but I've to say that
in order to understand the code you need to first understand how it
works at a high level (yes, it's sorta a chicken-and-egg problem),
otherwise you're just looking at the trees and not seeing the forest.


T

-- 
What are you when you run out of Monet? Baroque.
Dec 04 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 18:31:14 UTC, H. S. Teoh wrote:

 ...


Well I have found it to be somewhat reasonable in general but 
symbol emitting part is one of most arcane ones in whole 
compiler. Basically only few people know something about it and 
sometimes it feels like Walter is the only one to really knows it 
in details.
Dec 04 2013
parent Orvid King <blah38621 gmail.com> writes:
Just to put my 2cents into the conversation, you could also compile
vibe.d as a static library and then link against it, as that seems to
be enough to keep dmd's memory using within functional limits. (I have
a max of 800mb ram that I can easily allow a single program to use,
otherwise I start doing a whole lotta' paging)

On 12/4/13, Dicebot <public dicebot.lv> wrote:

 On Wednesday, 4 December 2013 at 18:31:14 UTC, H. S. Teoh wrote:

 ...


 Well I have found it to be somewhat reasonable in general but
 symbol emitting part is one of most arcane ones in whole
 compiler. Basically only few people know something about it and
 sometimes it feels like Walter is the only one to really knows it
 in details.
Dec 04 2013
prev sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
H. S. Teoh:


 Even without vibe.d, dmd eats way too much memory to be usable 
 on a machine with 512MB RAM.


On Windows32 to compile the whole Phobos DMD takes about 750-850 
MB. My D programs are much smaller, so usually I don't need more 
than 50-300 MB.

Bye,
bearophile
Dec 04 2013