www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - About compiler memory consumption

reply "Puming" <zhaopuming gmail.com> writes:
Hi, I'm learning D, and trying to develop a website using vibe.d.

I'm using Digital Ocean's VPS server, with 1GB memory. Many a 
time when I do `dub`, the memory consumption soars and forces dmd 
to quit, sometimes even causes the remote machine droplet to 
freeze.

I understand that due to template and static analysis in the 
compiler, the memory usage is huge for template heavy code. But I 
heard that there was effort going on and progress is already made 
in recent releases.

My question is, is there a plan for this effort? What is the 
general prediction of an ideal D compiler, some hundreds of MBs 
or less? I'd like to switch to 512MB VPS droplet, as it's much 
cheaper. When (which release) would the memory cost be in 
reasonable level?

As an alternative, I can build the project locally and rsync the 
binaries to the VPS, but a simple binary in vibe.d yields a ~12MB 
file, it's somewhat slow to upload it. So I wonder can we refer 
vibe.d/phobos as a dynamic library?

Thanks,
Puming.
Dec 04 2013
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Puming:

 My question is, is there a plan for this effort?
There are plans to reduce memory used by CTFE, but template bloat is less easy to face.
 What is the  general prediction of an ideal D compiler,
 some hundreds of MBs or less?
Non one knows, I think.
 but a simple binary in vibe.d yields a ~12MB file,
 it's somewhat slow to upload it.
There are programs to compress binaries a lot.
 So I wonder can we refer vibe.d/phobos as a dynamic library?
This is being worked on in the latest D releases. Bye, bearophile
Dec 04 2013
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Dec 04, 2013 at 05:12:26PM +0100, bearophile wrote:
 Puming:
[...]
but a simple binary in vibe.d yields a ~12MB file,
it's somewhat slow to upload it.
There are programs to compress binaries a lot.
Take a look at UPX: http://upx.sourceforge.net/ In my own projects, a typical D program is easily 5-10MB, or, if it's template-heavy, 15-20MB. But if I run 'strip' on the executable and then run 'upx' on it, I can quite easily get it down to less than 1MB. You won't be able to debug it, of course, since there would be no symbols, etc., left, but if you're deploying it to your production server anyway, presumably you've already debugged it beforehand, so this shouldn't be a problem. T -- Claiming that your operating system is the best in the world because more people use it is like saying McDonalds makes the best food in the world. -- Carl B. Constantine
Dec 04 2013
parent "Puming" <zhaopuming gmail.com> writes:
On Wednesday, 4 December 2013 at 16:26:43 UTC, H. S. Teoh wrote:
 On Wed, Dec 04, 2013 at 05:12:26PM +0100, bearophile wrote:
 Puming:
[...]
but a simple binary in vibe.d yields a ~12MB file,
it's somewhat slow to upload it.
There are programs to compress binaries a lot.
Take a look at UPX: http://upx.sourceforge.net/ In my own projects, a typical D program is easily 5-10MB, or, if it's template-heavy, 15-20MB. But if I run 'strip' on the executable and then run 'upx' on it, I can quite easily get it down to less than 1MB. You won't be able to debug it, of course, since there would be no symbols, etc., left, but if you're deploying it to your production server anyway, presumably you've already debugged it beforehand, so this shouldn't be a problem. T
Thanks, I'll try that :)
Dec 04 2013
prev sibling parent reply "Puming" <zhaopuming gmail.com> writes:
On Wednesday, 4 December 2013 at 16:12:28 UTC, bearophile wrote:
 Puming:

 My question is, is there a plan for this effort?
There are plans to reduce memory used by CTFE, but template bloat is less easy to face.
Vibe's diet templates would benefit from this I guess.
 What is the  general prediction of an ideal D compiler,
 some hundreds of MBs or less?
Non one knows, I think.
That is a pity.
 but a simple binary in vibe.d yields a ~12MB file,
 it's somewhat slow to upload it.
There are programs to compress binaries a lot.
Thanks for the tip, now I'll just need a little script to do the compress/rsync/decompress automation. Did you mean tar/zip or some specialized tool? I found that tar makes it to ~2.5MB, which is fairly good
 So I wonder can we refer vibe.d/phobos as a dynamic library?
This is being worked on in the latest D releases.
That is great news. Would it be out in 2.065? ore 2.066?
 Bye,
 bearophile
Thanks bearophile, So hopefully the situation might be much better earlier next year :-)
Dec 04 2013
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Puming:

 Did you mean tar/zip or some specialized tool?
I mean specialized programs that compile executables and keep them executables.
 I found that tar makes it to ~2.5MB, which is fairly good
I think that tar is not a compressor, it's a packager.
 That is great news. Would it be out in 2.065? ore 2.066?
It's already partially present. The next D release (2.065) will be about bug fixing, especially ICEs. Bye, bearophile
Dec 04 2013
parent reply "Puming" <zhaopuming gmail.com> writes:
On Wednesday, 4 December 2013 at 16:38:28 UTC, bearophile wrote:
 Puming:

 Did you mean tar/zip or some specialized tool?
I mean specialized programs that compile executables and keep them executables.
 I found that tar makes it to ~2.5MB, which is fairly good
I think that tar is not a compressor, it's a packager.
Sorry, I mean gzip.
 That is great news. Would it be out in 2.065? ore 2.066?
It's already partially present. The next D release (2.065) will be about bug fixing, especially ICEs.
Thanks, I'll try that :) are there docs about how to use phobos as a shared lib?
 Bye,
 bearophile
Dec 04 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 16:44:14 UTC, Puming wrote:
 Thanks, I'll try that :) are there docs about how to use phobos 
 as a shared lib?
$ cat test.d import std.stdio; void main() { writeln("Hello"); } $ dmd test.d -defaultlib=libphobos2.so $ ls -lah test -rwxr-xr-x 1 dicebot users 26K Dec 4 17:51 test $ ldd test linux-vdso.so.1 (0x00007fffc82e7000) libphobos2.so.0.64 => /usr/lib/libphobos2.so.0.64 (0x00007ffd253e0000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007ffd251c2000) libm.so.6 => /usr/lib/libm.so.6 (0x00007ffd24ebf000) librt.so.1 => /usr/lib/librt.so.1 (0x00007ffd24cb7000) libc.so.6 => /usr/lib/libc.so.6 (0x00007ffd2490c000) libdl.so.2 => /usr/lib/libdl.so.2 (0x00007ffd24708000) libcurl.so.4 => /usr/lib/libcurl.so.4 (0x00007ffd244a3000) /lib64/ld-linux-x86-64.so.2 (0x00007ffd25a4e000) libssh2.so.1 => /usr/lib/libssh2.so.1 (0x00007ffd2427a000) libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x00007ffd2400d000) libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x00007ffd23c05000) libz.so.1 => /usr/lib/libz.so.1 (0x00007ffd239ef000) (from my Arch Linux x64 box)
Dec 04 2013
parent "Puming" <zhaopuming gmail.com> writes:
Thanks, it works :-)

So I can modify package.json to let dub know how to use phobos as 
shared lib.

Can vibe.d also be used as a shared lib?

It would be great if we have a

```
dub --build=shared
```

or something similar.



On Wednesday, 4 December 2013 at 16:53:13 UTC, Dicebot wrote:
 On Wednesday, 4 December 2013 at 16:44:14 UTC, Puming wrote:
 Thanks, I'll try that :) are there docs about how to use 
 phobos as a shared lib?
$ cat test.d import std.stdio; void main() { writeln("Hello"); } $ dmd test.d -defaultlib=libphobos2.so $ ls -lah test -rwxr-xr-x 1 dicebot users 26K Dec 4 17:51 test $ ldd test linux-vdso.so.1 (0x00007fffc82e7000) libphobos2.so.0.64 => /usr/lib/libphobos2.so.0.64 (0x00007ffd253e0000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007ffd251c2000) libm.so.6 => /usr/lib/libm.so.6 (0x00007ffd24ebf000) librt.so.1 => /usr/lib/librt.so.1 (0x00007ffd24cb7000) libc.so.6 => /usr/lib/libc.so.6 (0x00007ffd2490c000) libdl.so.2 => /usr/lib/libdl.so.2 (0x00007ffd24708000) libcurl.so.4 => /usr/lib/libcurl.so.4 (0x00007ffd244a3000) /lib64/ld-linux-x86-64.so.2 (0x00007ffd25a4e000) libssh2.so.1 => /usr/lib/libssh2.so.1 (0x00007ffd2427a000) libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x00007ffd2400d000) libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x00007ffd23c05000) libz.so.1 => /usr/lib/libz.so.1 (0x00007ffd239ef000) (from my Arch Linux x64 box)
Dec 04 2013
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
While in general one may expect CTFE/template memory consumption 
to drop considerably, in vibe.d case it is unlikely to make that 
much of an impact. vibe.d does really lot of computations during 
compile-time, for example, all Diet templates get loaded and 
processed during that phase. I can't imagine optimizations that 
will make possible to fit compilation of any vibe.d project of 
notable size in 512Mb memory. To be honest, I'd hate to do any 
DMD + vibe.d development on anything less than 4Gb of RAM.

In that sense I think rsync'ing binary from development box is 
most reasonable approach in the long term. There is already 
working dynamic Phobos library for Linux, using it + stripping 
the binaries shall reduce their sizes quite a lot. Officially 
shared Phobos is still not very tested but I have not had any 
issues with it so far.
Dec 04 2013
next sibling parent reply "Puming" <zhaopuming gmail.com> writes:
On Wednesday, 4 December 2013 at 16:28:07 UTC, Dicebot wrote:
 While in general one may expect CTFE/template memory 
 consumption to drop considerably, in vibe.d case it is unlikely 
 to make that much of an impact. vibe.d does really lot of 
 computations during compile-time, for example, all Diet 
 templates get loaded and processed during that phase. I can't 
 imagine optimizations that will make possible to fit 
 compilation of any vibe.d project of notable size in 512Mb 
 memory. To be honest, I'd hate to do any DMD + vibe.d 
 development on anything less than 4Gb of RAM.
Other than diet, is there many other CTFE/template heavy parts? I hope there would be a non-CTFE version of diet template, similar to regex/CTRegex. If we use the non-CTFE version in dev mode, we would have a better memory & compile time turnaround in dev mode. And in the release mode the diet templates would be optimized with CTFE. Is that a viable approach?
 In that sense I think rsync'ing binary from development box is 
 most reasonable approach in the long term. There is already 
 working dynamic Phobos library for Linux, using it + stripping 
 the binaries shall reduce their sizes quite a lot. Officially 
 shared Phobos is still not very tested but I have not had any 
 issues with it so far.
Yes, after some trying and struggle, I came to the same conclusion. Will try the compression tools mentioned by Teoh :) Thanks, Puming.
Dec 04 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 16:41:14 UTC, Puming wrote:
 Other than diet, is there many other CTFE/template heavy parts?
Also vibe.http.rest and vibe.http.form create really lot of template instances because of heavy usage of reflection + code generation. This is one of domains though that can potentially improve a lot by better compiler implementation,
 I hope there would be a non-CTFE version of diet template, 
 similar to regex/CTRegex. If we use the non-CTFE version in dev 
 mode, we would have a better memory & compile time turnaround 
 in dev mode. And in the release mode the diet templates would 
 be optimized with CTFE. Is that a viable approach?
That should be possible and relatively simple (because of CTFE nature a lot of code can be reused). It may considerably impact Diet performance but I can see desire for that in development mode as project size increases. As usual, it is mostly matter of manpower - vibe.d has only one persistent contributor right now and probably 5-8 guys doing occasional pull request, definitely not enough for project of that scale.
Dec 04 2013
parent reply "Puming" <zhaopuming gmail.com> writes:
On Wednesday, 4 December 2013 at 16:50:45 UTC, Dicebot wrote:
 On Wednesday, 4 December 2013 at 16:41:14 UTC, Puming wrote:
 Other than diet, is there many other CTFE/template heavy parts?
Also vibe.http.rest and vibe.http.form create really lot of template instances because of heavy usage of reflection + code generation. This is one of domains though that can potentially improve a lot by better compiler implementation,
 I hope there would be a non-CTFE version of diet template, 
 similar to regex/CTRegex. If we use the non-CTFE version in 
 dev mode, we would have a better memory & compile time 
 turnaround in dev mode. And in the release mode the diet 
 templates would be optimized with CTFE. Is that a viable 
 approach?
That should be possible and relatively simple (because of CTFE nature a lot of code can be reused). It may considerably impact Diet performance but I can see desire for that in development mode as project size increases. As usual, it is mostly matter of manpower - vibe.d has only one persistent contributor right now and probably 5-8 guys doing occasional pull request, definitely not enough for project of that scale.
There maybe another traditional approach, by converting the .dt diet templates directly into D source code BEFORE the actuall compilation. That is the way many other HTML template systems use, and the performance is fairly good. The generated D code could be very C like, and does not use much CTFE and templates, so that it doesn't incur a lot of compilation resources. As as the matter of manpower, I'm very interested in D/Vibe.d, and will learn and try to contribute :-) I'm currently using Java/Netty&Friends at work. But found D/Vibe.d would be a better solution in the future.
Dec 04 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Thursday, 5 December 2013 at 04:02:58 UTC, Puming wrote:
 There maybe another traditional approach, by converting the .dt 
 diet templates directly into D source code BEFORE the actuall 
 compilation. That is the way many other HTML template systems 
 use, and the performance is fairly good.
It is somewhat complicated by the very tight integration between D handler host and Diet - you can expose D symbols for rendering context to template. It is still possible to do it that way but is likely to require more efforts and effectively will just to the same thing as good JIT-ed CTFE should. I don't like it because it is too effort-demanding for a quick hack and not suitable in the long term by design (one of D goals was to remove reliance on external build steps as much as possible)
Dec 05 2013
parent "Puming" <zhaopuming gmail.com> writes:
On Thursday, 5 December 2013 at 13:49:51 UTC, Dicebot wrote:
 On Thursday, 5 December 2013 at 04:02:58 UTC, Puming wrote:
 There maybe another traditional approach, by converting the 
 .dt diet templates directly into D source code BEFORE the 
 actuall compilation. That is the way many other HTML template 
 systems use, and the performance is fairly good.
It is somewhat complicated by the very tight integration between D handler host and Diet - you can expose D symbols for rendering context to template. It is still possible to do it that way but is likely to require more efforts and effectively will just to the same thing as good JIT-ed CTFE should. I don't like it because it is too effort-demanding for a quick hack and not suitable in the long term by design (one of D goals was to remove reliance on external build steps as much as possible)
Yes, I agree that for dev mode a pure runtime version would fit more and its implementation could take advantage of many functions in the CTFE version (or is that already possible?)
Dec 05 2013
prev sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Dec 04, 2013 at 05:28:06PM +0100, Dicebot wrote:
 While in general one may expect CTFE/template memory consumption to
 drop considerably, in vibe.d case it is unlikely to make that much
 of an impact. vibe.d does really lot of computations during
 compile-time, for example, all Diet templates get loaded and
 processed during that phase. I can't imagine optimizations that will
 make possible to fit compilation of any vibe.d project of notable
 size in 512Mb memory. To be honest, I'd hate to do any DMD + vibe.d
 development on anything less than 4Gb of RAM.
Even without vibe.d, dmd eats way too much memory to be usable on a machine with 512MB RAM. I was quite disappointed to discover this when I tried running dmd on a not-that-old machine with 512MB RAM (aka my office PC), and every time it would soak up all available RAM, force my browser (and probably significant parts of X11) to swap out, making the PC generally unresponsive for several seconds, before the dmd process exits and everything goes back to normal. For all of the good points of D and dmd's compile speeds, the one thing I can't recommend dmd for is efficient memory usage, sad to say. Of course, on a machine with more available RAM, dmd runs at lightning speed, so that at least is a redeeming quality. T -- Guns don't kill people. Bullets do.
Dec 04 2013
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 17:46:40 UTC, H. S. Teoh wrote:
 Even without vibe.d, dmd eats way too much memory to be usable 
 on a
 machine with 512MB RAM. I was quite disappointed to discover 
 this when I
 tried running dmd on a not-that-old machine with 512MB RAM (aka 
 my
 office PC), and every time it would soak up all available RAM, 
 force my
 browser (and probably significant parts of X11) to swap out, 
 making the
 PC generally unresponsive for several seconds, before the dmd 
 process
 exits and everything goes back to normal.
Heh, that was reason why I was forced to reconsider from setting up a CI suite on my 1Gb-worth VPS :) There are some live projects that require 2.5-3 Gb for routine build.
Dec 04 2013
parent reply "Dicebot" <public dicebot.lv> writes:
I want to note though that when it comes to template instance 
bloat which is one of huge memory consumers it is quite possible 
to design compiler in a much more efficient way, possibly with 
minor change to language spec.
Dec 04 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Dec 04, 2013 at 06:55:35PM +0100, Dicebot wrote:
 I want to note though that when it comes to template instance bloat
 which is one of huge memory consumers it is quite possible to design
 compiler in a much more efficient way, possibly with minor change to
 language spec.
I'm pretty sure there's a lot of room for optimization in dmd when it comes to handling templates. If only I had the time to dig into the dmd source code... T -- Why are you blatanly misspelling "blatant"? -- Branden Robinson
Dec 04 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 18:20:36 UTC, H. S. Teoh wrote:
 I'm pretty sure there's a lot of room for optimization in dmd 
 when it
 comes to handling templates. If only I had the time to dig into 
 the dmd
 source code...
I have wasted ~3 months on digging this with no success. And when Walter has finally paid attention to this issue and changed template instance emitting algorithm, I still was hardly able to understand why exactly the changes in pull were necessary to get that result. Just be warned ;)
Dec 04 2013
parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Dec 04, 2013 at 07:22:45PM +0100, Dicebot wrote:
 On Wednesday, 4 December 2013 at 18:20:36 UTC, H. S. Teoh wrote:
I'm pretty sure there's a lot of room for optimization in dmd when it
comes to handling templates. If only I had the time to dig into the
dmd source code...
I have wasted ~3 months on digging this with no success. And when Walter has finally paid attention to this issue and changed template instance emitting algorithm, I still was hardly able to understand why exactly the changes in pull were necessary to get that result. Just be warned ;)
Well, that's why I said I needed more time. :) I'm not expecting it to be a breeze to read through dmd code. I've actually looked into it briefly before when tracking down a compiler bug, but I've to say that in order to understand the code you need to first understand how it works at a high level (yes, it's sorta a chicken-and-egg problem), otherwise you're just looking at the trees and not seeing the forest. T -- What are you when you run out of Monet? Baroque.
Dec 04 2013
parent reply "Dicebot" <public dicebot.lv> writes:
On Wednesday, 4 December 2013 at 18:31:14 UTC, H. S. Teoh wrote:
 ...
Well I have found it to be somewhat reasonable in general but symbol emitting part is one of most arcane ones in whole compiler. Basically only few people know something about it and sometimes it feels like Walter is the only one to really knows it in details.
Dec 04 2013
parent Orvid King <blah38621 gmail.com> writes:
Just to put my 2cents into the conversation, you could also compile
vibe.d as a static library and then link against it, as that seems to
be enough to keep dmd's memory using within functional limits. (I have
a max of 800mb ram that I can easily allow a single program to use,
otherwise I start doing a whole lotta' paging)

On 12/4/13, Dicebot <public dicebot.lv> wrote:
 On Wednesday, 4 December 2013 at 18:31:14 UTC, H. S. Teoh wrote:
 ...
Well I have found it to be somewhat reasonable in general but symbol emitting part is one of most arcane ones in whole compiler. Basically only few people know something about it and sometimes it feels like Walter is the only one to really knows it in details.
Dec 04 2013
prev sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
H. S. Teoh:

 Even without vibe.d, dmd eats way too much memory to be usable 
 on a machine with 512MB RAM.
On Windows32 to compile the whole Phobos DMD takes about 750-850 MB. My D programs are much smaller, so usually I don't need more than 50-300 MB. Bye, bearophile
Dec 04 2013