www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Memory usage of dmd

reply Xavier Bigand <flamaros.xavier gmail.com> writes:
I develop a web site with vibe, but because I am using a Virtual Private 
Server I get some memory issues. The server only has 1Go of memory (> 
900Mo free) and it seems I can't compile directly on it a simple static 
page (70 lines).

I get the following message when building with dub :
Running dmd...
FAIL 
../../../.dub/packages/vibe-d-master/.dub/build/libevent-release-linux.posix-x86_64-dmd_2066-EB47C82EE359A
0A02828E314FCE5409/ 
vibe-d staticLibrary
Error executing command build: Orphan format specifier: %%s failed with 
exit code %s. This may indicate that the process has run out of memory.


So for the moment I build the web site on a physical machine, and I saw 
the compilation takes around 1.6Go of memory.

Is there some options can help me to reduce the memory consumption? As 
it's for production purpose I don't think that is a good idea to remove 
compiler optimizations.

Are gdc or ldc better in this case?
Nov 10 2014
next sibling parent reply Etienne <etcimon gmail.com> writes:
On 2014-11-10 11:32 AM, Xavier Bigand wrote:

 Is there some options can help me to reduce the memory consumption? As
 it's for production purpose I don't think that is a good idea to remove
 compiler optimizations.


The memory issues are probably related to diet templates.

LDC and GDC won't help. You should definitely work and build on a 
machine with 4GB of ram. The server application could use as low as 8mb 
of ram, but compiling requires a workstation.

Perhaps renting an amazon instance a few minutes for compilation would 
be a better idea?
Nov 10 2014
parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 10/11/2014 17:41, Etienne a écrit :

 On 2014-11-10 11:32 AM, Xavier Bigand wrote:

 Is there some options can help me to reduce the memory consumption? As
 it's for production purpose I don't think that is a good idea to remove
 compiler optimizations.


 The memory issues are probably related to diet templates.


Yes I think it too.


 LDC and GDC won't help. You should definitely work and build on a
 machine with 4GB of ram. The server application could use as low as 8mb
 of ram, but compiling requires a workstation.

 Perhaps renting an amazon instance a few minutes for compilation would
 be a better idea?


I already have a computer with a linux to build it, so amazon won't 
improve situation.
As I know to be able to have no down time with vibe we need to be able 
to build directly on the server where the program runs.

Maybe I just need to wait that I have some users to pay a better server 
with more memory.
Nov 10 2014
parent reply Etienne <etcimon gmail.com> writes:
On 2014-11-10 12:02 PM, Xavier Bigand wrote:

 As I know to be able to have no down time with vibe we need to be able
 to build directly on the server where the program runs.

 Maybe I just need to wait that I have some users to pay a better server
 with more memory.


With a low number of users, there's no reason to worry about a 1 second 
downtime from closing the process and replacing the application file. 
You should use a bash script to keep the process open though:


nohup ./autostart.sh > stdout.log 2> crash.log >/dev/null &


while true ; do
   if ! pgrep -f '{processname}' > /dev/null ; then
     sh /home/{mysitefolder}/start.sh
   fi
   sleep 1
done


nohup ./{yourapplication} --uid={user} --gid={group} >> stdout.log 2>> 
crash.log >> stdout.log &


pkill -f '{processname}'
/bin/cp -rf {yourapplication} /home/{mysitefolder}/


Using a console, run monitor.sh and the autostart.sh script will 
re-launch your server through start.sh into a daemon. Verifications will 
be made every second to ensure your server is never down because of a 
badly placed assert.

If you need to replace your server application with an update, run the 
install.sh script from the folder where the update is.
Nov 10 2014
next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 10/11/2014 18:17, Etienne a écrit :

 On 2014-11-10 12:02 PM, Xavier Bigand wrote:

 As I know to be able to have no down time with vibe we need to be able
 to build directly on the server where the program runs.

 Maybe I just need to wait that I have some users to pay a better server
 with more memory.


 With a low number of users, there's no reason to worry about a 1 second
 downtime from closing the process and replacing the application file.
 You should use a bash script to keep the process open though:


 nohup ./autostart.sh > stdout.log 2> crash.log >/dev/null &


 while true ; do
    if ! pgrep -f '{processname}' > /dev/null ; then
      sh /home/{mysitefolder}/start.sh
    fi
    sleep 1
 done


 nohup ./{yourapplication} --uid={user} --gid={group} >> stdout.log 2>>
 crash.log >> stdout.log &


 pkill -f '{processname}'
 /bin/cp -rf {yourapplication} /home/{mysitefolder}/


 Using a console, run monitor.sh and the autostart.sh script will
 re-launch your server through start.sh into a daemon. Verifications will
 be made every second to ensure your server is never down because of a
 badly placed assert.

 If you need to replace your server application with an update, run the
 install.sh script from the folder where the update is.


Thank you for tips. I'll start from your scripts.
Nov 10 2014
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
If your server runs systemd, I would strongly recommend to use 
that instead of a shell script. You can use "Restart=always" or 
"Restart=on-failure" in the unit file. It also provides socket 
activation, which will allow you to restart the program without 
downtime.
Nov 10 2014
next sibling parent Etienne <etcimon gmail.com> writes:
On 2014-11-10 2:52 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
<schuetzm gmx.net>" wrote:

 If your server runs systemd, I would strongly recommend to use that
 instead of a shell script. You can use "Restart=always" or
 "Restart=on-failure" in the unit file. It also provides socket
 activation, which will allow you to restart the program without downtime.


I totally agree, I couldn't find a good resource about this though. The 
shell script method is quite literally rudimentary but it took me a few 
minutes to put together and it's been working great during 6 months so 
far. I'll go out and read a systemd tutorial eventually to get this done 
the right way, if anyone had anything ready and compatible with vibe.d 
apps I'd be happy about it.

For the install script, I guess the best way would be to put together an 
RPM script and upgrade through that. I'd have to explore that solution, 
it's quite a lot more than 2 lines of code though :)
Nov 10 2014
prev sibling parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 10/11/2014 20:52, "Marc Schütz" <schuetzm gmx.net>" a écrit :

 If your server runs systemd, I would strongly recommend to use that
 instead of a shell script. You can use "Restart=always" or
 "Restart=on-failure" in the unit file. It also provides socket
 activation, which will allow you to restart the program without downtime.


Good to know, I wasn't systemd can do things like that.
Nov 12 2014
parent "olivier henley" <olivier.henley ubisoft.com> writes:
All this is very interesting.

Thx for the scripts, thx for pointing systemd.

We need to document that stuff somewhere... more indexable.
Nov 12 2014
prev sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Xavier Bigand:


 So for the moment I build the web site on a physical machine, 
 and I saw the compilation takes around 1.6Go of memory.


Compiling the whole Phobos as a single compilation unit on 32 bit 
DMD requires a little more than 1 GB.

Bye,
bearophile
Nov 10 2014