www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Heap memory limit?

reply bearophile <bearophileHUGS lycos.com> writes:
The Java VM has a limit of the amount of memory it allocates from the heap. So
is it useful to add a way to specify a similar limit for D2 programs
(especially ones made by safe module)? This limit can be given to the D GC heap
(and maybe to the C heap too).

This limit can just be a compile-time constant given to the compiler, this
produces a binary that has a limit of 200 MB of heap:
dmd -mlimit=200mb foo.d
The situation gets less simple if you want to change such limit after the
program is already compiled, I don't know of simple ways to do it. Maybe a
second program can be used for that:
gclimiter -mlimit=100mb foo.exe

Bye,
bearophile
Feb 23 2010
next sibling parent reply "Lars T. Kyllingstad" <public kyllingen.NOSPAMnet> writes:
bearophile wrote:

 The Java VM has a limit of the amount of memory it allocates from the heap. So
is it useful to add a way to specify a similar limit for D2 programs
(especially ones made by safe module)? This limit can be given to the D GC heap
(and maybe to the C heap too).
 
 This limit can just be a compile-time constant given to the compiler, this
produces a binary that has a limit of 200 MB of heap:
 dmd -mlimit=200mb foo.d
 The situation gets less simple if you want to change such limit after the
program is already compiled, I don't know of simple ways to do it. Maybe a
second program can be used for that:
 gclimiter -mlimit=100mb foo.exe


On Linux you can use the shell's built-in 'ulimit' command to set such a 
limit:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


-Lars
Feb 23 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Lars T. Kyllingstad:

 On Linux you can use the shell's built-in 'ulimit' command to set such a 
 limit


My idea was the opposite (and similar to how Java works on Windows): to set a
default limit that's always present unless you specify you want more memory. As
usual, if using too much memory is seen as a danger (and it can make a system
unstable, so I think it's a small safety thing, that's probably why Java has
such limit), then to avoid such danger you have make the default behaviour the
safer one.

Bye,
bearophile
Feb 23 2010
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 23 Feb 2010 08:36:04 -0500, bearophile <bearophileHUGS lycos.com>  
wrote:


 The Java VM has a limit of the amount of memory it allocates from the  
 heap. So is it useful to add a way to specify a similar limit for D2  
 programs (especially ones made by safe module)? This limit can be given  
 to the D GC heap (and maybe to the C heap too).

 This limit can just be a compile-time constant given to the compiler,  
 this produces a binary that has a limit of 200 MB of heap:
 dmd -mlimit=200mb foo.d
 The situation gets less simple if you want to change such limit after  
 the program is already compiled, I don't know of simple ways to do it.  
 Maybe a second program can be used for that:
 gclimiter -mlimit=100mb foo.exe


The GC is open source.  Submit a patch if you want.  DMD is not in charge  
of the GC, it is runtime code, I don't think we need a specific option for  
it.  I think it can be runtime-decided, and the option passed via your  
program initialization.  I.e.:

void main()
{
GC.memlimit = 200_000_000; // 200 MB
...
}

I also don't think that limits to memory are too important for most  
applications, only for specific ones.


-Steve
Feb 23 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
Steven Schveighoffer:


 The GC is open source.  Submit a patch if you want.


It can be nice if the the C heap too is counted in such max memory used. I
don't know if this is possible.



DMD is not in charge of the GC, it is runtime code, I don't think we need a
specific option for it.<


- D2 code has safe modules too, they are meant to turn D code into something a
little safer than usual C code.
- I think using too much memory is a unsafe situation, a small risk. You are
free to not agree.
- If it's seen as a real small risk, then if you want to do something concrete
to avoid this risk, you need such memory limit enforced by default (if it's not
a default, very few people will use it, and this small risk will be often
ignored). And it can be positive if all D implementations share have this
safety. That's why I was talking about a compiler argument (plus eventually a
small program to change at runtime the limit of an already compiled program.
Another silly possible solution is to use  small program to modify the max
value contained into the binary itself, a kind of binary patching).
- If you want the original behaviour you can compile with something like:
dmd -mlimit=none foo.d
This disables such limiting (it disables the run-time tests too, so the
allocations become a bit faster too).



 I also don't think that limits to memory are too important for most  
 applications, only for specific ones.


It's not just my desire to copy Java, I have made my PC unstable few times
because a bug in my code has caused a D proggy to eat too much memory, so
having a default limit of memory used seems useful to me.

Bye,
bearophile
Feb 23 2010
parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 23 Feb 2010 09:34:19 -0500, bearophile <bearophileHUGS lycos.com>  
wrote:


 Steven Schveighoffer:


 The GC is open source.  Submit a patch if you want.


 It can be nice if the the C heap too is counted in such max memory used.  
 I don't know if this is possible.


I don't know if this is possible.  D would have to replace C's malloc in  
order to do this, but I don't think that's a good idea.  Also, however,  
C's malloc isn't as likely to explode memory usage because items should be  
freed when they are no longer used.  GC-based languages tend to use more  
memory because there are at any one time a lot of unused pieces of memory  
waiting to be cleaned.


 DMD is not in charge of the GC, it is runtime code, I don't think we  
 need a specific option for it.<


 - D2 code has safe modules too, they are meant to turn D code into  
 something a little safer than usual C code.
 - I think using too much memory is a unsafe situation, a small risk. You  
 are free to not agree.


It is a risk, but it is entirely dependent on factors outside the  
program's control.  For example, how much memory is available.  If you  
limit to 200MB, what if you have to run 50 programs that were built from D  
with the same default limit, that's 10GB limit.  I just don't think it's  
realistic to come up with a simple per-program limit that solves the  
problem.


 - If it's seen as a real small risk, then if you want to do something  
 concrete to avoid this risk, you need such memory limit enforced by  
 default (if it's not a default, very few people will use it, and this  
 small risk will be often ignored). And it can be positive if all D  
 implementations share have this safety. That's why I was talking about a  
 compiler argument (plus eventually a small program to change at runtime  
 the limit of an already compiled program. Another silly possible  
 solution is to use  small program to modify the max value contained into  
 the binary itself, a kind of binary patching).


Again, this can be done in the GC code itself, it does not need to be a  
compiler-defined limit.  My preference is for the default to be unlimited,  
since this is the most common case.


 I also don't think that limits to memory are too important for most
 applications, only for specific ones.


 It's not just my desire to copy Java, I have made my PC unstable few  
 times because a bug in my code has caused a D proggy to eat too much  
 memory, so having a default limit of memory used seems useful to me.


This isn't a common case though.  An artificial limit that has no  
consideration of available memory seems like it would lead to more  
puzzlement than help.

-Steve
Feb 23 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Steven Schveighoffer:


Also, however, C's malloc isn't as likely to explode memory usage because items
should be freed when they are no longer used.<


Of course , if we assume perfect programmers, then there's no need of many
other kinds of safeties present in D code.



 Again, this can be done in the GC code itself, it does not need to be a  
 compiler-defined limit.  My preference is for the default to be unlimited,  
 since this is the most common case.


This is wrong; in the most common case you don't want a program to trash your
virtual memory and crash your other programs. Only for special programs you
want no limits.


ones created on dotnet and JVM) are designed the other way around compared to
D: they start from a safe situation and then add (usually safe) optimizations
to try to become faster. Because starting from C you have no hope of improving
certain things (some of such things are limits in the brain of people used to
C).

Thank you for your answers and explanations, I can see there's no use of the
limit I was talking about. Better to put an external limit to the program, as
Lars Kyllingstad has shown, or maybe to add a memory limit to the GC, that can
defined at run-time.

--------------

Justin Johansson:


Can you please explain why you think this is useful?<


To make D programs a bit safer. On Windows if you have 5 programs running, and
one of them eats too much RAM, the system starts swapping, everything starts
running very slowly, some other programs crash, etc. This is not nice. So if I
write a small buggy program to process a large amount of genomic data I'd like
a bug in my code to avoid that out of memory situation. But Steven
Schveighoffer has just explained why my idea can't work in D. So sorry for all
the noise :-)



Assuming the idea is useful, why would you have to be a compiler option rather
than, say, a linker option?<


I don't care of linkers. They are an implementation detail that in a modern
language I want to be able to ignore in most times, unless I'm doing something
special (and the situation I am discussing here is not special at all, I think
of it as the default one). To use a car designed in 2008 I don't want to do all
the things you have to do to start a car designed in 1940 :-) I want it to work
in a simpler way, even if inside it's probably more complex.

Bye,
bearophile
Feb 23 2010
prev sibling next sibling parent Justin Johansson <no spam.com> writes:
Hi bearophile,

bearophile wrote:

 The Java VM has a limit of the amount of memory it allocates from the heap. So
is it useful to add a way to specify a similar limit for D2 programs
(especially ones made by safe module)? This limit can be given to the D GC heap
(and maybe to the C heap too).


Can you please explain why you think this is useful?


 This limit can just be a compile-time constant given to the compiler, this
produces a binary that has a limit of 200 MB of heap:
 dmd -mlimit=200mb foo.d
 The situation gets less simple if you want to change such limit after the
program is already compiled, I don't know of simple ways to do it. Maybe a
second program can be used for that:
 gclimiter -mlimit=100mb foo.exe


Assuming the idea is useful, why would you have to be a compiler option
rather than, say, a linker option?

Regards,
Justin
Feb 23 2010
prev sibling parent reply "Robert Jacques" <sandford jhu.edu> writes:
On Tue, 23 Feb 2010 08:36:04 -0500, bearophile <bearophileHUGS lycos.com>  
wrote:

 The Java VM has a limit of the amount of memory it allocates from the  
 heap. So is it useful to add a way to specify a similar limit for D2  
 programs (especially ones made by safe module)? This limit can be given  
 to the D GC heap (and maybe to the C heap too).

 This limit can just be a compile-time constant given to the compiler,  
 this produces a binary that has a limit of 200 MB of heap:
 dmd -mlimit=200mb foo.d
 The situation gets less simple if you want to change such limit after  
 the program is already compiled, I don't know of simple ways to do it.  
 Maybe a second program can be used for that:
 gclimiter -mlimit=100mb foo.exe

 Bye,
 bearophile


My gut feeling is that you are confusing free heap memory with total heap  
memory, because I know of lots of Java programs where I seamlessly  
allocate more than 200mb on a regular basis. I mean, consider a user with  
gigs of memory free suddenly getting an out of memory error. Anyways, as  
others have mentioned, this seems like a fairly trivial addition to  
druntime's GC.
Feb 23 2010
parent bearophile <bearophileHUGS lycos.com> writes:
Robert Jacques:


Anyways, as others have mentioned, this seems like a fairly trivial addition to
druntime's GC.<


OK, and thank you.

Bye,
bearophile
Feb 23 2010