www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Memory detection

reply Charles Hixson <charleshixsn earthlink.net> writes:
Is it possible to detect when a program is using, say, 90% of the memory 
that is available, so that I can take steps to reduce usage?  I've got a 
method that would like to use probably more memory than is available, 
but which can (if it could tell) save stuff to files to free memory it 
isn't using at the moment.  But when I looked in 
http://dlang.org/phobos/core_memory.html all the methods seemed to have 
to do with garbage collection, which isn't quite the same thing.  I 
don't want the stuff to be garbage collected until after I've saved it's 
current state, but to know when to do that, I need to know how much 
memory is currently free.

I suppose that I could set an arbitrary limit on the number of various 
things I allow before compacting, but that's a bit ... arbitrary.  And 
doesn't adapt to different machines without recompiling.
Aug 23 2012
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Charles Hixson:


 Is it possible to detect when a program is using, say, 90% of 
 the memory that is available, so that I can take steps to 
 reduce usage?


With computers that have virtual memory this is not so easy to 
do. I think you have to use operating system-specific code to ask 
the OS about the physical available memory, about the virtual 
memory used, the rate of virtual memory swapping, and use those 
three values in some way.

Bye,
bearophile
Aug 23 2012
parent reply 1100110 <10equals2 gmail.com> writes:
On Thu, 23 Aug 2012 18:37:59 -0500, bearophile <bearophileHUGS lycos.com>  
wrote:


 Charles Hixson:


 Is it possible to detect when a program is using, say, 90% of the  
 memory that is available, so that I can take steps to reduce usage?


 With computers that have virtual memory this is not so easy to do. I  
 think you have to use operating system-specific code to ask the OS about  
 the physical available memory, about the virtual memory used, the rate  
 of virtual memory swapping, and use those three values in some way.

 Bye,
 bearophile


On linux this is not so difficult to do.

Those values are generally in /proc, and it seems to be portable across  
pretty much every distro with a relatively recent kernel.

I have an extremely half-assed bit of code that prints the load average  
and the totaly % of mem used to my tmux session.
It gives the exact same values that are seen in top, or htop.(without the  
overhead of parsing their output, cause that takes ~500ms, way too slow.)
It would make a decent starting point at least.

I would imagine that you wouldn't even need to know the % of memory YOUR  
program is using, just the general percentage overall.

After all, no matter who is eating all the memory, shit's about to hit the  
fan if *someone* doesn't free some memory.
-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/
Aug 23 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 24-Aug-12 07:03, 1100110 wrote:

 On linux this is not so difficult to do.

 Those values are generally in /proc, and it seems to be portable across
 pretty much every distro with a relatively recent kernel.

 I have an extremely half-assed bit of code that prints the load average
 and the totaly % of mem used to my tmux session.
 It gives the exact same values that are seen in top, or htop.(without
 the overhead of parsing their output, cause that takes ~500ms, way too
 slow.)


If parsing takes 500ms then something is seriously wrong. What is size 
of the input to parse and the machine specs?


-- 
Olshansky Dmitry
Aug 24 2012
parent reply 1100110 <10equals2 gmail.com> writes:
On Fri, 24 Aug 2012 15:42:21 -0500, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:


 On 24-Aug-12 07:03, 1100110 wrote:

 On linux this is not so difficult to do.

 Those values are generally in /proc, and it seems to be portable across
 pretty much every distro with a relatively recent kernel.

 I have an extremely half-assed bit of code that prints the load average
 and the totaly % of mem used to my tmux session.
 It gives the exact same values that are seen in top, or htop.(without
 the overhead of parsing their output, cause that takes ~500ms, way too
 slow.)


 If parsing takes 500ms then something is seriously wrong. What is size  
 of the input to parse and the machine specs?


No, you misunderstand.

Parsing the output of the `top` command takes ~500ms.

The snippet of code that reads /proc directly takes an order of magnitude  
less time.

I assume that top reads several times, and also reads the percentages of  
each process that is running, while I only
read the load average and total memory consumption.

The vast majority of the time spent parsing `top` was spent waiting for it  
to initialize, and print the values to be read.

TL;DR
Parsing was probably the wrong word to use.

Reading from /proc directly will take an order of magnitude less time.


-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/
Aug 24 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 25-Aug-12 00:52, 1100110 wrote:

 On Fri, 24 Aug 2012 15:42:21 -0500, Dmitry Olshansky
 <dmitry.olsh gmail.com> wrote:


 On 24-Aug-12 07:03, 1100110 wrote:

 On linux this is not so difficult to do.

 Those values are generally in /proc, and it seems to be portable across
 pretty much every distro with a relatively recent kernel.

 I have an extremely half-assed bit of code that prints the load average
 and the totaly % of mem used to my tmux session.
 It gives the exact same values that are seen in top, or htop.(without
 the overhead of parsing their output, cause that takes ~500ms, way too
 slow.)


 If parsing takes 500ms then something is seriously wrong. What is size
 of the input to parse and the machine specs?


 No, you misunderstand.

 Parsing the output of the `top` command takes ~500ms.


Then it's not parsing but the it's time to spawn process that does a 
fuckton of syscalls and read its output through the pipe :)
I just cringed at:
"the overhead of parsing their output, cause that takes ~500ms"


 The snippet of code that reads /proc directly takes an order of
 magnitude less time.

 I assume that top reads several times, and also reads the percentages of
 each process that is running, while I only
 read the load average and total memory consumption.

 The vast majority of the time spent parsing `top` was spent waiting for
 it to initialize, and print the values to be read.

 TL;DR
 Parsing was probably the wrong word to use.


Yes, sorry for being nit-picky.


 Reading from /proc directly will take an order of magnitude less time.



And yes.

-- 
Olshansky Dmitry
Aug 24 2012
parent 1100110 <10equals2 gmail.com> writes:
On Fri, 24 Aug 2012 16:05:28 -0500, Dmitry Olshansky  
<dmitry.olsh gmail.com> wrote:


 On 25-Aug-12 00:52, 1100110 wrote:

 On Fri, 24 Aug 2012 15:42:21 -0500, Dmitry Olshansky
 <dmitry.olsh gmail.com> wrote:


 On 24-Aug-12 07:03, 1100110 wrote:

 On linux this is not so difficult to do.

 Those values are generally in /proc, and it seems to be portable  
 across
 pretty much every distro with a relatively recent kernel.

 I have an extremely half-assed bit of code that prints the load  
 average
 and the totaly % of mem used to my tmux session.
 It gives the exact same values that are seen in top, or htop.(without
 the overhead of parsing their output, cause that takes ~500ms, way too
 slow.)


 If parsing takes 500ms then something is seriously wrong. What is size
 of the input to parse and the machine specs?


 No, you misunderstand.

 Parsing the output of the `top` command takes ~500ms.


 Then it's not parsing but the it's time to spawn process that does a  
 fuckton of syscalls and read its output through the pipe :)
 I just cringed at:
 "the overhead of parsing their output, cause that takes ~500ms"


 The snippet of code that reads /proc directly takes an order of
 magnitude less time.

 I assume that top reads several times, and also reads the percentages of
 each process that is running, while I only
 read the load average and total memory consumption.

 The vast majority of the time spent parsing `top` was spent waiting for
 it to initialize, and print the values to be read.

 TL;DR
 Parsing was probably the wrong word to use.


 Yes, sorry for being nit-picky.


 Reading from /proc directly will take an order of magnitude less time.



 And yes.


     size_t memTotal, memUsed;
     Stream memInfo  = new BufferedFile("/proc/meminfo");
     auto t          = memInfo.readLine();
     memTotal        = to!size_t(strip(t[9..$-3]));
     memUsed         = memTotal;

     auto f  = memInfo.readLine();
     memUsed -= to!size_t(strip(f[8..$-3]));

     auto c = memInfo.readLine();
     memUsed -= to!size_t(strip(c[8..$-3]));

     auto b = memInfo.readLine();
     memUsed -= to!size_t(strip(b[8..$-3]));

     memInfo.close();

Fair warning, I wrote this over the course of 5 minutes by directly  
translating some public domain code.
It works, so I've never bothered to clean it up.

But that'll give you the total memory used, and the total amount of memory.
-- 
Using Opera's revolutionary email client: http://www.opera.com/mail/
Aug 24 2012