www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - How to release memory? (D2.0.30)

reply AxelS <a_bothe gmx.net> writes:
Hello everyone,
I've got a problem with the following (very simple) code:

void foo()
{
      void[] dat=new void[50_000_000]; // allocate 50 MByte of dummy-data
      delete dat;
}

after I called foo() and watched the memory usage in the windows taskmanager,
the program blowed up to 50 MBytes although I deleted the allocated memory...

Why can't the GC remove that data and how CAN I remove it?

Thanks in advance!
Jul 03 2009
next sibling parent reply BCS <ao pathlink.com> writes:
Reply to AxelS,


 Hello everyone,
 I've got a problem with the following (very simple) code:
 void foo()
 {
 void[] dat=new void[50_000_000]; // allocate 50 MByte of
 dummy-data
 delete dat;
 }
 after I called foo() and watched the memory usage in the windows
 taskmanager, the program blowed up to 50 MBytes although I deleted the
 allocated memory...
 
 Why can't the GC remove that data and how CAN I remove it?
 
 Thanks in advance!
 


You can't. The D runtime (and most other runtimes) don't ever reduce the 
amount of memory they keep in the heap. If you where to allocate another 
25MB right after that function you would see no change in the memory usage. 
The good news is that with virtual memory, all of that has almost zero cost. 
What matters is how much ram you are actively using.
Jul 03 2009
next sibling parent reply AxelS <a_bothe gmx.net> writes:
BCS Wrote:


 You can't. The D runtime (and most other runtimes) don't ever reduce the 
 amount of memory they keep in the heap. If you where to allocate another 
 25MB right after that function you would see no change in the memory usage. 
 The good news is that with virtual memory, all of that has almost zero cost. 
 What matters is how much ram you are actively using.
 
 


I want to load and send a file via network...when I load the entire file into
memory it's very stupid that I can't release that memory again...

OK I'll try it with the C API but thanks for your help!
Jul 04 2009
next sibling parent reply Ary Borenszweig <ary esperanto.org.ar> writes:
AxelS escribió:

 BCS Wrote:
 

 You can't. The D runtime (and most other runtimes) don't ever reduce the 
 amount of memory they keep in the heap. If you where to allocate another 
 25MB right after that function you would see no change in the memory usage. 
 The good news is that with virtual memory, all of that has almost zero cost. 
 What matters is how much ram you are actively using.


 
 I want to load and send a file via network...when I load the entire file into
memory it's very stupid that I can't release that memory again...


Why not pipe it?
Jul 04 2009
parent reply AxelS <a_bothe gmx.net> writes:
 Ary Borenszweig: Good idea but I can't pipe data to a HTTP-server located
somewhere in the internet...


OK, I tried it with C's malloc and free - but everytime I access my array, D
puts the memory into its heap...I'm getting crazy because of this!

ubyte[] data=cast(ubyte[])malloc(50_000_000)[0 .. 50_000_000];

foreach(ref d;data)
{
   d=4; // Simulate data access
}

free(data.ptr);

Can't the D compiler handle such problems in future versions?
Of course it's right to release unused memory...but not just release it into
the program-internal heap...
Jul 04 2009
parent Daniel Keep <daniel.keep.lists gmail.com> writes:
AxelS wrote:

  Ary Borenszweig: Good idea but I can't pipe data to a HTTP-server located
somewhere in the internet...


I believe he means to read the file in chunks, sending them across the
network as you get them.


 OK, I tried it with C's malloc and free - but everytime I access my array, D
puts the memory into its heap...I'm getting crazy because of this!


D doesn't copy data like that; something else must be going on.
Jul 04 2009
prev sibling parent reply BCS <none anon.com> writes:
Hello AxelS,


 BCS Wrote:
 

 You can't. The D runtime (and most other runtimes) don't ever reduce
 the amount of memory they keep in the heap. If you where to allocate
 another 25MB right after that function you would see no change in the
 memory usage. The good news is that with virtual memory, all of that
 has almost zero cost. What matters is how much ram you are actively
 using.
 


 I want to load and send a file via network...when I load the entire
 file into memory it's very stupid that I can't release that memory
 again...
 
 OK I'll try it with the C API but thanks for your help!
 


C's runtime (malloc/free) doesn't return memory to the OS either. Unless 
your system starts complaining about wanting to make the swap file bigger, 
quit worrying about it.

OTOH another way to work this is to memory map the file and unmap it when 
you are done. That way the original file is used as the backing store rather 
than the swap file and the memory gets unmapped when you dump the file.
Jul 04 2009
parent reply downs <default_357-line yahoo.de> writes:
BCS wrote:

 Hello AxelS,
 

 BCS Wrote:


 You can't. The D runtime (and most other runtimes) don't ever reduce
 the amount of memory they keep in the heap. If you where to allocate
 another 25MB right after that function you would see no change in the
 memory usage. The good news is that with virtual memory, all of that
 has almost zero cost. What matters is how much ram you are actively
 using.


 I want to load and send a file via network...when I load the entire
 file into memory it's very stupid that I can't release that memory
 again...

 OK I'll try it with the C API but thanks for your help!


 
 C's runtime (malloc/free) doesn't return memory to the OS either.


Um .. yes it does. :)

#include <stdio.h>
#include <stdlib.h>

int main() {
  void *p = calloc(1024*1024*256, 1);
  system("sleep 10");
  free(p);
  system("sleep 10");
  return 0;
}

Run that, wait 10s, and watch the memory usage go down.
Jul 05 2009
next sibling parent reply AxelS <a_bothe gmx.net> writes:
 downs:

That's what I even had before....just allocate and release memory is not
difficult - just if you want to access the data the GC copies all the memory
into its heap...

I found a new way which is really good and easy:
Just use the Win32 Memory API for allocating and releasing your data
with functions like GlobalAlloc or GlobalFree

http://msdn.microsoft.com/en-us/library/ms810603.aspx
Jul 05 2009
parent Tom S <h3r3tic remove.mat.uni.torun.pl> writes:
AxelS wrote:

  downs:
 
 That's what I even had before....just allocate and release memory is not
difficult - just if you want to access the data the GC copies all the memory
into its heap...


Can you recommend a good crack dealer? ;)

Seriously, there are no GC calls associated with memory accesses. 
Perhaps malloc causes the OS to assign some virtual memory to the 
process, but only accessing it causes a page fault which the OS handles 
by committing physical memory.


-- 
Tomasz Stachowiak
http://h3.team0xf.com/
h3/h3r3tic on #D freenode
Jul 05 2009
prev sibling parent reply =?ISO-8859-1?Q?=22J=E9r=F4me_M=2E_Berger=22?= <jeberger free.fr> writes:
downs wrote:

 BCS wrote:

 Hello AxelS,


 BCS Wrote:


 You can't. The D runtime (and most other runtimes) don't ever reduce=









 the amount of memory they keep in the heap. If you where to allocate=









 another 25MB right after that function you would see no change in th=








e

 memory usage. The good news is that with virtual memory, all of that=









 has almost zero cost. What matters is how much ram you are actively
 using.


 I want to load and send a file via network...when I load the entire
 file into memory it's very stupid that I can't release that memory
 again...

 OK I'll try it with the C API but thanks for your help!


 C's runtime (malloc/free) doesn't return memory to the OS either.


=20
 Um .. yes it does. :)
=20


	No it doesn't (not always). Depending on your C runtime and the=20
size of the allocations, memory may or may not be returned to the OS.

	For example, on Unix systems, there are two system calls to get=20
memory from the OS: (s)brk and mmap. AFAIK, there is no way to=20
return memory allocated through (s)brk. Those calls are only able to=20
allocate blocs of a given size (the OS page size) and the runtime=20
manages smaller and larger sizes to balance speed, and efficiency.=20
Some C runtimes will allocate small blocs through (s)brk and large=20
blocs through mmap, some will allocate everything with (s)brk, some=20
will use mmap when the requested size is a multiple of the OS page=20
size and everything else through (s)brk. Depending on the scheme=20
used, the runtime may not be able to return the memory to the OS.

	Like BCS said, the only way to make sure that the memory will be=20
returned is to use mmap/munmap directly (or their equivalent on your=20
platform).

		Jerome
--=20
mailto:jeberger free.fr
http://jeberger.free.fr
Jabber: jeberger jabber.fr
Jul 05 2009
parent BCS <none anon.com> writes:
Hello Jérôme,


 Like BCS said, the only way to make sure that the memory will be
 returned is to use mmap/munmap directly (or their equivalent on your
 platform).


That maybe true, but it wasn't my point. What I was trying to get at was 
that if you want to load a file into a buffer then asking mmap to do it removes 
the need to allocate a buffer from the language runtime.
Jul 05 2009
prev sibling parent reply Daniel Keep <daniel.keep.lists gmail.com> writes:
BCS wrote:

 ... The good news is that with virtual memory, all of that has almost
 zero cost. What matters is how much ram you are actively using.


You've obviously never used a netbook with no swap file.  :)
Jul 04 2009
next sibling parent BCS <none anon.com> writes:
Hello Daniel,


 BCS wrote:
 

 ... The good news is that with virtual memory, all of that has almost
 zero cost. What matters is how much ram you are actively using.
 


 You've obviously never used a netbook with no swap file.  :)
 


Nope, my netbook has a swap file, and I've never heard of one befor (you 
would be better referring to embedded systems). And even if it didn't, unless 
you use that whole array, the system doesn't need to actually map ram for 
what isn't used.

As it happens, I ran across this discussion on swapfiles a few days back.

http://blog.stackoverflow.com/2009/06/podcast-59/
http://serverfault.com/questions/23621/any-benefit-or-detriment-from-removing-a-pagefile-on-an-8gb-ram-machine
Jul 04 2009
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 04 Jul 2009 05:11:39 -0400, Daniel Keep  
<daniel.keep.lists gmail.com> wrote:


 BCS wrote:

 ... The good news is that with virtual memory, all of that has almost
 zero cost. What matters is how much ram you are actively using.


 You've obviously never used a netbook with no swap file.  :)


Swap files are needed for hibernation, probably a very highly used feature  
for netbooks...

-Steve
Jul 07 2009
parent BCS <ao pathlink.com> writes:
Reply to Steven,


 On Sat, 04 Jul 2009 05:11:39 -0400, Daniel Keep
 <daniel.keep.lists gmail.com> wrote:
 

 BCS wrote:
 

 ... The good news is that with virtual memory, all of that has
 almost zero cost. What matters is how much ram you are actively
 using.
 


 You've obviously never used a netbook with no swap file.  :)
 


 Swap files are needed for hibernation, probably a very highly used
 feature  for netbooks...
 
 -Steve
 


Last I checked, windows doesn't use the swap file to hibernate (it seems 
to have to create a new file when you turn on the feature). I haven't a clue 
why it doesn't but that seems to be the case.
Jul 07 2009
prev sibling next sibling parent Jarrett Billingsley <jarrett.billingsley gmail.com> writes:
On Fri, Jul 3, 2009 at 4:30 PM, AxelS<a_bothe gmx.net> wrote:

 Hello everyone,
 I've got a problem with the following (very simple) code:

 void foo()
 {
 =A0 =A0 =A0void[] dat=3Dnew void[50_000_000]; // allocate 50 MByte of dum=


my-data

 =A0 =A0 =A0delete dat;
 }

 after I called foo() and watched the memory usage in the windows taskmana=


ger, the program blowed up to 50 MBytes although I deleted the allocated me=
mory...

 Why can't the GC remove that data and how CAN I remove it?

 Thanks in advance!


If you're using Tango or D2, you can use the GC.minimize() function
(in tango.core.Memory or core.memory) to release memory back to the
OS, but as BCS said, the amount of virtual memory is not really a
useful indicator anyway.
Jul 03 2009
prev sibling parent bearophile <bearophileHUGS lycos.com> writes:
AxelS:

       void[] dat=new void[50_000_000]; // allocate 50 MByte of dummy-data


I am not sure, but the GC may scan that chunk of voids. If you need a raw block
it may be better to work with an array of uint.



 Why can't the GC remove that data and how CAN I remove it?


If you really need to remove it, then you may not allocate it from the GC heap
in the first place, and use the C heap, with a malloc and then free. But that
requires a full manual management, and for safety it's then usually much better
to put in such memory chunk only data that doesn't contain references managed
by GC.

Bye,
bearophile
Jul 03 2009