www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - I want my Memory back ;-)

reply ABothe <info alexanderbothe.com> writes:
Hi everyone,

I don't know if it's a bug or not .. but D or at least its GC doesn't want to
free my allocated stuff! Also when I 'disable' the GC via GC.disable(); it
won't free it :-/

byte[] asdf=new byte[50_000_000]; // OK a bit much---but it's an example value

GC.free(&asdf); // It doesn't work!!
delete asdf; // Also, without GC.free, doesn't work!!!

So...please give me a solution for this, D people :-D


--Alex
Jul 12 2010
next sibling parent dsimcha <dsimcha yahoo.com> writes:
== Quote from ABothe (info alexanderbothe.com)'s article

 Hi everyone,
 I don't know if it's a bug or not .. but D or at least its GC doesn't want to
 free my allocated stuff! Also when I 'disable' the GC via GC.disable(); it
 won't free it :-/
 byte[] asdf=new byte[50_000_000]; // OK a bit much---but it's an example value
 GC.free(&asdf); // It doesn't work!!
 delete asdf; // Also, without GC.free, doesn't work!!!
 So...please give me a solution for this, D people :-D
 --Alex


GC.free(cast(void*) asdf.ptr);
Jul 12 2010
prev sibling next sibling parent reply Leandro Lucarella <luca llucax.com.ar> writes:
ABothe, el 12 de julio a las 23:44 me escribiste:

 Hi everyone,
 
 I don't know if it's a bug or not .. but D or at least its GC doesn't want to
 free my allocated stuff! Also when I 'disable' the GC via GC.disable(); it
 won't free it :-/


Disable that exactly that, force the GC *not* to run a collection.


 byte[] asdf=new byte[50_000_000]; // OK a bit much---but it's an example value


This is a clasical problem of the conservative GC. The problem is, false
pointers would keep your large chunks of memory alive (the bigger the
chunk, the higher the probability of a random word being confused as
a pointer into it).

There were some solutions proposed, but none accepted yet:
http://d.puremagic.com/issues/show_bug.cgi?id=2927
http://d.puremagic.com/issues/show_bug.cgi?id=3463


 GC.free(&asdf); // It doesn't work!!
 delete asdf; // Also, without GC.free, doesn't work!!!


Those 2 *must* work. What are you doing to check if they work?

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
There is no pain you are receding
A distant ship, smoke on the horizon.
You are only coming through in waves.
Your lips move but I can't hear what you're saying.
Jul 12 2010
next sibling parent BCS <none anon.com> writes:
Hello Leandro,


 ABothe, el 12 de julio a las 23:44 me escribiste:
 

 GC.free(&asdf); // It doesn't work!!
 delete asdf; // Also, without GC.free, doesn't work!!!


 Those 2 *must* work. What are you doing to check if they work?
 


If, by chance, you are looking at the memory usage of your program (i.e. 
the amount of space the OS has mapped) as reported by the OS or the like, 
then you won't see anything as the GC never returns memory to the OS, OTOH 
unless you are VERY tight on memory, this will have negligible effect on 
most modern computers.

-- 
... <IXOYE><
Jul 12 2010
prev sibling parent reply ABothe <info asdf.com> writes:
First, thanks for the response!

When executing the program at your own, look at the task
manager... It simply doesn't do it!


Is there somebody who can make it possible for the GC to give
allocated memory right back??
Jul 13 2010
next sibling parent reply bearophile <bearophileHUGS lycos.com> writes:
ABothe:

 Is there somebody who can make it possible for the GC to give
 allocated memory right back??


In Python generally this doesn't happen.

Bye,
bearophile
Jul 13 2010
parent reply ABothe <info asdf.com> writes:
So there isn't any other method than calling C's free(); or malloc
(); ?
Jul 13 2010
next sibling parent reply FeepingCreature <default_357-line yahoo.de> writes:
On 13.07.2010 12:28, ABothe wrote:

 So there isn't any other method than calling C's free(); or malloc
 (); ?


That's how it is.
Jul 13 2010
parent reply bearophile <bearophileHUGS lycos.com> writes:
FeepingCreature:

 On 13.07.2010 12:28, ABothe wrote:

 So there isn't any other method than calling C's free(); or malloc
 (); ?


 
 That's how it is.


But you can build some abstraction by yourself, for example inside a struct you
can keep the pointer to a memory block in the C heap, and in the ~this() of the
struct you can put the free(). Then D has to give you a way to disallow the
"new" on this struct, to be sure its destructor is called when it goes out of
scope (or the object it's contained into gets deallocated).

Bye,
bearophile
Jul 13 2010
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 07/13/2010 03:36 PM, bearophile wrote:

 FeepingCreature:

 On 13.07.2010 12:28, ABothe wrote:

 So there isn't any other method than calling C's free(); or malloc
 (); ?


 That's how it is.


 But you can build some abstraction by yourself, for example inside a struct
you can keep the pointer to a memory block in the C heap, and in the ~this() of
the struct you can put the free(). Then D has to give you a way to disallow the
"new" on this struct, to be sure its destructor is called when it goes out of
scope (or the object it's contained into gets deallocated).

 Bye,
 bearophile


Also refer to RefCounted:

http://www.dsource.org/projects/phobos/browser/trunk/phobos/std/typecons.d#L1991

Highly recommended.


Andrei
Jul 13 2010
parent reply ABothe <info asdf.com> writes:
So it's not planned by you or Walter to let the GC be able to free
some memory? I mean this would be very helpful and would make
large programs possible
(Especially those ones are often working with a huge data
amount...think of running that program for a while...the memory
consumption would be enormously high:-/ and mess up the whole
system)
Jul 13 2010
parent Leandro Lucarella <luca llucax.com.ar> writes:
ABothe, el 13 de julio a las 23:20 me escribiste:

 So it's not planned by you or Walter to let the GC be able to free
 some memory? I mean this would be very helpful and would make
 large programs possible
 (Especially those ones are often working with a huge data
 amount...think of running that program for a while...the memory
 consumption would be enormously high:-/ and mess up the whole
 system)


gc.minimize() is supposed to do that. But if you really need to work
with huge datasets, I'd recommend to do manual memory management
anyways, because of the current limitations of the GC.

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
Hoy estuvimos en el museo de antropología, pero yo voy a volver
para estar por lo menos un día ahí adentro... es una locura, como Disney pero
de indigenas
	-- Carla Lucarella (10/2008 contando de su viaje a México)
Jul 14 2010
prev sibling next sibling parent reply "Nick Sabalausky" <a a.a> writes:
"ABothe" <info asdf.com> wrote in message 
news:i1hf3h$7sd$1 digitalmars.com...

 So there isn't any other method than calling C's free(); or malloc
 (); ?


GC.minimize() should do it:

http://www.digitalmars.com/d/2.0/phobos/std_gc.html

If it doesn't, then there's probably something holding on a reference to it 
(like the LRU cache Steven mentioned), or something that the GC mistakenly 
thinks is a reference to it.

-------------------------------
Not sent from an iPhone.
Jul 13 2010
parent "Nick Sabalausky" <a a.a> writes:
"Nick Sabalausky" <a a.a> wrote in message 
news:i1iaan$1nbe$1 digitalmars.com...

 "ABothe" <info asdf.com> wrote in message 
 news:i1hf3h$7sd$1 digitalmars.com...

 So there isn't any other method than calling C's free(); or malloc
 (); ?


 GC.minimize() should do it:

 http://www.digitalmars.com/d/2.0/phobos/std_gc.html

 If it doesn't, then there's probably something holding on a reference to 
 it (like the LRU cache Steven mentioned), or something that the GC 
 mistakenly thinks is a reference to it.


But, in a GC'ed environment, you shouldn't normally be trying to release 
memory back to the OS every time you delete something. In most cases, the GC 
deliberately saves freed memory to return it back to you the next time you 
allocate. So if you want to see whether or not you're *really* leaking, try 
something more like this:

while(true)
{
    byte[] asdf=new byte[50_000_000];
    GC.free(cast(void*) asdf.ptr);
}

Then monitor the memory usage reported by the OS. If it keeps climbing 
higher and higher and higher, then something's going wrong. If it stays 
pretty much constant, then everything's working correctly.
Jul 13 2010
prev sibling parent =?UTF-8?B?IkrDqXLDtG1lIE0uIEJlcmdlciI=?= <jeberger free.fr> writes:
 boundary="------------070005050409000208000703"

This is a multi-part message in MIME format.
--------------070005050409000208000703
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

ABothe wrote:

 So there isn't any other method than calling C's free(); or malloc
 (); ?


	Note that there is no guarantee that C's malloc/free will return
the memory to the OS either (*). The only way to be sure that the
memory will be returned is to call mmap/munmap (or their equivalent
on whatever OS you're using).

(*) Most malloc implementations allocate large chunks of memory
using mmap, which can return the memory to the OS, but they allocate
small chunks with sbrk, which can't. As proof, see the attached C
code which prints:

At the beginning:
VmSize:	    3872 kB
After allocating the memory:
VmSize:	   36608 kB
After releasing the memory:
VmSize:	   36608 kB


		Jerome
--=20
mailto:jeberger free.fr
http://jeberger.free.fr
Jabber: jeberger jabber.fr

--------------070005050409000208000703
Content-Type: text/x-csrc;
 name="test.c"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline;
 filename="test.c"

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int startsWith (const char* s1, const char* s2)
{
   while ((*s1 !=3D 0) && (*s2 !=3D 0) && (*s1 =3D=3D *s2)) {
      ++s1;
      ++s2;
   }
   return (*s2 =3D=3D 0);
}

void printMemInfo (void)
{
   char*    filename;
   FILE*    status;
   char     buffer[128];

   asprintf (&filename, "/proc/%d/status", getpid());
   status =3D fopen (filename, "r");
   free (filename);

   while (fgets (buffer, sizeof (buffer), status) !=3D NULL)
      if (startsWith (buffer, "VmSize")) {
         printf ("%s", buffer);
         break;
      }

   fclose (status);
}

void** allocate (int n)
{
   void**   result =3D NULL;
   int      i;

   for (i =3D 0; i < n; ++i) {
      void**    next =3D malloc (sizeof (void**));
      *next          =3D result;
      result         =3D next;
   }
   return result;
}

void release (void** list)
{
   while (list !=3D NULL) {
      void**    next =3D *list;
      free (list);
      list           =3D next;
   }
}

int main (int argc, char** argv)
{
   void**   list;

   printf ("At the beginning:\n");
   printMemInfo();

   list =3D allocate (1024*1024);
   printf ("After allocating the memory:\n");
   printMemInfo();

   release (list);
   printf ("After releasing the memory:\n");
   printMemInfo();

   return 0;
}

--------------070005050409000208000703--
Jul 14 2010
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 13 Jul 2010 03:50:19 -0400, ABothe <info asdf.com> wrote:


 First, thanks for the response!

 When executing the program at your own, look at the task
 manager... It simply doesn't do it!


 Is there somebody who can make it possible for the GC to give
 allocated memory right back??


The GC does not release memory back to the OS in all circumstances, it may  
hold onto the memory, even though it isn't allocated.

Part of the reason for this is it may just not be possible.  IIRC, on  
Linux systems, all your heap-allocated memory must be contiguous, meaning  
that if you deallocate holes, you can't release them back to the OS.

There is also a design bug in the array allocator (which you probably have  
not encountered, since you did not append), where it will hold onto memory  
even if nothing is referencing it because the LRU cache is referencing  
it.  I am going to fix that in the near future.

-Steve
Jul 13 2010
prev sibling parent Leandro Lucarella <luca llucax.com.ar> writes:
ABothe, el 13 de julio a las 07:50 me escribiste:

 First, thanks for the response!
 
 When executing the program at your own, look at the task
 manager... It simply doesn't do it!
 
 Is there somebody who can make it possible for the GC to give
 allocated memory right back??


Calling gc.minimize() should free memory (as give it back to the OS) if
there are any pools completely free, but IFAIK is only implemented in
Tango and D2 (i.e., not implemented in Phobos/D1).

-- 
Leandro Lucarella (AKA luca)                     http://llucax.com.ar/
----------------------------------------------------------------------
GPG Key: 5F5A8D05 (F8CD F9A7 BF00 5431 4145  104C 949E BFB6 5F5A 8D05)
----------------------------------------------------------------------
La esperanza es una amiga que nos presta la ilusión.
Jul 13 2010
prev sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Tue, 13 Jul 2010 02:44:49 +0300, ABothe <info alexanderbothe.com> wrote:


 Hi everyone,

 I don't know if it's a bug or not .. but D or at least its GC doesn't  
 want to
 free my allocated stuff! Also when I 'disable' the GC via GC.disable();  
 it won't free it :-/


Alex, please have a look at the module I posted to D.announce. It allows  
working with unmanaged memory without having to worry about manual  
deallocation, etc.

For example:

import std.stdio;
import std.gc;
import data;

void main()
{
	Data d = new Data(50_000_000);
	readln(); // check my memory usage now
	d = null;
	fullCollect();
	readln(); // check my memory usage now
}

You can get my module from here: http://github.com/CyberShadow/data.d

-- 
Best regards,
  Vladimir                            mailto:vladimir thecybershadow.net
Jul 14 2010