www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - memory leakage in D

reply Tiago Gasiba <tiago.gasiba gmail.com> writes:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi all,

  The following code produces very strange behaviour in D.
  Try out the routine 1, routine 2, routine 3 or routine 4!
  Apparently there is a bug in the garbage collector and a memory leakage is
occouring.
  A test-case file goes in attachment.
  Is this a programmer bug or a bug in D, i.e. did I miss something? Thanks!

Best Regards,
Tiago

- -- 
Tiago Gasiba (M.Sc.) - http://www.gasiba.de
Everything should be made as simple as possible, but not simpler.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)

iD8DBQFDalOPDiEXU+4zMkURAhoVAJ9KxAES+UXuBMDXiZ3B4AryiOA5eACfefil
LW8wsPMhAa3Dq9fejjuY6Yc=
=HxwQ
-----END PGP SIGNATURE-----
Nov 03 2005
parent reply xs0 <xs0 xs0.com> writes:
Tiago Gasiba wrote:

 -----BEGIN PGP SIGNED MESSAGE-----
 Hash: SHA1
 
 Hi all,
 
   The following code produces very strange behaviour in D.
   Try out the routine 1, routine 2, routine 3 or routine 4!
   Apparently there is a bug in the garbage collector and a memory leakage is
occouring.
   A test-case file goes in attachment.
   Is this a programmer bug or a bug in D, i.e. did I miss something? Thanks!
 
 Best Regards,
 Tiago
 [snip]


I think that memory usage keeps increasing, because garbage collection 
doesn't kick in until the app runs out of system memory.. Try calling 
std.gc.fullCollect and see if it helps..


xs0
Nov 04 2005
parent reply Tiago Gasiba <tiago.gasiba gmail.com> writes:
xs0 schrieb:

 
 I think that memory usage keeps increasing, because garbage collection
 doesn't kick in until the app runs out of system memory.. Try calling
 std.gc.fullCollect and see if it helps..



Yes, it helps, but only in the function bar(), before the return's
Putting the fullCollect in the foo() routine, it does not do anything!!!
This must be a GC bug IMHO, because the "user" is not supposed to manually call
fullCollect()...
I have also posted another message entitled "Garbage Collector Bug?" whereby I
have reported a series of errors reported by Valgrind.
In this very same program, the same errors occour, i.e. I get some errors like:
(This sort of errors should definitly not occour...)

<snip>
==7748== Conditional jump or move depends on uninitialised value(s)
==7748==    at 0x8053406: _D3gcx3Gcx4markFPvPvZv (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x80535F7: _D3gcx3Gcx11fullcollectFPvZk (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x80534B9: _D3gcx3Gcx16fullcollectshellFZk (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x8053167: _D3gcx3Gcx8bigAllocFkZPv (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x8052693: _D3gcx2GC12mallocNoSyncFkZPv (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x8052536: _D3gcx2GC6mallocFkZPv (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x80526C4: _D3gcx2GC6callocFkkZPv (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x804ED04: _d_arraysetlength (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x804B720: _D3bug3fooFdZd (in /home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x804B773: _Dmain (in /home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x804B7FE: main (in /home/gasiba/PhD/D/bugs/bug1/bug)

==7748== Conditional jump or move depends on uninitialised value(s)
==7748==    at 0x8052FDD: _D3gcx3Gcx8findPoolFPvZPS3gcx4Pool (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x805340F: _D3gcx3Gcx4markFPvPvZv (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x80535F7: _D3gcx3Gcx11fullcollectFPvZk (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x80534B9: _D3gcx3Gcx16fullcollectshellFZk (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x8053167: _D3gcx3Gcx8bigAllocFkZPv (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x8052693: _D3gcx2GC12mallocNoSyncFkZPv (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x8052536: _D3gcx2GC6mallocFkZPv (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x80526C4: _D3gcx2GC6callocFkkZPv (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x804ED04: _d_arraysetlength (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x804B720: _D3bug3fooFdZd (in /home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x804B773: _Dmain (in /home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x804B7FE: main (in /home/gasiba/PhD/D/bugs/bug1/bug)

==7748== Use of uninitialised value of size 4
==7748==    at 0x8053D98: _D6gcbits6GCBits4testFkZk (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x8053457: _D3gcx3Gcx4markFPvPvZv (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x80535F7: _D3gcx3Gcx11fullcollectFPvZk (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x80534B9: _D3gcx3Gcx16fullcollectshellFZk (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x8052AC8: _D3gcx2GC11fullCollectFZv (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x804E7E0: _D3std2gc11fullCollectFZv (in
/home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x804B6B9: _D3bug3barFKlZd (in /home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x804B74C: _D3bug3fooFdZd (in /home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x804B773: _Dmain (in /home/gasiba/PhD/D/bugs/bug1/bug)
==7748==    by 0x804B7FE: main (in /home/gasiba/PhD/D/bugs/bug1/bug)

<snip>

etc...

Thanks for the reply,
Tiago

-- 
Tiago Gasiba (M.Sc.) - http://www.gasiba.de
Everything should be made as simple as possible, but not simpler.
Nov 04 2005
parent reply Tiago Gasiba <tiago.gasiba gmail.com> writes:
 
 xs0 schrieb:

 
 I think that memory usage keeps increasing, because garbage collection
 doesn't kick in until the app runs out of system memory.. Try calling
 std.gc.fullCollect and see if it helps..


 


std.gc.fullCollect() does NOT help in the following example:

<snip>
import std.gc;

double bar(inout long pvar){
  int          jj;
  long         kk;
  double       temp;
  static long  myvar2=123456789;
  static long  iy=0;
  static long  iv[32];


  kk    = (pvar/53668);
  pvar  = 40014*(pvar-kk*53668)-kk*12211;

  if( pvar<0 )
    pvar  += 2147483563;

  if( myvar2<0 )
    myvar2 += 2147483399;

  jj     = cast(int)(iy/(1+2147483562/32));
  jj     = jj&31;
  iy     = iv[jj]-myvar2;
  iv[jj] = pvar;

  // std.gc.fullCollect();  // here it helps.... why?
                            // program becomes much slower, though
  if( (temp=(1.0/2147483563)*iy) > (1.0-1.2e-7))
    return (1.0-1.2e-7);
  else
    return temp;
}


/*
  This routine has a memory leakage???
  Memory consumption keeps up increasing...
    even though I specifically call std.gc.fullCollect()
*/
double foo( double varx ){
  int        N = 1000000;
  double[]   X;
  int        ii;
  long       pvar = 0x7623627;

  X.length   = N;

  for( ii=0; ii<X.length; ii++ )
    X[ii] = bar(pvar);

  std.gc.fullCollect();
  return 0.0;
}


int main( ){

  while( 1==1 )
    foo(0);

  return 0;
}

<snip>

Tiago

-- 
Tiago Gasiba (M.Sc.) - http://www.gasiba.de
Everything should be made as simple as possible, but not simpler.
Nov 04 2005
parent reply =?ISO-8859-1?Q?Jari-Matti_M=E4kel=E4?= <jmjmak invalid_utu.fi> writes:
Tiago Gasiba wrote:

xs0 schrieb:


I think that memory usage keeps increasing, because garbage collection
doesn't kick in until the app runs out of system memory.. Try calling
std.gc.fullCollect and see if it helps..




 
 std.gc.fullCollect() does NOT help in the following example:


I made a simple mp3 player in D a while ago. First, it read 4000 files
and stored the key-value-pairs of the song tags in a linked list. Then
I decided to have multiple playlists. When I "deleted" a playlist and
tried the std.gc.fullCollect(), no memory was freed. I also tried
something like

#Playlist a = null;
#std.gc.fullCollect();

and

#Playlist a = null
#for (int i=0; i<1000; i++) { doSomethingElse(); std.gc.fullCollect(); }

but neither of those helped. I think the GC doesn't want to free memory
until it has waited for a while and we are actually allocating more
memory. But this is kind of stupid behaviour. Of course there are other
programs that may need some memory or at least use it for buffering, but
this GC seems to be a bit greedy, what do you think?
Nov 06 2005
parent Georg Wrede <georg.wrede nospam.org> writes:
Jari-Matti Mäkelä wrote:

 Tiago Gasiba wrote:
 

 xs0 schrieb:
 

 I think that memory usage keeps increasing, because garbage
 collection doesn't kick in until the app runs out of system
 memory.. Try calling std.gc.fullCollect and see if it helps..


 
 
 


 
 std.gc.fullCollect() does NOT help in the following example:


 
 
 I made a simple mp3 player in D a while ago. First, it read 4000
 files and stored the key-value-pairs of the song tags in a linked
 list. Then I decided to have multiple playlists. When I "deleted" a
 playlist and tried the std.gc.fullCollect(), no memory was freed. I
 also tried something like
 
 #Playlist a = null; #std.gc.fullCollect();
 
 and
 
 #Playlist a = null #for (int i=0; i<1000; i++) { doSomethingElse();
 std.gc.fullCollect(); }
 
 but neither of those helped. I think the GC doesn't want to free
 memory until it has waited for a while and we are actually allocating
 more memory. But this is kind of stupid behaviour. Of course there
 are other programs that may need some memory or at least use it for
 buffering, but this GC seems to be a bit greedy, what do you think?


I just had a look into the sources, and it looks like garbage collecting 
never does reduce the memory footprint of a D program. Seems it isn't 
even supposed to. GC just collects memory for _itself_ from dead objects.

For the situations where the footprint should be reduced (like right 
after a memory hungry operation is finished, before continuing), the 
programmer should first run a full collect, then run

std.gc.minimize();

That would be the first time the OS reports our D program as using less 
memory, i.e. that it actually has released some memory back.

---

Except that minimize() isn't implemented yet. (Don't tell anybody.)

---

OK, so what can one do when one needs something done that not only takes 
a long time but also needs tons of memory at the start, after which it 
needs very little but runs very long?
Nov 11 2005