www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.bugs - [Issue 1804] New: Severe GC leaks with repetitive array allocations

reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1804

           Summary: Severe GC leaks with repetitive array allocations
           Product: D
           Version: 1.026
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: major
          Priority: P2
         Component: DMD
        AssignedTo: bugzilla digitalmars.com
        ReportedBy: webmaster villagersonline.com


Linux (Fedora 6), 64 bit

The following program will run out of memory when the allocations reach about 8
MB.  The problem happens on *both* 1.026 and 2.010

FYI: I tried running fullCollect() periodically, just in case...no luck.

  import std.stdio;
  void main()
  {
    for(int i=1; ; i++)
    {
      auto tmp = new char[i*1024];
      if(i % 1_000 == 0)
        writefln("%d iterations, roughly %.1f MB", i,
tmp.length/(1024*1024.0));
    }
  }


--
Jan 24 2008
next sibling parent Sean Kelly <sean f4.ca> writes:
d-bugmail puremagic.com wrote:

 
 FYI: I tried running fullCollect() periodically, just in case...no luck.
 
   import std.stdio;
   void main()
   {
     for(int i=1; ; i++)
     {
       auto tmp = new char[i*1024];
       if(i % 1_000 == 0)
         writefln("%d iterations, roughly %.1f MB", i,
 tmp.length/(1024*1024.0));
     }
   }


Interestingly, this app crashes in the same place if 'tmp' is deleted on
each iteration.  If I had to guess, I'd say that empty pools aren't
being added to the free list after collections, even though there's code
in place that's supposed to do this.  Perhaps it's just pools dedicated
to big allocations that are the problem.  This portion of the GC isn't
really my area of expertise, but I'll submit a patch if I can figure
this one out.
Jan 24 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1804






This came up in a program where I had a large array, containing all of the
prime numbers through some value.  As I found more primes, I would append to
the array.  This works just fine, often (meaning that *sometimes* buffers are
getting correctly collected), but after we hit some point the memory usage
would spike because, whenever an append operation required a copy, the old
array wasn't getting GC'd.

I originally theorized that the prime numbers in the array were happening to
match addresses of arrays (and that the bug was that the GC was scanning
non-pointer memory).  But that doesn't play out, exactly.  My gut is still that
there is a false pointer out there somewhere.  Are the memory allocator
metadata structures scanned by the GC?  Maybe, after lots of allocations, we by
random get metadata that looks like a pointer, which pins some memory....and
then it gradually builds from there.  Dunno.

RECREATE UPDATE:

Update the writefln line to also trace the address:
   writefln("%d iterations, roughly %.1f MB at %x", i, ..... ,
cast(uint)tmp.ptr);

Then change the loop to start at 1014, 1015, or 1016 (maybe others) and the 1st
buffer to be allocated will only get garbage collected once: after it is
allocated the 2nd time, it doesn't appear that it is ever collected again.


--
Jan 24 2008
prev sibling next sibling parent reply d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1804






Okay, I think I know what's going on.  For big allocations, the GC will look
for a pre-existing set of contiguous pages within a single pool to hold the
block.  If one cannot be found, then it creates a new pool of the requested
size to hold the block.  However, in this app the allocation size just
continuously increases, so as time goes on the GC is unable to re-use existing
pools and so allocates successively larger pools to hold the requested block. 
The problem is that all these old pools which were only used once are held by
the GC for future use, so the total memory used by the application steadily
increases, with most of this memory going completely unused.

The simplest fix in this case would be for the GC to always release pools
obtained for big allocations back to the OS when they are no longer needed. 
This should address the pathological case demonstrated in this test app.  I'm
going to make this change in Tango and see if it helps.


--
Feb 05 2008
parent Sean Kelly <sean f4.ca> writes:
I've created a ticket for this in Tango:

http://www.dsource.org/projects/tango/ticket/878

The fix is done and will be checked in once DSource is back online.  The
diff is pretty succinct, but I'd be happy to supply a patch for Phobos
if that would help.
Feb 07 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1804






FYI: I hit this problem on Phobos, not Tango.


--
Mar 31 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1804






Sure.  But the same problem existed in Tango so I thought I'd diagnose it there
and report my findings to save Walter some legwork with Phobos.  Just trying to
be helpful here :-)


--
Apr 03 2008
prev sibling next sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1804


Martin Krejcirik <mk krej.cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
                 CC|                            |mk krej.cz
         Resolution|                            |FIXED



Looks like it has been fixed somewhere between v1.056 and current v1.076.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Apr 01 2013
prev sibling parent d-bugmail puremagic.com writes:
http://d.puremagic.com/issues/show_bug.cgi?id=1804


Martin Krejcirik <mk krej.cz> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|RESOLVED                    |REOPENED
          Component|DMD                         |Phobos
            Version|1.026                       |D1 & D2
         Resolution|FIXED                       |
         OS/Version|Linux                       |Windows



I'm reopening this, because I noticed the problem
(core.exception.OutOfMemoryError after about 20000 iterations) still exists on
Windows XP 32bit with latest dmd 2.063. Works ok on Linux though.

-- 
Configure issuemail: http://d.puremagic.com/issues/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
Jun 01 2013