www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Apparent problem with GC not collecting on Windows

reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
A friend of mine reports that the memory usage of the following program 
grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1 
home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).

If you are on Windows, could you please say whether you have the same 
problem. (I don't have Windows.) The following is how we reproduce:

Generate a file of about 1G by running the following program:

import std.stdio;

void foo()
{
     auto file = File("one_gigabyte_file", "w");
     auto data = new ubyte[](100 * 1024 * 1024);

     foreach (i; 0 .. 10) {
         file.rawWrite(data);
     }
}

void main()
{
     foo();
}

That part is not the problem. The problem is when reading such a large 
file by chunks. Could you please run the following program and report 
whether the memory consumption of it is increasing continuously:

import std.stdio;

void foo()
{
     auto file = File("one_gigabyte_file", "r");
     auto data = new ubyte[](100 * 1024 * 1024);

     foreach (i; 0 .. 10) {
         file.rawRead(data);
     }
}

void main()
{
     for (size_t i; true; ++i) {
         writeln(i);
         foo();
     }
}

The program gets terminated by a core.exception.OutOfMemoryError. (Same 
problem with std.stream.BufferedFile.)

I don't see any problem under Linux.

Thank you,
Ali
Nov 28 2012
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
11/28/2012 10:51 PM, Ali Çehreli пишет:

 A friend of mine reports that the memory usage of the following program
 grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1
 home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).

 If you are on Windows, could you please say whether you have the same
 problem. (I don't have Windows.) The following is how we reproduce:


[snip]


 The program gets terminated by a core.exception.OutOfMemoryError. (Same
 problem with std.stream.BufferedFile.)


Same here on more or less fresh 2.061. About 12 iterations. Win8 x64 
running 32-bit app.


-- 
Dmitry Olshansky
Nov 28 2012
parent reply 1100110 <0b1100110 gmail.com> writes:
On 11/28/2012 12:57 PM, Dmitry Olshansky wrote:

 11/28/2012 10:51 PM, Ali Çehreli пишет:

 A friend of mine reports that the memory usage of the following program
 grows continuously when compiled with dmd 2.060 and run on Windows 7 sp1
 home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).

 If you are on Windows, could you please say whether you have the same
 problem. (I don't have Windows.) The following is how we reproduce:


 [snip]


 The program gets terminated by a core.exception.OutOfMemoryError. (Same
 problem with std.stream.BufferedFile.)


 Same here on more or less fresh 2.061. About 12 iterations. Win8 x64
 running 32-bit app.


Isn't this a known problem with a conservative GC on 32bit?  The GC sees 
something that *Could* be a reference, and refuses to collect that data.
Nov 28 2012
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 11/28/2012 11:11 AM, 1100110 wrote:

 On 11/28/2012 12:57 PM, Dmitry Olshansky wrote:

 11/28/2012 10:51 PM, Ali Çehreli пишет:

 A friend of mine reports that the memory usage of the following program
 grows continuously when compiled with dmd 2.060 and run on Windows 






7 sp1

 home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).

 If you are on Windows, could you please say whether you have the same
 problem. (I don't have Windows.) The following is how we reproduce:


 [snip]


 The program gets terminated by a core.exception.OutOfMemoryError. (Same
 problem with std.stream.BufferedFile.)


 Same here on more or less fresh 2.061. About 12 iterations. Win8 x64
 running 32-bit app.


 Isn't this a known problem with a conservative GC on 32bit? The GC sees
 something that *Could* be a reference, and refuses to collect that data.


I think it is more like an integer value looking like a pointer to 
something. I don't see what the GC may be confused with in this case.

I've still created a bug:

   http://d.puremagic.com/issues/show_bug.cgi?id=9094

Ali
Nov 28 2012
parent 1100110 <0b1100110 gmail.com> writes:
On 11/28/2012 02:53 PM, Ali Çehreli wrote:

 On 11/28/2012 11:11 AM, 1100110 wrote:
  > On 11/28/2012 12:57 PM, Dmitry Olshansky wrote:
  >> 11/28/2012 10:51 PM, Ali Çehreli пишет:
  >>> A friend of mine reports that the memory usage of the following
 program
  >>> grows continuously when compiled with dmd 2.060 and run on Windows
 7 sp1
  >>> home premium 64 bit (also on Microsoft Windows [Version 6.1.7600]).
  >>>
  >>> If you are on Windows, could you please say whether you have the same
  >>> problem. (I don't have Windows.) The following is how we reproduce:
  >>
  >> [snip]
  >>
  >>> The program gets terminated by a core.exception.OutOfMemoryError.
 (Same
  >>> problem with std.stream.BufferedFile.)
  >>
  >> Same here on more or less fresh 2.061. About 12 iterations. Win8 x64
  >> running 32-bit app.
  >>
  >>
  >
  > Isn't this a known problem with a conservative GC on 32bit? The GC sees
  > something that *Could* be a reference, and refuses to collect that data.

 I think it is more like an integer value looking like a pointer to
 something. I don't see what the GC may be confused with in this case.

 I've still created a bug:

 http://d.puremagic.com/issues/show_bug.cgi?id=9094

 Ali


That's what I meant. =P
Nov 28 2012
prev sibling parent reply "Michael" <pr m1xa.com> writes:
1. Can be solved using allocators 
http://dlang.org/memory.html#newdelete and 
http://dlang.org/class.html#ClassAllocator (here deprecated)?
2. Why with
class Too
{
private uint[] pp;
    this(int s)
    {
       pp = new unit[s];
    }

alias pp this;
}

Only 3 iterations before out of memory exception?

3. Why it helps?

foreach (i; 0 .. 10) {
         file.rawRead(data);
     }

GC.free(data.ptr);


Win 8 Pro, 64 bit.
Nov 28 2012
parent reply "thedeemon" <dlang thedeemon.com> writes:
On Wednesday, 28 November 2012 at 22:02:35 UTC, Michael wrote:

 Only 3 iterations before out of memory exception?


Because you used uint instead of ubyte, array is bigger, memory 
exhausts faster.


 3. Why it helps?
 GC.free(data.ptr);


Initial leak happened because for some reason array allocated in 
previous iteration was not collected by GC when allocating new 
one, so the new one was allocated in another space growing the 
heap. If you place GC.free the array gets removed from heap on 
each iteration and each new allocation reuses the same memory, 
heap doesn't grow.
Nov 28 2012
parent reply "Michael" <pr m1xa.com> writes:
 Because you used uint instead of ubyte, array is bigger, memory 
 exhausts faster.


Oh, I see.


 3. Why it helps?
 GC.free(data.ptr);


 Initial leak happened because for some reason array allocated 
 in previous iteration was not collected by GC when allocating 
 new one, so the new one was allocated in another space growing 
 the heap. If you place GC.free the array gets removed from heap 
 on each iteration and each new allocation reuses the same 
 memory, heap doesn't grow.


If we do this manually it's works, but automatically is broken?
Nov 29 2012
parent reply =?UTF-8?B?QWxpIMOHZWhyZWxp?= <acehreli yahoo.com> writes:
On 11/29/2012 12:06 PM, Michael wrote:

 Because you used uint instead of ubyte, array is bigger, memory
 exhausts faster.


 Oh, I see.


 3. Why it helps?
 GC.free(data.ptr);


 Initial leak happened because for some reason array allocated in
 previous iteration was not collected by GC when allocating new one, so
 the new one was allocated in another space growing the heap. If you
 place GC.free the array gets removed from heap on each iteration and
 each new allocation reuses the same memory, heap doesn't grow.


 If we do this manually it's works, but automatically is broken?


Nothing is "broken." GC.free would have been applied by the GC only if 
there have been no more references to the allocated block of memory.

The fact is, dmd uses a conservative GC. The issue is due to the 
combination of conservative GC, 32-bit address space, and a large chunk 
of memory. When that happens, it is very likely that any other value in 
the system, including an innocent int, has the risk of looking like a 
reference into that memory. For that reason the GC keeps that memory 
block allocated.

The risk of that happening is grossly reduced in a 64-bit address space. 
Similarly, allocating a much smaller buffer helps as well.

Alternatively, we can use a better GC or a runtime that has a better GC.

Ali
Nov 29 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
11/30/2012 12:36 AM, Ali Çehreli пишет:

 On 11/29/2012 12:06 PM, Michael wrote:
  >> Because you used uint instead of ubyte, array is bigger, memory
  >> exhausts faster.
  > Oh, I see.
  >
  >>> 3. Why it helps?
  >>> GC.free(data.ptr);
  >>
  >> Initial leak happened because for some reason array allocated in
  >> previous iteration was not collected by GC when allocating new one, so
  >> the new one was allocated in another space growing the heap. If you
  >> place GC.free the array gets removed from heap on each iteration and
  >> each new allocation reuses the same memory, heap doesn't grow.
  > If we do this manually it's works, but automatically is broken?
  >

 Nothing is "broken." GC.free would have been applied by the GC only if
 there have been no more references to the allocated block of memory.

 The fact is, dmd uses a conservative GC. The issue is due to the
 combination of conservative GC, 32-bit address space, and a large chunk
 of memory. When that happens, it is very likely that any other value in
 the system, including an innocent int, has the risk of looking like a
 reference into that memory. For that reason the GC keeps that memory
 block allocated.


I'd just throw in that we have a (almost) precise GC that is used by  at 
least one large project (the VisualD apparently). Though there were some 
problems with it. Anyway I'd expect to see it in upstream by 2.062 at 
least. It should help the cases like this tremendously.

P.S. Either way the manual memory management (and reuse) is the way to 
go with big allocations.

-- 
Dmitry Olshansky
Nov 30 2012
parent reply "thedeemon" <dlang thedeemon.com> writes:
On Friday, 30 November 2012 at 18:46:08 UTC, Dmitry Olshansky 
wrote:


 I'd just throw in that we have a (almost) precise GC that is 
 used by  at least one large project (the VisualD apparently). 
 Though there were some problems with it. Anyway I'd expect to 
 see it in upstream by 2.062 at least. It should help the cases 
 like this tremendously.


I expect it wouldn't help in this particular case because stack 
scanning is still conservative there (correct me if I'm wrong), 
and here we've got some stack value looking like a pointer to the 
array.
Dec 01 2012
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
12/1/2012 2:53 PM, thedeemon пишет:

 On Friday, 30 November 2012 at 18:46:08 UTC, Dmitry Olshansky wrote:


 I'd just throw in that we have a (almost) precise GC that is used by
 at least one large project (the VisualD apparently). Though there were
 some problems with it. Anyway I'd expect to see it in upstream by
 2.062 at least. It should help the cases like this tremendously.


 I expect it wouldn't help in this particular case because stack scanning
 is still conservative there (correct me if I'm wrong), and here we've
 got some stack value looking like a pointer to the array.


It should help because:
1) Each allocation happens in a new stack frame thus the pointer is 
overwritten each time.
2) Precise heap scanning helps this case because it greatly reduces the 
amount of false pointers (note it's not the stack variables that hold 
these allocations in place)

It doesn't guarantee that it will collect all of these chunks but number 
of false pointers that could point to them is far lower and the chance 
of being collected is that much higher.

The stack is typically small, unlike the heap.

-- 
Dmitry Olshansky
Dec 01 2012
parent reply "thedeemon" <dlang thedeemon.com> writes:
On Saturday, 1 December 2012 at 12:55:19 UTC, Dmitry Olshansky 
wrote:

 It should help because:
 1) Each allocation happens in a new stack frame thus the 
 pointer is overwritten each time.


To probably the same stinky value (a false pointer).


 2) Precise heap scanning helps this case because it greatly 
 reduces the amount of false pointers (note it's not the stack 
 variables that hold these allocations in place)


I'm pretty sure in this case the false pointers are indeed in 
stack or data segment, not in the heap. Data arrays must be 
allocated with NO_SCAN flag so they're not scanned for pointers.
Dec 01 2012
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
12/1/2012 10:46 PM, thedeemon пишет:

 On Saturday, 1 December 2012 at 12:55:19 UTC, Dmitry Olshansky wrote:

 It should help because:
 1) Each allocation happens in a new stack frame thus the pointer is
 overwritten each time.


 To probably the same stinky value (a false pointer).


Pointer is different every time. So the initial pointer itself is 
overwritten. Everything else depends on luck. Some other values up the 
stack might still "point" to the block.


 2) Precise heap scanning helps this case because it greatly reduces
 the amount of false pointers (note it's not the stack variables that
 hold these allocations in place)


 I'm pretty sure in this case the false pointers are indeed in stack or
 data segment, not in the heap. Data arrays must be allocated with
 NO_SCAN flag so they're not scanned for pointers.


I've meant the actual array variable. Basically it's some other data 
that looks like a pointer to it elsewhere. Depending on how much runtime 
does dynamically allocate vs statically it may be the case that it is 
not in heap but in data segment. Or the used length of stack that should 
be quite small here.

Bottom line is that the bigger array you have the greater chance it 
won't be collected (because it's a nice target for random values in 
stack/data segment/heap). The more precise the scanner the better are 
chances of it being collected.


-- 
Dmitry Olshansky
Dec 01 2012