• torhu (24/24) Mar 06 2007 I haven't dealt directly with the garbage collector much before, but
• Sean Kelly (16/28) Mar 06 2007 When I run the test on Tango, memory juse jumps to 467k and stays steady...
• torhu (12/31) Mar 07 2007 Uh... but 1024^2*10 ints is 40 MB exactly. Are you talking about a
• Sean Kelly (8/41) Mar 07 2007 No, you're right. I misread the output to contain one more zero than it...
• torhu (16/19) Mar 07 2007 I assume you mean 467MB? Anyway, the point is not how fast it happens.
• Lutger (7/36) Mar 07 2007 Not being exactly an expert on garbage collection myself, I did
• torhu (4/15) Mar 07 2007 I tried minimize, but now the memory usage goes even higher, up to 469
• Paul Findlay (5/7) Mar 08 2007 Using plain phobos, my testing stabilises at ~51MB. When minimize was
• torhu (5/10) Mar 08 2007 Ok, I have 896 MB. I have a feeling that doesn't affect the GC, but I
• Paul Findlay (4/5) Mar 08 2007 Exactly the same..
• Derek Parnell (8/14) Mar 08 2007 There is quite a large portion of the fullCollect source which is commen...
• torhu (14/20) Mar 08 2007 Do you mean gcx.d, line 1619? That would explain some things. There's
• Sean Kelly (4/29) Mar 08 2007 fullCollect performs a full collection of orphaned memory. With the
• torhu (6/9) Mar 08 2007 It's supposed to recover memory for future use, but it doesn't do that.
• Sean Kelly (4/6) Mar 08 2007 Yeah. This is weird because I'd expect it to, but then I didn't write

torhu <fake address.dude> writes:

I haven't dealt directly with the garbage collector much before, but 
just used delete and scope to deal with deallocation.  And left some of 
the minor allocations to be freed by the GC.

Now I want to work more closely with the GC.  Can anyone explain to me 
why fullCollect doesn't seem to do anything here?  This example just 
keeps allocating memory, until the GC seems to kick in to stop it from 
growing.  It stabilizes at 426MB.  The same example using Tango instead 
stops at 597MB.

---
import std.gc;

void main()
{
     int[] a;

     for (;;) {
         a = new int[1024 * 1024 * 10];
         a = null;
         assert(a.ptr is null);
         fullCollect();
     }
}
---

Removing the call to fullCollect() doesn't change the behavior.  I 
expected fullCollect to collect as much memory as it can, and then reuse 
it for the next allocation.  Am I on the wrong track here?

Sean Kelly <sean f4.ca> writes:

torhu wrote:
 I haven't dealt directly with the garbage collector much before, but 
 just used delete and scope to deal with deallocation.  And left some of 
 the minor allocations to be freed by the GC.
 
 Now I want to work more closely with the GC.  Can anyone explain to me 
 why fullCollect doesn't seem to do anything here?  This example just 
 keeps allocating memory, until the GC seems to kick in to stop it from 
 growing.  It stabilizes at 426MB.  The same example using Tango instead 
 stops at 597MB.

When I run the test on Tango, memory juse jumps to 467k and stays steady 
there.  Doing some quick math with a calculator shows an array 1024^2*10 
integers should occupy ~409k of memory, so the rest is just program 
overhead.  For what it's worth, running this app on Tango without the 
call to gc.collect() brings memory use up to 594k.  I'd have to debug 
the GC to figure out why.

 Removing the call to fullCollect() doesn't change the behavior.  I 
 expected fullCollect to collect as much memory as it can, and then reuse 
 it for the next allocation.  Am I on the wrong track here?

Nope.  That's what it should do.  But be aware that the current GC 
allocates memory in pages and those pages are devoted to a specific size 
of data.  And once so assigned, I don't think the GC ever attempts to 
reclaim empty pages if another size allocation is required.  It doesn't 
matter in this case because anything over a page (4k) is the same "size" 
as far as the GC is concerned, but it may matter in an app that 
allocates a ton of small objects and discards them, then allocates a ton 
of larger objects and discards them, etc.


Sean

torhu <fake address.dude> writes:

Sean Kelly wrote:
 When I run the test on Tango, memory juse jumps to 467k and stays steady 
 there.  Doing some quick math with a calculator shows an array 1024^2*10 
 integers should occupy ~409k of memory, so the rest is just program 
 overhead.  For what it's worth, running this app on Tango without the 
 call to gc.collect() brings memory use up to 594k.  I'd have to debug 
 the GC to figure out why.

Uh... but 1024^2*10 ints is 40 MB exactly.  Are you talking about a 
different test program?

 
 Removing the call to fullCollect() doesn't change the behavior.  I 
 expected fullCollect to collect as much memory as it can, and then reuse 
 it for the next allocation.  Am I on the wrong track here?

 
 Nope.  That's what it should do.  But be aware that the current GC 
 allocates memory in pages and those pages are devoted to a specific size 
 of data.  And once so assigned, I don't think the GC ever attempts to 
 reclaim empty pages if another size allocation is required.  It doesn't 
 matter in this case because anything over a page (4k) is the same "size" 
 as far as the GC is concerned, but it may matter in an app that 
 allocates a ton of small objects and discards them, then allocates a ton 
 of larger objects and discards them, etc.


Are you on Windows or Linux?  I'm on windows, and I'm not seeing the 
behavior you describe.  The test I posted allocates the same amount of 
memory over and over again.  But the GC doesn't seem to start reusing it 
until somewhere around the eleventh time through the loop.  Until then 
it just grows.

Just for the record, the number I'm looking at is 'private bytes' in 
process explorer, which is the same as 'VM size' in task manager.  Which 
shows allocated heap memory, and probably some other stuff.  And I use 
dmd 1.007.

Sean Kelly <sean f4.ca> writes:

torhu wrote:
 Sean Kelly wrote:
 When I run the test on Tango, memory juse jumps to 467k and stays 
 steady there.  Doing some quick math with a calculator shows an array 
 1024^2*10 integers should occupy ~409k of memory, so the rest is just 
 program overhead.  For what it's worth, running this app on Tango 
 without the call to gc.collect() brings memory use up to 594k.  I'd 
 have to debug the GC to figure out why.

 
 Uh... but 1024^2*10 ints is 40 MB exactly.  Are you talking about a 
 different test program?

No, you're right.  I misread the output to contain one more zero than it 
does.

 Removing the call to fullCollect() doesn't change the behavior.  I 
 expected fullCollect to collect as much memory as it can, and then 
 reuse it for the next allocation.  Am I on the wrong track here?

 Nope.  That's what it should do.  But be aware that the current GC 
 allocates memory in pages and those pages are devoted to a specific 
 size of data.  And once so assigned, I don't think the GC ever 
 attempts to reclaim empty pages if another size allocation is 
 required.  It doesn't matter in this case because anything over a page 
 (4k) is the same "size" as far as the GC is concerned, but it may 
 matter in an app that allocates a ton of small objects and discards 
 them, then allocates a ton of larger objects and discards them, etc.

 
 
 Are you on Windows or Linux?  I'm on windows, and I'm not seeing the 
 behavior you describe.  The test I posted allocates the same amount of 
 memory over and over again.  But the GC doesn't seem to start reusing it 
 until somewhere around the eleventh time through the loop.  Until then 
 it just grows.

Windows.  The app jumps to 467k almost immediately and stays there.  I 
haven't tried stepping through the loop, so it may just be happening too 
fast for me to see.

 Just for the record, the number I'm looking at is 'private bytes' in 
 process explorer, which is the same as 'VM size' in task manager.  Which 
 shows allocated heap memory, and probably some other stuff.  And I use 
 dmd 1.007.

Thanks, I'll check that number to be sure.


Sean

torhu <fake address.dude> writes:

Sean Kelly wrote:
 Windows.  The app jumps to 467k almost immediately and stays there.  I 
 haven't tried stepping through the loop, so it may just be happening too 
 fast for me to see.

I assume you mean 467MB?  Anyway, the point is not how fast it happens. 
  The problem is that it happens at all.  I expected to be able to use 
the garbage collector to keep the test app's memory at ~40MB.  If 
std.gc.minimize doesn't do that, then how is it different from 
std.gc.fullCollect?

By the way, the limit for how much memory size grows when running the 
test app depends on the size of the array.  So the GC seems to set a 
limit based on the app's typical allocation sizes.  Which makes sense 
and probably keeps allocation speeds up.

It might be that I should start viewing the GC more as a leak preventor 
than a simpler way of keeping memory usage down.  Simpler than explicit 
deallocation, that is.  The GC might decide to let memory usage grow to 
what the programmer or the user consider unreasonable sizes.  So 
basically, you should always use delete or scope to free memory, since 
when the GC does it, it will be a bit late.

Lutger <lutger.blijdestijn gmail.com> writes:

torhu wrote:
 I haven't dealt directly with the garbage collector much before, but 
 just used delete and scope to deal with deallocation.  And left some of 
 the minor allocations to be freed by the GC.
 
 Now I want to work more closely with the GC.  Can anyone explain to me 
 why fullCollect doesn't seem to do anything here?  This example just 
 keeps allocating memory, until the GC seems to kick in to stop it from 
 growing.  It stabilizes at 426MB.  The same example using Tango instead 
 stops at 597MB.
 
 ---
 import std.gc;
 
 void main()
 {
     int[] a;
 
     for (;;) {
         a = new int[1024 * 1024 * 10];
         a = null;
         assert(a.ptr is null);
         fullCollect();
     }
 }
 ---
 
 Removing the call to fullCollect() doesn't change the behavior.  I 
 expected fullCollect to collect as much memory as it can, and then reuse 
 it for the next allocation.  Am I on the wrong track here?

Not being exactly an expert on garbage collection myself, I did 
encounter the same behaviour which could be changed with a call to 
std.gc.minimize. This led me to think fullCollect does in fact not 
always clean up all unused memory, but just forces a GC scan, which may 
decide there is no need to collect if enough memory is available. Can 
anybody confirm this?

torhu <fake address.dude> writes:

Lutger wrote:
 torhu wrote:
 Removing the call to fullCollect() doesn't change the behavior.  I 
 expected fullCollect to collect as much memory as it can, and then reuse 
 it for the next allocation.  Am I on the wrong track here?

 
 Not being exactly an expert on garbage collection myself, I did 
 encounter the same behaviour which could be changed with a call to 
 std.gc.minimize. This led me to think fullCollect does in fact not 
 always clean up all unused memory, but just forces a GC scan, which may 
 decide there is no need to collect if enough memory is available. Can 
 anybody confirm this?

I tried minimize, but now the memory usage goes even higher, up to 469 
MB instead of 426.  So none of these functions seem to do what the docs 
say.  Same goes for tango.core.Memory.gc.collect().

Paul Findlay <r.lph50+d gmail.com> writes:

Unfortunately this is unlikely to help but..

 This example just keeps allocating memory,  until the GC seems to kick
 in to stop it from growing. It stabilizes at 426MB.

Using plain phobos, my testing stabilises at ~51MB. When minimize was 
used instead of fullCollect the memory usage stabilised at ~93MB

This is with 512MB of total RAM

  - Paul

torhu <fake address.dude> writes:

Paul Findlay wrote:
 Unfortunately this is unlikely to help but..

No, it's very interesting to hear what results other people get.

 Using plain phobos, my testing stabilises at ~51MB. When minimize was 
 used instead of fullCollect the memory usage stabilised at ~93MB
 
 This is with 512MB of total RAM

Ok, I have 896 MB.  I have a feeling that doesn't affect the GC, but I 
can't be sure.

I use dmd 1.007 on winxp sp2.  What are you using?

Paul Findlay <r.lph50+d gmail.com> writes:

 I use dmd 1.007 on winxp sp2.  What are you using?

Exactly the same..

My virtual memory is 768MB (I have no idea if this is revelevant, but in 
my spurious thinking, it may effect how the GC gives memory back to the OS)

  - Paul

Derek Parnell <derek psych.ward> writes:

On Wed, 07 Mar 2007 06:15:16 +0100, torhu wrote:

 I haven't dealt directly with the garbage collector much before, but 
 just used delete and scope to deal with deallocation.  And left some of 
 the minor allocations to be freed by the GC.
 
 Now I want to work more closely with the GC.  Can anyone explain to me 
 why fullCollect doesn't seem to do anything here? 

There is quite a large portion of the fullCollect source which is commented
out in phobos, mainly in the section which reclaims RAM (I think).

-- 
Derek Parnell
Melbourne, Australia
"Justice for David Hicks!"
skype: derek.j.parnell

torhu <fake address.dude> writes:

Derek Parnell wrote:
 On Wed, 07 Mar 2007 06:15:16 +0100, torhu wrote:
 Now I want to work more closely with the GC.  Can anyone explain to me 
 why fullCollect doesn't seem to do anything here? 

 
 There is quite a large portion of the fullCollect source which is commented
 out in phobos, mainly in the section which reclaims RAM (I think).

Do you mean gcx.d, line 1619?  That would explain some things. There's 
also something  else of interest in that file:

void minimize()	// minimize physical memory usage
{
     // Not implemented, ignore
}

Is this really the one being called as std.gc.minimize?  Looks like it. 
  I wish I had known this before.  I guess I should file a bug report; 
either implement it or make the docs state that it is currently not 
implemented.

If I knew what fullCollect() really was supposed to do, I could include 
that in the bug report too.  I'm hoping someone with a better 
understanding of this than me can help out here.

Sean Kelly <sean f4.ca> writes:

torhu wrote:
 Derek Parnell wrote:
 On Wed, 07 Mar 2007 06:15:16 +0100, torhu wrote:
 Now I want to work more closely with the GC.  Can anyone explain to 
 me why fullCollect doesn't seem to do anything here? 

 There is quite a large portion of the fullCollect source which is 
 commented
 out in phobos, mainly in the section which reclaims RAM (I think).

 
 Do you mean gcx.d, line 1619?  That would explain some things. There's 
 also something  else of interest in that file:
 
 void minimize()    // minimize physical memory usage
 {
     // Not implemented, ignore
 }
 
 Is this really the one being called as std.gc.minimize?  Looks like it. 
  I wish I had known this before.  I guess I should file a bug report; 
 either implement it or make the docs state that it is currently not 
 implemented.
 
 If I knew what fullCollect() really was supposed to do, I could include 
 that in the bug report too.  I'm hoping someone with a better 
 understanding of this than me can help out here.

fullCollect performs a full collection of orphaned memory.  With the 
current GC, this memory is recovered for future use by the GC, but it 
not returned to the OS.

torhu <fake address.dude> writes:

Sean Kelly wrote:
 fullCollect performs a full collection of orphaned memory.  With the 
 current GC, this memory is recovered for future use by the GC, but it 
 not returned to the OS.

It's supposed to recover memory for future use, but it doesn't do that. 
  Is that a correct description of the current situation?  I'm not sure 
what you're really saying, you seem to imply that it does actually does 
recover memory.  With my test app it sure doesn't reuse it, at least not 
for the next allocation that is made.

Sean Kelly <sean f4.ca> writes:

torhu wrote:
 With my test app it sure doesn't reuse it, at least not 
 for the next allocation that is made.

Yeah.  This is weird because I'd expect it to, but then I didn't write 
the GC.  When I get some time I'm going to debug the thing and see 
what's going on.