www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - is it a bug or what?

reply "collerblade" <cccc cc.cc> writes:
Dear D user, i have this code:

import core.sys.windows.windows;

real[] a;
while(1) {
   a.length=4096*4096;
   a=null;
		
   Sleep(2000);
}

It allocates memory, but its never gets freed, just keep going 
up, and after 10 secs, memoryexception is thrown. I checked the 
pointer it is GC safe (attributes is 10). Am i missing something?

TY: collerblade
Jan 22 2015
next sibling parent reply "collerblade" <cccc cc.cc> writes:
Also forget to mention, that im using DMD 2.066.1.
If i alloc memory with GC.malloc(..), it is handled correcty (and 
freed).
Why GC doesn't frees dynamic arrays?
Jan 22 2015
parent "collerblade" <cccc cc.cc> writes:
i tested malloc, and it also run out of mem, except when i 
manually free the pointer.
There is no GC im DMD?
Jan 22 2015
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 1/22/15 8:44 AM, collerblade wrote:

 Dear D user, i have this code:

 import core.sys.windows.windows;

 real[] a;
 while(1) {
    a.length=4096*4096;
    a=null;

    Sleep(2000);
 }

 It allocates memory, but its never gets freed, just keep going up, and
 after 10 secs, memoryexception is thrown. I checked the pointer it is GC
 safe (attributes is 10). Am i missing something?


Are you compiling 64-bit code or 32-bit? It makes a huge difference.

A real is going to be 16-byte aligned. This means real[4096*4096] is 
allocating 1/4GB per allocation. This consumes 1/16th of your total 
address space on 32-bit address space.

The GC in D is conservative. This means that on 32-bit architecture, any 
4-byte segment scanned by the GC (whether it's a pointer or not) that 
happens to point into the array you allocated is going to keep that 
memory pinned. Because any 4-byte garbage has a 1:16 chance of pointing 
there, odds are good that this will happen. Then it keeps happening, you 
eventually run out of memory.

Now, some possible explanation of why GC.malloc might work while this 
does not:

GC.malloc allocates exactly 4096*4096*16 bytes. Setting a.length goes 
through the append code. Because extending length is considered an 
appending event, it *over-allocates* by a certain factor, expecting you 
to keep appending. (this is how appending can be amortized O(1) 
performance). This means the amount of memory allocated is larger than 
1/4GB. Therefore the chance of failing by setting a.length is going to 
be higher than with using GC.malloc directly, but it could fail with 
either case.

Answers?

1. Manually free such a large array when done with it.
2. Do not allocate such large arrays.
3. Use 64-bit code generation (-m64 on DMD windows, even if you are on 
Win64) if possible.

-Steve
Jan 22 2015
parent reply "collerblade" <cccc cc.cc> writes:
On Thursday, 22 January 2015 at 14:19:10 UTC, Steven 
Schveighoffer wrote:

 On 1/22/15 8:44 AM, collerblade wrote:

 Dear D user, i have this code:

 import core.sys.windows.windows;

 real[] a;
 while(1) {
   a.length=4096*4096;
   a=null;

   Sleep(2000);
 }

 It allocates memory, but its never gets freed, just keep going 
 up, and
 after 10 secs, memoryexception is thrown. I checked the 
 pointer it is GC
 safe (attributes is 10). Am i missing something?


 Are you compiling 64-bit code or 32-bit? It makes a huge 
 difference.

 A real is going to be 16-byte aligned. This means 
 real[4096*4096] is allocating 1/4GB per allocation. This 
 consumes 1/16th of your total address space on 32-bit address 
 space.

 The GC in D is conservative. This means that on 32-bit 
 architecture, any 4-byte segment scanned by the GC (whether 
 it's a pointer or not) that happens to point into the array you 
 allocated is going to keep that memory pinned. Because any 
 4-byte garbage has a 1:16 chance of pointing there, odds are 
 good that this will happen. Then it keeps happening, you 
 eventually run out of memory.

 Now, some possible explanation of why GC.malloc might work 
 while this does not:

 GC.malloc allocates exactly 4096*4096*16 bytes. Setting 
 a.length goes through the append code. Because extending length 
 is considered an appending event, it *over-allocates* by a 
 certain factor, expecting you to keep appending. (this is how 
 appending can be amortized O(1) performance). This means the 
 amount of memory allocated is larger than 1/4GB. Therefore the 
 chance of failing by setting a.length is going to be higher 
 than with using GC.malloc directly, but it could fail with 
 either case.

 Answers?

 1. Manually free such a large array when done with it.
 2. Do not allocate such large arrays.
 3. Use 64-bit code generation (-m64 on DMD windows, even if you 
 are on Win64) if possible.

 -Steve


TY for your answer.
I checked malloc and it is the same. It runs out if memory too.
Also tried with int[]->result is the same.
Memory is initialized to 0-s, so there is no pointer pointing to 
memory. At least at my side. Are u saiying, that most likely 
other GC allocated memory points to my arrays? Hm thats maybe 
true. If thats a case i have to manage memory myself.
Also what about scpoed alloc?

scoped int[] a=new int[whatever_big_size];

It sould free memory but it does not :( :(.

D is soo convinient. i never wanna use malloc again :D :D :D
Jan 22 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 1/22/15 9:40 AM, collerblade wrote:

 On Thursday, 22 January 2015 at 14:19:10 UTC, Steven Schveighoffer wrote:

 On 1/22/15 8:44 AM, collerblade wrote:

 Dear D user, i have this code:

 import core.sys.windows.windows;

 real[] a;
 while(1) {
   a.length=4096*4096;
   a=null;

   Sleep(2000);
 }

 It allocates memory, but its never gets freed, just keep going up, and
 after 10 secs, memoryexception is thrown. I checked the pointer it is GC
 safe (attributes is 10). Am i missing something?


 Are you compiling 64-bit code or 32-bit? It makes a huge difference.

 A real is going to be 16-byte aligned. This means real[4096*4096] is
 allocating 1/4GB per allocation. This consumes 1/16th of your total
 address space on 32-bit address space.

 The GC in D is conservative. This means that on 32-bit architecture,
 any 4-byte segment scanned by the GC (whether it's a pointer or not)
 that happens to point into the array you allocated is going to keep
 that memory pinned. Because any 4-byte garbage has a 1:16 chance of
 pointing there, odds are good that this will happen. Then it keeps
 happening, you eventually run out of memory.

 Now, some possible explanation of why GC.malloc might work while this
 does not:

 GC.malloc allocates exactly 4096*4096*16 bytes. Setting a.length goes
 through the append code. Because extending length is considered an
 appending event, it *over-allocates* by a certain factor, expecting
 you to keep appending. (this is how appending can be amortized O(1)
 performance). This means the amount of memory allocated is larger than
 1/4GB. Therefore the chance of failing by setting a.length is going to
 be higher than with using GC.malloc directly, but it could fail with
 either case.

 Answers?

 1. Manually free such a large array when done with it.
 2. Do not allocate such large arrays.
 3. Use 64-bit code generation (-m64 on DMD windows, even if you are on
 Win64) if possible.

 -Steve


 TY for your answer.


Before I respond, first question to answer is if you are using 32-bit 
compiled code. If not, then it's not typical for what you are doing to 
cause problems.


 I checked malloc and it is the same. It runs out if memory too.
 Also tried with int[]->result is the same.
 Memory is initialized to 0-s, so there is no pointer pointing to memory.
 At least at my side. Are u saiying, that most likely other GC allocated
 memory points to my arrays? Hm thats maybe true. If thats a case i have
 to manage memory myself.


Most likely it's a stack variable or global/TLS variable. Those are 
scanned too :)

Also note that reals are NOT initialized to 0, but NaN.


 Also what about scpoed alloc?

 scoped int[] a=new int[whatever_big_size];

 It sould free memory but it does not :( :(.


I would not use this, I think it's scheduled for deprecation, or at 
least the meaning is scheduled to change.

Note that a must go out of scope pointing at the array that was 
allocated in order to free the memory. You can't set it to null, because 
then nothing would be freed!

-Steve
Jan 22 2015
next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Steven Schveighoffer:


 scoped int[] a=new int[whatever_big_size];
 ...


 I would not use this, I think it's scheduled for deprecation, 
 or at least the meaning is scheduled to change.


As D introduces some limited static management of memory 
ownership, the semantics of scoped arrays could be fixed.

Bye,
bearophile
Jan 22 2015
prev sibling parent "collerblade" <cccc cc.cc> writes:
Thank all of u,
im going to use malloc.
Jan 22 2015