www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Memory allocation failed. Why?

reply MGW <mgw yandex.ru> writes:
import core.sys.windows.windows: MessageBoxA;

void test() {
	for(int i; i != 10; i++) {
		ubyte[] buf;
		for(int j; j != 100000000; j++) buf ~= 65;
		MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
		// delete buf; // if ON - then memory delete in step
	}
	MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
}

void main() {
	test();
	MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
}

----------------
core.exception.OutOfMemoryError src\core\exception.d(693): Memory 
allocation failed
----------------

Simple program and error. Why?  Windows 7 (32) dmd 2.072.0
Nov 20 2016
next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:

 import core.sys.windows.windows: MessageBoxA;

 void test() {
 	for(int i; i != 10; i++) {
 		ubyte[] buf;
 		for(int j; j != 100000000; j++) buf ~= 65;
 		MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
 		// delete buf; // if ON - then memory delete in step
 	}
 	MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
 }

 void main() {
 	test();
 	MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
 }

 ----------------
 core.exception.OutOfMemoryError src\core\exception.d(693): 
 Memory allocation failed
 ----------------

 Simple program and error. Why?  Windows 7 (32) dmd 2.072.0


For me there's no exception. Maybe the GC is poluted. Try to add 
this after each iteration in the first test loop:

         import core.memory: GC;
         GC.collect();

Also note that the text you pas in the message box should be null 
terminated, eg:

         MessageBoxA(null, "--The end--\0".ptr, "".ptr, 0);
Nov 20 2016
next sibling parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote:

 On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:

 [...]


 For me there's no exception. Maybe the GC is poluted. Try to 
 add this after each iteration in the first test loop:

         import core.memory: GC;
         GC.collect();

 Also note that the text you pas in the message box should be 
 null terminated, eg:

         MessageBoxA(null, "--The end--\0".ptr, "".ptr, 0);


string literals are implicitly null terminated explicitly for the 
interoperation with C.
there is no need.
Nov 20 2016
prev sibling parent reply MGW <mgw yandex.ru> writes:
On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote:

 On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:

 import core.sys.windows.windows: MessageBoxA;

 void test() {
 	for(int i; i != 10; i++) {
 		ubyte[] buf;
 		for(int j; j != 100000000; j++) buf ~= 65;
 		MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
 		// delete buf; // if ON - then memory delete in step
 	}
 	MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
 }

 void main() {
 	test();
 	MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
 }

 ----------------
 core.exception.OutOfMemoryError src\core\exception.d(693): 
 Memory allocation failed
 ----------------




This short program doesn't work at Windows 32. Why GC doesn't 
release memory in case of its shortage in Windows! This is bug?
Nov 20 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 21 November 2016 at 03:58:00 UTC, MGW wrote:

 On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote:

 On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:

 import core.sys.windows.windows: MessageBoxA;

 void test() {
 	for(int i; i != 10; i++) {
 		ubyte[] buf;
 		for(int j; j != 100000000; j++) buf ~= 65;
 		MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
 		// delete buf; // if ON - then memory delete in step
 	}
 	MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
 }

 void main() {
 	test();
 	MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
 }

 ----------------
 core.exception.OutOfMemoryError src\core\exception.d(693): 
 Memory allocation failed
 ----------------




 This short program doesn't work at Windows 32. Why GC doesn't 
 release memory in case of its shortage in Windows! This is bug?


You are appending 100_000_000 Times.
Note that this will take up much more then 100 MB.
Because the allocator cannot always grow the array.
All previously allocated blocks are still in scope and could be 
reached.
Therefore the memory is not released.

At least this is what I think happens.
it would help more if you could post the specs of your machine 
and what else is running on it.
Nov 20 2016
parent reply ag0aep6g <anonymous example.com> writes:
On 11/21/2016 07:36 AM, Stefan Koch wrote:

 On Monday, 21 November 2016 at 03:58:00 UTC, MGW wrote:

 On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote:

 On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:

 import core.sys.windows.windows: MessageBoxA;

 void test() {
     for(int i; i != 10; i++) {
         ubyte[] buf;
         for(int j; j != 100000000; j++) buf ~= 65;
         MessageBoxA(null, "--on for--".ptr, "".ptr, 0);
         // delete buf; // if ON - then memory delete in step
     }
     MessageBoxA(null, "--off for--".ptr, "".ptr, 0);
 }

 void main() {
     test();
     MessageBoxA(null, "--The end--".ptr, "".ptr, 0);
 }








[...]

 You are appending 100_000_000 Times.
 Note that this will take up much more then 100 MB.
 Because the allocator cannot always grow the array.
 All previously allocated blocks are still in scope and could be reached.
 Therefore the memory is not released.


How could they be reached? The only reference is `buf', and when 
appending triggers relocation, `buf` should point to the new location, no?
Nov 20 2016
parent reply Stefan Koch <uplink.coder googlemail.com> writes:
On Monday, 21 November 2016 at 06:45:04 UTC, ag0aep6g wrote:

 On 11/21/2016 07:36 AM, Stefan Koch wrote:

 On Monday, 21 November 2016 at 03:58:00 UTC, MGW wrote:

 On Sunday, 20 November 2016 at 18:58:04 UTC, Basile B. wrote:

 On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:

 [...]








 [...]

 You are appending 100_000_000 Times.
 Note that this will take up much more then 100 MB.
 Because the allocator cannot always grow the array.
 All previously allocated blocks are still in scope and could 
 be reached.
 Therefore the memory is not released.


 How could they be reached? The only reference is `buf', and 
 when appending triggers relocation, `buf` should point to the 
 new location, no?


Someone could still be hanging on to an old Reference of buf.
Nov 20 2016
parent reply ag0aep6g <anonymous example.com> writes:
On 11/21/2016 08:27 AM, Stefan Koch wrote:

 Someone could still be hanging on to an old Reference of buf.


Who could "someone" be? It's a self-contained example, and buf doesn't 
leave the test function.
Nov 21 2016
parent reply Kagamin <spam here.lot> writes:
On Monday, 21 November 2016 at 11:22:40 UTC, ag0aep6g wrote:

 Who could "someone" be? It's a self-contained example, and buf 
 doesn't leave the test function.


Anything in .data and .bss sections and stack. See 
https://issues.dlang.org/show_bug.cgi?id=15723

As for GC compaction: 
https://issues.dlang.org/show_bug.cgi?id=3284
Nov 21 2016
parent reply ag0aep6g <anonymous example.com> writes:
On Monday, 21 November 2016 at 16:37:32 UTC, Kagamin wrote:

 Anything in .data and .bss sections and stack. See 
 https://issues.dlang.org/show_bug.cgi?id=15723


Ok, not an actual reference then, but a false pointer.
Nov 21 2016
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 11/21/16 11:53 AM, ag0aep6g wrote:

 On Monday, 21 November 2016 at 16:37:32 UTC, Kagamin wrote:

 Anything in .data and .bss sections and stack. See
 https://issues.dlang.org/show_bug.cgi?id=15723


 Ok, not an actual reference then, but a false pointer.


Yes. 100 million bytes is 1/40 of all addressable space on 32-bits. 
There only needs to be one 4-byte segment somewhere on the stack that 
points at this, and it won't be collected.

Assuming you have a quite large segment that can't be extended or 
collected (due to false pointer), this means you have to allocate 
another large one to satisfy the next allocation (which then could be 
pinned). And it gets worse from there.

-Steve
Nov 22 2016
parent MGW <mgw yandex.ru> writes:
On Tuesday, 22 November 2016 at 15:53:39 UTC, Steven 
Schveighoffer wrote:

 On 11/21/16 11:53 AM, ag0aep6g wrote:


Thank you very much for explaining such a difficult and slippery 
situation.
Nov 23 2016
prev sibling parent thedeemon <dlang thedeemon.com> writes:
On Sunday, 20 November 2016 at 17:47:50 UTC, MGW wrote:


 ----------------
 core.exception.OutOfMemoryError src\core\exception.d(693): 
 Memory allocation failed
 ----------------

 Simple program and error. Why?  Windows 7 (32) dmd 2.072.0


Making a 100 million bytes array by appending one byte at a time 
creates a lot of intermediate-size arrays. Ideally these should 
be garbage collected away but GC in D is not only slow but also 
leaky. In 32 bits if you have 1000 random int values on the stack 
or data segment, with uniform distribution, this is 1000 random 
locations in memory pinned and seen by GC as live, i.e. one per 4 
MB of address space. Which means if your array is 4 MB or larger 
it's almost doomed to be never collected by GC in this scenario. 
Your program creates a lot of large arrays and they don't get 
collected because of false pointers and not precise enough GC. 
Moral of the story: in 32 bits don't allocate anything big (1 MB 
or more) in GC heap, otherwise there are good chances it will 
create a memory leak. Use std.container.array  or something 
similar.
Nov 21 2016