www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Running out of memory ctfe 16GB

reply Nierjerson <Nierjerson somewhere.com> writes:
I am running out of memory trying to generate CTFE code. It is 
quite large generation but just repetitive loop acting on an 
array item.

Surely 16GB should be enough to compile such a thing? I am using 
64-bit dmd. It climes to about 12GB then eventually drops down to 
around 1GB and then back up to 16GB and then quits.

I cannot generate the array in parts and stick them all together 
because they are all interdependent(but not much, nothing that 
should actually cost memory).

Seems that dmd not freeing up memory for speed is gonna start 
causing problems with complex template generation.

Is there any way to fix this? A special switch that will enable 
the compiler to reduce memory consumption(free unused stuff) or 
use the swap file?

https://github.com/IllusionSoftware/COM2D/

At the very least have something to give feedback on how to 
reduce memory consumption. Leaving things up in the air for 
programmers to stumble upon after a lot of work is not good.

On the "Error: Out of Memory" at least report some statistics on 
functions and lines and how much memory they have used and how 
many times they have been called.
Apr 06 2017
next sibling parent Jack Stouffer <jack jackstouffer.com> writes:
On Thursday, 6 April 2017 at 20:49:00 UTC, Nierjerson wrote:

 I am running out of memory trying to generate CTFE code. It is 
 quite large generation but just repetitive loop acting on an 
 array item.


CTFE is a memory hog, and there's not much you can do about it. A 
new CTFE engine is in the works, and it's "coming soon".

I can tell you that if the array is actually an AliasSeq or a 
compiler tuple then what the compiler is doing is generating a 
new chunk of code for every loop iteration and then interpreting 
that, leading to O(n) memory used for the loop. Also, try to 
reduce your template instantiations as much as possible.
Apr 06 2017
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 4/6/17 10:49 PM, Nierjerson wrote:

 I am running out of memory trying to generate CTFE code. It is quite
 large generation but just repetitive loop acting on an array item.

 Surely 16GB should be enough to compile such a thing? I am using 64-bit
 dmd. It climes to about 12GB then eventually drops down to around 1GB
 and then back up to 16GB and then quits.


I know how it feels.
CTFE allocating lots of objects and never bothering to free them is a 
known problem that has plagued D for years.
The new hope is the NewCTFE engine that ought to be reasonable on memory 
footprint and actually faster as an interpreter.


 At the very least have something to give feedback on how to reduce
 memory consumption. Leaving things up in the air for programmers to
 stumble upon after a lot of work is not good.


I just did a cursory look on the source but a few things you may try:

1) Use plain string instead of wstring for codegen, since this is simply 
source code which is ASCII I don't see a need for UTF-16. Technically 
should reduce the size of arrays involved x2.

2) Instead of building out out big string try building each interface 
separately. If I'm not mistaken current CTFE will actually allocate a 
whole new array on each append and copy things.
Since it never deallocates building up a huge string by bits and pieces 
is a bad idea as it will leak the entire thing on each append.


----
Dmitry Olshansky
Apr 06 2017
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
Dmitry Olshansky wrote:


 2) Instead of building out out big string try building each interface 
 separately. If I'm not mistaken current CTFE will actually allocate a 
 whole new array on each append and copy things.
 Since it never deallocates building up a huge string by bits and pieces 
 is a bad idea as it will leak the entire thing on each append.


or use `char[]` buffer instead, manually increasing it's size by some step. 
assigning to such array won't do any copies, so growing the array by 32kb 
(or even several mb) steps will reduce memory footprint for string builders 
considerably.
Apr 06 2017
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Apr 07, 2017 at 01:16:20AM +0300, ketmar via Digitalmars-d wrote:

 Dmitry Olshansky wrote:
 

 2) Instead of building out out big string try building each
 interface separately. If I'm not mistaken current CTFE will actually
 allocate a whole new array on each append and copy things.  Since it
 never deallocates building up a huge string by bits and pieces is a
 bad idea as it will leak the entire thing on each append.


 
 or use `char[]` buffer instead, manually increasing it's size by some
 step.  assigning to such array won't do any copies, so growing the
 array by 32kb (or even several mb) steps will reduce memory footprint
 for string builders considerably.


Are you sure?  AFAIK, the current CTFE engine will copy values every
time they are assigned, regardless of whether they are technically
"mutable" or not.


T

-- 
What's a "hot crossed bun"? An angry rabbit.
Apr 06 2017
next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 4/7/17 12:23 AM, H. S. Teoh via Digitalmars-d wrote:

 On Fri, Apr 07, 2017 at 01:16:20AM +0300, ketmar via Digitalmars-d wrote:

 Dmitry Olshansky wrote:


 2) Instead of building out out big string try building each
 interface separately. If I'm not mistaken current CTFE will actually
 allocate a whole new array on each append and copy things.  Since it
 never deallocates building up a huge string by bits and pieces is a
 bad idea as it will leak the entire thing on each append.


 or use `char[]` buffer instead, manually increasing it's size by some
 step.  assigning to such array won't do any copies, so growing the
 array by 32kb (or even several mb) steps will reduce memory footprint
 for string builders considerably.


 Are you sure?  AFAIK, the current CTFE engine will copy values every
 time they are assigned, regardless of whether they are technically
 "mutable" or not.


Yeah, imagine the worst possible way to implement an operation and
you have an idea of what CTFE will actually do.

Copying the whole array on element assignment is not out of question, 
though I recall there was some work done to prevent it at least in most 
cases.

----
Dmitry Olshansky
Apr 06 2017
prev sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
H. S. Teoh wrote:


 On Fri, Apr 07, 2017 at 01:16:20AM +0300, ketmar via Digitalmars-d wrote:

 Dmitry Olshansky wrote:
 

 2) Instead of building out out big string try building each
 interface separately. If I'm not mistaken current CTFE will actually
 allocate a whole new array on each append and copy things.  Since it
 never deallocates building up a huge string by bits and pieces is a
 bad idea as it will leak the entire thing on each append.


 or use `char[]` buffer instead, manually increasing it's size by some
 step.  assigning to such array won't do any copies, so growing the
 array by 32kb (or even several mb) steps will reduce memory footprint
 for string builders considerably.


 Are you sure?  AFAIK, the current CTFE engine will copy values every
 time they are assigned, regardless of whether they are technically
 "mutable" or not.


i'm sure: i'm using this trick alot. without that, CTFE won't be able to 
correctly work with arrays at all, 'cause assigning to mutable array 
element should be visible with all other slices of that array.

sure, CTFE can technically allocate "hidden array object" and re-assign it 
each time, but it actually has a concept of "owned by CTFE" objects, and if 
the array was created in CTFE, it becomes owned by CTFE, and copies will be 
made only on array resize.
Apr 06 2017
parent reply Nierjerson <Nierjerson somewhere.com> writes:
On Thursday, 6 April 2017 at 22:42:28 UTC, ketmar wrote:

 H. S. Teoh wrote:


 On Fri, Apr 07, 2017 at 01:16:20AM +0300, ketmar via 
 Digitalmars-d wrote:

 Dmitry Olshansky wrote:
 

 [...]


 or use `char[]` buffer instead, manually increasing it's size 
 by some
 step.  assigning to such array won't do any copies, so 
 growing the
 array by 32kb (or even several mb) steps will reduce memory 
 footprint
 for string builders considerably.


 Are you sure?  AFAIK, the current CTFE engine will copy values 
 every
 time they are assigned, regardless of whether they are 
 technically
 "mutable" or not.


 i'm sure: i'm using this trick alot. without that, CTFE won't 
 be able to correctly work with arrays at all, 'cause assigning 
 to mutable array element should be visible with all other 
 slices of that array.

 sure, CTFE can technically allocate "hidden array object" and 
 re-assign it each time, but it actually has a concept of "owned 
 by CTFE" objects, and if the array was created in CTFE, it 
 becomes owned by CTFE, and copies will be made only on array 
 resize.


How to implement trick is this and are you 100% sure it works?

e.g.,

char[] x;

x.length = 65536;
x.length = 0;

?

If if what you say is correct it should solve my memory problem. 
I do a lot of string appending and memory usage seems to grows 
exponentially.
Apr 07 2017
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
Nierjerson wrote:


 How to implement trick is this and are you 100% sure it works?

 e.g.,

 char[] x;

 x.length = 65536;
 x.length = 0;


this won't work. the moment you did `.length = 0;`, you are returned to 
point zero.

what you have to do is to maintain "shadow length" yourself, like this:

	x.length = 65536;
	size_t xpos = 0;
	
	void put (const(char)[] s...) {
		foreach (immutable char ch; s) {
			if (xpos == x.length) x.length += 65536; // or anything
			x[xpos++] = ch;
		}
	}
Apr 07 2017
parent reply Jethro <qyzz gr.ff> writes:
On Friday, 7 April 2017 at 21:02:33 UTC, ketmar wrote:

 Nierjerson wrote:


 How to implement trick is this and are you 100% sure it works?

 e.g.,

 char[] x;

 x.length = 65536;
 x.length = 0;


 this won't work. the moment you did `.length = 0;`, you are 
 returned to point zero.

 what you have to do is to maintain "shadow length" yourself, 
 like this:

 	x.length = 65536;
 	size_t xpos = 0;
 	
 	void put (const(char)[] s...) {
 		foreach (immutable char ch; s) {
 			if (xpos == x.length) x.length += 65536; // or anything
 			x[xpos++] = ch;
 		}
 	}


thanks, I'll try it... seems like it is basically appender though?
Apr 07 2017
parent ketmar <ketmar ketmar.no-ip.org> writes:
Jethro wrote:


 On Friday, 7 April 2017 at 21:02:33 UTC, ketmar wrote:

 Nierjerson wrote:


 How to implement trick is this and are you 100% sure it works?

 e.g.,

 char[] x;

 x.length = 65536;
 x.length = 0;


 this won't work. the moment you did `.length = 0;`, you are returned to 
 point zero.

 what you have to do is to maintain "shadow length" yourself, like this:

 	x.length = 65536;
 	size_t xpos = 0;
 	
 	void put (const(char)[] s...) {
 		foreach (immutable char ch; s) {
 			if (xpos == x.length) x.length += 65536; // or anything
 			x[xpos++] = ch;
 		}
 	}


 thanks, I'll try it... seems like it is basically appender though?


yeah. but the key here is to not use any fancy data structures, it is 
important. the more intermediaries you have, the less control over copying 
is left for you. also, no slice assigns too -- it is dangerous, as it can 
easy go out of control.
Apr 07 2017
prev sibling next sibling parent Stefan Koch <uplink.coder googlemail.com> writes:
On Thursday, 6 April 2017 at 20:49:00 UTC, Nierjerson wrote:

 I am running out of memory trying to generate CTFE code. It is 
 quite large generation but just repetitive loop acting on an 
 array item.

 [...]


Avoid function calls
Avoid Ranges (since those have alot of function calls 3 per 
iteration)
avoid AliasSeq

newCTFE is going to fix this!
And it should work for your case in 2-5 months.

I am sorry, but newCTFE work does take alot of time.
Apr 06 2017
prev sibling parent Alexander Breckel <example localhost.com> writes:
On Thursday, 6 April 2017 at 20:49:00 UTC, Nierjerson wrote:

 I am running out of memory trying to generate CTFE code. It is 
 quite large generation but just repetitive loop acting on an 
 array item.

 Surely 16GB should be enough to compile such a thing? I am 
 using 64-bit dmd. It climes to about 12GB then eventually drops 
 down to around 1GB and then back up to 16GB and then quits.

 I cannot generate the array in parts and stick them all 
 together because they are all interdependent(but not much, 
 nothing that should actually cost memory).

 Seems that dmd not freeing up memory for speed is gonna start 
 causing problems with complex template generation.

 Is there any way to fix this? A special switch that will enable 
 the compiler to reduce memory consumption(free unused stuff) or 
 use the swap file?

 https://github.com/IllusionSoftware/COM2D/

 At the very least have something to give feedback on how to 
 reduce memory consumption. Leaving things up in the air for 
 programmers to stumble upon after a lot of work is not good.

 On the "Error: Out of Memory" at least report some statistics 
 on functions and lines and how much memory they have used and 
 how many times they have been called.


Some years ago I managed to force DMD to collect memory during 
CTFE by loading the malloc replacement of Boehm GC using 
LD_PRELOAD on Linux. It sounds totally crazy, but it worked...

Check the last part called "Simplified leak detection under 
Linux" of this link:

https://www.hboehm.info/gc/leak.html

You can ignore the leak detection aspect and just build and 
preload libgc.so. It will (very conservatively) collect memory if 
a malloc fails. Lets hope DMD drops old references during CTFE...

Please report back if this still works.
Apr 07 2017