www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - whatever I do, things get slower and bigger

reply BCS <ao pathlink.com> writes:
I'm working on my parser generator and trying to shoe horn it into a smaller 
memory foot print. One approach I'm trying, is to replace recursive templates 
with CTFE function and string mixins. This is in an effort to reduce the 
number of types that get declared (mostly I'm using struct templates). The 
problem is that the CTFE functions seem to actually use MORE ram than the 
templates do. I've tried a number of things like converting to an O(n log 
n) concatenation system rather than a O(n^2) and other such tricks but
everything 
seems to be going backwards. Any advice? (D v1 BTW) I can post code if anyone 
cares (it's kind of large: 1.2KLOC, takes a while to compile: ~6min, and 
use a lot of RAM: ~2GB).

(after a bit more fiddling)

I've got it down to 1GB of ram (Yeah!). My conclusions: don't build strings 
with CTFE unless it's a lot harder to do it with templates. It seems that 
CTFE is a bigger memory hog than templates even with temples memoizing
everything 
into the symbol tables.
Mar 13 2008
parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
BCS wrote:

 I'm working on my parser generator and trying to shoe horn it into a 
 smaller memory foot print. One approach I'm trying, is to replace 
 recursive templates with CTFE function and string mixins. This is in an 
 effort to reduce the number of types that get declared (mostly I'm using 
 struct templates). The problem is that the CTFE functions seem to 
 actually use MORE ram than the templates do. I've tried a number of 
 things like converting to an O(n log n) concatenation system rather than 
 a O(n^2) and other such tricks but everything seems to be going 
 backwards. Any advice? (D v1 BTW) I can post code if anyone cares (it's 
 kind of large: 1.2KLOC, takes a while to compile: ~6min, and use a lot 
 of RAM: ~2GB).
 
 (after a bit more fiddling)
 
 I've got it down to 1GB of ram (Yeah!). My conclusions: don't build 
 strings with CTFE unless it's a lot harder to do it with templates. It 
 seems that CTFE is a bigger memory hog than templates even with temples 
 memoizing everything into the symbol tables.
 
 


Yes I believe Don has previously reported that CTFE manipulations on 
strings basically *never* free the intermediate strings.  Basically the 
CTFE engine needs a garbage collector (or equiv).

--bb
Mar 13 2008
next sibling parent Moritz Warning <moritzwarning _nospam_web.de> writes:
On Fri, 14 Mar 2008 08:42:27 +0900, Bill Baxter wrote:


 BCS wrote:

 I'm working on my parser generator and trying to shoe horn it into a
 smaller memory foot print. One approach I'm trying, is to replace
 recursive templates with CTFE function and string mixins. This is in an
 effort to reduce the number of types that get declared (mostly I'm
 using struct templates). The problem is that the CTFE functions seem to
 actually use MORE ram than the templates do. I've tried a number of
 things like converting to an O(n log n) concatenation system rather
 than a O(n^2) and other such tricks but everything seems to be going
 backwards. Any advice? (D v1 BTW) I can post code if anyone cares (it's
 kind of large: 1.2KLOC, takes a while to compile: ~6min, and use a lot
 of RAM: ~2GB).
 
 (after a bit more fiddling)
 
 I've got it down to 1GB of ram (Yeah!). My conclusions: don't build
 strings with CTFE unless it's a lot harder to do it with templates. It
 seems that CTFE is a bigger memory hog than templates even with temples
 memoizing everything into the symbol tables.
 
 
 


 Yes I believe Don has previously reported that CTFE manipulations on
 strings basically *never* free the intermediate strings.  Basically the
 CTFE engine needs a garbage collector (or equiv).
 
 --bb


Looks like D needs a script engine for CTFE
that makes use of a GC. Would be a good feature imho.

Reminds me of Virgil when I heard about in a post some time ago.
Virgil uses a compile time GC that includes reachable memory
into the binary for some kind of memory pre-allocation.
http://compilers.cs.ucla.edu/virgil/ref/pubs.html

my2cents
Mar 13 2008
prev sibling next sibling parent reply "Jarrett Billingsley" <kb3ctd2 yahoo.com> writes:
"Bill Baxter" <dnewsgroup billbaxter.com> wrote in message 
news:frce57$1vf5$1 digitalmars.com...

 Yes I believe Don has previously reported that CTFE manipulations on 
 strings basically *never* free the intermediate strings.  Basically the 
 CTFE engine needs a garbage collector (or equiv).

 --bb


You know, if we had a D compiler made in D, it'd be GCed.






<_<

_>


...

tiny print: dil
Mar 13 2008
parent BCS <ao pathlink.com> writes:
Reply to Jarrett,


 "Bill Baxter" <dnewsgroup billbaxter.com> wrote in message
 news:frce57$1vf5$1 digitalmars.com...
 

 Yes I believe Don has previously reported that CTFE manipulations on
 strings basically *never* free the intermediate strings.  Basically
 the CTFE engine needs a garbage collector (or equiv).
 
 --bb
 


 You know, if we had a D compiler made in D, it'd be GCed.
 


Odd, that's what the project the problem showed up on is.

My plan for CTFE is to compile all code to a high level IL byte code that 
can be converted to machine code for object files but can also be executed 
directly on some sort of VM, I should be able to shoehorn in a GC (or better) 
somewhere in that .
Mar 13 2008
prev sibling parent reply Robert Fraser <fraserofthenight gmail.com> writes:
Bill Baxter wrote:

 BCS wrote:

 I'm working on my parser generator and trying to shoe horn it into a 
 smaller memory foot print. One approach I'm trying, is to replace 
 recursive templates with CTFE function and string mixins. This is in 
 an effort to reduce the number of types that get declared (mostly I'm 
 using struct templates). The problem is that the CTFE functions seem 
 to actually use MORE ram than the templates do. I've tried a number of 
 things like converting to an O(n log n) concatenation system rather 
 than a O(n^2) and other such tricks but everything seems to be going 
 backwards. Any advice? (D v1 BTW) I can post code if anyone cares 
 (it's kind of large: 1.2KLOC, takes a while to compile: ~6min, and use 
 a lot of RAM: ~2GB).

 (after a bit more fiddling)

 I've got it down to 1GB of ram (Yeah!). My conclusions: don't build 
 strings with CTFE unless it's a lot harder to do it with templates. It 
 seems that CTFE is a bigger memory hog than templates even with 
 temples memoizing everything into the symbol tables.


 
 Yes I believe Don has previously reported that CTFE manipulations on 
 strings basically *never* free the intermediate strings.  Basically the 
 CTFE engine needs a garbage collector (or equiv).
 
 --bb


grep for 'delete' in the front-end code. It's not there very much :-).
Mar 13 2008
parent reply Christopher Wright <dhasenan gmail.com> writes:
Robert Fraser wrote:

 grep for 'delete' in the front-end code. It's not there very much :-).


Maybe Walter should use the Boehm collector for the frontend?
Mar 14 2008
parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Christopher Wright wrote:

 Robert Fraser wrote:

 grep for 'delete' in the front-end code. It's not there very much :-).


 
 Maybe Walter should use the Boehm collector for the frontend?


Doesn't he already do that? (He has often stated he uses a collector, 
and I'm not aware of any others for C++)
Mar 14 2008
parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Frits van Bommel wrote:

 Christopher Wright wrote:

 Robert Fraser wrote:

 grep for 'delete' in the front-end code. It's not there very much :-).


 Maybe Walter should use the Boehm collector for the frontend?


 
 Doesn't he already do that? (He has often stated he uses a collector, 
 and I'm not aware of any others for C++)


There's a mem.h that looks designed to potentially wrap a GC, but the 
actual implementation just calls system malloc and free.
Contains the comment:
/* This implementation of the storage allocator uses the standard C 
allocation package.
  */

That package's mem.malloc is used in constfold.c where a lot of the CTFE 
magic happens, so maybe wiring up mem.c to Boehm would fix it.  Looks 
like it wouldn't be too difficult to try out for someone who is able to 
build a working compiler.  So far that does not include me. :-(

--bb
Mar 14 2008
parent Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Bill Baxter wrote:

 Frits van Bommel wrote:

 Christopher Wright wrote:

 Robert Fraser wrote:

 grep for 'delete' in the front-end code. It's not there very much :-).


 Maybe Walter should use the Boehm collector for the frontend?


 Doesn't he already do that? (He has often stated he uses a collector, 
 and I'm not aware of any others for C++)


 
 There's a mem.h that looks designed to potentially wrap a GC, but the 
 actual implementation just calls system malloc and free.
 Contains the comment:
 /* This implementation of the storage allocator uses the standard C 
 allocation package.
  */
 
 That package's mem.malloc is used in constfold.c where a lot of the CTFE 
 magic happens, so maybe wiring up mem.c to Boehm would fix it.  Looks 
 like it wouldn't be too difficult to try out for someone who is able to 
 build a working compiler.  So far that does not include me. :-(


Oh, I guess that file (mem.c) was modified by Lindquist for LLVMDC (the 
derivative of DMDFE I'm most familiar with).
Anyone who wants to hook DMDFE up to Boehm can probably just use that 
version (<http://www.dsource.org/projects/llvmdc/browser/trunk/dmd/mem.c>).

Still, Walter claims to use a GC (see 
<http://www.digitalmars.com/d/1.0/faq.html#q7_3>). Though that doesn't 
specify he uses it in DMD itself, so maybe that's the source of the 
misunderstanding?
Mar 14 2008