www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - dmd memory usage

reply Steven Schveighoffer <schveiguy gmail.com> writes:
I'm fighting some out of memory problems using DMD and some 
super-template heavy code.

I have ideas on how to improve the situation, but it involves 
redesigning a large portion of the design. I want to do it 
incrementally, but I need to see things improving.

Is there a straightforward way to figure out how much memory the 
compiler uses during compilation? I though maybe /usr/bin/time, but I 
feel like I don't trust the output to be the true max resident size to 
be what I'm looking for (or that it's 100% accurate). Is there a 
sure-fire way to have DMD print it's footprint?

-Steve
Nov 17 2019
next sibling parent reply Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer 
wrote:

 Is there a straightforward way to figure out how much memory 
 the compiler uses during compilation?


So this isn't a great solution but what I have done in the past 
is just have a little script running

ps aux | grep dmd


in a loop running in the background to gather some data as it 
goes. If combined with dmd -v you can interpolate the -v and ps 
output to get an idea of what the memory use is at a certain 
point of what the compiler is working on.

again i know it isn't great but it is easy to play with and 
better than nothing
Nov 17 2019
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/17/19 7:33 PM, Adam D. Ruppe wrote:

 On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer wrote:

 Is there a straightforward way to figure out how much memory the 
 compiler uses during compilation?


 
 So this isn't a great solution but what I have done in the past is just 
 have a little script running
 
 ps aux | grep dmd

 
 in a loop running in the background to gather some data as it goes. If 
 combined with dmd -v you can interpolate the -v and ps output to get an 
 idea of what the memory use is at a certain point of what the compiler 
 is working on.
 
 again i know it isn't great but it is easy to play with and better than 
 nothing


Yeah, I started trying out valgrind, which seems to give a decent 
report. It slows down the compiler a bit, but that's ok.

In any case, I realized that I need to write some more targeted tests, 
because the template I want to replace is used all over the place, so I 
would need to change them all in order to reduce memory usage.

-Steve
Nov 18 2019
prev sibling next sibling parent reply ikod <igor.khasilev gmail.com> writes:
On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer 
wrote:

 I'm fighting some out of memory problems using DMD and some 
 super-template heavy code.

 I have ideas on how to improve the situation, but it involves 
 redesigning a large portion of the design. I want to do it 
 incrementally, but I need to see things improving.

 Is there a straightforward way to figure out how much memory 
 the compiler uses during compilation? I though maybe 
 /usr/bin/time, but I feel like I don't trust the output to be 
 the true max resident size to be what I'm looking for (or that 
 it's 100% accurate). Is there a sure-fire way to have DMD print 
 it's footprint?

 -Steve


Hello,
You can look at
http://man7.org/linux/man-pages/man2/getrusage.2.html
But, probably /usr/bin/time already use it.
Nov 17 2019
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/17/19 7:48 PM, ikod wrote:

 On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer wrote:

 I'm fighting some out of memory problems using DMD and some 
 super-template heavy code.

 I have ideas on how to improve the situation, but it involves 
 redesigning a large portion of the design. I want to do it 
 incrementally, but I need to see things improving.

 Is there a straightforward way to figure out how much memory the 
 compiler uses during compilation? I though maybe /usr/bin/time, but I 
 feel like I don't trust the output to be the true max resident size to 
 be what I'm looking for (or that it's 100% accurate). Is there a 
 sure-fire way to have DMD print it's footprint?


 
 Hello,
 You can look at
 http://man7.org/linux/man-pages/man2/getrusage.2.html
 But, probably /usr/bin/time already use it.
 


Yeah, that looks like what /usr/bin/time is doing.

-Steve
Nov 18 2019
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer 
wrote:

 I'm fighting some out of memory problems using DMD and some 
 super-template heavy code.


Try to enable the GC using the `-lowmem` flag [1].

[1] https://dlang.org/changelog/2.086.0.html#lowmem

—
/Jacob Carlborg
Nov 18 2019
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/18/19 3:16 AM, Jacob Carlborg wrote:

 On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer wrote:

 I'm fighting some out of memory problems using DMD and some 
 super-template heavy code.


 
 Try to enable the GC using the `-lowmem` flag [1].
 
 [1] https://dlang.org/changelog/2.086.0.html#lowmem
 


I don't think this will work for me, as this isn't CTFE, and I'm 
assuming the instantiated templates won't be collected because they stay 
referenced for the whole program.

-Steve
Nov 18 2019
parent reply Jacob Carlborg <doob me.com> writes:
On Monday, 18 November 2019 at 15:31:02 UTC, Steven Schveighoffer 
wrote:

 I don't think this will work for me, as this isn't CTFE, and 
 I'm assuming the instantiated templates won't be collected 
 because they stay referenced for the whole program.


You could also combine that flag with the GC profiling flags.

—
/Jacob Carlborg
Nov 18 2019
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/18/19 10:36 AM, Jacob Carlborg wrote:

 On Monday, 18 November 2019 at 15:31:02 UTC, Steven Schveighoffer wrote:

 I don't think this will work for me, as this isn't CTFE, and I'm 
 assuming the instantiated templates won't be collected because they 
 stay referenced for the whole program.


 
 You could also combine that flag with the GC profiling flags.
 


That's a good point. I tried the profiling flags, but realized really 
quickly that it doesn't use the GC. But if you told it to use the GC, 
then that info becomes relevant.

I also noticed that dub suppresses the output of the profiler somehow (I 
had to run it with -v, and then run the compile line by hand to get it 
to do the profiling).

-Steve
Nov 18 2019
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/18/19 11:15 AM, Steven Schveighoffer wrote:

 On 11/18/19 10:36 AM, Jacob Carlborg wrote:

 On Monday, 18 November 2019 at 15:31:02 UTC, Steven Schveighoffer wrote:

 I don't think this will work for me, as this isn't CTFE, and I'm 
 assuming the instantiated templates won't be collected because they 
 stay referenced for the whole program.


 You could also combine that flag with the GC profiling flags.


 
 That's a good point. I tried the profiling flags, but realized really 
 quickly that it doesn't use the GC. But if you told it to use the GC, 
 then that info becomes relevant.
 
 I also noticed that dub suppresses the output of the profiler somehow (I 
 had to run it with -v, and then run the compile line by hand to get it 
 to do the profiling).
 


Yeah, as I expected it still gets killed by the OOM killer.

But still, good to know that switch exists.

-Steve
Nov 18 2019
prev sibling next sibling parent Dejan Lekic <dejan.lekic gmail.com> writes:
On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer 
wrote:

 I'm fighting some out of memory problems using DMD and some 
 super-template heavy code.

 I have ideas on how to improve the situation, but it involves 
 redesigning a large portion of the design. I want to do it 
 incrementally, but I need to see things improving.

 Is there a straightforward way to figure out how much memory 
 the compiler uses during compilation? I though maybe 
 /usr/bin/time, but I feel like I don't trust the output to be 
 the true max resident size to be what I'm looking for (or that 
 it's 100% accurate). Is there a sure-fire way to have DMD print 
 it's footprint?

 -Steve


You can wrap the whole thing in a shell script that takes PID of 
the compiler, and uses psrecord [1] Python tool to give you CPU 
and memory chart.

[1] https://pypi.org/project/psrecord/
Nov 18 2019
prev sibling next sibling parent Max Haughton <maxhaton gmail.com> writes:
On Monday, 18 November 2019 at 00:20:12 UTC, Steven Schveighoffer 
wrote:

 I'm fighting some out of memory problems using DMD and some 
 super-template heavy code.

 I have ideas on how to improve the situation, but it involves 
 redesigning a large portion of the design. I want to do it 
 incrementally, but I need to see things improving.

 Is there a straightforward way to figure out how much memory 
 the compiler uses during compilation? I though maybe 
 /usr/bin/time, but I feel like I don't trust the output to be 
 the true max resident size to be what I'm looking for (or that 
 it's 100% accurate). Is there a sure-fire way to have DMD print 
 it's footprint?

 -Steve


Massif is good for this. ms_print will give you a graphical 
summary, and the data is human and machine readable.

The only setback being that massif can make the execution slower 
however I can't give exact numbers but it can be terrible.
Nov 18 2019
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2019-11-18 01:20, Steven Schveighoffer wrote:

 I'm fighting some out of memory problems using DMD and some 
 super-template heavy code.
 
 I have ideas on how to improve the situation, but it involves 
 redesigning a large portion of the design. I want to do it 
 incrementally, but I need to see things improving.
 
 Is there a straightforward way to figure out how much memory the 
 compiler uses during compilation? I though maybe /usr/bin/time, but I 
 feel like I don't trust the output to be the true max resident size to 
 be what I'm looking for (or that it's 100% accurate). Is there a 
 sure-fire way to have DMD print it's footprint?


You can also try the memory profiler in Instruments (shipped with Xcode).

-- 
/Jacob Carlborg
Nov 24 2019
parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 11/24/19 10:34 AM, Jacob Carlborg wrote:

 On 2019-11-18 01:20, Steven Schveighoffer wrote:

 I'm fighting some out of memory problems using DMD and some 
 super-template heavy code.

 I have ideas on how to improve the situation, but it involves 
 redesigning a large portion of the design. I want to do it 
 incrementally, but I need to see things improving.

 Is there a straightforward way to figure out how much memory the 
 compiler uses during compilation? I though maybe /usr/bin/time, but I 
 feel like I don't trust the output to be the true max resident size to 
 be what I'm looking for (or that it's 100% accurate). Is there a 
 sure-fire way to have DMD print it's footprint?


 
 You can also try the memory profiler in Instruments (shipped with Xcode).
 


I've actually been using --DRT-gcopt=profile:1 -lowmem with great 
success. And I'm also doing this on my Linux VM, so no xcode.

The only issue is that dub somehow removes the output from the profile, 
so I have to dub -v then copy the command line. Not the end of the 
world, but it would be nice if it just output the result correctly.

-Steve
Nov 24 2019