• Walter Bright (5/5) Jun 22 2013 Compiling std.algorithm for unittests consumes all the memory on many ma...
• Geancarlo Rocha (4/10) Jun 22 2013 Maybe Dr. Memory could work out well as an alternative to
• Walter Bright (3/5) Jun 22 2013 The web page indicates it is designed for looking for memory allocation ...
• Kiith-Sa (3/3) Jun 22 2013 With valgrind; did you use massif?
• Walter Bright (2/4) Jun 22 2013 I've never heard of massif. Will check it out.
• Martin Nowak (6/11) Jun 24 2013 I tried it, works nice. No graphical output though.
• Dmitry Olshansky (4/16) Jun 24 2013 There is a massif-visualizer tool that works nicely for me.
• Kiith-Sa (3/17) Jun 24 2013 sudo apt-get install massif-visualizer
• Jacob Carlborg (7/12) Jun 23 2013 There's a tool called dtrace on Mac OS X and *BSD which possibly could
• qznc (3/19) Jun 23 2013 That would be SystemTap on Linux. However, I wonder if it is the
• Jacob Carlborg (6/8) Jun 23 2013 Mac OS X has Instruments as well, which is used for these type of tasks....
• Nick B (3/10) Jun 27 2013 here is a comparion of Systemtap and DTrace
• Nick B (6/15) Jun 27 2013 here is the SystemTap FAQ
• dennis luehring (7/12) Jun 23 2013 there are "sometimes" better tools on windows or commercials - you
• Walter Bright (2/15) Jun 23 2013 Thanks!
• Richard Webb (4/5) Jun 24 2013 I've used that at work, though mainly for looking for leaks
• Richard Webb (8/15) Jun 25 2013 I had a go at running the VC build of DMD through Memory Validator,
• dennis luehring (2/19) Jun 25 2013 MTuner works - but i can't find a export or report feature
• monarch_dodra (6/12) Jun 23 2013 Just wanted to add that with the latest release, I can't compile
• Kagamin (2/2) Jun 25 2013 Another possibility is to comment some tests and see if valgrind
• Don (16/22) Jul 02 2013 I did a check on the CTFE part of it. You can do this by setting
• Richard Webb (10/22) Jul 04 2013 I had another go at running memory validator on an MSVC build of the

Walter Bright <newshound2 digitalmars.com> writes:

Compiling std.algorithm for unittests consumes all the memory on many machines. 
I've been looking into what is allocating all that memory, and it isn't so easy 
without adding instrumentation code anywhere.

Anyone know of a convenient tool to do this on Linux?

(valgrind just hangs, or at least I gave up on it after 6 hours)

"Geancarlo Rocha" <asdf mailinator.com> writes:

Maybe Dr. Memory could work out well as an alternative to 
valgrind with this issue?
http://www.drmemory.org/

On Saturday, 22 June 2013 at 21:41:15 UTC, Walter Bright wrote:
 Compiling std.algorithm for unittests consumes all the memory 
 on many machines. I've been looking into what is allocating all 
 that memory, and it isn't so easy without adding 
 instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 hours)

Walter Bright <newshound2 digitalmars.com> writes:

On 6/22/2013 4:18 PM, Geancarlo Rocha wrote:
 Maybe Dr. Memory could work out well as an alternative to valgrind with this
issue?
 http://www.drmemory.org/

The web page indicates it is designed for looking for memory allocation bugs. 
That isn't the issue I'm having.

"Kiith-Sa" <kiithsacmp gmail.com> writes:

With valgrind; did you use massif?
That would be the right tool to use instead of memcheck (the 
default tool).

Walter Bright <newshound2 digitalmars.com> writes:

On 6/22/2013 4:24 PM, Kiith-Sa wrote:
 With valgrind; did you use massif?
 That would be the right tool to use instead of memcheck (the default tool).

I've never heard of massif. Will check it out.

Martin Nowak <code dawg.eu> writes:

On 06/23/2013 03:19 AM, Walter Bright wrote:
 On 6/22/2013 4:24 PM, Kiith-Sa wrote:
 With valgrind; did you use massif?
 That would be the right tool to use instead of memcheck (the default
 tool).

 I've never heard of massif. Will check it out.

I tried it, works nice. No graphical output though.

1. compile dmd with debug symbols (-g)
2. valgrind --tool=massif ../dmd/src/dmd -main -unittest std/algorithm
3. ms_print massif.out.%PID% > outfile

http://valgrind.org/docs/manual/ms-manual.html

Dmitry Olshansky <dmitry.olsh gmail.com> writes:

24-Jun-2013 22:47, Martin Nowak пишет:
 On 06/23/2013 03:19 AM, Walter Bright wrote:
 On 6/22/2013 4:24 PM, Kiith-Sa wrote:
 With valgrind; did you use massif?
 That would be the right tool to use instead of memcheck (the default
 tool).

 I've never heard of massif. Will check it out.

 I tried it, works nice. No graphical output though.

 1. compile dmd with debug symbols (-g)
 2. valgrind --tool=massif ../dmd/src/dmd -main -unittest std/algorithm
 3. ms_print massif.out.%PID% > outfile

 http://valgrind.org/docs/manual/ms-manual.html

There is a massif-visualizer tool that works nicely for me.

-- 
Dmitry Olshansky

"Kiith-Sa" <kiithsacmp gmail.com> writes:

On Monday, 24 June 2013 at 18:47:17 UTC, Martin Nowak wrote:
 On 06/23/2013 03:19 AM, Walter Bright wrote:
 On 6/22/2013 4:24 PM, Kiith-Sa wrote:
 With valgrind; did you use massif?
 That would be the right tool to use instead of memcheck (the 
 default
 tool).

 I've never heard of massif. Will check it out.

 I tried it, works nice. No graphical output though.

 1. compile dmd with debug symbols (-g)
 2. valgrind --tool=massif ../dmd/src/dmd -main -unittest 
 std/algorithm
 3. ms_print massif.out.%PID% > outfile

 http://valgrind.org/docs/manual/ms-manual.html

sudo apt-get install massif-visualizer

(on a newish debian/ubuntu/mint/etc)

Jacob Carlborg <doob me.com> writes:

On 2013-06-22 23:41, Walter Bright wrote:
 Compiling std.algorithm for unittests consumes all the memory on many
 machines. I've been looking into what is allocating all that memory, and
 it isn't so easy without adding instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 hours)

There's a tool called dtrace on Mac OS X and *BSD which possibly could 
be used for this. Don't know if there's a corresponding tool for Linux. 
Hmm, seems to be available on Oracle Linux.

http://en.wikipedia.org/wiki/DTrace

-- 
/Jacob Carlborg

"qznc" <qznc web.de> writes:

On Sunday, 23 June 2013 at 09:37:26 UTC, Jacob Carlborg wrote:
 On 2013-06-22 23:41, Walter Bright wrote:
 Compiling std.algorithm for unittests consumes all the memory 
 on many
 machines. I've been looking into what is allocating all that 
 memory, and
 it isn't so easy without adding instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 
 hours)

 There's a tool called dtrace on Mac OS X and *BSD which 
 possibly could be used for this. Don't know if there's a 
 corresponding tool for Linux. Hmm, seems to be available on 
 Oracle Linux.

 http://en.wikipedia.org/wiki/DTrace

That would be SystemTap on Linux. However, I wonder if it is the 
right tool for the job.

Jacob Carlborg <doob me.com> writes:

On 2013-06-23 15:12, qznc wrote:

 That would be SystemTap on Linux. However, I wonder if it is the right
 tool for the job.

Mac OS X has Instruments as well, which is used for these type of tasks. 
It's built on top of Dtrace:

http://en.wikipedia.org/wiki/Instruments_(application)

-- 
/Jacob Carlborg

"Nick B" <nick.barbalich gmail.com> writes:

On Sunday, 23 June 2013 at 15:22:22 UTC, Jacob Carlborg wrote:
 On 2013-06-23 15:12, qznc wrote:

 That would be SystemTap on Linux. However, I wonder if it is 
 the right
 tool for the job.

 Mac OS X has Instruments as well, which is used for these type 
 of tasks. It's built on top of Dtrace:

 http://en.wikipedia.org/wiki/Instruments_(application)


here is a comparion of Systemtap and DTrace


http://sourceware.org/systemtap/wiki/SystemtapDtraceComparison

"Nick B" <nick.barbalich gmail.com> writes:

On Thursday, 27 June 2013 at 22:12:49 UTC, Nick B wrote:
 On Sunday, 23 June 2013 at 15:22:22 UTC, Jacob Carlborg wrote:
 On 2013-06-23 15:12, qznc wrote:

 That would be SystemTap on Linux. However, I wonder if it is 
 the right
 tool for the job.



[snip]
 here is a comparion of Systemtap and DTrace


 http://sourceware.org/systemtap/wiki/SystemtapDtraceComparison

here is the SystemTap FAQ

http://sourceware.org/systemtap/wiki/SystemTapFAQ

it looks to be a very usefull tool.

Nick

dennis luehring <dl.soluz gmx.net> writes:

Am 22.06.2013 23:41, schrieb Walter Bright:
 Compiling std.algorithm for unittests consumes all the memory on many machines.
 I've been looking into what is allocating all that memory, and it isn't so easy
 without adding instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 hours)

there are "sometimes" better tools on windows or commercials - you 
should try these

http://www.softwareverify.com/cpp-memory.php - i like this one

http://mtuner.net/

http://www.puredevsoftware.com/MemPro.htm

all for memory leak detection, but also doing memory profiling

Walter Bright <newshound2 digitalmars.com> writes:

On 6/23/2013 5:21 AM, dennis luehring wrote:
 Am 22.06.2013 23:41, schrieb Walter Bright:
 Compiling std.algorithm for unittests consumes all the memory on many machines.
 I've been looking into what is allocating all that memory, and it isn't so easy
 without adding instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 hours)

 there are "sometimes" better tools on windows or commercials - you should try
these

 http://www.softwareverify.com/cpp-memory.php - i like this one

 http://mtuner.net/

 http://www.puredevsoftware.com/MemPro.htm

 all for memory leak detection, but also doing memory profiling

Thanks!

"Richard Webb" <webby beardmouse.org.uk> writes:

On Sunday, 23 June 2013 at 12:23:14 UTC, dennis luehring wrote:
 http://www.softwareverify.com/cpp-memory.php - i like this one


I've used that at work, though mainly for looking for leaks 
rather than memory profiling.
Might try running DMD thruogh it if i get some time.

Richard Webb <richard.webb boldonjames.com> writes:

On 24/06/2013 13:11, Richard Webb wrote:
 On Sunday, 23 June 2013 at 12:23:14 UTC, dennis luehring wrote:
 http://www.softwareverify.com/cpp-memory.php - i like this one


 I've used that at work, though mainly for looking for leaks rather than
 memory profiling.
 Might try running DMD thruogh it if i get some time.


I had a go at running the VC build of DMD through Memory Validator, 
using "-release -unittest -c 
D:\DTesting\dmd.2.063\src\phobos\std\algorithm.d".

It ran for about 45 mins, completed, and then locked up :-(


I've put a screen shot of the object types list   
<http://tinyurl.com/qybudb4> as an example of what it shows - didn't get 
a chance to look at anything else before it fell over.

dennis luehring <dl.soluz gmx.net> writes:

Am 25.06.2013 16:18, schrieb Richard Webb:
 On 24/06/2013 13:11, Richard Webb wrote:
 On Sunday, 23 June 2013 at 12:23:14 UTC, dennis luehring wrote:
 http://www.softwareverify.com/cpp-memory.php - i like this one


 I've used that at work, though mainly for looking for leaks rather than
 memory profiling.
 Might try running DMD thruogh it if i get some time.


 I had a go at running the VC build of DMD through Memory Validator,
 using "-release -unittest -c
 D:\DTesting\dmd.2.063\src\phobos\std\algorithm.d".

 It ran for about 45 mins, completed, and then locked up :-(


 I've put a screen shot of the object types list  
 <http://tinyurl.com/qybudb4> as an example of what it shows - didn't get
 a chance to look at anything else before it fell over.

MTuner works - but i can't find a export or report feature

"monarch_dodra" <monarchdodra gmail.com> writes:

On Saturday, 22 June 2013 at 21:41:15 UTC, Walter Bright wrote:
 Compiling std.algorithm for unittests consumes all the memory 
 on many machines. I've been looking into what is allocating all 
 that memory, and it isn't so easy without adding 
 instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 hours)

Just wanted to add that with the latest release, I can't compile 
algorithm in unittest mode with -inline -O -release.

This is actually an important test to run, as algorithm has a 
tendency to create "unreachable statement" error due to better 
flow analysis.

"Kagamin" <spam here.lot> writes:

Another possibility is to comment some tests and see if valgrind 
finishes.

"Don" <turnyourkidsintocash nospam.com> writes:

On Saturday, 22 June 2013 at 21:41:15 UTC, Walter Bright wrote:
 Compiling std.algorithm for unittests consumes all the memory 
 on many machines. I've been looking into what is allocating all 
 that memory, and it isn't so easy without adding 
 instrumentation code anywhere.

 Anyone know of a convenient tool to do this on Linux?

 (valgrind just hangs, or at least I gave up on it after 6 hours)

I did a check on the CTFE part of it. You can do this by setting 
SHOWPERFORMANCE to 1 in the top of interpret.c. This produces:

$ dmd -c -unittest std.algorithm
         ---- CTFE Performance ----
max call depth = 20	max stack = 63
array allocs = 356	assignments = 45356

That's actually not so terrible. The number of assignments gives 
a rough idea of how many CTFE statements are executed (almost 
everything interesting is an assignment). Most of the assignments 
are IntegerExpressions, of about 8 bytes, so it's under a 
megabyte in total. Note that there are very few array 
allocations, they are the thing that can really chew up memory 
quickly in CTFE.

So although CTFE generally leaks memory like the Exxon Valdez 
leaks oil, I don't think it's to blame in this particular case.

Richard Webb <richard.webb boldonjames.com> writes:

On 02/07/2013 11:30, Don wrote:
 $ dmd -c -unittest std.algorithm
          ---- CTFE Performance ----
 max call depth = 20    max stack = 63
 array allocs = 356    assignments = 45356

 That's actually not so terrible. The number of assignments gives a rough
 idea of how many CTFE statements are executed (almost everything
 interesting is an assignment). Most of the assignments are
 IntegerExpressions, of about 8 bytes, so it's under a megabyte in total.
 Note that there are very few array allocations, they are the thing that
 can really chew up memory quickly in CTFE.

 So although CTFE generally leaks memory like the Exxon Valdez leaks oil,
 I don't think it's to blame in this particular case.


I had another go at running memory validator on an MSVC build of the 
latest source, and that says that there are 619008 instances of 
IntegerExp, at 30 bytes each (it's the 3rd most numerous named object, 
though only uses 2.3% of the memory).

I'm not sure how accurate that is as the wrapping of malloc etc in DMD 
confuse the results - e.g. it just lists 2159119 instances of 
'rmem.c:46' rather than saying what they actually are.

fwiw, it says there are 14,828,139 allocations totalling 814.7MB from 
'-c 'unittest' algorithm.d .