digitalmars.D - Tracing allocations with "new" in dmd
- Andrei Alexandrescu (5/5) Jun 19 2020 dmd does most of its allcoation with new. I wonder what would be the
- Guillaume Piolat (3/9) Jun 19 2020 -profile=gc ?
- Andrei Alexandrescu (3/13) Jun 19 2020 Thanks, I'll look into it. Does it provide type information or only
- Stefan Koch (3/18) Jun 19 2020 gc profiling will not work UNLESS you use the `-lowmem` switch.
- Guillaume Piolat (11/15) Jun 19 2020 Here is an example of a profilegc.log file:
- Andrei Alexandrescu (4/24) Jun 19 2020 Thanks. A quick experiment (built dmd itself with -profile=gc, ran it on...
- Walter Bright (2/5) Jun 19 2020 Please post to bugzilla.
- Andrei Alexandrescu (2/8) Jun 19 2020 https://issues.dlang.org/show_bug.cgi?id=20960
- Stefan Koch (4/10) Jun 19 2020 I have a tracing tool.
- Stefan Koch (15/27) Jun 19 2020 it's here:
- Andrei Alexandrescu (2/13) Jun 19 2020 Cool, some links to docs would be great!
- Stefan Koch (5/20) Jun 19 2020 There are no docs unfortunately.
- Tim (16/22) Jun 19 2020 You can also use heaptrack for tracing allocations under Linux.
- Andrei Alexandrescu (2/30) Jun 19 2020 Thanks. Does it provide type information about the objects allocated?
- Tim (5/7) Jun 19 2020 No, it does not save type information.
- kinke (4/5) Jun 19 2020 All (?) `new` allocations in DMD are redirected to
- Andrei Alexandrescu (3/9) Jun 19 2020 Thank you! Just to clarify: this is advice on where to insert my own
- kinke (3/13) Jun 19 2020 Yes, instrumentation to be inserted.
- FeepingCreature (4/30) Nov 10 2022 Reply to bump. This is one of the best "secret weapons" in the D
- Tejas (3/8) Nov 11 2022 And of course it's not on `dub`, hurting adoption...
dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created. There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed). Would be in your debt for any ideas. Thanks!
Jun 19 2020
On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created. There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed). Would be in your debt for any ideas. Thanks!-profile=gc ?
Jun 19 2020
On 6/19/20 9:48 AM, Guillaume Piolat wrote:On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:Thanks, I'll look into it. Does it provide type information or only low-level allocated size information?dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created. There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed). Would be in your debt for any ideas. Thanks!-profile=gc ?
Jun 19 2020
Stefan Koch <uplink.coder googlemail.com> On Friday, 19 June 2020 at 14:19:36 UTC, Andrei Alexandrescu wrote:On 6/19/20 9:48 AM, Guillaume Piolat wrote:gc profiling will not work UNLESS you use the `-lowmem` switch.On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:Thanks, I'll look into it. Does it provide type information or only low-level allocated size information?dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created. There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed). Would be in your debt for any ideas. Thanks!-profile=gc ?
Jun 19 2020
On Friday, 19 June 2020 at 14:19:36 UTC, Andrei Alexandrescu wrote:Here is an example of a profilegc.log file: --------------- profilegc.log ----------- bytes allocated, allocations, type, function, file:line 4096 1 char[] D main source\main.d:36 240 10 main.main.D D main source\main.d:44 64 1 int[] D main source\main.d:37 56 1 double[] D main source\main.d:55 ------------------------------------------profile=gc ?Thanks, I'll look into it. Does it provide type information or only low-level allocated size information?
Jun 19 2020
On 6/19/20 10:34 AM, Guillaume Piolat wrote:On Friday, 19 June 2020 at 14:19:36 UTC, Andrei Alexandrescu wrote:Thanks. A quick experiment (built dmd itself with -profile=gc, ran it on a few simple files) seems to show that only array allocations are traced, but not allocations of individual objects.Here is an example of a profilegc.log file: --------------- profilegc.log ----------- bytes allocated, allocations, type, function, file:line 4096 1 char[] D main source\main.d:36 240 10 main.main.D D main source\main.d:44 64 1 int[] D main source\main.d:37 56 1 double[] D main source\main.d:55 ------------------------------------------profile=gc ?Thanks, I'll look into it. Does it provide type information or only low-level allocated size information?
Jun 19 2020
On 6/19/2020 8:36 AM, Andrei Alexandrescu wrote:A quick experiment (built dmd itself with -profile=gc, ran it on a few simple files) seems to show that only array allocations are traced, but not allocations of individual objects.Please post to bugzilla.
Jun 19 2020
On 6/19/20 4:59 PM, Walter Bright wrote:On 6/19/2020 8:36 AM, Andrei Alexandrescu wrote:https://issues.dlang.org/show_bug.cgi?id=20960A quick experiment (built dmd itself with -profile=gc, ran it on a few simple files) seems to show that only array allocations are traced, but not allocations of individual objects.Please post to bugzilla.
Jun 19 2020
On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created. There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed). Would be in your debt for any ideas. Thanks!I have a tracing tool. USE IT.
Jun 19 2020
On Friday, 19 June 2020 at 14:23:10 UTC, Stefan Koch wrote:On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:it's here: https://github.com/UplinkCoder/dmd/tree/dmd_tracing_2092 Currently I am still hunting down a bug which seems to be caused, By tracing a temporary allocated, expression which will then lead to a segfault while writing the trace. But I would very much like for others to use it, it's designed to allow you to say which source_code takes the most time out of the front-end process and where. You will have to modify the source code of dmd to investigate particular issues, But it comes with a number of likely candidates pre-selected for tracing. I am happy to work with someone on up-streaming this and making it better.dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created. There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed). Would be in your debt for any ideas. Thanks!I have a tracing tool. USE IT.
Jun 19 2020
On 6/19/20 10:23 AM, Stefan Koch wrote:On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:Cool, some links to docs would be great!dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created. There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed). Would be in your debt for any ideas. Thanks!I have a tracing tool. USE IT.
Jun 19 2020
On Friday, 19 June 2020 at 14:42:27 UTC, Andrei Alexandrescu wrote:On 6/19/20 10:23 AM, Stefan Koch wrote:There are no docs unfortunately. I am happy to provide assistance and in the process of exploration the docs will be created as well.On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:Cool, some links to docs would be great!dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created. There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed). Would be in your debt for any ideas. Thanks!I have a tracing tool. USE IT.
Jun 19 2020
On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created. There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed). Would be in your debt for any ideas. Thanks!You can also use heaptrack for tracing allocations under Linux. It uses LD_PRELOAD to replace allocation functions with wrappers. I have modified heaptrack to also track allocations on the D GC heap: https://github.com/tim-dlang/heaptrack/tree/heaptrack-d When using it, the application has to be linked with druntime dynamically. It also only works, when the application is launched by heaptrack, and not, when heaptrack is attached later. The modified heaptrack will only work with D applications. dmd can be linked dynamically with phobos/druntime by passing DFLAGS="-defaultlib=phobos2 -debuglib=phobos2" to build.d. For dmd you can call it like this: heaptrack dmd -lowmem test.d It will then generate a file like heaptrack.dmd.*.zst, which can be analyzed with heaptrack --analyze.
Jun 19 2020
On 6/19/20 12:03 PM, Tim wrote:On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:Thanks. Does it provide type information about the objects allocated?dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created. There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed). Would be in your debt for any ideas. Thanks!You can also use heaptrack for tracing allocations under Linux. It uses LD_PRELOAD to replace allocation functions with wrappers. I have modified heaptrack to also track allocations on the D GC heap: https://github.com/tim-dlang/heaptrack/tree/heaptrack-d When using it, the application has to be linked with druntime dynamically. It also only works, when the application is launched by heaptrack, and not, when heaptrack is attached later. The modified heaptrack will only work with D applications. dmd can be linked dynamically with phobos/druntime by passing DFLAGS="-defaultlib=phobos2 -debuglib=phobos2" to build.d. For dmd you can call it like this: heaptrack dmd -lowmem test.d It will then generate a file like heaptrack.dmd.*.zst, which can be analyzed with heaptrack --analyze.
Jun 19 2020
On Friday, 19 June 2020 at 16:09:00 UTC, Andrei Alexandrescu wrote:Thanks. Does it provide type information about the objects allocated?No, it does not save type information. It generates a stack trace for every allocation and can combine those into a bottom-up or top-down tree.
Jun 19 2020
On Friday, 19 June 2020 at 16:09:00 UTC, Andrei Alexandrescu wrote:Does it provide type information about the objects allocated?All (?) `new` allocations in DMD are redirected to https://github.com/dlang/dmd/blob/b6b0c0f41a476c4eaa88ba106fb4de1175d40440/src/dmd/ro t/rmem.d#L222-L310. There you have the TypeInfos.
Jun 19 2020
On 6/19/20 6:05 PM, kinke wrote:On Friday, 19 June 2020 at 16:09:00 UTC, Andrei Alexandrescu wrote:Thank you! Just to clarify: this is advice on where to insert my own tracing, not a pointer to an existing instrumentation, right?Does it provide type information about the objects allocated?All (?) `new` allocations in DMD are redirected to https://github.com/dlang/dmd/blob/b6b0c0f41a476c4eaa88ba106fb4de1175d40440/src/dmd/ro t/rmem.d#L222-L310. There you have the TypeInfos.
Jun 19 2020
On Saturday, 20 June 2020 at 01:20:02 UTC, Andrei Alexandrescu wrote:On 6/19/20 6:05 PM, kinke wrote:Yes, instrumentation to be inserted.On Friday, 19 June 2020 at 16:09:00 UTC, Andrei Alexandrescu wrote:Thank you! Just to clarify: this is advice on where to insert my own tracing, not a pointer to an existing instrumentation, right?Does it provide type information about the objects allocated?All (?) `new` allocations in DMD are redirected to https://github.com/dlang/dmd/blob/b6b0c0f41a476c4eaa88ba106fb4de1175d40440/src/dmd/ro t/rmem.d#L222-L310. There you have the TypeInfos.
Jun 19 2020
On Friday, 19 June 2020 at 16:03:55 UTC, Tim wrote:On Friday, 19 June 2020 at 13:45:23 UTC, Andrei Alexandrescu wrote:Reply to bump. This is one of the best "secret weapons" in the D ecosystem. Just yesterday we used it to find the reason why a service was ballooning to 11GB.dmd does most of its allcoation with new. I wonder what would be the easiest way to track how many instances of each type are created. There are over 3000 call sites, so ideally the changes to the source would be entirely automatic (e.g. with sed). Would be in your debt for any ideas. Thanks!You can also use heaptrack for tracing allocations under Linux. It uses LD_PRELOAD to replace allocation functions with wrappers. I have modified heaptrack to also track allocations on the D GC heap: https://github.com/tim-dlang/heaptrack/tree/heaptrack-d When using it, the application has to be linked with druntime dynamically. It also only works, when the application is launched by heaptrack, and not, when heaptrack is attached later. The modified heaptrack will only work with D applications. dmd can be linked dynamically with phobos/druntime by passing DFLAGS="-defaultlib=phobos2 -debuglib=phobos2" to build.d. For dmd you can call it like this: heaptrack dmd -lowmem test.d It will then generate a file like heaptrack.dmd.*.zst, which can be analyzed with heaptrack --analyze.
Nov 10 2022
On Friday, 11 November 2022 at 07:26:37 UTC, FeepingCreature wrote:On Friday, 19 June 2020 at 16:03:55 UTC, Tim wrote:And of course it's not on `dub`, hurting adoption...[...]Reply to bump. This is one of the best "secret weapons" in the D ecosystem. Just yesterday we used it to find the reason why a service was ballooning to 11GB.
Nov 11 2022