www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - voldemort stack traces (and bloat)

reply Steven Schveighoffer <schveiguy yahoo.com> writes:
I have a library where I was using very many voldemort types a la std.range.

While debugging I had an exception triggered, but I found that the 
library *pauses* significantly while printing the exception.

What I found is essentially that using voldemort types results in 
horrible stack traces.

To demonstrate the problem:

struct S(T)
{
     void foo(){ throw new Exception("1");}
}

auto s(T)(T t)
{
     struct Result
     {
         void foo(){ throw new Exception("2");}
     }
     return Result();
}

void main(string[] args)
{
     version(bad)
         auto x = 1.s.s.s.s.s;
     else
         S!(S!(S!(S!(S!(int))))) x;

     x.foo;
}

Building without bad version, and running, I get this as the stack frame 
for the foo call:

4   testexpansion                       0x0000000103c3fc14 pure  safe 
void 
testexpansion.S!(testexpansion.S!(testexpansion.S!(testexpansion.S!(testexpansion.S!(int)
S).S).S).S).S.foo() 
+ 144


Now, if I compile with version=bad:

4   testexpansion                       0x000000010fb5dbec pure  safe 
void 
testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).s(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).Result).s(testexpansion.s!(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).s(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).Result).Result).s(testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).s(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).Result).s(testexpansion.s!(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).s(testexpansion.s!(testexpansion.s!(int).s(int).Result).s(testexpansion.s!(int).s(int).Result).Result).Result).Result).Result.foo
() 
+ 144

I believe what is happening is both the template parameter and the 
argument type are being printed, but both are the same! And each level 
of nesting results in another doubling of the printouts. So you have an 
exponential effect, and the resulting stack trace is horrendously useless.

what's more, the template bloat factor skyrockets:

dmd -c testexpansion.d
ls -l testexpansion.o
-rw-r--r--+ 1 steves  staff  5664 Feb  7 00:06 testexpansion.o

dmd -c -version=bad testexpansion.d
ls -l testexpansion.o
-rw-r--r--+ 1 steves  staff  15312 Feb  7 00:07 testexpansion.o

as a final test, I tried this:

auto s(T)(T t)
{
     return S!(T)();
}

And the resulting .o file:
-rw-r--r--+ 1 steves  staff  7104 Feb  7 00:11 testexpansion.o

With obviously the exception code printing in the less verbose form. So 
the cost in template bloat of using a voldemort type over a private type 
is 8k here, more than double the existing size. With more nesting, I'm 
sure that factor gets worse.

Is there a better way we should be doing this? I'm wondering if 
voldemort types are really worth it. They offer a lot of convenience, 
and are much DRYer than separate private template types. But the bloat 
cost is not really worth the convenience IMO.

Thoughts?

-Steve
Feb 06 2016
next sibling parent reply deadalnix <deadalnix gmail.com> writes:
On Sunday, 7 February 2016 at 05:18:39 UTC, Steven Schveighoffer 
wrote:

 Thoughts?


And no line number. But hey, these are convenience for 
youngsters. We real program, who type on the keyboard using our 
balls, don't need such distractions.
Feb 07 2016
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/7/16 5:20 AM, deadalnix wrote:

 On Sunday, 7 February 2016 at 05:18:39 UTC, Steven Schveighoffer wrote:

 Thoughts?


 And no line number. But hey, these are convenience for youngsters. We
 real program, who type on the keyboard using our balls, don't need such
 distractions.


Remind me never to borrow your laptop.

But really, if you can't figure out what +144 is, *eyeroll*.

-Steve
Feb 07 2016
prev sibling next sibling parent reply Iakh <iaktakh gmail.com> writes:
On Sunday, 7 February 2016 at 05:18:39 UTC, Steven Schveighoffer 
wrote:

 4   testexpansion                       0x000000010fb5dbec pure 
  safe void 
 testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!


Why "bad" foo is void?


 Is there a better way we should be doing this? I'm wondering if


Yeah would by nice to auto-repacle with 
testexpansion.S!(...)(...).Result.foo
or even with ...Result.foo
Feb 07 2016
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/7/16 10:42 AM, Iakh wrote:

 On Sunday, 7 February 2016 at 05:18:39 UTC, Steven Schveighoffer wrote:

 4   testexpansion                       0x000000010fb5dbec pure  safe
 void
 testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!


 Why "bad" foo is void?


Huh? foo returns void in both instances.


 Yeah would by nice to auto-repacle with
 testexpansion.S!(...)(...).Result.foo
 or even with ...Result.foo


A possible fix for the stack printing is to use the template parameter 
placeholders:

testexpansion.s!(T = testexpansion.s!...)(T).Result.foo

But this doesn't fix the object-file bloat.

-Steve
Feb 07 2016
parent Nicholas Wilson <iamthewilsonator hotmail.com> writes:
On Monday, 8 February 2016 at 01:48:32 UTC, Steven Schveighoffer 
wrote:

 On 2/7/16 10:42 AM, Iakh wrote:

 On Sunday, 7 February 2016 at 05:18:39 UTC, Steven 
 Schveighoffer wrote:

 4   testexpansion                       0x000000010fb5dbec 
 pure  safe
 void
 testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!(testexpansion.s!


 Why "bad" foo is void?


 Huh? foo returns void in both instances.


 Yeah would by nice to auto-repacle with
 testexpansion.S!(...)(...).Result.foo
 or even with ...Result.foo


 A possible fix for the stack printing is to use the template 
 parameter placeholders:

 testexpansion.s!(T = testexpansion.s!...)(T).Result.foo

 But this doesn't fix the object-file bloat.

 -Steve


I think it was Manu who was complaining about symbol length some 
time ago and we ended up discussing symbol compression as a 
possible solution. Did anything ever come of that? If so this 
seems like an obvious candidate for recursive compression.

Nic
Feb 07 2016
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/7/16 12:18 AM, Steven Schveighoffer wrote:

 I have a library where I was using very many voldemort types a la
 std.range.


[snip]


 Is there a better way we should be doing this? I'm wondering if
 voldemort types are really worth it. They offer a lot of convenience,
 and are much DRYer than separate private template types. But the bloat
 cost is not really worth the convenience IMO.


I modified all my voldemort-returning functions to return module-level 
types.

One of my example programs (compiled optimized/inline) went from 10MB to 
1MB. The other example program went from 2.1MB to 900k.

-Steve
Feb 08 2016
parent reply wobbles <grogan.colin gmail.com> writes:
On Monday, 8 February 2016 at 13:01:44 UTC, Steven Schveighoffer 
wrote:

 On 2/7/16 12:18 AM, Steven Schveighoffer wrote:

 I have a library where I was using very many voldemort types a 
 la
 std.range.


 [snip]


 Is there a better way we should be doing this? I'm wondering if
 voldemort types are really worth it. They offer a lot of 
 convenience,
 and are much DRYer than separate private template types. But 
 the bloat
 cost is not really worth the convenience IMO.


 I modified all my voldemort-returning functions to return 
 module-level types.

 One of my example programs (compiled optimized/inline) went 
 from 10MB to 1MB. The other example program went from 2.1MB to 
 900k.

 -Steve


Just to be sure, you replaced something like this:

auto myFunc(int x){
    struct MyStruct{ int a; }
    return MyStruct(x);
}

with?

private struct MyStruct{ int a; }
auto myFunc(int x){
     return MyStruct(x);
}
Feb 08 2016
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/8/16 8:19 AM, wobbles wrote:

 On Monday, 8 February 2016 at 13:01:44 UTC, Steven Schveighoffer wrote:

 On 2/7/16 12:18 AM, Steven Schveighoffer wrote:

 I have a library where I was using very many voldemort types a la
 std.range.


 [snip]


 Is there a better way we should be doing this? I'm wondering if
 voldemort types are really worth it. They offer a lot of convenience,
 and are much DRYer than separate private template types. But the bloat
 cost is not really worth the convenience IMO.


 I modified all my voldemort-returning functions to return module-level
 types.

 One of my example programs (compiled optimized/inline) went from 10MB
 to 1MB. The other example program went from 2.1MB to 900k.


 Just to be sure, you replaced something like this:

 auto myFunc(int x){
     struct MyStruct{ int a; }
     return MyStruct(x);
 }

 with?

 private struct MyStruct{ int a; }
 auto myFunc(int x){
      return MyStruct(x);
 }



Yes, but with template parameters. It's not so much the moving of the 
struct that reduced the bloat, but the nature of how the template 
parameters are used. Each function that returns one of these structs 
wraps another such struct, so the template bloat is exponential because 
of the repeat of the template parameter for the function argument. By 
moving the struct into the module, there is only one specification of 
the template parameter for the name mangling. Basically, I changed a.b.c 
bloat factor from 2^3 to 1^3.

-Steve
Feb 08 2016