www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Stack traced exceptions

reply "Maxime Larose" <mlarose broadsoft.com> writes:
Walter,
(or anyone else who can respond)

I finally got the time to investigate built-in stack tracing in the last few
days. FYI, I want to deal with Windows first.

(I'll say bluntly and off the bat that I'm probably not the best person to
tackle this issue. I have no particular knowledge of stack frames, debugging
schemes, symbol tables, omf/coff, etc. I'd whish someone more knowledgeable
than me took up the task and maybe what I say below will sound naive. At any
rate, I believe built-in stack tracing is very important for any new
language. I've made my case in another thread, I will not repeat myself
here.)

So, my initial approach went like this:
- I use MSVC to debug D executables and it usually works fine (templates are
the exception, somtimes the debugger will have a hard time with them -
understandable).
- So, I figured that MSVC probably eat its own dog food, i.e. use
IMAGEHLP.DLL to do its stack walking and symbol referencing.
- The idea is to use StalkWalk() in imagehlp.dll to walk the stack frames.
Digged up old articles on the net about imagehlp.dll and tried to implement
something in phobos.
- There are some undocumented requirements for StackWalk and I need a
CONTEXT. Apparently, a valid CONTEXT can only be obtained when: (1) the
thread is sleeping or (2) an exception is raised.
- Function _d_framehandler in deh.c receives a CONTEXT as a parameter.So, I
hacked Object.d so that a specific (new) exception was raised (w/
RaiseException) in Exception.this(). The _d_framehandler  function traps
that exception and deals with it in a special manner, .i.e. call StackWalk
etc.

Unfortunately, at this point it's not working. The first stack frame I get
back from the library is wrong somehow. I didn't have time to investigate
more yesterday, but I was a bit worried that I'd wasted my time. What if D
(or the digital mars C++ compiler) don't follow the standard stack frame
format. But than how can MSVC debug D apps?  So, yes, D must follow the
standard stack frame format, and MSVC can work with it. However, perhaps
MSVC doesn't use imagehlp.


Again, I may sound naive and ignorant, but I would like to know if imagehlp
_should_ work with D apps. If not, I'd rather know now than engulf countless
hours in a futile attempt. If imagehlp can't work, than that's a whole
different story for me. I'd have to walk the stack myself and that involves
a *lot* more effort (mainly because the learning curve would be as steep as
it gets.)

So:
1. Should imagehlp work with D apps?
2. Perhaps with some tweakings?
3. Anyone (Walter?) that can provide pointers on how stack frames/symbols
are implemented in D, the overall design (ex: this file contains methods
that do this, this other file does this), etc.

The feeling I have is it would be so much simpler for Walter to implement
this... Anyway, I'm not complaining, just trying to save myself a lot of
time (which I don't have - but aren't we all in this same situation? ;).

Thanks,

Max
Apr 27 2005
next sibling parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
I think it's just done "by hand". On Windows you get the EBP register (or 
EIP) register for the stack frame. That then has the address of the calling 
function and the pointer to the parent stack frame. I'm not sure how to map 
the function address to a string to print. I'd recommend Googling for "Win32 
stack trace" or something for more info. I expect the code to be pretty 
platform-dependent. See for example 
http://www.eptacom.net/pubblicazioni/pub_eng/except.html

"Maxime Larose" <mlarose broadsoft.com> wrote in message 
news:d4ntes$tm5$1 digitaldaemon.com...

 Walter,
 (or anyone else who can respond)

 I finally got the time to investigate built-in stack tracing in the last 
 few
 days. FYI, I want to deal with Windows first.

 (I'll say bluntly and off the bat that I'm probably not the best person to
 tackle this issue. I have no particular knowledge of stack frames, 
 debugging
 schemes, symbol tables, omf/coff, etc. I'd whish someone more 
 knowledgeable
 than me took up the task and maybe what I say below will sound naive. At 
 any
 rate, I believe built-in stack tracing is very important for any new
 language. I've made my case in another thread, I will not repeat myself
 here.)

 So, my initial approach went like this:
 - I use MSVC to debug D executables and it usually works fine (templates 
 are
 the exception, somtimes the debugger will have a hard time with them -
 understandable).
 - So, I figured that MSVC probably eat its own dog food, i.e. use
 IMAGEHLP.DLL to do its stack walking and symbol referencing.
 - The idea is to use StalkWalk() in imagehlp.dll to walk the stack frames.
 Digged up old articles on the net about imagehlp.dll and tried to 
 implement
 something in phobos.
 - There are some undocumented requirements for StackWalk and I need a
 CONTEXT. Apparently, a valid CONTEXT can only be obtained when: (1) the
 thread is sleeping or (2) an exception is raised.
 - Function _d_framehandler in deh.c receives a CONTEXT as a parameter.So, 
 I
 hacked Object.d so that a specific (new) exception was raised (w/
 RaiseException) in Exception.this(). The _d_framehandler  function traps
 that exception and deals with it in a special manner, .i.e. call StackWalk
 etc.

 Unfortunately, at this point it's not working. The first stack frame I get
 back from the library is wrong somehow. I didn't have time to investigate
 more yesterday, but I was a bit worried that I'd wasted my time. What if D
 (or the digital mars C++ compiler) don't follow the standard stack frame
 format. But than how can MSVC debug D apps?  So, yes, D must follow the
 standard stack frame format, and MSVC can work with it. However, perhaps
 MSVC doesn't use imagehlp.


 Again, I may sound naive and ignorant, but I would like to know if 
 imagehlp
 _should_ work with D apps. If not, I'd rather know now than engulf 
 countless
 hours in a futile attempt. If imagehlp can't work, than that's a whole
 different story for me. I'd have to walk the stack myself and that 
 involves
 a *lot* more effort (mainly because the learning curve would be as steep 
 as
 it gets.)

 So:
 1. Should imagehlp work with D apps?
 2. Perhaps with some tweakings?
 3. Anyone (Walter?) that can provide pointers on how stack frames/symbols
 are implemented in D, the overall design (ex: this file contains methods
 that do this, this other file does this), etc.

 The feeling I have is it would be so much simpler for Walter to implement
 this... Anyway, I'm not complaining, just trying to save myself a lot of
 time (which I don't have - but aren't we all in this same situation? ;).

 Thanks,

 Max
Apr 27 2005
parent reply brad beveridge <brad nowhere.com> writes:
I might be completely naive here, but why can't we build stack tracking 
functionality into the language?  For example, alter the D compiler to 
have an internal stack of strings (or an index into a string table if 
you are really concerned about speed), and make it so the D compiler 
emits a "push(function_name)" when you call a function and "pop" when 
you return.
That way you get stack tracing for every architechture without having to 
know anything about the backend binary format.

Brad
Apr 27 2005
parent reply "Ben Hinkle" <bhinkle mathworks.com> writes:
"brad beveridge" <brad nowhere.com> wrote in message 
news:d4ords$1tth$1 digitaldaemon.com...

I might be completely naive here, but why can't we build stack tracking 
functionality into the language?  For example, alter the D compiler to have 
an internal stack of strings (or an index into a string table if you are 
really concerned about speed), and make it so the D compiler emits a 
"push(function_name)" when you call a function and "pop" when you return.
 That way you get stack tracing for every architechture without having to 
 know anything about the backend binary format.

 Brad


It would slow down function calls uniformly instead of just paying the price 
of determining the stack at throw-time. The performance hit would be too big 
(I assume).

Now that I look more closely at the MSDN doc for StackWalk64 I think Maxime 
is on the right track. MSDN talks about DebugHlp.dll instead of 
ImageHlp.dll, though.
Apr 27 2005
parent reply Brad Beveridge <brad somewhere.net> writes:
Ben Hinkle wrote:

 It would slow down function calls uniformly instead of just paying the price 
 of determining the stack at throw-time. The performance hit would be too big 
 (I assume).


Now that sounds like premature optimisation :)
I think you'd get away with
  ** function start **
stack_trace[depth++] = function_ID;
....
  ** function return **
depth--;
Where function_ID is a pointer to a string, or something that can be 
looked up later.
Which isn't much overhead.


Pros of inbuilt stack tracing
1) Shouldn't be too hard to implement in the compiler (I could be 
completly wrong here!)
2) Will work on all platforms.
3) Can be enabled/disabled with a compiler switch
4) Don't have to go poking into stack frames, etc at throw time.

Cons of inbuilt stack tracing
1) Small uniform overhead for each function called.

Brad
Apr 27 2005
parent reply Kevin Bealer <Kevin_member pathlink.com> writes:
In article <d4p1u1$24mm$1 digitaldaemon.com>, Brad Beveridge says...

Ben Hinkle wrote:

 It would slow down function calls uniformly instead of just paying the price 
 of determining the stack at throw-time. The performance hit would be too big 
 (I assume).


Now that sounds like premature optimisation :)
I think you'd get away with
  ** function start **
stack_trace[depth++] = function_ID;
....
  ** function return **
depth--;
Where function_ID is a pointer to a string, or something that can be 
looked up later.
Which isn't much overhead.


Pros of inbuilt stack tracing
1) Shouldn't be too hard to implement in the compiler (I could be 
completly wrong here!)
2) Will work on all platforms.
3) Can be enabled/disabled with a compiler switch
4) Don't have to go poking into stack frames, etc at throw time.

Cons of inbuilt stack tracing
1) Small uniform overhead for each function called.

Brad


While this isn't too much overhead for heavy functions, I think for getters and
setters it would swamp you.

But, the functions always return where they are supposed to, right?  That is due
to the fact that the "stack" already does what you are proposing, only its idea
of a "function id" is the function pointer.

I am guessing (from a POV of no facts) that the complex part of this would be
trying to work out which "inlined" function you are in based on ranges of insns.
This can be done if you have debug data (I think) because C++ programs in gdb
show this.

Funny thing is, the code seems to go in-out-in of inlined and optimized methods
because it all gets mixed and reordered by the optimizer.

If you want to do this portably, though, a better way that inserting "push/pop"
on every call, might be (for the compiler) to insert something like:

try {
.. function contents
}
catch(TracedException e) {
e.push_frame_name("function name");
throw e;
}

..around every function body.  Then you only pay the piper in the sense that
you have increased the bloat factor, and neutered the inlining phase of the
optimizer.  Of course if you can (at compile time) determine that this function
can't/won't throw a traced exception, then you can omit the try{}catch{}.

Another downside is that you only get stack tracing from throw-point to
catch-point, but I suspect if you want a stack trace, it is probably thrown "all
the way" to main or further, right?

Kevin
Apr 28 2005
parent reply "Maxime Larose" <mlarose broadsoft.com> writes:
Pushing function names on the stack and the try-catch method have one
important drawback: you lose granularity. You know in which function the
exception occured, but you don't know where inside the function. The only
way to know where exactly the exception was thrown is by inspecting EIP/EBP
and walking the stack from there.

As Ben said, walking the stack is not that difficult really. I was mentally
preparing myself to do it manually, but in the process of stepping through
my imagehlp code, I found out that some structures were badly filled by
Windows. I now fill them myself with some tricks. Anyways, to make a long
story short, I made a lot of progress yesterday and the overall design will
be a lot more elegant.

I now have a stack trace that lists:
Method name 1 + displacement
Method name 2 + displacement
Method name 3 + displacement
...

My idea is to not limit such stack traces to instances where Exception is
created manually, but also to cases where you have null pointers and other
such "system" exceptions. (BTW Kevin, that would eliminate the possibility
for the compiler to eliminate the try-catch at compile time, since any code
can throw). I little nudging in deh.c will do the trick.

The problem now is the displacement. I don't want a displacement, I want a
line number. I had trouble making it work though as the function that
resolves line numbers on Windows seems flaky.

I'm going to spend more time on it, but Ben mentionned something good:
debughlp vs imagehlp. Debughlp is only available on Windows XP and Windows
2000 professional. I assumed that was not acceptable, but now I'm
reconsidering. I guess everyone is on Windows XP or 2000 by now (time
flies!).

So, I will use debughlp and support AMD64 out of the box.

Thanks for your comments,

Max




"Kevin Bealer" <Kevin_member pathlink.com> wrote in message
news:d4q251$l5$1 digitaldaemon.com...

 In article <d4p1u1$24mm$1 digitaldaemon.com>, Brad Beveridge says...

Ben Hinkle wrote:

 It would slow down function calls uniformly instead of just paying the






price

 of determining the stack at throw-time. The performance hit would be






too big

 (I assume).


Now that sounds like premature optimisation :)
I think you'd get away with
  ** function start **
stack_trace[depth++] = function_ID;
....
  ** function return **
depth--;
Where function_ID is a pointer to a string, or something that can be
looked up later.
Which isn't much overhead.


Pros of inbuilt stack tracing
1) Shouldn't be too hard to implement in the compiler (I could be
completly wrong here!)
2) Will work on all platforms.
3) Can be enabled/disabled with a compiler switch
4) Don't have to go poking into stack frames, etc at throw time.

Cons of inbuilt stack tracing
1) Small uniform overhead for each function called.

Brad


 While this isn't too much overhead for heavy functions, I think for


getters and

 setters it would swamp you.

 But, the functions always return where they are supposed to, right?  That


is due

 to the fact that the "stack" already does what you are proposing, only its


idea

 of a "function id" is the function pointer.

 I am guessing (from a POV of no facts) that the complex part of this would


be

 trying to work out which "inlined" function you are in based on ranges of


insns.

 This can be done if you have debug data (I think) because C++ programs in


gdb

 show this.

 Funny thing is, the code seems to go in-out-in of inlined and optimized


methods

 because it all gets mixed and reordered by the optimizer.

 If you want to do this portably, though, a better way that inserting


"push/pop"

 on every call, might be (for the compiler) to insert something like:

 try {
 .. function contents
 }
 catch(TracedException e) {
 e.push_frame_name("function name");
 throw e;
 }

 ..around every function body.  Then you only pay the piper in the sense


that

 you have increased the bloat factor, and neutered the inlining phase of


the

 optimizer.  Of course if you can (at compile time) determine that this


function

 can't/won't throw a traced exception, then you can omit the try{}catch{}.

 Another downside is that you only get stack tracing from throw-point to
 catch-point, but I suspect if you want a stack trace, it is probably


thrown "all

 the way" to main or further, right?

 Kevin
Apr 28 2005
parent reply "Ben Hinkle" <ben.hinkle gmail.com> writes:
 I'm going to spend more time on it, but Ben mentionned something good:
 debughlp vs imagehlp. Debughlp is only available on Windows XP and Windows
 2000 professional. I assumed that was not acceptable, but now I'm
 reconsidering. I guess everyone is on Windows XP or 2000 by now (time
 flies!).

 So, I will use debughlp and support AMD64 out of the box.


I searched for StackWalk on MSDN and didn't find anything except the 
debughlp version. I notice, though, that the include headers in 
dm/include/win32 only talk about StackWalk and not StackWalk64. I don't know 
which is better to use - maybe the 64 version would be fine behind a 
version(X86_64) or something.
Apr 28 2005
parent "Maxime Larose" <mlarose broadsoft.com> writes:
The StalkWalk64 version deals with 32-bit apps also.

My statement regarding StalkWalk64 as only being available under
Windows2000+ is false. You can have it for previous versions as well if you
get your hands on the dbghelp.dll redistributable. So, that's settled!

I don't want to jump the gun too much, but if all goes well, we will have
stack traced D in a few days/weeks. For windows that is. For Linux, I
unfortunately don't have it installed at home. I was considering getting a
unix server up for other reasons, but I pretty much decided against that for
the moment. So, if anyone wants to step in for linux...





"Ben Hinkle" <ben.hinkle gmail.com> wrote in message
news:d4qj84$kjr$1 digitaldaemon.com...

 I'm going to spend more time on it, but Ben mentionned something good:
 debughlp vs imagehlp. Debughlp is only available on Windows XP and




Windows

 2000 professional. I assumed that was not acceptable, but now I'm
 reconsidering. I guess everyone is on Windows XP or 2000 by now (time
 flies!).

 So, I will use debughlp and support AMD64 out of the box.


 I searched for StackWalk on MSDN and didn't find anything except the
 debughlp version. I notice, though, that the include headers in
 dm/include/win32 only talk about StackWalk and not StackWalk64. I don't


know

 which is better to use - maybe the 64 version would be fine behind a
 version(X86_64) or something.
Apr 28 2005
prev sibling parent reply Sean Kelly <sean f4.ca> writes:
In article <d4ntes$tm5$1 digitaldaemon.com>, Maxime Larose says...

- There are some undocumented requirements for StackWalk and I need a
CONTEXT. Apparently, a valid CONTEXT can only be obtained when: (1) the
thread is sleeping or (2) an exception is raised.


Look at std.thread and/or internal.gc.gcx.  One of those grabs a CONTEXT
structure in Windows builds.  This is used to get stack pointers for garbage
collection.


Sean
Apr 30 2005
parent "Maxime Larose" <mlarose broadsoft.com> writes:
"Sean Kelly" <sean f4.ca> wrote in message
news:d50f1t$s4m$1 digitaldaemon.com...

 In article <d4ntes$tm5$1 digitaldaemon.com>, Maxime Larose says...

- There are some undocumented requirements for StackWalk and I need a
CONTEXT. Apparently, a valid CONTEXT can only be obtained when: (1) the
thread is sleeping or (2) an exception is raised.


 Look at std.thread and/or internal.gc.gcx.  One of those grabs a CONTEXT
 structure in Windows builds.  This is used to get stack pointers for


garbage

 collection.


 Sean



I checked std.thread: nothing. In internal.gc.gcx, the CONTEXT is gotten
with GetThreadContext. Problem there is that the thread must be stopped for
the CONTEXT to be valid...
May 02 2005