www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.debugger - Natvis, CV8, dAssocArray

reply Void-995 <void995 gmail.com> writes:
I almost finished my -gc for Natvis experiment (to use MS C++ 
Debugger from VS Code, fully translating type names from what 
comes from DMD frontend to valid C++ type name so Natvis works), 
but then I encountered dAssocArray, which has form of: void* as 
CodeView structure member, KeyType __key_t and ValueType __val_t 
as nested types.

As for someone who just found about CodeView today: what is 
nested type, what is the deal with void*, what to do with __key_t 
and __val_t to get values.
Sep 25 2018
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 25/09/2018 22:08, Void-995 wrote:

 I almost finished my -gc for Natvis experiment (to use MS C++ Debugger 
 from VS Code, fully translating type names from what comes from DMD 
 frontend to valid C++ type name so Natvis works), but then I encountered 
 dAssocArray, which has form of: void* as CodeView structure member, 
 KeyType __key_t and ValueType __val_t as nested types.
 
 As for someone who just found about CodeView today: what is nested type, 
 what is the deal with void*, what to do with __key_t and __val_t to get 
 values.


Nested types are types declared in another type, e.g. in C++

struct dAssocArray
{
	typedef int __key_t;
	typedef long __val_t;
	void* ptr;
};

The AA data structures are not exposed by the compiler, but the mago 
debugger rebuilds them from these types.
Sep 26 2018
parent reply Void-995 <void995 gmail.com> writes:
On Wednesday, 26 September 2018 at 07:37:26 UTC, Rainer Schuetze 
wrote:

 On 25/09/2018 22:08, Void-995 wrote:

 I almost finished my -gc for Natvis experiment (to use MS C++ 
 Debugger from VS Code, fully translating type names from what 
 comes from DMD frontend to valid C++ type name so Natvis 
 works), but then I encountered dAssocArray, which has form of: 
 void* as CodeView structure member, KeyType __key_t and 
 ValueType __val_t as nested types.
 
 As for someone who just found about CodeView today: what is 
 nested type, what is the deal with void*, what to do with 
 __key_t and __val_t to get values.


 Nested types are types declared in another type, e.g. in C++

 struct dAssocArray
 {
 	typedef int __key_t;
 	typedef long __val_t;
 	void* ptr;
 };

 The AA data structures are not exposed by the compiler, but the 
 mago debugger rebuilds them from these types.


Makes sense. I've tried to move them to "member" but they 
obviously did not point anywhere, which is logical if that is 
just "typedef". Is there any way to expose real key/values from 
backend or get them via some Natvis magic? That's basically only 
thing that stops from using MS C/C++ plug-in for debugging from 
VS Code. Natvis support may be valuable for Linux and Mac as 
well, as they've added minimal support for Natvis to GDB and LLDB 
via their plug-in.
Sep 26 2018
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 26/09/2018 09:45, Void-995 wrote:

 On Wednesday, 26 September 2018 at 07:37:26 UTC, Rainer Schuetze wrote:

 On 25/09/2018 22:08, Void-995 wrote:

 I almost finished my -gc for Natvis experiment (to use MS C++ 
 Debugger from VS Code, fully translating type names from what comes 
 from DMD frontend to valid C++ type name so Natvis works), but then I 
 encountered dAssocArray, which has form of: void* as CodeView 
 structure member, KeyType __key_t and ValueType __val_t as nested types.

 As for someone who just found about CodeView today: what is nested 
 type, what is the deal with void*, what to do with __key_t and 
 __val_t to get values.


 Nested types are types declared in another type, e.g. in C++

 struct dAssocArray
 {
     typedef int __key_t;
     typedef long __val_t;
     void* ptr;
 };

 The AA data structures are not exposed by the compiler, but the mago 
 debugger rebuilds them from these types.


 
 Makes sense. I've tried to move them to "member" but they obviously did 
 not point anywhere, which is logical if that is just "typedef". Is there 
 any way to expose real key/values from backend or get them via some 
 Natvis magic? That's basically only thing that stops from using MS C/C++ 
 plug-in for debugging from VS Code. Natvis support may be valuable for 
 Linux and Mac as well, as they've added minimal support for Natvis to 
 GDB and LLDB via their plug-in.


Not sure if you can define new structs in NatVis, but the void pointer 
points to rt.aaA.Impl which contains an array of Buckets, and their 
entry member points to a std::pair<__key_t,__val_t>.
Sep 26 2018
parent reply Void-995 <void995 gmail.com> writes:
On Wednesday, 26 September 2018 at 18:06:48 UTC, Rainer Schuetze 
wrote:

 On 26/09/2018 09:45, Void-995 wrote:

 On Wednesday, 26 September 2018 at 07:37:26 UTC, Rainer 
 Schuetze wrote:

 [...]


 
 Makes sense. I've tried to move them to "member" but they 
 obviously did not point anywhere, which is logical if that is 
 just "typedef". Is there any way to expose real key/values 
 from backend or get them via some Natvis magic? That's 
 basically only thing that stops from using MS C/C++ plug-in 
 for debugging from VS Code. Natvis support may be valuable for 
 Linux and Mac as well, as they've added minimal support for 
 Natvis to GDB and LLDB via their plug-in.


 Not sure if you can define new structs in NatVis, but the void 
 pointer points to rt.aaA.Impl which contains an array of 
 Buckets, and their entry member points to a 
 std::pair<__key_t,__val_t>.


Thanks, I did something like that meanwhile: 
https://www.dropbox.com/s/h70m4v63jxp79g1/d.natvis?dl=0

Need to take a look at other ideas as well. Of course it won't 
hurt to clean that up, but this one works.
Sep 26 2018
parent reply Void-995 <void995 gmail.com> writes:
On Wednesday, 26 September 2018 at 18:48:27 UTC, Void-995 wrote:

 On Wednesday, 26 September 2018 at 18:06:48 UTC, Rainer 
 Schuetze wrote:

 On 26/09/2018 09:45, Void-995 wrote:

 On Wednesday, 26 September 2018 at 07:37:26 UTC, Rainer 
 Schuetze wrote:

 [...]


 
 Makes sense. I've tried to move them to "member" but they 
 obviously did not point anywhere, which is logical if that is 
 just "typedef". Is there any way to expose real key/values 
 from backend or get them via some Natvis magic? That's 
 basically only thing that stops from using MS C/C++ plug-in 
 for debugging from VS Code. Natvis support may be valuable 
 for Linux and Mac as well, as they've added minimal support 
 for Natvis to GDB and LLDB via their plug-in.


 Not sure if you can define new structs in NatVis, but the void 
 pointer points to rt.aaA.Impl which contains an array of 
 Buckets, and their entry member points to a 
 std::pair<__key_t,__val_t>.


 Thanks, I did something like that meanwhile: 
 https://www.dropbox.com/s/h70m4v63jxp79g1/d.natvis?dl=0

 Need to take a look at other ideas as well. Of course it won't 
 hurt to clean that up, but this one works.


My C++ naming for NatVis experiment has been failed right now. 
Somewhere around memory allocations which are all that kind of 
spaghetti right now in backend of DMD. But! For x86_mscoff and 
x86_64 associative arrays are named just as `dAssocArray`, while 
slices are using full/real name from D. It's pretty easy to get 
it to be named as `dArray`, just as it was before. Of course, 
adding additional nested type to allow NatVis to know about 
actual type would be great as well, as for no reason it ignores 
pointer type and thinks it's `void*`, so we would need an 
additional cast in NatVis. With that simple change, next NatVis 
file would totally work as you expect: 
https://www.dropbox.com/s/t5yejma3w7cbna9/dlang.natvis?dl=0

Which might be a best we can get to make VS Code experience on 
Windows with DMD and debugging more comfortable. I hope it will 
be useful addition to D ecosystem for others as well.
Sep 27 2018
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 27/09/2018 18:52, Void-995 wrote:

 On Wednesday, 26 September 2018 at 18:48:27 UTC, Void-995 wrote:

 On Wednesday, 26 September 2018 at 18:06:48 UTC, Rainer Schuetze wrote:

 Not sure if you can define new structs in NatVis, but the void 
 pointer points to rt.aaA.Impl which contains an array of Buckets, and 
 their entry member points to a std::pair<__key_t,__val_t>.


 Thanks, I did something like that meanwhile: 
 https://www.dropbox.com/s/h70m4v63jxp79g1/d.natvis?dl=0

 Need to take a look at other ideas as well. Of course it won't hurt to 
 clean that up, but this one works.


 
 My C++ naming for NatVis experiment has been failed right now. Somewhere 
 around memory allocations which are all that kind of spaghetti right now 
 in backend of DMD. But! For x86_mscoff and x86_64 associative arrays are 
 named just as `dAssocArray`, while slices are using full/real name from 
 D. It's pretty easy to get it to be named as `dArray`, just as it was 
 before. Of course, adding additional nested type to allow NatVis to know 
 about actual type would be great as well, as for no reason it ignores 
 pointer type and thinks it's `void*`, so we would need an additional 
 cast in NatVis. With that simple change, next NatVis file would totally 
 work as you expect: 
 https://www.dropbox.com/s/t5yejma3w7cbna9/dlang.natvis?dl=0
 
 Which might be a best we can get to make VS Code experience on Windows 
 with DMD and debugging more comfortable. I hope it will be useful 
 addition to D ecosystem for others as well.


Reminds me of autoexp.dat, the predecessor of NatVis. With a little help 
from cv2pdb I managed to show slices and AAs in the C++ debugger aswell, 
but it was horrible to figure out as the debugger crashed if there was 
something unexpected in the visualizer macros, needing a bunch of 
workarounds. I guess this has improved with NatVis, has it?

You could also capture the type of template arguments in the type name. 
Doesn't NatVis feature this, too? That should allow casting the slice 
pointer (though the debug info should be correct, too).

autoexp.dat macros were also pretty slow, so showing a larger AA could 
cause the debugger to grind to a halt. How does NatVis perform?

For a more advanced problem: show the value of `aa["7"]` in the watch 
window ;-)
Sep 28 2018
parent Void-995 <void995 gmail.com> writes:
On Friday, 28 September 2018 at 16:32:36 UTC, Rainer Schuetze 
wrote:

 On 27/09/2018 18:52, Void-995 wrote:

 On Wednesday, 26 September 2018 at 18:48:27 UTC, Void-995 
 wrote:

 On Wednesday, 26 September 2018 at 18:06:48 UTC, Rainer 
 Schuetze wrote:

 Not sure if you can define new structs in NatVis, but the 
 void pointer points to rt.aaA.Impl which contains an array 
 of Buckets, and their entry member points to a 
 std::pair<__key_t,__val_t>.


 Thanks, I did something like that meanwhile: 
 https://www.dropbox.com/s/h70m4v63jxp79g1/d.natvis?dl=0

 Need to take a look at other ideas as well. Of course it 
 won't hurt to clean that up, but this one works.


 
 My C++ naming for NatVis experiment has been failed right now. 
 Somewhere around memory allocations which are all that kind of 
 spaghetti right now in backend of DMD. But! For x86_mscoff and 
 x86_64 associative arrays are named just as `dAssocArray`, 
 while slices are using full/real name from D. It's pretty easy 
 to get it to be named as `dArray`, just as it was before. Of 
 course, adding additional nested type to allow NatVis to know 
 about actual type would be great as well, as for no reason it 
 ignores pointer type and thinks it's `void*`, so we would need 
 an additional cast in NatVis. With that simple change, next 
 NatVis file would totally work as you expect: 
 https://www.dropbox.com/s/t5yejma3w7cbna9/dlang.natvis?dl=0
 
 Which might be a best we can get to make VS Code experience on 
 Windows with DMD and debugging more comfortable. I hope it 
 will be useful addition to D ecosystem for others as well.


 Reminds me of autoexp.dat, the predecessor of NatVis. With a 
 little help from cv2pdb I managed to show slices and AAs in the 
 C++ debugger aswell, but it was horrible to figure out as the 
 debugger crashed if there was something unexpected in the 
 visualizer macros, needing a bunch of workarounds. I guess this 
 has improved with NatVis, has it?

 You could also capture the type of template arguments in the 
 type name. Doesn't NatVis feature this, too? That should allow 
 casting the slice pointer (though the debug info should be 
 correct, too).

 autoexp.dat macros were also pretty slow, so showing a larger 
 AA could cause the debugger to grind to a halt. How does NatVis 
 perform?

 For a more advanced problem: show the value of `aa["7"]` in the 
 watch window ;-)


NatVis pretty fast from my experience, at least for my current 
needs.

After some additional digging into the topic I'm thinking about 
DIP or something to "standardize" debug type names. CodeView for 
`x86_mscoff` and `x86_64` on Windows works in one way, DWARF code 
path on Linux seems to generate totally another names, LDC2 on 
Windows says that AA is `unsigned char *`.

With help from WebFreak, the author of VS Code plug-in for D, we 
found that the only thing that was really missing on MIEngine, 
that powers GDB/LLDB debugging in Microsoft's plug-in, is the 
support for `CustomListItems` tag that I'm using for AA. I'm even 
willing to add `CustomListItems` support for MIEngine by getting 

+ DWARF:
- `int[]` - `_Array_int`
- `int[int]` - `_AArray_int_int`
+ CV8 (CV4?):
- `int[]` - `int[]`
- `int[int]` - `dAssocArray`
+ What actually works if we'll change CV8 (CV4?):
- `int[]` - `darray.ptrid:423` (423 is an internal type ID for 
example, can work as `darray.int[]` as well, whatever, needs 
prefix `whatever.` to make NatVis happy)
- `int[int]` - `dassocarray.keyid:423_valueid:423` (again, 423 is 
an internal type ID for example, it would be great to do 
dassocarray.int[int], but setting identifier inside of DMD's 
backend for AA's type crashes DMD, while working perfectly fine 
when same is done to Slice)

And now to the fun part... If I will change CV8 to what works - I 
will break Mago in VisualD. And I would have to change naming in 
DWARF as well. Which means to make D debugging with DMD working 
in VS Code everywhere (Windows, Linux, Mac), I need to:
- Change DWARF naming in DMD
- Add support of `CustomListItems` to MIEngine
- Change CV8 naming (CV4?) in DMD
- Make sure Mago would use prefix checking with new names instead 
of total matching for AA (and same for Slices).

So my proposal (at least at this moment) for `int[]` and 
`int[int]` is to generate/use `darray.int[]` and 
`dassocarray.int[int]` with `dassocarray.keyid:423_valueid:423` 
as temporary workaround (that will work as well) for DMD. 
LDC2/GDC may use normal variant without workaround. This one will 
allow NatVis be happy on Windows and with patching MIEngine - on 
Linux and Mac as well. I guess it shouldn't interfere with how 
Pretty Printer is working for native GDB as well (LLDB too?).

Why the prefix + suffix? More like why adding suffix for AA, you 
might ask. Because NatVis is caching how it handles the type, and 
second time he will see `dAssocArray` - he will think that this 
AA has the same type as previous one, while __key_t and __val_t 
could be different.
Sep 28 2018