Getting another view on thread stacks with ClrMD
This post of the series details how to look into your threads stack with ClrMD.
It’s been a long time (see the resources at the end) since I’ve been discussing what ClrMD could bring to .NET developers/DevOps! My colleague Kevin just wrote an article about how to emulate SOS DumpStackObjects command both on Windows and Linux with ClrMD. This implementation lists the objects on the stack but without their values (like strings content for example) nor the stack frames corresponding to the method calls.
The rest of the post will show you, with ClrMD, how to get an higher view, closer to what the SOS ClrStack command could provide.
Let’s take this simple application as an example:
As you can see, I’ve mixed value and reference types as parameters and local variables up to the call to the
Third method that displays the procdump command line to execute in order to generate a memory dump of the process.
Use WinDBG + SOS Luke!
When you open it with WinDBG and load SOS, here is the result of the dso command:
The clrstack command shows the stacked method calls:
And if you use the -a parameter, you will get methods with their parameters and local variables (or -p for parameters only and -l for local variables only):
It is weird that SOS implementation does not give the type of both the parameters and locals. But wait! While researching for this post, I looked at the SOS implementation (now in the strike.cs file moved from the coreclr to the diagnostics repository) to find this nice comment:
So I tried clrstack with -i and I got the types for parameters (and locals unlike what the comments implies):
Even though clrstack supports the -all flag to dump the call stack of all managed threads, you might need to do your own automatic analysis on hundreds of threads and this is where ClrMD shines.
Merging methods and parameters/locals
When I read Kevin’s post, I immediately thought about adding the method call on the stack based on the work I’ve done in March 2019 to implement the pstacks tool. At that time, my goal was to aggregate the call stacks of a large number of threads in order to find out pattern of blocked threads, sharing the “same” call stacks. Visual Studio provides a great “Parallel Stacks” pane but I needed it for both Windows and Linux.
To list all the call stack with ClrMD, you simply enumerate the managed threads and for each one, its
StackTrace property contains the list of
StackFrame objects corresponding to each method call.
StackPointer property of each frame contains the address of the frame in the call stack, allowing a mapping of the method call with its parameters and locals:
As always with stacks, lower addresses correspond to the last things added to the stack (i.e. last called method). While checking between what is shown by SOS, the parameters/locals addresses and frame stack pointers, you realize that all objects at an address in the stack equal or below the
StackPointer of a frame are either parameters or local variables of the frame method.
Even better, for non static method, you can guess what is the this implicit parameter if the address is the same as the frame
StackPointer; shown with the green = sign in the previous screenshot and prefixed by > in Kevin’s updated code that merges the method calls to the parameters and locals:
FormatFrame helper method simply prefix static methods with # instead of | for instance methods:
Unfortunately, I did not find any way with ClrMD to make the difference between parameters and locals. Based on what you can see in SOS implementation of this part of the clrstack command, it relies on the
EnumetateLocalVariables methods of
ICorDebugILFrame which is not exposed by ClrMD. There is another undocumented implementation based on private interfaces I could not leverage neither. For a larger discussion around stack walking in .NET, read this great post by Matt Warren.
Also, without any explicit access to specific parameter or local, I did not find a way to get the value of primitive and value type instances stored on the stack. However, it is still possible to get them for boxed ones and reference type instances such as string for example.
Getting instances from the stack
In the last code excerpt, I did not describe the
DumpObject helper method used to display an object on the stack. The implementation provided by Kevin was used to show the address and the type of the object:
The next step would be to display value for primitive types such as numbers, boolean, string and even array size:
Most of this code is based on the
GetValue helper from
ClrType: it returns the right “thing” for simple types. Look at ClrMD implementation details to get a better understanding of how the value is rebuilt.
GetArrayAsString simply returns the number of elements in the array:
And the call stack is now complete!
Note that you may even get more locals or parameters than with WinDBG+SOS but don’t ask me why…
For more advanced object formatting cases such as dumping structs or enumerating fields and their value, I would highly recommend to look at the related ClrMD documentation page (just replace
ClrType and you’ll be safe).
- Dumping stack objects with ClrMD by Kevin Gosse
- Stack walking in the .NET Runtime by Matt Warren
- Part 1: Bootstrap ClrMD to load a dump.
- Part 2: Find duplicated strings with ClrMD heap traversing.
- Part 3: List timers by following static fields links.
- Part 4: Identify timers callback and other properties.
- Part 5: Use ClrMD to extend SOS in WinDBG.
- Part 6: Manipulate memory structures like real objects.
- Part 7: Manipulate nested structs using dynamic.
- Part 8: Spelunking inside the .NET Thread Pool
- Part 9: Deciphering Tasks and Thread Pool items