Criteo R&D Blog
Published in

Criteo R&D Blog

Getting another view on thread stacks with ClrMD

This post of the series details how to look into your threads stack with ClrMD.


It’s been a long time (see the resources at the end) since I’ve been discussing what ClrMD could bring to .NET developers/DevOps! My colleague Kevin just wrote an article about how to emulate SOS DumpStackObjects command both on Windows and Linux with ClrMD. This implementation lists the objects on the stack but without their values (like strings content for example) nor the stack frames corresponding to the method calls.

The rest of the post will show you, with ClrMD, how to get an higher view, closer to what the SOS ClrStack command could provide.

Let’s take this simple application as an example:

As you can see, I’ve mixed value and reference types as parameters and local variables up to the call to the Third method that displays the procdump command line to execute in order to generate a memory dump of the process.

Use WinDBG + SOS Luke!

When you open it with WinDBG and load SOS, here is the result of the dso command:

The clrstack command shows the stacked method calls:

And if you use the -a parameter, you will get methods with their parameters and local variables (or -p for parameters only and -l for local variables only):

It is weird that SOS implementation does not give the type of both the parameters and locals. But wait! While researching for this post, I looked at the SOS implementation (now in the strike.cs file moved from the coreclr to the diagnostics repository) to find this nice comment:

So I tried clrstack with -i and I got the types for parameters (and locals unlike what the comments implies):

Even though clrstack supports the -all flag to dump the call stack of all managed threads, you might need to do your own automatic analysis on hundreds of threads and this is where ClrMD shines.

Merging methods and parameters/locals

When I read Kevin’s post, I immediately thought about adding the method call on the stack based on the work I’ve done in March 2019 to implement the pstacks tool. At that time, my goal was to aggregate the call stacks of a large number of threads in order to find out pattern of blocked threads, sharing the “same” call stacks. Visual Studio provides a great “Parallel Stacks” pane but I needed it for both Windows and Linux.

To list all the call stack with ClrMD, you simply enumerate the managed threads and for each one, its StackTrace property contains the list of StackFrame objects corresponding to each method call.

The StackPointer property of each frame contains the address of the frame in the call stack, allowing a mapping of the method call with its parameters and locals:

As always with stacks, lower addresses correspond to the last things added to the stack (i.e. last called method). While checking between what is shown by SOS, the parameters/locals addresses and frame stack pointers, you realize that all objects at an address in the stack equal or below the StackPointer of a frame are either parameters or local variables of the frame method.

Even better, for non static method, you can guess what is the this implicit parameter if the address is the same as the frame StackPointer; shown with the green = sign in the previous screenshot and prefixed by > in Kevin’s updated code that merges the method calls to the parameters and locals:

The FormatFrame helper method simply prefix static methods with # instead of | for instance methods:

Unfortunately, I did not find any way with ClrMD to make the difference between parameters and locals. Based on what you can see in SOS implementation of this part of the clrstack command, it relies on the EnumerateArguments and EnumetateLocalVariables methods of ICorDebugILFrame which is not exposed by ClrMD. There is another undocumented implementation based on private interfaces I could not leverage neither. For a larger discussion around stack walking in .NET, read this great post by Matt Warren.

Also, without any explicit access to specific parameter or local, I did not find a way to get the value of primitive and value type instances stored on the stack. However, it is still possible to get them for boxed ones and reference type instances such as string for example.

Getting instances from the stack

In the last code excerpt, I did not describe the DumpObject helper method used to display an object on the stack. The implementation provided by Kevin was used to show the address and the type of the object:

The next step would be to display value for primitive types such as numbers, boolean, string and even array size:

Most of this code is based on the GetValue helper from ClrType: it returns the right “thing” for simple types. Look at ClrMD implementation details to get a better understanding of how the value is rebuilt.

The GetArrayAsString simply returns the number of elements in the array:

And the call stack is now complete!

Note that you may even get more locals or parameters than with WinDBG+SOS but don’t ask me why…

For more advanced object formatting cases such as dumping structs or enumerating fields and their value, I would highly recommend to look at the related ClrMD documentation page (just replace GCHeapType by ClrType and you’ll be safe).




Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Christophe Nasarre

Christophe Nasarre

Loves to understand how things work (MVP Developer Technologies)