How to extend dotnet-dump (1/2) — What are the new commands?
To ease our troubleshooting sessions at Criteo, new high level commands for WinDBG have been written and grouped in the gsose extension. As we moved to Linux, Kevin implemented the plumbing to be able to load gsose into LLDB. In both cases, our extension commands are based on ClrMD to dig into a memory dump.
As Microsoft pushed for dotnet-dump as the new cross-platform way to deal with memory dump, it was obvious that we would have to be able to use our extension commands in dotnet-dump. Unfortunately dotnet-dump does not support any extension mechanism. In May this year, Kevin updated a minimum of code to load a list of extension assemblies at the startup of dotnet-dump. I followed another direction by adding a “load” command to dynamically add extensions commands.
However, the Diagnostics team was focusing on supporting .NET Core 5 release and adding extensibility was not planned before next year. Due to our own time constraints at Criteo, gsose extension commands were really needed so we agreed with the Diagnostics team to implement these commands directly into dotnet-dump as pull requests.
This first post describes the new commands, when to use them, and the git setup I used to implement them.
Commands details and challenges
Here is the list of extension commands as shown by the help command:
As of today with the last 3.1.141901 official version, only timerinfo is available but feel free to clone the repository and rebuild it to get all commands.
pstacks: almost Parallel Stacks a la Visual Studio
When you start an investigation, you are usually interested in getting a high level view of what the running threads are doing and this is what pstacks provides. When hundreds of threads are running, commands such as clrthreads are useless. Instead, pstacks merges the common parts of each call stack and present them in a tree:
Don’t be scared by the layout; here is a description of each section:
If you look at the threads 9c6c and 4020 (after the last ~~~~), it means that 2 threads (only the first 4 threads id are shown except if you pass — all as a parameter) share the same callstack:
Compared to the “list” representation, the “tree” representation is useful when you have a lot of threads to deal with and see the different branches in the groups of stack frames.
ti: see all your timers
If you don’t find anything interesting in the running threads (i.e. not hundreds of threads with the same code stack blocked on a Wait call for example), you should look at what will run as timer callbacks with the ti command.
Here is example of what you will get:
The first part of the output lists each instance of Timer with start/repeat information followed by the callback parameter and callback if available. The second list groups timer so you could identify cases where the same timers have been created many times.
tpq: what is in the ThreadPool queues
If no culprit is identified in timers, it is time to look at what is in the ThreadPool with the tpq command.
Both work items from the global queue and the local queues are displayed with each identified callback. The final summary spits work items and tasks.
Two other helper commands are also available.
tks: Task State
Pass a Task object reference to tks to get its human readable state.
dcq : Dump ConcurrentQueue
Dump elements stored in a ConcurrentQueue. Due to implementation details, more manual steps are required in case of value types.
Note that for reference type items, the dumpobj command is provided to get the value of the item fields: you just copy and paste them to get instance fields.
Before digging into the implementation details, I want to spend some time on how to properly create a pull request. Since the Diagnostics repository is hosted on Github, you have to create a fork and push your changes on a dedicated branch to then submit a pull request.
Due to my previous Team Foundation Server experience, it seems that I have problems with Git commands: too many different ways to do simple things maybe. So I was faithfully relying on the documentation to configure/sync a fork and I ended up merging the Microsoft Master branch to our Criteo fork Master before creating my own branch dedicated to my pull request:
After a while, in the next pull requests, I started to have unrelated commits in the pull requests:
It was due to the fact that I was merging the Criteo fork master with Diagnostics master to stay in sync.
My coworker Guillaume explained to me that I should follow a different path. This time, I’m just creating a branch on the Microsoft repository (so no need to merge) and I’m passing by the Criteo fork just to push upstream:
Now I can change my local source code before pushing upstream to Criteo fork
That way, I can synchronize the Criteo fork without “impacting” the history of commits brought with my dedicated pull request branch.
The next episode will cover command implementation details.
What’s next? Read the second part of this article:
How to write commands for dotnet-dump
This post describes the different steps, tips and tricks to write your own commands for dotnet-dump
Like what you are reading? Check out more articles from Christophe on our medium account.
The .NET Core Journey at Criteo
This post shows the challenges we faced during the migration to .NET Core on containerized Linux for our main…
Are you interested to work with Christophe and other talented Engineers at Criteo? Take a look at our open positions: