Windows Debugger API — The End of Versioned Structures

Published in

The Startup

9 min readAug 14, 2020

Some time ago I was introduced to the Windows debugger API and found it incredibly useful for projects that focus on forensics or analysis of data on a machine. This API allows us to open a dump file taken on any windows machine and read information from it using the symbols that match the specific modules contained in the dump.

This API can be used in live debugging as well, either user-mode debugging of a process or kernel debugging. This post will show how to use it to analyze a memory dump, but this can be converted to live debugging relatively easily.

The main benefit of the debugger API is that it uses the specific symbols for the Windows version that it is running against, letting us write code that will work with any Windows version without having to keep an ever-growing header of structures for different versions, and needing to choose the right one and update our code every time the structure changes. For example, a common data structure to look at on Windows is the process, represented in the kernel by the EPROCESS structure. This structure changes almost every Windows build, meaning that fields inside it keep moving around. A field we are interested in might be at offset 0x100 in one Windows version, 0x120 in another, 0x108 in another, and so on. If we use the wrong offset the driver will not work properly and is very likely to accidentally crash the system. By using the symbols, we also receive the correct size and type of each structure and its sub-structures, so a nested structure getting larger, or a field changing its type, for example being a push lock in one version and a spin lock another, will be handled correctly by the debugger API without and code changes on our side.

The debugger API avoids this problem entirely by using symbols, so we can write our code once and it will run successfully on dumps taken from every possible Windows version without any need for updates when new builds are released. Also, it runs in user-mode so it doesn’t have all the inherent risks that kernel mode code carries with it, and since it can operate on a dump file, it doesn’t have to run on the machine that it analyzes. Which can be a huge benefit, as sometimes we can’t run our debugging tools on the machine we are interested in. This also lets us do extremely complicated things on much faster machines, such as analyzing a dump — or many dumps — in the cloud.

The main disadvantage of it is that the interface is not as easy as just using the types directly, and it takes some effort to get used to it. It also means slightly uglier, less readable code, unless you create macros around some of the calls.

In this post we’ll learn how to write a simple program that opens a memory dump iterates over all the processes and prints the name and PID of each one. For anyone not familiar with process representation in the Windows kernel, all the processes are linked together by a linked list (that is a LIST_ENTRY structure that points to the next entry and the previous entry). This list is pointed to by the nt!PsActiveProcessHead symbol and the list is found at the ActiveProcessLinks field of the EPROCESS structure. Of course, the symbol is not exported and the EPROCESS structure is not available in any of the public headers so implementing this in a driver will require some hard coded offsets and version checks to get the right offsets for each. Or we can use the debugger API instead!

To access all of this functionality we’ll need to include DbgEng.h and link against DbgEng.lib. And this is the right time for an important tip shared by Alex Ionescu — the debugging-related DLLs supplied by Windows are unstable and will often simply not work at all and leave you confused and wondering what you did wrong and why your code that was perfectly good yesterday is suddenly failing. WinDbg comes with its own versions of all the DLLs required for this functionality, that are way better. So you’ll want to copy Dbgeng.dll, Dbghelp.dll and Symsrv.dll from the directory where windbg.exe is into your output directory of this project. Do whatever you need to remember to always use the DLLs that come with WinDbg, this will save you a lot of time and frustration later.

Now that we have that covered we can start writing the code. Before we can access the dump file, we need to initialize 4 basic variables:

IDebugClient* debugClient;
IDebugSymbols* debugSymbols;
IDebugDataSpaces* dataSpaces;
IDebugControl* debugControl;

These will let us open the dump, access its memory and the symbols for all the modules in it and use them to parse the contents of the dump. First, we call DebugCreate to initialize the debugClient variable:

DebugCreate(__uuidof(IDebugClient), (PVOID*)&debugClient);

Note that all the functions we’ll use here return an HRESULT that should be validated using SUCCEEDED(result). In this post I will skip those validations to keep the code smaller and easier to read, but in any real program these should not be skipped.

After we initialized debugClient we can use it to initialize the other 3:

debugClient->QueryInterface(__uuidof(IDebugSymbols), 
                            (PVOID*)&debugSymbols);
debugClient->QueryInterface(__uuidof(IDebugDataSpaces), 
                            (PVOID*)&dataSpaces);
debugClient->QueryInterface(__uuidof(IDebugControl), 
                            (PVOID*)&debugControl);

There, setup done. We can open our dump file with debugClient->OpenDumpFile and then wait until all symbol files are loaded:

debugClient->OpenDumpFile(DumpFilePath);
debugControl->WaitForEvent(DEBUG_WAIT_DEFAULT, 0);

Once the dump is loaded we can start reading it. The module we are most interested in here is nt — we are going to use the PsActiveProcessHead symbol as well as the EPROCESS structure that belong to it. So we need to get the base of the module using dataSpaces->ReadDebuggerData. This function receives 4 arguments — Index, Buffer, BufferSize and DataSize. The last one is an optional output parameter, telling us how many bytes were written, or if the buffer wasn’t large enough, how many bytes are needed. To keep things simple we will always pass nullptr as DataSize, since we know in advance the needed sizes for all of our data. The second and third arguments are pretty clear so no need to say much about them. And for the first argument we need to look at the list of options found at DbgEng.h:

// Indices for ReadDebuggerData interface
#define DEBUG_DATA_KernBase 24
#define DEBUG_DATA_BreakpointWithStatusAddr 32
#define DEBUG_DATA_SavedContextAddr 40
#define DEBUG_DATA_KiCallUserModeAddr 56
#define DEBUG_DATA_KeUserCallbackDispatcherAddr 64
#define DEBUG_DATA_PsLoadedModuleListAddr 72
#define DEBUG_DATA_PsActiveProcessHeadAddr 80
#define DEBUG_DATA_PspCidTableAddr 88
#define DEBUG_DATA_ExpSystemResourcesListAddr 96
#define DEBUG_DATA_ExpPagedPoolDescriptorAddr 104
#define DEBUG_DATA_ExpNumberOfPagedPoolsAddr 112
...

These are all commonly used symbols, so they get their own index to make querying their value faster and easier. Later in this post we’ll see how we can get the value of a symbol that is less common and isn’t on this list.

The first index on this list is, conveniently, DEBUG_DATA_KernBase. So we create a variable to get the base address of the nt module and call ReadDebuggerData:

ULONG64 kernBase;
dataSpaces->ReadDebuggerData(DEBUG_DATA_KernBase,
                             &kernBase,
                             sizeof(kernBase),
                             nullptr);

Next, we want to iterate over all the processes and print information about them. To do that we need the EPROCESS type. One annoying thing about the debugger API is that it doesn’t allow us to use types like we would if they were in a header file. We can’t declare a variable of type EPROCESS and access its fields. Instead we need to access memory through a type ID and the offsets inside the type. Foe example, if we want to access the ImageFileName field inside a process we will need to read the information that’s found in processAddr + imageFileNameOffset. But this is getting a bit ahead. First we need to get the type ID of _EPROCESS using debugSymbols->GetTypeId, which receives the module base, type name and an output argument for the type ID. As the name suggests, this function doesn’t give us the type itself, only an identifier that we’ll use to get offsets inside the structure:

ULONG EPROCESS;
debugSymbols->GetTypeId(kernBase, “_EPROCESS”, &EPROCESS);

Now let’s get the offsets of the fields inside the EPROCESS so we can easily access them. Since we want to print the name and PID of each process we’ll need the ImageFileName and UniqueProcessId fields, in addition to ActiveProcessLinks so we iterate over the processes. To get those we’ll call debugSymbols->GetFieldOffset, which receives the module base, type ID, field name and an output argument that will receive the field offset:

ULONG imageFileNameOffset;
ULONG uniquePidOffset;
ULONG activeProcessLinksOffset;debugSymbols->GetFieldOffset(kernBase,
                             EPROCESS,
                             “ImageFileName”,
                             &imageFileNameOffset);
debugSymbols->GetFieldOffset(kernBase,
                             EPROCESS,
                             “UniqueProcessId”,
                             &uniquePidOffset);
debugSymbols->GetFieldOffset(kernBase,
                             EPROCESS,
                             “ActiveProcessLinks”,
                             &activeProcessLinksOffset);

To start iterating the process list we need to read PsActiveProcessHead. You might have noticed earlier that this symbol has an index in DbgEng.h so it can be read directly using ReadDebuggerData. But for this example we won’t read it that way, and instead show how to read it like a symbol that doesn’t have an index. So first we need to get the symbol offset in the dump file, using debugSymbols->GetOffsetByName:

ULONG64 activeProcessHead;
debugSymbols->GetOffsetByName(“nt!PsActiveProcessHead”,  
                              &activeProcessHead);

This doesn’t give us the actual value yet, only the offset of this symbol. To get the value we’ll need to read the memory that this address points to from the dump using dataSpaces->ReadVirtual, which receives an address to read from, Buffer, BufferSize and an optional output argument BytesRead. We know that this symbol points to a LIST_ENTRY structure so we can just define a local linked list and read the variable into it. In this case we got lucky — the LIST_ENTRY structure is documented. If this symbol contained a non-documented structure this process would require a couple more steps and be a bit more painful.

LIST_ENTRY activeProcessLinks;
dataSpaces->ReadVirtual(activeProcessHead,
                        &activeProcessLinks,
                        sizeof(activeProcessLinks),
                        nullptr);

Now we have almost everything we need to start iterating the process list! We’ll define a local process variable and use it to store the address of the current process we’re looking at. In each iteration, activeProcessLinks.Flink will point to the first process in the system, but it won’t point to the beginning of the EPROCESS. It points to the ActiveProcessLinks field, so to get to the beginning of the structure we’ll need to subtract the offset of ActiveProcessLinks field from the address (basically what the CONTAINING_RECORD macro would do if we could use it here). Notice that we are using a ULONG64 here on purpose, instead of a ULONG_PTR to save us the pain of using pointer arithmetic and avoiding casts in future function calls, since most debugger API functions receive arguments as ULONG64:

ULONG64 process;
process = (ULONG64)activeProcessLinks.Flink — activeProcessLinksOffset;

The process iteration is pretty simple — for each process we want to read the ImageFileName value and UniqueProcessId value, and then read the next process pointer from ActiveProcessLinks. Notice that we cannot access any data in the debugger directly. The addresses we have are meaningless in the context of our current process (they are also kernel addresses, and our application is running in user mode and not necessarily on the right machine), and we need to call dataSpaces->ReadVirtual, or any of the other debugger functions that let us read data, to access any of the memory and will have to read these values for each process.

Generally we don’t have to read each value separately, we can also read the whole EPROCESS structure with debugSymbols->ReadTypedDataVirtual for each process and then access the fields by their offsets. But the EPROCESS structure is very large and we only need a few specific fields, so reading the whole structure is pretty wasteful and not necessary in this case.

We now have everything we need to implement our process iteration:

UCHAR imageFileName[15];
ULONG64 uniquePid;
LIST_ENTRY activeProcessLinks;do
{
    //
    // Read process name, pid and activeProcessLinks 
    // for the current process
    //
    dataSpaces->ReadVirtual(process + imageFileNameOffset,
                            &imageFileName,
                            sizeof(imageFileName),
                            nullptr);
    dataSpaces->ReadVirtual(process + uniquePidOffset,
                            &uniquePid,
                            sizeof(uniquePid),
                            nullptr);
    dataSpaces->ReadVirtual(process + activeProcessLinksOffset,
                            &activeProcessLinks,
                            sizeof(activeProcessLinks),
                            nullptr);
    printf(“Current process name: %s, pid: %d\n”,
           imageFileName,
           uniquePid);
    //
    // Get the next process from the list and
    // subtract activeProcessLinksOffset
    // to get to the start of the EPROCESS.
    //
    process = (ULONG64)activeProcessLinks.Flink — activeProcessLinksOffset;
} while ((ULONG64)activeProcessLinks.Flink != activeProcessHead);

That’s it, that’s all we need to get this nice output:

Some of you might notice that a few of these process names look incomplete. This is because the ImageFileName field only has the first 15 bytes of the process name, while the full name is saved in an OBJECT_NAME_INFORMATION structure (which is actually just a UNICODE_STRING) in SeAuditProcessCreationInfo.ImageFileName. But in this post I wanted to keep things simple so we’ll use ImageFileName here.

Now we only have one last part left — being good developers and cleaning up after ourselves:

if (debugClient != nullptr)
{
    debugClient->EndSession(DEBUG_END_ACTIVE_DETACH);
    debugClient->Release();
}
if (debugSymbols != nullptr)
{
    debugSymbols->Release();
}
if (dataSpaces != nullptr)
{
    dataSpaces->Release();
}
if (debugControl != nullptr)
{
    debugControl->Release();
}

This was a very brief, but hopefully helpful, introduction to the debugger API. There are endless more options available with this, looking at DbgEng.h or at the official documentation should reveal a lot more. I hope you all find this as useful as I do and will find new and interesting things to use it for.

Windows Debugger API — The End of Versioned Structures

Written by Yarden Shafir