Technology was a mistake
So I had a day.
In my day job I work with Unreal. Today I encountered a crash in our dedicated server build which dumped an anonymous callstack — that is, just a bunch of memory addresses, no function names.
But that’s okay, because it’s a development build and we have a .pdb file with all the debug symbols sitting right next to it, so it should be trivial to just load up the crash dump in Visual Studio, point it at the .pdb, and let Visual Studio resolve the stack into something usable.
Visual Studio, however, threw an error about being unable to find the executable associated with the crash dump. It displays the full path that it’s searching, which… is 100% correct. And I manually verified the file exists, and all the case is correct, and all the slashes go in the right direction (because LOL Windows), and it is byte-for-byte the correct version of the executable, and so on and so forth. But still: “executable not found”.
After some digging, I found this, which suggests that Unreal’s crash dump generation was broken by the Windows 10 Fall Creators Update, with no current ETA for a fix.
Windows 10 strikes again!
The suggested workaround is to source some system DLLs from before the Creators Update, which I neither have access to nor am willing to download from some random, probably-shady third-party website, because accidentally embedding secret spyware/ransomware/loli porn in our game sounds like a generally poor career choice.
Fortunately, Unreal does have its own built-in tool for resolving these call stacks, called SymbolDebugger. I’d never used it before, but no time like the present, right?
Turns out, SymbolDebugger requires that Perforce be installed, and requires that your project be versioned in Perforce.
We use git.
After some more digging, I found this, in which an Epic staffer confirms that SymbolDebugger “is only designed to work with Perforce at the moment”, so basically us git users are just SOL. Which I guess is fine, really, seeing as git is super niche and used by almost no one, and you definitely want to put your development resources where they’ll have the greatest positive impact, and all that.
That thread also led me here where some enterprising and wonderful soul figured out that SymbolDebugger actually just more-or-less wraps another built-in utility called MinidumpDiagnostics, which may have been the actual workhorse for resolving these callstacks all along. I had to do some voodoo to get it configured and runnable, because it doesn’t work right out of the box (because of course it doesn’t) but finally I had it set up and the moment of truth arrived!
And it crashed on launch.
This tool, though, actually gave me a real callstack for its crash. Keep in mind, this isn’t the server crash I was originally trying to debug; this is the crash in the tool that’s supposed to resolve the server crash into a usable callstack.
But at least the tool’s callstack might lead me to some explanation for why the tool isn’t working, and if I could fix that then I could use the now-working tool to figure out why the server isn’t working.
So I loaded up the engine source and navigated to the top of the tool’s callstack, where I found the useless assert that I’d hit, accompanied by a single, lonesome comment:
// Something wrong.
I finally said “screw it” and rebuilt the whole damn engine/project in debug — which took hours, because siiiiigh — repro’d the crash, got a sensible callstack as god intended and then fixed the stupid server bug five minutes later.
Technology was a mistake.