Fixing the Arch-Vile fire bug in Doom II

I thought I would share this interesting exercise that I did last summer. I patched the Doom II executable with a hex editor in order to fix a bug.

Why would someone fix a cosmetic bug in a game that’s about 25 years old? Especially that this bug doesn’t affect the gameplay and has rarely been seen by anyone? Well, I wanted to test my skills and see how hard it would be to accomplish this. It’s also an occasion to learn a bit about patching executable files in an hex editor because it’s good knowledge for a reverse engineer and as someone who likes to participate in CTFs (Capture The Flag competitions).

Doom II has a bug where the fire from an Arch-Vile attack will be spawned at the wrong location. Usually, the fire will appear in the face of the targeted player or monster, but a typo in the game’s source code causes this bug.

Let’s look at the source code that was released under the GNU GPL in 1999:

   //
// A_VileTarget
// Spawn the hellfire
//
void A_VileTarget (mobj_t* actor)
{
mobj_t* fog;

if (!actor->target)
return;

A_FaceTarget (actor);

fog = P_SpawnMobj (actor->target->x,
actor->target->x,
actor->target->z, MT_FIRE);

actor->tracer = fog;
fog->target = actor;
fog->tracer = actor->target;
A_Fire (fog);
}

As you probably noticed, the game passes the values “x,x,z” instead of the expected “x,y,z” to the function that spawns the fire. This bug was probably never noticed by the developers because the fire gets correctly positioned in the face of the targeted game object once it’s spawned. The bug only occurs if the player moves out of the line of sight of the Arch-Vile in the very short moment after the monster has started its attack animation.

The bug will go unnoticed most of the time because the fire will spawn to a location hidden from the player, such as outside of the map. Though, other times, it’s completely apparent because it will spawn close to the player and inside his field of view.

A demonstration of this bug

As you can see in the video above, the first time, the player moved out of the Arch-Vile’s line of sight before it spawned the fire and it spawned at the wrong place. The second time, the player stayed long enough in the Arch-Vile’s line of sight for the fire to be readjusted to his position. This readjustment is done continuously while the Arch-Vile’s target moves but remains in its line of sight.

Arch-Vile fire spawned at the wrong location, but inside the player’s field of view.

To fix this bug in the original game’s executable, we have to open it in IDA Pro. Lucky for me, someone already documented the function offsets and made them available online as IDC/map files. Finding the offsets of these functions is a long processes that I gladly won’t have to go through before I fix this bug.

I can then open DOOM2.EXE in IDA and update my database with these files which will give a name to different offsets so I can match the functions with the source code.

DOOM2.EXE in IDA updated with the function names (A_VileTarget is shown)

I’ve activate the opcodes and the offsets from the Options/General menu to help me understand how I’ll have to change the code. The opcodes are helpful because this is what will allow me to find the function in an hex editor and change the code’s behavior. An opcode is a byte of machine code that will get executed on the CPU. A single line of assembly (the human-readable representation of machine code) is converted to one or many opcodes by an assembler.

I have to change the arguments that are passed to the P_SpawnMobj function. I first have to be aware of the Watcom compiler’s convention when passing arguments to functions. Wikipedia says “Up to 4 registers are assigned to arguments in the order eax, edx, ebx, ecx. Arguments are assigned to registers from left to right.”

The function expects “x”, “y”, “z” and a number that represents the thing to spawn (“MT_FIRE” is “4”). I will explain the five assembly lines that come after the call to A_FaceTarget from the IDA screenshot above. The first puts the player’s data structure into “edx”. It then assigns the function arguments: “ebx” is for the “z” (offset 0x14 in the player’s struct) and “edx” is for “x” (offset 0x0C). The number 4 is then put into “ecx” and “edx” gets copied into “eax” before P_SpawnMobj is called.

At first glance, it looks easy, right? It’s only one parameter that we have to change. We only have to change this last copy so we copy “y” into it instead of “z”. It’s should only be a one-line code change. Well, no…

“edx” was replaced with “x”, so it no longer contains the address of the player’s structure. Furthermore, notice that copying a variable from an offset takes three opcodes and copying to registers from another takes two. We don’t have enough space to patch the code.

Luckily, Watcom left some padding between our A_VileTarget function and the next function, so we can remove one byte from there so the executable remains aligned correctly. The rest of the function will be shifted positively by one byte, so I’m going to have to fix the two next “call” instructions, which jump to another part of the code using a relative offset. I’ll have to decrease the offset at which they jump by one.

Next, I have to know how to copy “y” correctly so it’s used by the function. I know “eax” is the target register and I’ll have to copy “y” from the player’s structure before the “edx” register is replaced. I know “y” is between “x” and “z”, so I know its offset. The code to assemble will be “mov eax, [edx + 10h]” which results in the opcodes “8B 42 10”. I have all the information I need to know to start patching.

I’ll save you the details of the manipulations that I did in my hex editor.

Here the summary of what I did:

  • Copied “y” into “eax” first.
  • Removed the instruction that copies “edx” into “eax”.
  • The remaining of the function was pushed by one byte, so I decreased the relative address of the two “call” instructions to P_SpawnMobj and A_Fire by one. Note that a “call” starts with opcode E8 and is followed by an address in little-endian format, so it’s the first byte of this address that’s the least significant part of the address.
  • Removed a padding byte after the “retn” instruction so the rest of the executable remains aligned.

Here’s a picture of the end result with the code patched:

The patched code as displayed in Radare2 (r2 -b 32 DOOM2.EXE)

I added the patch that fixes this bug into my patcher for Doom. It can change the original opcodes for the new ones inside of this function.

I hope you liked this article and that it can help you. Of course, this example was simple. We didn’t need to change the space of arrays allocated on the stack and other such things. An example that I like is when Microsoft manually patched their equation solver in Excel because they didn’t have the source code. A next step for me would be to learn how to do things like this.


I would like to thank Randy87 from Doomworld for his IDC/map files.

Alexandre-Xavier Labonté-Lamoureux

Written by

Software Engineering student at École de Technologie Supérieure

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade