Mr. Robot Disassembled: eps3.5_kill-process.inc

Ryan Kazanciyan
6 min readNov 16, 2017

--

Hello friend. I’m Ryan Kazanciyan, Technical Consultant for Mr. Robot. I’ve been working with Kor Adana — Writer, Producer, and mastermind behind the ARG — and the rest of the Mr. Robot team since the second half of Season 2. Throughout Season 3, I’ll be writing about the hacks depicted in the show, how they came together, and their basis in reality.

Spoiler Alert! This post discusses events from Season 3 Episode 6.

Throughout the first half of this season, Elliot and Mr. Robot have pursued their own competing agendas, with the fate of the Dark Army’s “Stage 2” against E Corp hanging in the balance. “eps3.5_kill-process.inc” brings this conflict to a head, with Elliot losing his ability to retain control as he races to stop the attack.

To recap how we got here: last week, Elliot discovered that the Dark Army tried to load their malicious firmware onto the UPS at the NY recovery building. His patch stopped them, but shortly afterwards, they pivoted to compromise a member of E Corp’s Code Signing Architecture team. Elliot connects the dots and realizes that the Dark Army will go after the HSMs to steal E Corp’s code-signing certificates. Once he runs into Angela, he realizes he’s too late to stop them.

An HSM compromise is a major security incident — but the damage isn’t necessarily permanent. E Corp can go through the process of revoking the stolen keys and certificates, ensuring that they are no longer trusted by systems that rely on them. This is a massive undertaking, and Elliot no longer has the access nor time to set it in motion. He needs to delay the attack — if not halt it outright — as soon as possible.

Round One

Once online within the recovery building, Elliot SSH’s to a host named “K6F7241” (192.251.68.242). This is a *nix-based system used to managing various building facilities, including the network-connected UPS in the server room. It’s the same system that the Dark Army backdoored at the end of last season.

Elliot needs to confirm what happened after the HSM attack, and take whatever steps he can to temporarily undo the damage. We see him execute a long series of shell commands (and get a nice clear shot of the screen in the process).

File integrity verification 101

Let’s break down what each of these commands do:

gpg --verify update.bin.asc update.bin

Elliot uses Gnu Privacy Guard (GPG) to check the digital signature for firmware update files in the “~/ups_update” directory. The results show a valid signature belonging to the “E Corp Power Systems Hardware Development Team”. This indicates that the HSM attack worked, enabling the Dark Army to sign their malware in order to load it onto the UPS.

shred -uzn3 update*

He securely deletes the malicious firmware update files (from the staging host — remember, he’s not logged on to the UPS itself). The options “uzn3” instruct shred to zero out the file contents three times before deleting the file records.

wget -q https://192.251.68.232/files/ups_640_patch.zip

This downloads the same patch that we saw Elliot deploy in episode 302 using Tanium. He had the foresight to keep the patch files staged on another accessible web server, just in case they were needed again. (Having a photographic memory to recall IPs and URLs also helps!)

The ZIP contains three files:

  • apc_hw05_aos_640_patch.bin — patched, clean firmware
  • apc_hw05_aos_640_patch.bin.asc — digital signature for the firmware
  • hashes.asc — SHA256 hashes for the firmware binary and signature file

How can Elliot trust that the Dark Army hasn’t tampered with these files? Check out the next two commands:

gpg --verify hashes.asc

sha256sum --check hashes.asc

The “gpg” command confirms the integrity of the hash list: it’s signed by Elliot himself. That means he can trust the file hashes in “hashes.asc”, which he subsequently checks against the other patch files using the “sha256sum” command.

scp * upsadmin@192.251.68.229/upsfw

Almost there. Elliot uses Secure Copy Protocol (SCP) to transfer the patched firmware to the UPS. The final step is to log on to the UPS itself and run the command upsfwupdate. A few keystrokes short of doing so, Elliot loses control to Mr. Robot…

Round Two

Elliot returns to the building after an unexpected cab ride, and gets back on a computer. He immediately connects to the UPS device (192.251.68.229) via SSH. I based its command-line interface on the management operating system used by APC, a popular UPS manufacturer. It’s briefly shown in episode 302 as well.

The logon banner displays the uptime and loaded firmware version. Based on this information, he realizes that he wasn’t able to finish installing the patch — the UPS would have rebooted. Time to start all over again.

UPS logon banner

Once connected back to .242, he sees that Mr. Robot deleted the clean firmware files. He starts to re-download the patch ZIP with wget, only to lose control once more.

Downloading the firmware patch yet again

Round Three

Patch downloaded, hashes look good. Everything checks out. Elliot SCP’s the files to the UPS, connects to run the firmware update command, and loses control moments before finishing the job…

One last try…

Round Four

Elliot awakens to see a markedly different screen, showing the tail end of a series of commands that Mr. Robot executed on the UPS device:

ups> firewall -f fw.fwl

ups> firewall -r

ups> exit

These commands configured the UPS host-based firewall to deny all inbound connections from the entire subnet. Connection closed — and Elliot can’t SSH back in. They’re both locked out.

Locked out by the UPS host-based firewall

After a failed attempt to negotiate via notepad.exe, Elliot realizes that triggering the fire suppression system in the battery room might be the last remaining option…

Stage 2 is complete

This episode’s hacks were much less complex than last week’s HSM heist. However, the actual execution of these scenes — with Elliot glitching out as he regains and loses control — posed its own set of challenges. I originally wrote out the complete series of steps that Elliot was trying to perform, mindful of the need to include enough breakpoints where Mr. Robot would take over and knock him back. Kor and I then worked through the progression of each phase, making refinements to keep up the momentum.

“eps3.5_kill-process.inc” ends on a somber note as we see the Dark Army execute their true plan for stage 2. On a personal level, it marks a rewarding conclusion to one of the technical story arcs that Kor and I have painstakingly crafted over the course of two seasons.

When we first started working on the UPS firmware hack for episode 212 (way back in early 2016), I already had started thinking about what the next steps might be. How would E Corp defend itself? How would the Dark Army respond? It was a promising foundation for future attacks.

As this season took shape, I saw the Dark Army’s efforts to install their firmware as a parallel to Elliot’s own story. Once digitally signed, malicious code looks like authentic and trustworthy software — unless you closely monitor its execution over time. Likewise, Elliot initially believes he can be a force for good by taking a job at E Corp — unaware that he remains “backdoored” by Mr. Robot.

--

--