Why I started learning Reverse Engineering

Reverse engineering is not a term inherent to cybersecurity. In general, it simply means to de-construct something —software or hardware — for analysis and research. In the world of Information Security (often shortened to InfoSec), it usually means to take a compiled executable, disassemble it, and analyze the assembly code.

There are two phases of analysis, static and dynamic, and while I will go into more detail about these in another post, the idea is that sometimes you just examine the existing code for any useful information (IP addresses, hash lookups, domains, etc.) and other times you execute the code with tools like debuggers to get more insight on its purpose.

Overview

There are many applications of RE, like modifying video games (e.g. using CFF Explorer to allow your game executable to utilize more RAM), bypassing copyright protections, understanding how malware operates, and finding vulnerabilities in software.

Such tools for RE include disassemblers/decompilers (like Ida Pro, Ghidra, Binary Ninja, or Radare2), debuggers (GDB, Ollydbg, WinDbg), hex editors (010 Hex Editor, vim), scripting tools (Python), and virtual machines to create a safe “sandbox” to play in without the fear of damaging your native operating system. In short, there are a lot of things to become familiar with, and it can be overwhelming when you first start learning. But rest assured, over time you will get more used to these tools as you play around with them, ask others questions, and read articles by experts in the field.

My Research

Although I may have tricked you into believing I am one such expert, I can assure you I’m basically a beginner as well. I’m a Computer Science student with a work background in Linux system administration and software engineering, and while InfoSec has always been my main area of study, I only recently started to learn RE because of my research. About six months ago I joined a research team which focuses on security and virtual proctoring in the world of Computer Science education. Proctoring exams at a university level has always been a problem for test administrators, and with the global pandemic, virtual proctoring services like HonorLock and Respondus LockDownBrowser have grown exponentially in their use, leading to a new series of concerns. Questions arise of how secure these services are to use and if they are the most effective method of cheating prevention.

While most of the members in my group are working on developing technical solutions to protect against cheating, my advisor and I are focusing on how virtual proctoring services like HonorLock, ProctorU, or LockDownBrowser exploit the systems of the students utilizing them. How do these virtual proctoring services (VPS) use memory and the kernel? What permissions are they allowed? Are there ways to easily bypass the cheating restrictions they put in place?

Unfortunately, the literature on the topic is limited. And since most VPS are proprietary, one cannot simple contact the vendors to obtain the source code. Over the months, I’ve had to use less “reliable” resources like blogposts or Reddit threads as well as my past experiences as a student to gain a general understanding of how they work. And what I’ve gathered is that a lot of these software applications are(unsurprisingly) like malware. They’re glitchy, invasive, and leave a lot of a student’s computer vulnerable to attack.

Note that my experience with RE before I started was minimal. I’d taken courses on penetration testing and enterprise security, but these generally involved little to no use of the practice. I’d done CTFs before which always had some challenge involving RE but whenever I attempted them, I tended to get confused right at the start. My point is that I didn’t start my research as a tech-savy hacker. I started as a n00b.

So how did I start? First I set up a virtual machine (if you don’t know how to do this, I promise my next post will cover this topic!). Then I obtained a copy of the Respondus LockDownBrowser executable and, over the past few months, I have been analyzing its contents in Ghidra (NSA’s open source disassmbler) and debuggers like WinDbg and x64dbg. I started with LDB because executable programs are typically more analyzable than the browser plugins, especially since these plugins require knowledge of JavaScript (ew, gross).

LDB is a web browser software used by schools to put students’ computer in “lockdown” when taking an online quiz or exam. In theory, it prevents the user from opening new tabs or files outside of the exam or from using hotkeys like ‘alt+tab’ to move out of the browser or take screenshots. It also is not possible to run in a virtual machine.

Except… that’s not true. Don’t just take my word for it, a simple google search of “running Respondus Lockdownbrowser in a vm” will pull up a myriad of tutorials on how to bypass its virtualization detection. While it does involve a certain level of technical finesse, it’s still doable for the modern high school or college student. And if the average student can manipulate these VPS to do their bidding, what could a skilled attacker accomplish?

Legality

Now all this time you may have been wondering: “Isn’t reverse-engineering these proprietary proctoring services illegal?”

Well actually, no. While the world of hacking often times blurs the boundaries of legality, the United States Supreme Court has ruled that state trade secret laws such as those of VPS may not preclude “discovery by fair and honest means,” like reverse engineering (Kewanee Oil Co. v. Bicron Corp., 416 U.S. 470, 476 (1971)). The Supreme Court further confirmed the legitimacy of reverse engineering in the case of Bonito Boats, Inc. And while in the license agreement of these services you may agree to not “modify” the code, it is still possible to analyze them without modification. It is also possible to obtain a copy of an executable without agreeing to its Terms of Service, and it’s even easier to analyze the source code of the Chrome Plugins for HonorLock or ProctorU by installing the Chrome extension source viewer (link here).

Conclusion

All that being said, reverse engineering VPS software is not a task only achievable by elite hackers. Reverse engineering as a whole is a practice anyone interested can begin if they have time, determination, and a working computer. The research I do mainly involves a lot of puttering around in a vm while watching Youtube tutorials to learn about different hacker tools. It’s tedious and time-consuming, and of course there’s a lot to learn, but I mostly enjoy it. So much so, in fact, I hope to secure a RE-related job post graduation to continue improving my skills. My research advisor and peers in my school’s InfoSec Discord have been extraordinarily helpful in directing my path of learning, inspiring me to create this blog and pass on what I’ve learned to other beginners. I hope you all find it useful, educational, and perhaps even entertaining, since we all could use a break from the monotony that is analyzing assembly code.

And remember, you know you’re doing RE right when you’re frustrated as hell. Let this not dissuade you, but encourage you to keep trying if you find yourself overwhelmed and exasperated. There’s a lot of opportunities out there in this field, employment or otherwise, so in time you will be rewarded one way or another. Just don’t give up, be patient with yourself and your computer, never be afraid to ask questions, and have fun!