An Overview of Virtual Machines
For Reverse Engineering and beyond
If you are completely new to the world of reverse engineering, malware analysis, penetration testing, or InfoSec as a whole, your first step to learning should be setting up a virtual machine. A VM is a computer within your computer, an emulation of another operating system which provides the functionality of a physical machine without the actual hardware (e.g. allowing you to have access to a Windows machine even if you have a Mac laptop). It essentially gives you a sandbox to play in with little to no risk of damaging your native operating system.
And while it’s true that any of the tutorials founds on my blog or most any site out there can be followed on your native OS, it is good practice to have a VM to work in, especially when tinkering with things like malware or kernel modification which could seriously wreak havoc.
Note that VMs can take up a lot of disk space, so make sure you have plenty of storage freed before you try to set up a VM. Your computer will let you know if you need to free up space if at any point in the process there isn’t enough, but it’s still best to do so beforehand.
Also note that this is NOT a tutorial on how to set up a VM, since there are already hundreds all over the internet which explain in detail every step to take. This is instead an overview of virtualization’s importance in the world of RE, and some helpful tips for getting started once you have one set up.
1. Choosing a Virtualization Hypervisor
To set up a VM, you must first install a hypervisor like VMware Fusion or VirtualBox. This will be the library on your host machine which will contain all your guest machines. Think of it as the playground where your sandbox(es) will be, allowing you to have multiple VMs on your computer at once. Having more than one VM can be really helpful if you need access to different OS types or wish to emulate a network (e.g. a host machine and a server).
There are two types of hypervisors, Type1 being the kind that run directly on your physical hardware (think a kernel VM) and Type2 that run as an application. For RE purposes, I suggest using a Type2. The main Type2 hypervisors to choose from are ones from VMware (like VMware Workstation Player), Oracle’s VirtualBox, and Docker. These are all open source (so you shouldn’t have to pay money for them) and relatively easy to install and configure.
VMware apps are the most widely used hypervisors in my experience, but ultimately it comes down to a matter of preference. I mainly use VMware Fusion but also have VirtualBox and Docker from past endeavors that I keep around for good measure.
2. Choosing Your VM’s Operating System
Now that you have a virtualization library, you must choose an operating system! I currently have a Windows 10 VM and RemNux VM in my VMware library, as well as an Ubuntu 10 VM in my VirtualBox library (remnants of a horrid OS class I took a few semesters back. Oh, the horrors). The Windows 10 VM is what I mainly use for my RE research since it has all the necessary software I need (disassemblers, debuggers, hex editors, etc.).
If you want to get involved in Penetration Testing as well, I would highly suggest Linux’s Kali VM. I’ve used it in the past for a pentesting class and found it very user-friendly (and cool!), since it came with a lot of preexisting tools of the trade.
Note: in any sort of VM set-up tutorial, they’ll talk about obtaining a disc image (.ISO) file. Make sure you download the 64-bit version, and also realize these files are very large and may take a while to download depending on your internet speed.
3. Picking Which Software to Install
So you set up your hypervisor, you picked an operating system and downloaded the .ISO, you set it up in your virtualization library, and you got it to boot. Great job! This means you can finally start puttering around in your VM.
Depending on the OS you chose, there may already be a lot of useful tools installed which can help you achieve your goals. For RE in particular, I would suggest making sure you have the following software:
- A Disassembler/Decompiler: Ghidra, Binary Ninja, IDA Pro, Radare2, etc.
- A Debugger: Windbg, x64dbg (and x32dbg), Ollydbg, etc.
- A Hex editor: 010 Hex Editor, WinHex, vim, etc.
- A Portable Executable (PE) Analyzer: PE Explorer, CFF Explorer, PEview, etc.
- Network Analyzers: WireShark, nmap, traceroute, etc.
Some are command line tools which means just running a few commands in your terminal/command prompt to install. Others are nifty apps you will find yourself wasting hours in (I speak from experience).
4. Playing Around
Now that you have a hefty arsenal of tools, it’s time to start the fun stuff! Chances are if you’ve made it this far, you’re dedicated to the idea of learning reverse engineering, which means you’re willing to put in the time and effort.
All these tools can get confusing, so I suggest just picking one and playing around in it. Look up YouTube walkthroughs and blogposts about it. Open a simple program with the app you’re trying to get experience with, and try out some of its features. Join a Discord server or Reddit thread and ask questions. I started with Ghidra, and the first thing I realized when I opened up the executable was that I desperately needed to review the x86 assembly language. I was also extremely befuddled by the wide variety of buttons and configurations shown in the program, but with practice I became more comfortable navigating it.
(By the way, if you don’t know what assembly is… don’t worry, I’ll be writing a beginner’s guide later on. But do realize it is something you will have to learn if you really want to do RE stuff)
And remember, if you’re tired, confused, and frustrated, you’re on the right track! This takes a LOT of time and practice, so whenever you’re overwhelmed, it’s okay to take a little break and come back to it later. You can’t learn everything at once, and it’s okay to be a n00b and ask questions. How do you think the pros got where they are now?
As a whole, reverse engineering isn’t necessarily complex, it can just be tedious and confusing. But if you’re persistent and put in the time and effort, you’ll get the hang of it. I’ve included some additional resources below if you want further reading and assistance on the matter.
Additional Resources:
Dennis Yurichev, Reverse Engineering for Beginners, https://beginners.re/RE4B-EN.pdf
Michael Sikorski, Andrew Honig, Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software, (2012)
