Decoding the Craft: Essential Skills for Aspiring Reverse Engineers

Introduction

Oftentimes I am asked how I got into reverse engineering and the steps one should take to do the same. I wanted to take some time to create a resource that will serve as guidance for entering the world of software reverse engineering. At the heart of cybersecurity, reverse engineering is an art form that demystifies software and hardware to understand their essence and behavior. Whether it’s for improving security, uncovering vulnerabilities, or simply satisfying a deep-seated curiosity about how complex systems operate, reverse engineering is a critical skill in the tech arena.

In this blog we will navigate through the required foundational knowledge, delve into the various specialties within the field, and explore the methodologies and mindsets that pave the way to mastery. Drawing from my own experiences in the trenches of binary analysis and exploit development, I aim to arm you with the insights and resources needed to embark on your own journey into the realm of reverse engineering.

My Experience

Before we continue, I believe it is important to outline the professional milestones that have shaped my expertise in the field. My career began in the Army National Guard as a Signal Officer in 2013, equipped with a Security+ certification and a security clearance. In 2016 I entered the IT realm with a help desk position, marking my first step into the industry.

In 2017, I advanced to IT operations support, focusing on firewall administration, while transitioning to the Army Reserves. The following year, I pivoted towards cybersecurity, securing a role as a vulnerability analyst in patch management.

My commitment to deepening my cybersecurity acumen led me to achieve the OSCP certification in 2018, which opened doors to my first penetration testing job in 2019. By 2020, I had set my sights on exploit development and reverse engineering, attaining the OSCE and GREM certifications, and soon after, a position as a vulnerability researcher with a focus on avionics systems.

I completed several advanced offensive security exams (OSWE, OSEP, OSED) and the SANS SEC760 course between 2021 and 2022. These efforts culminated in my current role at Blackpoint Cyber as a malware reverse engineer and passing the OSEE certification in 2023. I also ended a 10+ year Army reservist career as a Cyber Officer.

This progression through various roles and continuous education has been pivotal to my development as a reverse engineering professional.

Foundational Knowledge

Here, I outline the core competencies that I consider essential for anyone aspiring to excel in this field.

Reverse engineers should be proficient in programming concepts. Programming forms a critical pillar in the foundation of reverse engineering, requiring a firm grasp of core principles in a chosen language. This proficiency is essential for understanding software construction, automating tasks, developing analytical tools, and dissecting code during the reverse engineering process.

A reverse engineer must have a robust understanding of operating system internals, whether it’s Windows or Linux. This includes grasping the nuances of user mode versus kernel mode, the intricacies of virtual memory, and the structure of Windows registry settings. One should be adept at reading event logs and tracing API calls from user mode down to the kernel mode. It’s also crucial to comprehend how an operating system executes a program and manages processes and process threads.

Networking knowledge is another pillar of reverse engineering. Familiarity with the network stack is fundamental, as is the ability to read and capture network packets. This understanding allows one to navigate through the layers of network communication with precision.

The anatomy of an executable file is also part of this foundational knowledge. A reverse engineer should be able to dissect the header information of an executable and understand the purpose of various sections like .text, .data, and .rdata. Recognizing the permissions associated with these sections-read, write, and execute-is equally important.

Command line proficiency is non-negotiable. Whether it’s using Bash in Linux or PowerShell in Windows, a reverse engineer should navigate these environments with ease. Familiarity with default operating system command line tools is expected, as these are often the first line of interaction with the system’s deeper layers.

Research skills are often overlooked but are vital. The ability to efficiently find answers and gather information through open-source intelligence can save invaluable time and resources. This skill is as much about knowing where to look as it is about what to look for.

Lastly, an inquisitive attitude is the thread that ties all these skills together. It’s the relentless pursuit of understanding ‘how’ and ‘why’ things work the way they do. We’ll delve deeper into this mindset later, but it’s worth noting that curiosity is the engine that drives all discovery in reverse engineering.

Reverse Engineering Specialties

The field of reverse engineering is diverse, offering various specialization paths that, while distinct, share a common foundation of skills. Here we explore two primary roles that a reverse engineer might pursue, each with its unique responsibilities and contributions to the cybersecurity landscape.

Malware Reverse Engineer

A Malware Reverse Engineer is akin to a digital forensic expert, dissecting and analyzing malicious software to understand its mechanisms and intentions. This role involves breaking down malware samples, which can range from compiled executables to scripts laced with malicious intent. The insights gleaned are crucial-they inform customers or security teams about the nature of the threat and its potential impact. Creating detections from discovered artifacts is a key duty, as is the ability to analyze malware at scale through automation. A particularly challenging yet vital aspect of this role may include decrypting ransomware to mitigate its damaging effects.

Vulnerability Researcher

On the other side of the spectrum is the Vulnerability Researcher, whose primary objective is to identify and explore vulnerabilities within both commercial and open-source software. This role requires a proactive approach to security, using reverse engineering to uncover zero-day vulnerabilities and develop corresponding proof-of-concept exploits. These researchers play a critical role in cybersecurity by reporting vulnerabilities to vendors, facilitating the creation of patches, and enhancing overall software integrity. Additionally, they may craft exploits for known vulnerabilities, contributing to a deeper understanding of potential threats. The end goal varies-some may seek monetary rewards through bug bounty programs, while others might operate within government parameters in an offensive operation.

Methodology and Mindset

The bedrock of my success in reverse engineering is not just rooted in technical skill, but also in the cultivation of a disciplined methodology and the right mindset. Here’s how these two elements intertwine to form the backbone of effective reverse engineering practices.

Methodology

A methodical approach is critical. It begins with thorough research into the target system or software. Understanding the work that has already been done can provide a springboard for further investigation and prevent redundant efforts. Configuring tools and environments specifically tailored to the task at hand can greatly enhance efficiency and reduce the tedium of repetitive tasks.

Documentation is another pillar of sound methodology. Maintaining detailed records of targets and findings not only aids in the current project but also serves as a valuable reference for future endeavors. Knowing when to pivot is equally important. The ability to recognize when a particular avenue or approach is yielding diminishing returns can save precious time and resources.

Assumptions are inevitable, but they must be validated. Experience can guide intuition, but in the thick of reverse engineering, unchecked assumptions can lead to missteps.

Mindset

The mindset of a reverse engineer is characterized by an insatiable curiosity. It’s about continuously questioning the how and why behind the systems we engage with. This curiosity drives the relentless pursuit of knowledge, pushing the boundaries of what is known and understood.

Keeping an open mind is vital. The technological landscape is ever-changing, and openness to new ideas and techniques is essential for growth and adaptation. Humility complements this mindset; it’s a recognition that there is always more to learn, and that each challenge, whether overcome or not, is an opportunity to expand one’s expertise.

Together, a robust methodology and a resilient mindset form the dual core of a successful reverse engineer’s approach to problem-solving. It’s a blend of strategic planning and a philosophical outlook that values continuous learning and adaptability.

Resources

You’ve reached the section that many of you may consider the most practical and actionable part of this guide. Below, I’ve compiled a selection of resources that have not only been instrumental in my own development but are also well-regarded within the reverse engineering community. While this compilation is not exhaustive, it serves as a robust starting point for both novices and those at an intermediate level, offering tools and knowledge to advance your journey in reverse engineering.

Programming

Learn to Code — for Free | Codecademy HackerRank — Online Coding Tests and Technical Interviews C Programming Language, 2nd Edition

Operating Systems

Windows Internals, Part 1: System architecture, processes, threads, memory management, and more, 7th Edition | Microsoft Press Store The Linux Command Line, 2nd Edition: A Complete Introduction

Books

Mastering Malware Analysis (re-and-more.com) Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software

Challenges

Challenges.re — Reverse Engineering challenges

Blogs

Cybersecurity Blog (x0rb3l.github.io) Home (connormcgarr.github.io) Hasherezade’s projects Exploit Reversing — A blog about reverse engineering

Training and Exams

FOR610: Reverse Engineering Malware Training | Malware Tools & Techniques | SANS Institute PEN-200: Penetration Testing with Kali Linux | OffSec EXP-301: Windows User Mode Exploit Development | OffSec Practical Malware Analysis & Triage | TCM Security, Inc. (tcm-sec.com) SEC760: Advanced Exploit Development for Pen Testers | SANS Institute

YouTube

LaurieWired — YouTube Low Level Learning — YouTube Anuj Soni — YouTube John Hammond — YouTube Off By One Security — YouTube

Conclusion

To sum up, becoming a skilled reverse engineer is about building a strong base of technical know-how and pairing it with the right approach to problem-solving. This post has given you the basics, the areas to focus on, and a set of tools and resources to get started or to get better. Remember, it’s about persistence, curiosity, and being ready to tackle tough challenges.

Keep learning, stay curious, and use the resources shared here to sharpen your skills. Your role in reverse engineering is crucial, whether it’s analyzing harmful software or finding weak spots in systems to make them more secure. Take what you’ve learned here and apply it to make a difference in the tech world.

Originally published at https://reverencecyber.com on November 6, 2023.