WinAPI, and “Cheap” Malware Analysis Using AI — Part 1

Yiftah Perl
4 min readDec 20, 2022

--

Understanding the Windows API is a key concept in the journey of understanding the Windows Operating system.
In order to understand how programs run on Windows, one must get a good understanding of Win internals concepts such as memory, processes, threads, and more.
At the backbone of all of these functionalities, stands the Windows API.

In this series of articles, we will look at the core concepts of the WinAPI,
shed light on the Import Address Table (don’t worry, we’ll get to that),
and how all of that can help us perform “cheap” malware analysis.

What is the WinAPI?

According to the WinAPI Wikipedia page :

The Windows API, informally WinAPI, is Microsoft’s core set of application programming interfaces (APIs) available in the Microsoft Windows operating systems. The name Windows API collectively refers to several different platform implementations that are often referred to by their own names (for example, Win32 API); see the versions section. Almost all Windows programs interact with the Windows API. On the Windows NT line of operating systems, a small number (such as programs started early in the Windows startup process) use the Native API.[1]

In regular human language — the WinAPI is an application program interface that lets programmers interact with the basic functionalities of the OS.
Basically, it’s a set of functions exposed by Windows that allows the software to interact with the operating system.
This system utilizes a chain of function calls from various origins (DLLs), that also benefits memory and privilege management in Windows.

Due to the massive amount of information and documentation on this subject, I will provide practical examples that serve the purpose of this paper.

Hello, Kernel!

Your “Hello World” program, written in Python (or any other language), can’t acquire resources directly from the Windows kernel.
In order to do that, a chain of function calls is made, crossing the User Land by triggering a syscall.
This call can access privileged resources that reside in the kernel itself.
Only then, your program can access the kernel function that triggers the execution of the desired action.

For example — here is the journey of the OpenProcess() function :

https://alice.climent-pommeret.red/posts/a-syscall-journey-in-the-windows-kernel/

The Malicious Connection

So now that we got a basic understanding of the Windows API and its purpose, it’s time to look at the connection to real-world use cases (or malware).

But first — let’s define Malware :

Malware (a portmanteau for malicious software)[1] is any software intentionally designed to cause disruption to a computer, server, client, or computer network, leak private information, gain unauthorized access to information or systems, deprive access to information, or which unknowingly interferes with the user’s computer security and privacy.

The thing about malware that many don’t understand; is that many elements of regular software can be found in a malicious one, and vice-versa.

Let’s take the average game developer as an example. In order to get input from the player, he will have to use certain API methods that listen to keystrokes made by the player.
On the other hand, as a threat actor developing keylogger malware, I will find myself using the same method for malicious activity.

This function call will make a similar path on its way to the kernel, and will eventually get access to the low-level keyboard input events made by the victim/player.

Getting familiar with the API

The WinAPI set of functions is enormous, and many pages of documentation cover its functionality.

Luckily, we don’t have to memorize each and every function within this system — we can use resources that make our life easier :

  • MSDN — the official Microsoft documentary, covering all the functions, their syntax, and functionality.
    Although it’s aimed for programmers, analysts can get value from the description of the functions, and understand the syntax better.
  • MalAPI — created by security researcher mrd0x, the MalAPI library categorizes the various API functions to different stages of the cyberattack and provides a brief overview of the function.
    A very good resource to determine the role of the function and connecting the dots while analyzing a malicious binary.

Coming up next

The first part of the series focused mainly on the background and theoretical knowledge required to understand WinAPI.

In the next part, we’ll have a deeper look inside the executable file, specifically at the Import Address Table.
Using all this knowledge, we would then have the ability to use a tool that leverage AI to perform malware analysis.

I hope you enjoyed this reading, stay tuned for part 2!

--

--