Intro to Syscalls & Windows internals for malware development Pt.1

Amit Moshel
9 min readOct 2, 2023

--

Hello Everyone, over the years I’ve written many articles/summaries that for some reason I kept to myself and didn’t think to share them.
The summaries are from all over the spectrum of cyber security, from Windows privilege escalation & Active Directory, web application exploitation, HackTheBox machines writeups, all the way to heap exploitation, reverse engineering and Windows Internals. Once in a while I’ll publish more posts.

The Windows Operating System is divided to 2 modes: user-mode and kernel-mode and are described by using 4 rings as the following:

https://upload.wikimedia.org/wikipedia/commons/2/2f/Priv_rings.svg

User-Mode — resides on rings 1–3. Ring 3 is where applications are being executed from.

Kernel Mode — resides on ring 0. Kernel Mode is the most privileged and it is the area where the kernel operates. Most of the programs are not and should not operate directly on Ring 0.

Every program will somehow need to interact with the kernel in order to run properly for things such as memory allocation, File Creation, Memory permissions modificiation, etc. In order to safely do that, Windows have some kind of bridge that will safely transfer instructions to the kernel.
That bridge is consisted out of 3 steps:
Win32 API → Native API → Kernel

Win32 API is the main API that is being used as a bridge to the Native API and from there, to the kernel. The WIN32 API is constructed from DLLs (Dynamic Link Librarys — such as kernel32.dll) and each DLL includes functions inside them that are being called and invokes the Native API in order to transfer execution to kernel mode.

The Native API (“ntdll.dll”) is the closest user-mode functionality to the kernel and is responsible for making the transition between user-mode and kernel-mode. The transition from user-mode to kernel-mode is being performed through the “syscall” instruction that resides in each Native API fucntion found in NTDLL.dll:

https://www.youtube.com/watch?v=I_nJltUokE0&t=2208s

The “syscall” instruction needs to be specified with a special number (stored in the eax 32-bit register) that the kernel requires in order to execute the correct routine. This number is called System Service Number (SSN) and using that number, the kernel mode performs some mathematical operations and knows what kernel subroutine it needs to execute in order to fill the program’s requirement.

System Service Dispatch Table (SSDT) is a special table that resides in the kernel mode and is responsible to match the syscall number received from the Native API to the kernel subroutine that was required to fulfill the application’s request. SSDT matches the SSN to the appropriate kernel subroutine through the following mathematical operation:
KiServiceTableAddress = Address of the start of the SSDT table.
routineOffset = System Service Number that was passed from the Native API.

https://www.ired.team/miscellaneous-reversing-forensics/windows-kernel-internals/glimpse-into-ssdt-in-windows-x64-kernel

When a process is created, one of the first things that are being loaded and allocated into the virtual address space of the process is the ntdll.dll which is the DLL that includes all the Native API functions. During process initialization in an environment that includes EDR (Endpoint-Detection-Response), the EDR injects a code to some functions inside the ntdll.dll that redirects the flow of execution into an EDR’s DLL, which will allow it to intercept each Native API request, inspect it’s content and it’s context to the flow of execution, and occording to some rules or signatures, it will decide whether the program attempts to execute some kind of malicious action such as process injection. This defense mechanism is called API Hooking also known as EDR Hooking.

That’s how the flow of execution looks like with EDR Hooking:

https://redops.at/en/blog/direct-syscalls-vs-indirect-syscalls

From the image above, we can see how the flow of execution is being redirected when attempting to execute a Hooked Native API function called “NtAllocateMemory()”, which is a NativeAPI function that is used to allocate additional virtual memory to a process.

This is how a Native API hooked function looks like in a disassembler:

https://www.riskinsight-wavestone.com/en/2023/10/process-injection-using-ntsetinformationprocess/

The “jmp” instruction at the start of the function is the instruction that redirects the flow of execution to an EDR code section.

In an unhooked environment, there won’t be any unconditional “jmp” instruction.
The following image is an example of an unhooked Native API function:

One of the techniques malware developers use to avoid the EDR Hooks is called “Direct Syscalls”. The “Direct syscalls” is a technique that is used to bypass EDR Hooking through manually crafting the NativeAPI function that the program should go through, manually performing the “syscall” instruction and transferring the flow of execution to the kernel-mode without getting to hooked NativeAPI functions.

https://redops.at/en/blog/direct-syscalls-vs-indirect-syscalls

Since we already know how a NTDLL.dll function’s assembly instructions look like, we can craft the instructions by ourselves and find a way to resolve the SSN number of the operation we want the kernel to execute:
mov r10, rcx
mov eax, <SSN to reslove>
syscall
ret

The only problem with that is that the SSNs (System Service Numbers) are not the same between the windows versions:

https://j00ru.vexillium.org/syscalls/nt/64/

This can make the “direct syscalls” technique harder to implement if we can’t tell what SSN we need to use for our specific function.

This means that we need to find a way to execute function’s in a way that won’t be detected by AVs and won’t be.

One way we can get to the syscall number is by getting the base address of loaded “ntdll.dll” module using a WinAPI function called “GetProcAddress()” and specify a handle to “ntdll.dll” using the “GetModuleHandle()” function. The “GetProcAddress()” will return the base address of the loaded “ntdll.dll”. After that we can find the offset to the to the address of the required function jump to it, and resolve the syscall number. The only problem with that, is that probably the 2 functions GetProcAddress() and GetModuleHandle() are hooked functions and implementing this technique will make the EDR alert.

We need to remember that the NTDLL.dll is being hooked only during process initialization. At runtime, the EDR injects it’s code to the ntdl.dll that was loaded to the process. This means that in order to bypass it, we can read a clean unhooked NTDLL.dll from the disk and overwrite the .text section of the NTDLL.dll of our running process with the clean .text section of the NTDLL.dll from the disk but unfortunately, if we’ll try to implement this technique it would also get detected because to implement that, we’ll need to use an API functions that might be hooked such as “GetModuleHandle()”.

One of the tools that was created to dynamically resolve SSNs is called “syswhispers”.
The “SysWhispers” generates all the SSNs for all of the windows versions and in runtime, the program will dynamically choose the right syscall stubs depending on the OS version on the target computer. “syswhisper” extracts the OS Version at runtime.

Link to the “syswhispers” tools on GitHub:
https://github.com/jthuraisamy/SysWhispers

The second technique, we’re going to talk about is more sophisticated and is called “HellsGate”.

HellsGate” is a technique in which an attacker uses a data structure called “Process Environment Block” (PEB) to find the base address of the NTDLL.dll using the loaded module list information contained inside the PEB. The PEB is a data structure created during process initialization and it contains process related information such as:
— Environment variables
— Command line used for initalizing the process
— Working directory
— module (DLLs) list
— heap pointer
and more…

After getting the base address of “ntdll.dll” from the PEB, it parses the “Export Address Table” of the NTDLL.dll to get the address of the function we want to execute. Then, we navigate inside the address space where the desired Native API function resides and extract the syscall number. The extract syscall number is being used to implement a direct function call from the “HellsGate” code.

The following image describes how the PEB is being accessed to extract the base address of the NTDLL.dll:

https://blog.christophetd.fr/dll-unlinking/

Explanation:
The access to the PEB (Process Environment Block) is performed through another data structure called the TEB (Thread Environment Block) which is being initialized when a thread starts.

The TEB contains thread-related information such as:
— stack range (base & limit)
— “GetLastError()” value
TLS (Thread Local Storage)
— pointer address to the PEB which resides at gs:[0x60] on 64 bit architecture and fs:[0x30] on 32 bit architecture. “gs” and “fs” are the registers that are used to access the TEB. On 64-bit architecture the “gs” register is being used, and on 32-bit architecture the “fs” register is being used.

After getting a hold of the PEB, there is another pointer to a data structure inside the PEB called: “PPEB_LDR_DATA Ldr”, which is a pointer to a data structure that holds the loaded DLL information. Inside the “Ldr” structure, there is another data structure that of type “LIST_ENTRY” called “InMemoryOrderModuleList”. The “InMemoryOrderModuleList” is a data structure in a form of a doubly-linked list that allows iterating over loaded modules, it iterates over the loaded modules based on their memory order inside the process. Iterating over that module list will lead to the location that holds information about the loaded NTDLL.dll library.

After getting into the NTDLL.dll’s base address, we need to get to it’s Export Address Table (Described also as “Export Directory”), which contains information about all of the exported functions of the DLL:

After getting into the Export Address Table, we’ll run through the exported functions until we find the base address of the function we want to execute, we extract the SSN (System Service Number) from it, and use it in our own implementation of the NTDLL.dll function we want.

But there is one problem with the HellsGate technique, if we’re trying to extract the SSN value from a hooked function, the offset from the base function address to the SSN might be different between an unhooked function and a hooked function. To overcome that, a new technique was created called “HalosGate”.

SSNs are performed in an incremental order between each NativeAPI function. This means that the first function will probably have the SSN of 1, the second function might have the SSN of 2 etc… This means that if we can parse the SSN from the neighbor’s function, we can infer the required function’s SSN.

The direct syscall is a clever technique but unfortunately is getting detected by AVs and EDR. The main reason for it’s detection is because calling directly to the “syscall” instruction from the main program is being flagged as an IOC (Indicator Of Compromise) by AV vendors. This leads us to another similar techinque called “Indirect Syscalls”.

Indirect syscalls” is similar to direct syscalls but with one difference. Instead of directly calling the “syscall” instruction from our program, we resolve the address of the “syscall” instruction inside the NTDLL.dll and then, make the program jump to the address of the “syscall” instruction inside the NTDLL.dll and execute “syscall” from there. This will bypass the IOC (Indicator of Compromise) AV’s alerts on when executing “syscall” directly from our program.

The following image describes the “indirect syscall” implementation:

https://redops.at/en/blog/direct-syscalls-vs-indirect-syscalls

That will be it for this article. The article touched mainly on the theoretic side of the concept as it’s only an introduction.

The next article talks about the implementation of the “Indirect Syscall” technique in C++ and can be found here:
https://medium.com/@amitmoshel70/intro-to-syscalls-windows-internals-for-malware-development-pt-2-b8d88bb10eb9

--

--