Writing a Custom Shellcode Encoder

Mar 19, 2019 · 3 min read

In this post, we will learn about shellcode encoders and explore how to write a custom encoder and decoder in plain assembly.

Photo by Kevin Horvat on Unsplash

The Target Payload

Let’s get a payload first. We’ll use the reverse TCP shell for Linux x64. You can get the payload by issuing:

msfvenom -a x64 --platform linux -p linux/x64/shell_reverse_tcp -f hex

I choose this one in particular because it has null-bytes, therefore, we could test if our custom encoding scheme removes the null bytes from it. Here you can see the disassembly of the msfvenom payload:

have you spotted the null-byte on line 10?

If you are interested in learning more about this payload the following articles might be of your interest:

Photo by Arian Darvishi on Unsplash

The Encoder

Allow me to introduce the encoder source code.

How does this work? We basically go through the whole target payload from the data section and we transform each byte of it using the 4 bytes key. After the encoding process is done we proceed to write the result to standard output and exit. The point of writing to standard output is to give the user the ability to easily save the encoded payload to another file or process it using any other command tools, like in the following example:

The actual encoding process happens around lines 23–30 where we take each byte of the payload and load it into the al register, pass it through a series of conversions (xor, add, not, add then xor again) an finally we replace the original payload byte with the result:

mov al, byte [rsi+rcx]
xor al, keys.xor1
add al, keys.add1
not al
add al, keys.add2
xor al, keys.xor2
mov byte [rsi+rcx], al
inc rcx
cmp rcx, payload.len
jne encode

We repeat the process until the whole payload is encoded and then we output the result to standard output.

The rsi register is used as a pointer to the payload and rcx as an index to access every byte. The rsi register is loaded using rip-relative addressing (a neat x64 feature) and the keys used in each part of the encoding process are defined as constants at the beginning of the file.

Photo by Benjamin Lotterer on Unsplash

The Decoder

The decoder stub is pretty similar, we just take the encoded payload and run the process backward, then jump into the decoded payload for maximum pwneage.

The main difference here is that, as we are running shellcode this time, the code should be position-independent and it can’t have a .data section, therefore, the whole encoded payload must be placed into the .text section at the beginning to avoid having null-bytes on relative address references.

And here’s how you can test this decoder:

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification
Student ID: SLAE64–1326
Source code can be found here and here


Hacking/Infosec writeups and articles


Written by


Twitter: @syscall59 | medium.syscall59.com | syscall59@protonmail.com



Hacking/Infosec writeups and articles

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade