Writing a Custom Shellcode Encoder
In this post, we will learn about shellcode encoders and explore how to write a custom encoder and decoder in plain assembly.
The Target Payload
Let’s get a payload first. We’ll use the reverse TCP shell for Linux x64. You can get the payload by issuing:
msfvenom -a x64 --platform linux -p linux/x64/shell_reverse_tcp -f hexI choose this one in particular because it has null-bytes, therefore, we could test if our custom encoding scheme removes the null bytes from it. Here you can see the disassembly of the msfvenom payload:
If you are interested in learning more about this payload the following articles might be of your interest:
The Encoder
Allow me to introduce the encoder source code.
How does this work? We basically go through the whole target payload from the data section and we transform each byte of it using the 4 bytes key. After the encoding process is done we proceed to write the result to standard output and exit. The point of writing to standard output is to give the user the ability to easily save the encoded payload to another file or process it using any other command tools, like in the following example:
The actual encoding process happens around lines 23–30 where we take each byte of the payload and load it into the al register, pass it through a series of conversions (xor, add, not, add then xor again) an finally we replace the original payload byte with the result:
mov al, byte [rsi+rcx]
xor al, keys.xor1
add al, keys.add1
not al
add al, keys.add2
xor al, keys.xor2
mov byte [rsi+rcx], alinc rcx
cmp rcx, payload.len
jne encode
We repeat the process until the whole payload is encoded and then we output the result to standard output.
The rsi register is used as a pointer to the payload and rcx as an index to access every byte. The rsi register is loaded using rip-relative addressing (a neat x64 feature) and the keys used in each part of the encoding process are defined as constants at the beginning of the file.
The Decoder
The decoder stub is pretty similar, we just take the encoded payload and run the process backward, then jump into the decoded payload for maximum pwneage.
The main difference here is that, as we are running shellcode this time, the code should be position-independent and it can’t have a .data section, therefore, the whole encoded payload must be placed into the .text section at the beginning to avoid having null-bytes on relative address references.
And here’s how you can test this decoder:
The encoder and decoder source code can be found on ExploitDB:
