TAMUCTF2019 Keygenme — Write-up

Hello everybody! This is how I solved one of the many reversing problems at TAMUCTF 2019. This one was Keygenme, a pretty standard kind of challenge but yet very fun to solve it, let’s go.

Let’s start by running the binary that was given to us and see what it says.

Basically, when we run the binary it asks for a product key and if we give an incorrect one it simply shuts itself down. Then I searched for the strings inside the binary, sometimes they help us understand what the program is doing.

By using rabin2 I was able to identify all the strings in the binary, all of them are very important so let’s take a look at them one by one


A pretty odd string that at the start don’t get our attention as the others, but it’s gonna be important in a moment.

\nPlease Enter a product key to continue:

That’s the string we just saw.


A string that everybody likes to see, but I don’t have a flag.txt? Do you?

Too bad the flag is only on the remote server!

string, even if we bypassed the checking functions using the disassembler locally we would not be able to retrieve our flag because it’s been hosted remotely.
So with this quick overview of the strings, we now know that we have to create our own keygen to beat this challenge.

Analyzing the binary with hopper disassembler there are 3 functions that get our attention

0x92a P enc
0x9da P verify_key
0xa46 P main

It’s pretty clear what each function is doing so I’m gonna skip that whole explanation and focus on the verify_key function first.

That part is at the beginning of the function verify_key and its job is to get our input, our “KEY”, and checks if its size is bigger than 0x9 in hexa aka 9 in decimal xd.

So, important information, we must supply a key that is longer than 9 digits, noted.

Right after this block, there’s a check to see if our input is larger than 0x40 bytes, since we will no be providing a key this long, I’m gonna skip it.

There are a few import things we must consider on this part of the function.

  1. The enc function is called by verify_key and not by main.
  2. Right after enc function is called our modified key is compared to [OIonU2_<__nK<KsK

Now it’s clear that we have to provide some input that after it passes the enc function it has to be equal to that strange string we saw early on.

Now we have to go through the enc function and figure what it is doing with our input and reverse it.

0000000000000960         mov        eax, dword [rbp+var_10]                     ; CODE XREF=enc+168
0000000000000963 movsxd rdx, eax
0000000000000966 mov rax, qword [rbp+var_28]
000000000000096a add rax, rdx
000000000000096d movzx eax, byte [rax]
0000000000000970 movsx eax, al
0000000000000973 lea edx, dword [rax+0xc]
0000000000000976 movzx eax, byte [rbp+var_11]
000000000000097a imul eax, edx
000000000000097d lea ecx, dword [rax+0x11]
0000000000000980 mov edx, 0xea0ea0eb
0000000000000985 mov eax, ecx
0000000000000987 imul edx
0000000000000989 lea eax, dword [rdx+rcx]
000000000000098c sar eax, 0x6
000000000000098f mov edx, eax
0000000000000991 mov eax, ecx
0000000000000993 sar eax, 0x1f
0000000000000996 sub edx, eax
0000000000000998 mov eax, edx
000000000000099a imul eax, eax, 0x46
000000000000099d sub ecx, eax
000000000000099f mov eax, ecx
00000000000009a1 lea ecx, dword [rax+0x30]
00000000000009a4 mov eax, dword [rbp+var_10]
00000000000009a7 movsxd rdx, eax
00000000000009aa mov rax, qword [rbp+var_8]
00000000000009ae add rax, rdx
00000000000009b1 mov edx, ecx
00000000000009b3 mov byte [rax], dl
00000000000009b5 mov eax, dword [rbp+var_10]
00000000000009b8 movsxd rdx, eax
00000000000009bb mov rax, qword [rbp+var_8]
00000000000009bf add rax, rdx
00000000000009c2 movzx eax, byte [rax]
00000000000009c5 mov byte [rbp+var_11], al
00000000000009c8 add dword [rbp+var_10], 0x1
00000000000009cc mov eax, dword [rbp+var_10] ; CODE XREF=enc+52
00000000000009cf cmp eax, dword [rbp+var_C]
00000000000009d2 jl loc_960

using this part and the hopper pseudo code:

int enc(int arg0) {
var_28 = arg0;
var_8 = malloc(0x40);
var_C = strlen(var_28);
var_11 = 0x48;
for (var_10 = 0x0; var_10 < var_C; var_10 = var_10 + 0x1) {
rcx = (var_11 & 0xff) * (sign_extend_64(*(int8_t *)(var_28 + sign_extend_64(var_10)) & 0xff) + 0xc) + 0x11;
*(int8_t *)(var_8 + sign_extend_64(var_10)) = (rcx - ((SAR(HIDWORD(rcx * 0xffffffffea0ea0eb) + rcx, 0x6)) - (SAR(rcx, 0x1f))) * 0x46) + 0x30;
var_11 = *(int8_t *)(var_8 + sign_extend_64(var_10)) & 0xff;
rax = var_8;
return rax;

I managed to write this whole process in python and I noticed that the next key’s value always depends on its previous neighbor. So to discover what key generates this [OIonU2_<__nK<KsK, I simply brute forced it XD.

By running this script I got


Let’s send to the server and get our FLAG!

Well…Maybe not 😢.

Let’s use gdb-PEDA to find out why we did not get our deserved flag.

I set up a breakpoint right before my input and that awful string are compared and I got this.

So the program adds this “\n” and I’m not sure why.
That thing made me wonder for a while why that’s there and how to bypass it.
Then I found the solution to be very simple (and a bit of luck maybe?). This “\n” is changing into “i” because it’s after a “B”, so I have to find a character that change “\n” into “K”.

And the “R” was there all along to save me.

Hope you guys enjoyed it!

by me