Finding a password for a protected file: The very basics

Bilal Khan
6 min readMay 20, 2016

--

We are going to find a password for a very simple file. If you are an advanced user, please go back and read something else. The file I will use in this project can be found on my github repo.

Let’s simply run the file and see if we can gather some information from our first run.

$ ./crackme3
Usage: ./crackme3 password

So the file tells us we have to pass the password as an argument. That’s a good start. Lets try a few random passwords now.

$ ./crackme3 heyyy
ko
$ ./crackme3 yooo
ko
$ ./crackme3 fdsf
ko

Using ltrace:

$ ltrace ./crackme3 password
__libc_start_main(0x40068c, 2, 0x7ffd29008948, 0x400710 <unfinished …>
strlen("password") = 8
puts("ko"ko
) = 3
+++ exited (status 1) +++

Lets jump into gdb and find what’s happening. Let’s list all the functions in our file.

$ gdb ./crackme3
(gdb) info functions
All defined functions:
Non-debugging symbols:
0x0000000000400498 _init
0x00000000004004d0 puts@plt
0x00000000004004e0 strlen@plt
0x00000000004004f0 __libc_start_main@plt
0x0000000000400500 fprintf@plt
0x0000000000400510 __gmon_start__@plt
0x0000000000400520 _start
0x0000000000400550 deregister_tm_clones
0x0000000000400580 register_tm_clones
0x00000000004005c0 __do_global_dtors_aux
0x00000000004005e0 frame_dummy
0x000000000040060d check_password
0x000000000040068c main
0x0000000000400710 __libc_csu_init
0x0000000000400780 __libc_csu_fini
0x0000000000400784 _fini

A function with the name check_password? I know, I know. But then this is just a basic tutorial. The functions strlen and check_password are important to us here.

(gdb) b *main
Breakpoint 1 at 0x40068c
(gdb) run psswrd
Starting program: /home/vagrant/password/new/dont_hate_the_hacker_hate_the_code/crackme3 psswrd
Breakpoint 1, 0x000000000040068c in main ()
(gdb) disass
Dump of assembler code for function main:
=> 0x000000000040068c <+0>: push %rbp
0x000000000040068d <+1>: mov %rsp,%rbp
0x0000000000400690 <+4>: sub $0x20,%rsp
0x0000000000400694 <+8>: mov %edi,-0x14(%rbp)
0x0000000000400697 <+11>: mov %rsi,-0x20(%rbp)
0x000000000040069b <+15>: cmpl $0x2,-0x14(%rbp)

0x000000000040069f <+19>: je 0x4006c8 <main+60>
0x00000000004006a1 <+21>: mov -0x20(%rbp),%rax
0x00000000004006a5 <+25>: mov (%rax),%rdx
0x00000000004006a8 <+28>: mov 0x2009a1(%rip),%rax # 0x601050 <stderr@@GLIBC_2.2.5>
0x00000000004006af <+35>: mov $0x400794,%esi
0x00000000004006b4 <+40>: mov %rax,%rdi
0x00000000004006b7 <+43>: mov $0x0,%eax
0x00000000004006bc <+48>: callq 0x400500 <fprintf@plt>
0x00000000004006c1 <+53>: mov $0x1,%eax
0x00000000004006c6 <+58>: jmp 0x400704 <main+120>
0x00000000004006c8 <+60>: mov -0x20(%rbp),%rax
0x00000000004006cc <+64>: add $0x8,%rax
0x00000000004006d0 <+68>: mov (%rax),%rax
0x00000000004006d3 <+71>: mov %rax,%rdi
0x00000000004006d6 <+74>: callq 0x40060d <check_password>
0x00000000004006db <+79>: mov %eax,-0x4(%rbp)
0x00000000004006de <+82>: cmpl $0x1,-0x4(%rbp)
0x00000000004006e2 <+86>: jne 0x4006f5 <main+105>
0x00000000004006e4 <+88>: mov $0x4007a8,%edi
0x00000000004006e9 <+93>: callq 0x4004d0 <puts@plt>
0x00000000004006ee <+98>: mov $0x0,%eax
0x00000000004006f3 <+103>: jmp 0x400704 <main+120>
0x00000000004006f5 <+105>: mov $0x4007b9,%edi
0x00000000004006fa <+110>: callq 0x4004d0 <puts@plt>
0x00000000004006ff <+115>: mov $0x1,%eax
0x0000000000400704 <+120>: leaveq
0x0000000000400705 <+121>: retq

We see that right upon entering the main function, first some memory is reserved, then $edi register content is moved to $rbp with an offset of -0x14, and it is compared with a constant i.e. 0x2. This is actually comparing the number of arguments that we passed to the program (the first “./crackme3” is also included in this total number, that is why the value is 2, and not 1). Then after that there is a ‘jump if equal’ (je) instruction. Note here that, if the jump is not made, we see that the program jumps to fprintf, and then exits with a return value of 1. Let’s put a breakpoint on check_password.

(gdb) b *check_password
Breakpoint 2 at 0x40060d
(gdb) continue
Continuing.
Breakpoint 2, 0x000000000040060d in check_password ()
(gdb) disass
Dump of assembler code for function check_password:
=> 0x000000000040060d <+0>: push %rbp
0x000000000040060e <+1>: mov %rsp,%rbp
0x0000000000400611 <+4>: sub $0x20,%rsp
0x0000000000400615 <+8>: mov %rdi,-0x18(%rbp)
0x0000000000400619 <+12>: mov -0x18(%rbp),%rax
0x000000000040061d <+16>: mov %rax,%rdi
0x0000000000400620 <+19>: callq 0x4004e0 <strlen@plt>
0x0000000000400625 <+24>: cmp $0x4,%rax

0x0000000000400629 <+28>: je 0x400632 <check_password+37>
0x000000000040062b <+30>: mov $0x0,%eax
0x0000000000400630 <+35>: jmp 0x40068a <check_password+125>
0x0000000000400632 <+37>: movl $0x4434241,-0x4(%rbp)
0x0000000000400639 <+44>: movb $0xff,-0x5(%rbp)
0x000000000040063d <+48>: movb $0x0,-0x6(%rbp)
0x0000000000400641 <+52>: jmp 0x40067f <check_password+114>
0x0000000000400643 <+54>: movzbl -0x6(%rbp),%edx
0x0000000000400647 <+58>: mov -0x18(%rbp),%rax
0x000000000040064b <+62>: add %rdx,%rax
0x000000000040064e <+65>: movzbl (%rax),%eax
0x0000000000400651 <+68>: movzbl %al,%eax
0x0000000000400654 <+71>: movzbl -0x6(%rbp),%edx
0x0000000000400658 <+75>: shl $0x3,%edx
0x000000000040065b <+78>: mov -0x4(%rbp),%esi
0x000000000040065e <+81>: mov %edx,%ecx
0x0000000000400660 <+83>: shr %cl,%esi
0x0000000000400662 <+85>: mov %esi,%ecx
0x0000000000400664 <+87>: movzbl -0x5(%rbp),%edx
0x0000000000400668 <+91>: and %ecx,%edx
0x000000000040066a <+93>: cmp %edx,%eax
0x000000000040066c <+95>: je 0x400675 <check_password+104>
0x000000000040066e <+97>: mov $0x0,%eax
0x0000000000400673 <+102>: jmp 0x40068a <check_password+125>
0x0000000000400675 <+104>: movzbl -0x6(%rbp),%eax
0x0000000000400679 <+108>: add $0x1,%eax
0x000000000040067c <+111>: mov %al,-0x6(%rbp)
0x000000000040067f <+114>: cmpb $0x3,-0x6(%rbp)
0x0000000000400683 <+118>: jbe 0x400643 <check_password+54>
0x0000000000400685 <+120>: mov $0x1,%eax
0x000000000040068a <+125>: leaveq
0x000000000040068b <+126>: retq
End of assembler dump.
(gdb)

On first look we can see these instructions:

callq 0x4004e0 <strlen@plt>: Calling strlen to calculate string length of the argument passed.

cmp $0x4,%rax: Comparing the output of strlen, with a constant 0x4. That most likely means the password is 4 characters long (if we see further, that becomes certain).

je 0x400632 <check_password+37>: jumps if equal. Most likely we need that jump, let’s find out whether this whole function should return 0 or 1 to work properly.

(gdb) nexti 10
0x0000000000400630 in check_password ()
(gdb) set $eax=1
(gdb) continue
Continuing.
Congratulations!
[Inferior 1 (process 2685) exited normally]
(gdb)

Basically what we did here was flip the eax value, it was going to return 0, we set it to 1 before leaving check_password, and it worked! We are one step closer to finding the password. Let’s go back to the check_password and investigate further.

movl $0x4434241,-0x4(%rbp): moving a constant 0x4434241 to -0x4(%rbp). The ‘l’ in movl means a long (32 bit integer or 64-bit floating point). We know 41, 42 and 43 correspond to A, B and C (See ASCII table), but the remaining 4 is not a printable character (4 is mentioned as ‘EOT’ which stands for End of Transmission).

movb $0xff,-0x5(%rbp): If we look further, ff is actually a mask, which extracts two least significant digits, when an AND logic is used with another value. For example, 0x6cdf AND 0xff = df, see how 6c gets discarded? Remember we need two hex digits to represent one ASCII character, therefore this is how we can extract 1 character at a time. Cool, isn’t it?

movb $0x0,-0x6(%rbp): If we look further in the code, we see two instructions near the end: “cmpb $0x3,-0x6(%rbp)”,jbe 0x400643”. jbe is jump if less than or equal to. This is actually a loop with a counter, until unless the counter is greater than 3, it will keep looping (of course we can have a jmp instruction inside to get out of this loop too).

shl $0x3,%edx, shr %cl,%esi: shl and shr are shift left and shift right instructions. The first operand to shl or shr tells how many bytes to shift in the second operand. Shifting left once, means multiplying by 2. Shifting right once, means dividing by 2. If you left shift a number, x, c number of times, then the resulting number will be equal to x*2^c; for right shift, it will be x/2^c. What this program does is that it takes the current value of the counter, left shifts it by 3 bits, which means (counter*2³). Lets calculate the corresponding values after shl for all the counter values.

0 -> 0*2³ = 0 decimal = 0x0
1 -> 1*2³ = 8 decimal = 0x8
2 -> 2*2³ = 16 decimal = 0x10
3 -> 3*2³ = 24 decimal = 0x18

Similarly, right shift corresponds to dividing, and most precisely the number of rights shifts corresponds to dividing by 2^x, where x is the number of right shifts. We see from our code that each corresponding right shifts takes the number of shifts to be made from the output of our left shift; and correspondingly right shifts the constant 0x4434241 by that many bits.

2⁰ = 1 decimal -> 0x4434241/1=0x4434241 → 0x4434241 AND 0xFF = 41

2⁸ = 256 decimal -> 0x4434241/0x100 = 0x44342 → 0x44342 AND 0xFF = 42

2¹⁶ = 65536 decimal -> 0x4434241/0x10000=0x443 → 0x443 AND 0xFF=43

2²⁴ = 16777216 decimal -> 0x4434241/0x1000000=0x4 → 0x4 AND 0xFF=4

So what we are seeing here is that from the constant 0x4434241, at every loop iteration, two least significant digits are taken (after shifting), and compared with the corresponding input password character.

Turning these numbers to ASCII, 41->‘A’, 42 -> ‘B’, 43 -> ‘C’, and 4 -> an unprintable character. We should manually input that last ASCII value and let bash pass it to the program.

$ ./crackme3 $'ABC\x4'
Congratulations!
$ ./crackme3 `printf "ABC\x4"`
Congratulations!

For more information on how to directly pass ASCII characters as arguments, follow this link.

This article is for educational purposes only.

--

--