xorpd assembly riddle 0x05

xorpd has some riddle-like pieces of assembly code here. In this post, I’ll analyze this one.

I present to you the sixth riddle, which is made of just two simple instructions:

sub      rax,5
cmp rax,4

First instruction subtracts 5 from rax and then it compares its value against 4. Now, what’s interesting about this? We need to have a talk about flags!

well.. not this kind of flag, actually

According to the Intel manuals, both instructions modify CF, OF, SF, ZF, AF, and PF flags depending on the result obtained. But what are these flags anyway?

The CF (or Carry Flag) is set to 1 if a carry or borrow happened as a result of the last operation (i.e when subtracting 4-5 => CF = 1 since 5 is bigger than 4).

The OF(or Overflow Flag) is set to 1 if an overflow occurred.

The SF (or Sign Flag) mimics the highest bit of the result. Since negative values start with a ‘1’ this flag is set to 1 when the result is a negative value.

The ZF (or Zero Flag) indicates if the last result was zero.

The AF (or Adjust Flag) is set to 1 if during an “add” operation there is a carry from the lowest four bits to the upper four bits, or a borrow occurs in the same way but during a subtraction.

The PF (or Parity Flag) is set to 1 if the result of the operation has an even number of bits set to 1 i.e : for 111 the parity flag would be 0 since it has 3 bits set and 101 would have a parity flag set to 1 since it has 2 bits set to 1.

Now, in the context of our riddle these are the only flags we care about:

; cf: carry flag (for sub/cmp, it indicates a borrow)                       ; this will be set to 1 when rax = 5,6,7 or 8                     
; zf: zero flag (set if sub/cmp produces a zero value) ; this will be set to 1 when rax = 9
; sf: sign flag (set if sub/cmp produced a negative result) ; this will be set to 1 when rax is lower than 9

Let’s try some cases and see how the flags are modified:

--------------------------------------------------------------------
Case 0:
mov rax,0
sub rax,5
cmp rax,4
flags: 0x0000000000000282 [cf:0, zf:0, of:0, sf:1, pf:0, af:0, df:0]--------------------------------------------------------------------
Case 1:
mov rax,1
sub rax,5
cmp rax,4
flags: 0x0000000000000282 [cf:0, zf:0, of:0, sf:1, pf:0, af:0, df:0]--------------------------------------------------------------------
Case 2:
mov rax,5
sub rax,5
cmp rax,4
flags: 0x0000000000000297 [cf:1, zf:0, of:0, sf:1, pf:1, af:1, df:0]--------------------------------------------------------------------
Case 3:
mov rax,6
sub rax,5
cmp rax,4
flags: 0x0000000000000293 [cf:1, zf:0, of:0, sf:1, pf:0, af:1, df:0]--------------------------------------------------------------------
Case 4:
mov rax,8
sub rax,5
cmp rax,4
flags: 0x0000000000000297 [cf:1, zf:0, of:0, sf:1, pf:1, af:1, df:0]--------------------------------------------------------------------
Case 5:
mov rax,9
sub rax,5
cmp rax,4
flags: 0x0000000000000246 [cf:0, zf:1, of:0, sf:0, pf:1, af:0, df:0]--------------------------------------------------------------------
Case 6:
mov rax,10
sub rax,5
cmp rax,4
flags: 0x0000000000000202 [cf:0, zf:0, of:0, sf:0, pf:0, af:0, df:0]--------------------------------------------------------------------
Case 7:
mov rax,-2
sub rax,5
cmp rax,4
flags: 0x0000000000000286 [cf:0, zf:0, of:0, sf:1, pf:1, af:0, df:0]

Displaying the results into a single table makes it easier to detect a pattern:

    Nº  hex     cf  zf  sf
------------------------------ lower numbers
-1 0xFF 0 0 1
00 0x00 0 0 1
01 0x01 0 0 1
02 0x02 0 0 1
03 0x03 0 0 1
04 0x04 0 0 1
------------------------------ greater or equal to 5
05 0x05 1 0 1
06 0x06 1 0 1
07 0x07 1 0 1
08 0x08 1 0 1
------------------------------ lower than 9
09 0x09 0 1 0
10 0x0A 0 0 0

From this table, we can clearly see there are two breaks on values 5 and 9 that define a range in which the carry flag is set to 1. Any value outside that range sets CF to 0. This setup let us with the ability to check if any given value on rax is between 5 and 8 with just one conditional instruction:

  sub rax,5
cmp rax,4
jb is_inside_the_range
is_outside:
... ; will be executed if rax's value is out of range
is_inside_the_range:
... ; will be executed if rax's value is in range

This is equivalent to the following high-level code:

if(a >= 5 && a <= 8){
doSomething();
} else {
doSomethingElse();
}

that’s it! hope you enjoy this one, see you around!


syscall59

Shellcode for the masses

Syscall59 — by Alan Vivona

Written by

Twitter: @syscall59 | medium.syscall59.com | syscall59@protonmail.com

syscall59

syscall59

Shellcode for the masses

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade