Basics of Anti Reverse Engineering

Recently while making my reversing challenges I have discovered some tricks that I found interesting to make the reverse engineering process harder. The reason I decided to write this is mainly because after doing hours of research I was unable to find any articles that covered this topic for Linux well. This is the example code that I am going to be using.

First thing you will probably notice is what I have commented out but I will get to that later first lets look at the assembly of this main function to use as a reference down the road.

As you can see it looks just as expected and it is pretty clear what is going on but lets change that by uncommenting the line __asm__(“int $3”); using __asm__(); I am able to execute any assembly instruction I want so I went with int3 here.

As you can see the int3 instruction has appeared to remove the rest of the main function but when we load up the binary into GDB we are still able to see the missing instructions. This method forces the user to use GDB for obvious reasons.

This trick will work with other disassemblers like IDA too but it does not always work so the use case is really situational.

Instead of using int3 we could also do this.

It is important to remember when using assembly like this in you C code you have to use AT&T syntax if you are using GCC.

The next trick is very well know and basic so I am not gonna spend much time on that is use ptrace to detect if a debugger is attached. This is not a very good method to rely on because it can easily be skipped. When researching anti debugging for Linux this is pretty much the only answer you will get that works. This function can only be called once so choose where you call it wisely.

In the initial disassembly is it very easy to look at cmp dword [rbp — local_8h], 0xff and know it is comparing a variable to 255 but to make that not as obvious I can do.

Now after compiling we can see.

Which is much harder to tell right off of the bat what is going on then this.

Another thing if you want to make things harder to figure out what is going on you should do as many things as you can manually like for example.

As you can see here instead of using the strlen function I made a for loop that calculated the string length then using that I made another for loop that looks for the newline character and replaces it with a null byte. Doing simple things like this will make the job of the reverse engineer harder.

We are also able to execute code before the main function for example.

You can not find this function right off the bat with Radare you have to use IDA.

Even though using this method it can easily be found by using IDA the reverse engineer still have to identify that this function is executing before main. A practical use of this is calling ptrace here. Using GDB I set a break point on the very first instruction in main and ran it.

You also want to make sure that you always strip your binaries and doing so can be very easy to forget so I came up with a solution that prevents that and that is just do everything thing at once like so gcc level03.c -o level03 && strip level03 && ./level03 now you do not have to worry about it anymore.

The last tip I am gonna give is make sure you do things as complicated as you possibly can for example instead of having a handler function that calls all your other functions and returns to that function you should just daisy chain your functions. What I mean by that is have each function call the next function do not have them in an organized fashion. You should also add code that is used as filler like so.

The second for loop will only get executed if the length of the input is not what I want it to be so it really does not matter what I do with it after that. This example is still rather simple but if it was not just an example I would of went a lot farther with it then I did.