If you know what’s a hook, you know they aren’t user friendly and may be used for many different (good or bad) things…
So what’s a hook?
In simple terms a hook is a deviation from the standard flow of a program, usually induced by an external party.
If that sounds like malware to you, well that’s one of the kinds of software that make extensive use of hooking…
If you want to deal with the coding/memory layout details I recommend checking this.
But not all the hooks are bad, have you ever heard of “binary hot-patching”, that’s one of the enterprise names for “We are hooking our own software to fix/improve something” same reason for which “hot-patching” is a temporary fix for most pieces of software which are still under maintenance.
For old rusty versions which are in-use but not maintained, well if you can figure out the issue and write a hook for it you’ll be saved and in case you do not know Microsoft does this for a huge amount of old software so they can still run on newer versions of Windows and it is called “Application Compatibility Database” aka the “Shim Database” (you can see most of the APIs have the sdb prefix or Shim Data Base).
There are some crafty uses of this Windows feature ranging from regular application monitoring to more advanced forensic/security style analysis (great in depth series on the topic here by Alex Ionescu)
But in general the top users of hooks in normal operating systems are:
Anti-Virus
Software Monitoring tools
Virtualization extensions (very nice write-up of how Citrix deals with this)
But why?
Windows has documented ways to register system hooks that will be called by the OS when required, and as you have just read that is partly why these vendors tend to use handmade hooks instead of the properly documented APIs from Microsoft.
If you want to hook a crucial system call, let’s say “NtOpenFile” how much overhead will you incur by running the OS path plus your call? This may not be acceptable performance-wise and for most complex cases (AVs) you’ll need to make a lot of extra work like calling to your core process in charge of the actual analysis of the event in question.
What can go wrong?
Well many, many things…
In the simplest of the scenarios you’ll crash the hooked application.
In case of two or more pieces of software hooking some of the same functions (let’s say you run +1 AV/Security Software from different vendors)… You’ll (usually) also crash but with a more complex scenario left behind for debugging.
You may also not crash, but breaking the normal flow of execution can lead to many other complex and obscure issues.
Let’s say an AV hooks NtOpenFile, makes a driver call to its analytics driver, this in turn will invoke the User Space App that will actually carry out the analysis.
The chain is long and potentially complex, and at any of those steps you could hang or worse.
Think about what will happen if in any of those code paths you (or some code you are calling) ends up calling back to your hook?
A multi-process deadlock that most probably only Microsoft itself will be able to debug is a highly potential (and very common) outcome.
Later I will write some articles [1]more focused on the disassembly and explaining some in-market pieces of software that could be affected by this, we will be looking at their hooks and which functions are the ones more prone to be hooked.
But this is all for the intro into the topic.