Beyond Sanitizers: Guided fuzzing and security hardening

Kostya Serebryany from Google came to give a talk about his experience building dynamic testing tools. I’ll present an overview of his talk, which includes work from various papers. To learn more about each individual tool, you can read the corresponding paper for more detail. These papers are available on his Google webpage. He also gave a similar talk at CppCon in 2015, so a link to a video of that talk is available here.

At Google, there are various dynamic tools such as sanitizers (AddressSanitizer + LeakSanitizer, ThreadSanitizer, MemorySanitizer, UndefinedBehaviorSanitizer), fuzzing (libFuzzer), kernel testers (KASAN, KTSAN, Syzkaller), and hardening for control flow integrity (SafeStack). I’ll discuss a bit about these tools below.

AddressSanitizer is a dynamic testing tool for C++ based on compile-time instrumentation focused on bugs related to addressing memory.

Example of an ASan report for global buffer overflow

ThreadSanitizer is a tool to find bugs related to concurrency. MemorySanitizer is a tool to find bugs related to contents of memory. Undefined Behavior Sanitizer is a tool to find many other kinds of undefined behavior.

A report for undefined behavior sanitizer for int overflow.

These sanitizers have found bugs in many places. However, sanitizers are not enough because they are “best effort tools,” and they are only as good as the tests. They also do not prove correctness, so beyond sanitizers, Google uses fuzzing to improve test coverage and quality. They also protect from security-sensitive bugs in production using hardening.

Fuzzing automatically generates many test inputs to crash your code and improve test coverage. Some fuzzing strategies are the following:

• Grammar-based generation — Generate random inputs according to a grammar
• Blind mutations — Collect a corpus of representative inputs, apply random mutations to them
• Guided mutations — Build the target code with coverage instrumentation, run the target on the initial test corpus and collect coverage, run the target on random mutations of the elements of the corpus, and if new coverage is discovered add the mutation back to the corpus

LLVM libFuzzer is a lightweight in-process control-flow guided fuzzer. It provides status in-process and can be combined with any of the sanitizers. However, it is targeted at libraries/APIs not large applications.

Finally, the last tool is code hardening. Consider the following threat: Buffer-overflow/use-after-free overwrites a vptr or a function pointer by an attacker-controlled value. The solution is to use control flow integrity (CFI).

Consider another threat: Stack-buffer-overflow overwrites return address by an attacker-controlled value. The solution is to use SafeStack. SafeStack places local variables on a separately mmaped region. Stack-buffer-overflow can’t touch the return addresses, but VPTRs and function pointers can still be affected. However, you can combine SafeStack with CFI to solve this issue.

Finally, there is Scudo, a hardened malloc. It is based on the Sanitizer’s malloc. There is a 16 bit header with a checksum, delayed freelist, and randomness (not just ASLR). It results in only a 2–3% slowdown on malloc-intensive applications.

Kostya presented some interesting tools that they use at Google to find and mitigate bugs at Google. Here are the slides that he used. It was very interesting to learn more about what dynamic tools are used in practice and in what context.