Finding a Fuzzer: Peach Fuzzer vs. Sulley

You can also read this post on my blog!

A few months ago, I started looking at fuzzing tools. Finding a decent open source tool to use was more trouble than I expected, so I decided to write about it here.

tl;dr

I went with Sulley because it is open source and much more usable than Peach. However, it is less mature, bugs and all, so much so that I needed to fork the project to keep using it. The new project is called boofuzz.

Mutational or Generational

First, there are mutational and generational fuzzers. This post will focus on generational fuzzers. If you’re looking for a mutational fuzzer, I’d recommend looking into American Fuzzy Lop, though I haven’t personally used it yet.

Generational fuzzers target specific protocols or file formats, and are often created using a framework or tool. A nice framework can make the process much easier. A poor framework can make you regret ever thinking about making a fuzzer.

Prepackaged vs Custom

There are many prepackaged, for-sale protocol fuzzers out there. The main proprietors are Codenomicon Defensics (acquired by Synopsys) and Peach Fuzzer. These are likely the best option for a company or business, as it will take a lot of development time to match what is already for sale.

You might need to look into a custom fuzzer if:

1. You are interested in fuzzing and want to try it out yourself, or
2. You need to test a protocol or format for which no fuzzing test is available.

The rest of this article looks at frameworks for writing your own generational fuzzer.

Peach vs Sulley

A fair amount of reading led me to two contenders, Peach and Sulley. There’s a lot on the internet about SPIKE, but Sulley seemed to be SPIKE’s successor.

Open Source

At the time that paper was written, Peach was an open source framework. The initial business model followed by Deja vu Security seemed to be:

1. Make cool open source fuzzing framework.
2. Create proprietary protocol implementations to sell.

Somewhere around the end of 2014, however, they forked the tool to a closed source version 2.0.

Sustainability

When I first started looking for a tool, I stumbled upon a paper called Demystifying Fuzzers, 2009, by Michael Eddington, the creator of the Peach Fuzzing Framework. He gives some points for comparing fuzzing tools. The basic points have to do with features like monitoring, pre-test validation, code coverage, Virtual Machine control, etc. Some of the more interesting points of comparison are usability and sustainability, that is, how long is the project likely to last, and what kind of support can you expect?

From my perspective, paying money to use a tool that I don’t have source code access to presents a pretty big risk. What if the company goes defunct? What if they just decide not to implement a feature I want? While Peach Fuzzer may be great if you’re buying pre-built fuzzers, I was very apprehensive to use a closed source, proprietary framework.

Usability

I downloaded both Peach Fuzzer and Sulley to try them out.

In Peach, ever since the 2.0 release, all configuration and protocol definitions are in XML. There’s a popular saying that, “XML is like violence — if it doesn’t solve your problems, you are not using enough of it.” Suffice it to say that I didn’t enjoy slogging through XML, even for the basic example provided on the project’s web site. Maybe others have had a better experience.

Sulley is written as a set of Python libraries, and configuration and protocol definitions happen in Python. This seems to have worked out much better than Peach’s XML option. This basic example for FTP is very easy to understand, and the overhead for defining a protocol is minimal.

To make the comparison concrete, a basic Peach file fuzzer that did nothing but trivial random modifications takes 110 lines of boilerplate XML. This is as trivial an example as you could get. The FTP example I linked above intelligently fuzzes three separate messages in FTP, and takes 60 lines of code, 40 of which are real content instead of boilerplate. That’s only 20 lines of boilerplate. And it’s not XML.

Comparison

• Usability - Sulley is more usable, requiring less code and less XML.
• Sustainability - While Peach Fuzzer has corporate support, Sulley is open source. This means that you can, at the cost of development time, take control of Sulley as much as you want. You need to decide if paying for Peach Fuzzer is worth it, and if the risks of using a proprietary platform are acceptable in your case.
• Open Source - Sulley is open source; the latest Peach Fuzzer is not.
• Maturity - Peach Fuzzer is much more mature. Sulley’s latest release is buggy, to say the least.

Conclusion

Peach Fuzzer and Sulley are both excellent and notable fuzzing frameworks. In my estimation, Sulley is more usable than the latest Peach Fuzzer, and more sustainable since it is open source — but it comes with many more issues related to its immaturity.

Since I decided to go with Sulley, I needed to make countless bug fixes and patches for my own use case. The pace of fixes was outpacing the maintainers’ responses, and I was forced to fork it into a new tool, boofuzz.