Fuzzy Testing: Taming the Unpredictable in Software Testing

9 min readNov 3, 2023

In today’s digital landscape, vulnerabilities serve as gateways for attackers to exploit system weaknesses, posing a significant threat to network security.

These vulnerabilities can manifest in hardware, software, protocol implementations, or system security policies, allowing unauthorized access and compromising the integrity of a system.

The surge in vulnerability disclosures, as tracked by Common Vulnerabilities and Exposures (CVE), is nothing short of astonishing…

Their count began to explode back in 2017, from a previous number of 6.4k to 14.6k, but today, as of the writing of this article, it has surpassed a staggering 24.2k.

Open source software is not exempt from this trend, as a report by Synopsys revealed that 84% of 1,481 examined code bases contained at least one open source vulnerability!

What’s even more alarming is the exponential growth of cybercrime. According to Cybersecurity Ventures, the global cost of cybercrime is projected to reach $8 trillion in 2023, with estimates soaring to $10.5 trillion by 2025.

If it were measured as a country, cybercrime would be the world’s third-largest economy after the U.S. and China.

These numbers certainly sound scary, and nowadays, no company is immune to the threat, regardless of size. Every organization, big or small, operates with the risk of a potential breach, jeopardizing operations, brand reputation, and revenue pipelines.

As a prevention measure, vulnerability assessment appears as an indispensable component of software security.

However, manual identification of vulnerabilities is an arduous and time-consuming process, particularly in the face of increasingly complex software. Moreover, certain scenarios lack direct access to the software source code, rendering manual review nearly impossible.

The solution lies in intelligent tools capable of automatically detecting vulnerabilities. One such method is fuzzing, an efficient and effective approach to vulnerability discovery.

What is Fuzz Testing?
What would be a simple and practical use case?
How does a fuzzer generates input data?
What main types of Fuzzy Testing exist?
What are examples of Fuzz Testing tools?
What are the benefits and limitations of this test type?
Conclusion

What is Fuzz Testing?

Fuzz testing, also known as application fuzzing, serves as a valuable software testing technique enabling teams to uncover security vulnerabilities and bugs within the source code of software applications.

Distinguished from conventional software testing methods like, for example Static Application Security Testing (SAST), fuzzing adopts a unique approach.

It essentially bombards the code with random inputs, aiming to provoke crashes that unveil hidden flaws that might remain undetected otherwise. These code imperfections, including issues related to business logic, mark potential high-risk areas susceptible to security threats.

When a fault or vulnerability emerges, a fuzzer, a tool designed to pinpoint the potential causes of these crashes, steps in to target specific weaknesses within the source code.

Fuzzers excel in exposing vulnerabilities exploitable by attacks like SQL injection and cross-site scripting, tactics employed by malicious actors to compromise security, steal information, or disrupt systems.

What would be a simple and practical use case?

Suppose you have an input field on a web form where users are expected to enter their email addresses.

In classic testing, you would typically validate the input for correctness, ensuring it adheres to the expected format (e.g., contains “@” and a domain).

Here’s some classic test cases:

Enter a valid email address (e.g., “user@example.com”). The test passes because it meets the expected format.
Enter an invalid email address (e.g., “user@”). The test fails due to unexpected format.

In some cases, you might even add one extra test case to cover the empty state, and that’s it.

Now, let’s introduce fuzz testing to the mix.

Fuzzing involves injecting a wide range of unexpected and often malicious inputs to identify vulnerabilities or unexpected behavior.

In this case, fuzzing might include inputs like:

A very long email address.
An email address with special characters.
An email address with an incomplete domain (e.g., “user@”).
An email address with non-standard characters (e.g., “user@ex@mple.com”).
Non-standard encodings or encoding combinations (e.g., URL encoding, HTML entities).

During fuzz testing, the goal is to identify how the system handles these unexpected inputs. Fuzzing would reveal potential issues that classic testing might miss, such as:

The system’s inability to handle excessively long input, leading to potential buffer overflows.
Security vulnerabilities due to special characters causing injection attacks.
System crashes or unexpected behavior when encountering incomplete or malformed email addresses.
Encoding-related vulnerabilities that could lead to cross-site scripting (XSS) or other security flaws.

In this example, fuzz testing helps uncover vulnerabilities and weaknesses that classic testing might overlook, ultimately improving the overall robustness and security of the system. Fuzz testing is especially valuable when dealing with unanticipated or unexpected inputs that can arise in real-world scenarios.

And if you find this example useful, imagine combining the test cases of several input fields that accept different input data types…

How does a fuzzer generates input data?

A fuzzer aims to test various attack combinations across different data types, including:

Numbers: This encompasses signed and unsigned integers, floats, and other numerical data types.
Characters: Fuzzing involves manipulating character data, such as URLs, command-line inputs, and metadata like user-input text, including ID3 tags.
Pure binary sequences: Fuzzers also operate on raw binary data.

A commonly adopted fuzzing approach involves defining sets of “known-to-be-dangerous values” or fuzz vectors for each data type and then injecting or combining these values. For instance:

For integers, this might involve using zero, potentially negative values, or extremely large numbers.
When dealing with character data, fuzzers may introduce escaped or interpretable characters and instructions. For example, in the case of SQL requests, fuzzing may involve testing quotes, commands, and other sensitive characters.
In the realm of binary data, fuzzers frequently employ random binary sequences to uncover vulnerabilities and weaknesses.

What main types of Fuzzy Testing exist?

Fuzz testing encompasses several main types, each tailored to target different aspects of the software being tested, namely Application, Protocol and File Format Fuzzing.

Application fuzzing focuses on the inputs and outputs of the system under test. For desktop applications, this includes testing various aspects such as:

User interface (UI) elements, examining button sequences and text inputs.
Command-line options to ensure proper handling of different input parameters.
Import and export capabilities, particularly file format handling (see file format fuzzing below). For web applications, the attack vectors extend to URLs, forms, user-generated content, and remote procedure call (RPC) requests.

On the other hand, protocol fuzzing involves sending manipulated or forged packets to the tested application. In some cases, the fuzzer may act as a proxy, altering requests in real-time before replaying them.

This method aims to uncover vulnerabilities in network protocols and their implementations.

Finally, file format fuzzing is employed to assess how an application handles different file formats. It generates multiple malformed samples and sequentially opens them. When the application crashes, debug information is preserved for further analysis.

This file format fuzzing can target two main layers:

The parser layer (container layer): This assesses file format constraints, structure, conventions, field sizes, flags, and other aspects related to the format itself.
The codec/application layer: This delves into lower-level attacks, probing the program’s deeper internals.

This fuzzing type is somewhat less common, but it has gained traction in recent times. Some tools and examples in this category include:

A generic file format fuzzer, like Ilja van Sprundel’s “mangle.c,” which alters header data.
Zzuf, which can function as a fuzzed file generator.
Tools like Hachoir, which serve as generic parsers for developing file format fuzzers.

These types of fuzz testing play a crucial role in identifying vulnerabilities and strengthening the security of software systems.

What are examples of Fuzz Testing tools?

There are some free and open-source fuzzing tools provide essential resources for identifying and mitigating vulnerabilities in software applications:

Google OSS-Fuzz

Google’s OSS-Fuzz project, born from their extensive use of fuzzing in developing Chrome OS and Chrome browser, is an open-source initiative.
It leverages various fuzzing engines, including AFL++, libFuzzer, and Honggfuzz.
Compatible with languages like C, C++, Rust, Go, Python, Java/JVM, and potentially others.
Supports x86–64 and i386 builds, with strong community support.

FuzzDB

FuzzDB is an extensive library of attack payloads and injection techniques, designed to expose vulnerabilities in applications.
Categorized by platform type, potential issues, source exposure, and more.
Ideal for use with programmable fuzzing engines, combining known and unknown attack patterns.

Ffuf (Fuzz Faster U Fool)

Ffuf, a fuzzing engine written in Go, is a powerful, command-line-based tool.
Versatile for common fuzzing tasks, like testing application responses to unknown GET and POST requests.
Regularly updated with new features; supports a sponsorship model for immediate access to updates.

Google ClusterFuzz

Google’s ClusterFuzz, used to discover bugs in Chrome, integrates with the OSS-Fuzz project and can fuzz any program or application.
Notable for its ability to scale, even running on 100,000 virtual machines for large-scale testing.

go-fuzz

Highly regarded fuzzing platform for testing Go language packages, especially those parsing complex inputs.
Valuable for strengthening systems that parse inputs from potentially malicious sources, such as web-based applications.

Jazzer.js

A creation of Code Intelligence, it is a coverage-guided, in-process fuzzer tailored for the Node.js platform.
Building upon libFuzzer, it introduces instrumentation-powered mutation capabilities to the JavaScript ecosystem.
Ideal for enhancing the security of Node.js applications by systematically identifying vulnerabilities.

What are the benefits and limitations of this test type?

There are several advantages of using Fuzz Testing, namely:

Automation: Fuzz testing can be configured to run tests automatically with minimal human intervention, saving time and resources.
Systematic Approach: Fuzz testing’s random input eliminates human bias and uncovers vulnerabilities that might be missed by human testers.
Suitable for Closed Systems: In closed systems, where inner workings are obscure, fuzz testing is often the only viable option. For example, it’s essential in applications like artificial intelligence and deep learning where input-output relationships are complex and unclear.

But, there are also some limitations associated with it:

Misses Some Attack Types: Fuzz testing is less effective at identifying security threats that do not lead to program crashes, such as spyware, viruses, worms, Trojans, and keyloggers.
Complex Setup: Managing and monitoring fuzz tests requires coding expertise and effective bug handling.
Limited Scope: Fuzz testing alone provides bug detection but doesn’t offer a comprehensive evaluation of security, quality, and effectiveness. Additional testing methods like functional testing and beta testing are necessary in the software development process.

Conclusion

Software vulnerabilities pose a formidable threat to network security, with exponential growth in CVE reports and open source vulnerabilities. The surge in cybercrime further amplifies the urgency of robust security measures.

Vulnerability assessment is essential, but manual identification is challenging, especially in complex software. Fuzz testing, as an automated, efficient approach, offers a compelling solution.

It excels at uncovering hidden flaws and enhancing security, and encompasses various types, such as application, protocol, and file format fuzzing, bolstering software security.

As you saw, there are several free and open-source fuzzing tools that provide essential resources for identifying and mitigating vulnerabilities, offering automation, systematic testing, and compatibility with multiple languages and platforms.

It may miss certain attack types, require coding expertise for setup and monitoring, and provide only a partial assessment of software security, but, overall, fuzz testing is vital for securing software applications and mitigating risks in today’s evolving threat landscape.

It complements other testing methods, forming a holistic approach to modern software security.

Stay tuned for a possible upcoming article, where I’ll delve deeper into the practical use of a specific fuzz testing tool and provide a step-by-step guide on how to use it. 😃

I hope you enjoyed reading this article!

My name is João Coelho, and I am currently a QA Automation Engineer at Talkdesk. Lately, I have been writing articles regarding automation, QA and software engineering topics, that might not be known by the community.

If you want to follow my work, check my Linkedin and my author profile at Medium!

Furthermore, if you’re interested in further supporting me and my content creation efforts, you can do so by buying me a coffee! 😄👇

Your support goes a long way in helping me dedicate more time to researching and sharing valuable insights about automation, QA, and software engineering.