Fuzz Testing With Microsoft’s RESTler

An overview on how to use RESTler against QRI REST API

Introduction ℹ️

In this post, I’ll explore how to test QRI API with Microsoft RESTler tool. I’ll explain in later sections what is QRI, RESTler and how the process works, but for now, the image below outlines the details on the workflow this tutorial will cover:

What is RESTler ❓

RESTler is a brand new tool from Microsoft to run fuzz tests in a given REST API. It was released DAYS ago (November 16th, 2020), so why not give it a try?

The tool requires the API to be specified using OpenAPI, formerly known as Swagger, so that it can derive a grammar that describes requests structures. Later, this grammar can be used to form valid HTTP requests that can be sent to the API and, by monitoring responses, find bugs and effectively testing the service. After executing the fuzz test command, RESTler will generate a lot of logs, separating requests that produced errors in ‘buckets’ for inspection.

If you are looking for deeper details about RESTler, please refer to Microsoft’s post about it, and its associated article here.

What is QRI ❔

QRI is the an open source software that I personally picked to test its API. It seems to be an important project: with almost 3k commits and 1k stars on Github:

“It’s a global dataset version control system (GDVCS) built on the distributed web.”

This means that everyone can run nodes that makes up the P2P network that holds the dataset. Also means that every time people change a given dataset, previous versions are not lost, just like in any version control system.

Besides, being relevant and somewhat popular, QRI has its web API specified in a OpenAPI file, just like RESTler requires (but not quite, as we’ll see later on). Also, it has an Docker image, which will allow us to run its API fairly easily.

Hands on 🔧

In this section, let’s get to it: install the softwares, and use them together.

Installing RESTler (dependencies) 📦

RESTler, by the time of writing, requires:

• Python 3.8.2 🐍;
• .NET SDK 3.1 💻;

People usually install Python through their system package manager, but chances are (as in my case) the package manager current Python version is not 3.8.2. Because of that, you might need to install Python 3.8.2 from source code. The process is quite simple: after downloading and extracting the archive, all you need in your system is Make. Then, open the README file and it will describe what to do:

Instruction on how to install python from source code

Make sure to have libssl-dev module installed and you are good to go.

Currently, installing .NET SDK version 3.1 is extremely simple and the process is described in details on Microsoft’s website.

With dependencies installed, let’s proceed to install the actual software.

Installing the RESTler itself 💾

This process is almost disposable, since installation is basically:

• Cloning the repository from GitHub;
• Move inside the repository folder;
• Creating a folder to place the binaries:

mkdir bins/

• Running a python module:

python ./build-restler.py --dest_dir ./bins

And that’s it! bins directory should be filled with binary files to run RESTler. Next, let’s setup QRI.

Installing the QRI 📡

Since QRI has a Docker image, running its API is extremely simple:

1. Clone the repository from GitHub;
2. Move inside the repository folder;
3. Run:

docker build . -t qri

4. Run:

docker run qri

You should see an output like:

And now you have the API running on port 2503. Pay attention to the Docker container IP address, it won’t be localhost. In my case, the QRI container is on 172.17.0.2. Remember to use your container IP instead. With both softwares ready, let the fun begin!

Generating a Grammar for the API 📖

To be able to generate inputs to fuzz test the API, RESTler will need to know the structure of the input the API takes. For that, we will feed the OpenAPI specification file, mentioned in the beginning of this post, to RESTler and it will take care of generating a Grammar for us. After properly installing RESTler in a previous section, it will have generated the file bins/restler/Restler. That’s the binary we will be using to run Restler.

But before generating the Grammar, a caveat: OpenAPI specs can be written in JSON or YAML formats. RESTler, at the time of writing, only accepts the JSON format and QRI specifies its API in YAML. Hence we will need to convert the specification from YAML to JSON. Luckily, all that is needed is to paste the content of the YAML file (located at qri/api/open_api_3.yaml) in this OpenAPI editor tool, click in ‘File’ dropdown and then in ‘Convert and save as JSON’. Place the file inside RESTler repository for ease of use.

Now, all left is to run the binary specifying ‘compile’ word and the path to the OpenAPI specification file. In my case, that translates to:

bins/restler/Restler compile --api_spec open_api_3.json

The ‘compile’ word instructs RESTler to compile the specification into a grammar. We will soon use the ‘fuzz’ word to instruct it to start fuzz testing the API. After running the command shown above, a folder ‘Compile’ should have been created at the repository root folder. It contains a python file called ‘grammar.py’ and a few other configurations files. We will use it soon.

Fuzz testing 🎆

Now that RESTler knows how to generate inputs for tests, let’s start running them! Make sure to have QRI running as instructed in a previous section.

Now, we will use the same RESTler executable again. This time instead of ‘compile’ we will specify ‘fuzz’ word. Run:

bins/restler/Restler fuzz --grammar_file Compile/grammar.py --dictionary_file Compile/dict.json --settings Compile/engine_settings.json --time_budget 0.05 --target_ip 172.17.0.2 --target_port 2503 --no_ssl

I’ll go through a few flags specified in this command:

1. grammar_file, dictionary_file, settings: those are files created in the previous ‘compile’ step, when generating the grammar. This will instruct RESTler how to run the fuzz tests.
2. time_budget: This is the time fuzz testing will last in hours. Here, we specify 0.05, hence, 3 minutes. We don’t want it to take too long.
3. target_ip, target_port: specifies the IP address and port of the process processing the generated requests, i.e., the API address.
4. no_ssl: QRI API is not running under SSL, it’s plain HTTP, hence we should disable SSL, otherwise an error will occur.

Currently, the generated grammar contains a bug! Hence, the process we are launching to do the fuzz testing will not be successful unless Microsoft has fixed the bug. Let’s check it out.

Grammar Bug 🐛

After running the command described in the previous section, an error is reported:

RESTler error report

After inspecting ‘EngineStdOut.txt’ file under the generated ‘Fuzz’ directory, here is what we see:

Python Grammar error

So, let’s see what lies on line 2010 in the generated ‘grammar.py’ file under ‘Compile’ directory:

Grammar code bug

We can see why RESTler couldn’t import this grammar: there is an incomplete string at the end of line 2010 (it doesn’t have a closing ‘). Also, between every closing and opening curly brackets in the following lines, there’s a misplaced string that will also cause errors. And at line 2018, also another not closed string. But this code intention is quite clear: it’s a list of python dictionaries. So we can manually fix this as:

Corrected python grammar

And now we have valid python code! Running again:

bins/restler/Restler fuzz --grammar_file Compile/grammar.py --dictionary_file Compile/dict.json --settings Compile/engine_settings.json --time_budget 0.05 --target_ip 172.17.0.2 --target_port 2503 --no_ssl

And switching to QRI console should yield:

You can see the requests from RESTler being processed by QRI!

After the three minutes of execution, you should see an output like:

Fuzz output

Even though the execution didn’t exercise every API endpoint (only 4 out of 43 because we allowed it to run for just a small amount of time) it successfully found bugs! Lets dig deeper into that.

API bugs found with RESTler 🔎

After running the binary with the ‘fuzz’ task, it should have generated a ‘Fuzz’ folder. Navigate to:

Fuzz/RestlerResults/experiment<id>/bug_buckets

where <id> will be some auto generated number. Listing this directory we will see a lot of files describing problems found:

Bugs found reported in files

Each of these files will have information about the request made and also about the response. We can see that the server responded with a lot of 500 status code. This is HTTP for internal server error, which mean that the requests unveiled error not caught by the programmers.

Taking a deeper look into PayloadBodyChecker_500_2.txt describes the request made as:

POST /save HTTP/1.1
Accept: application/json
Host:
Content-Type: application/json
{“commit”:{“path”:”fuzzstring”}}

But when we look at the OpenAPI specification file, the ‘/save’ endpoint we see that it requires a body with an object with a lot more fields than just “commit”, and that field itself has a lot more fields than just “path”!

So what’s happening here? RESTler generated this input because the missing fields were not marked as required in the specification. But while they were not marked as required in the API specification, the actual endpoint code was expecting them to be present! Hence, an internal server error was returned. So we could easily spot a discrepancy between the specification and implementation!

That is proof of how useful RESTler can be!

Conclusions

RESTler seems to be an extremely useful fuzz testing tool. We could easily spot problems in a large and important codebase. By the way, we didn’t even look too deep into the problems reported by RESTler! Not to mentioned we only let it run for three minutes, which is close to nothing for fuzz testing. It also didn’t have the chance to test a myriad of endpoints. ✅

Since it’s a brand new tool, we had a chance to spot a bug in the process. Luckily, it was a simple problem, so I was able to fix by myself. It also could support YAML format for the API specification, but currently it only supports JSON. ❌

I will definitely consider using RESTler in future services I might build.