Smoke Testing with Swagger

At RetailMeNot, we deploy changes to the microservices that power our API multiple times a day. Because of this, we need an efficient way to smoke test these deployments to make sure we haven’t broken something horribly.

In its most basic form, smoke testing takes static input and compares it to a known output or a subset of characteristics. This approach works for a while. However, nearly all systems deal with data that changes over time, which mean tests relying on fixed data fail eventually.

Take the http://petstore.swagger.io/v2/swagger.json API, for example. Suppose we want to test the endpoint defined by the swagger document:

"/pet/{petId}": {
"get": {
"parameters": [
{
"name": "petId",
"in": "path",
"description": "ID of pet to return",
"required": true,
"type": "integer",
"format": "int64"
}
],
"responses": {
"200": {
"description": "successful operation",
"schema": {
"$ref": "#/definitions/Pet"
}
}
}
}
}

This endpoint takes a petId as a parameter and returns an object with the “Pet” schema, if successful. Using this endpoint, we could write a smoke test that makes GET requests to “/pet/123” that verifies the deployment hasn’t done anything to break the endpoint!

That is until the same smoke test begins to “fail” when deploying unrelated changes because another user deleted pet “123”. Avoiding these and other related types of false positives lead us to look for a more robust manner to write our tests.

Our Solution

Since all of our services produce Swagger documentation, we knew we could safely rely on the exposed information about path, parameter, response type, etc. to build our tests. In many cases, this documentation is automatically generated and updated with every new version. It suddenly dawned on us that Swagger documentation provides all the information needed to start creating system level tests using dynamic inputs.

We’ve long been proponents of unit testing at RetailMeNot for its ability to prevent regressions. However, the unit testing model of “given static input X, we should get static output Y” didn’t hold up as well for smoke tests.

On another project, we had begun to incorporate a style of testing called generative testing to great effect.

With generative testing you define how to generate parameters at runtime and the properties that the output should satisfy. The testing library — QuickCheck, or its Scala implementation, ScalaCheck, which we use — is then able to generate random test data and test the system’s functionality against its defined properties.

Defining inputs in this way allows for a highly configurable and generalized solution to smoke testing. If you wanted, you could still define static sets as input, but using a small fixed set of possible inputs means that edge cases never get surfaced and must be added manually. When the tests themselves are dynamic, edge cases are tested without human intervention, resulting in less developer time required to write and maintain.

Combining this type of parameter generation with a swagger definition provides a simple and effective way of smoke testing. Using the example from above, we’ll compare seeding a smoke test of the pet endpoint with a static set of inputs, then compare its implementation to one that uses a dynamically generated set of inputs.

{
type: "StaticGenerator"
param: "petId"
values: [
"1",
"123",
"41"
]
}

Testing with this kind of input makes it easy to see what is being tested. However, the same things that make this method easy to understand also cause it to break when any of the response data changes. It’s also useless for discovering edge cases.

Here’s what a typical configuration for creating a dynamic set of inputs for the “petId” path parameter might look like:

{
type: "RequestGenerator"
param: "petId"
endpoint: "/pet/findByStatus:
generators = [ {
type: "StaticGenerator"
param: "status"
values: [
"available",
"pending",
"sold"
]
}]
mapping: [
"id"
]
}

We make a request to the RequestGenerator’s “pet/findByStatus” endpoint with the goal of creating a list of petIds that can be used later. A couple of things need to happen to make this possible.

We define our “dynamic” RequestGenerator to use a StaticGenerator. The key here is that the call to the StaticGenerator returns a potentially exhaustive set of inputs to the dynamic RequestGenerator. The mapping object specifies what part of the response to use. In our example, the parameter generation to the “findByStatus” endpoint is simple because it is a static list of values, but also the exhaustive set values accepted by the endpoint. Finally, the request is made and a mapping is applied to the output which determines the fields that should be extracted, in this case “id”.

But How Do I Create My Own Generators?

Every service has its own unique business logic, so in order to make this generative testing stuff work, you need a custom way to define parameters.

This is easily done by extending the base Generator class. Since Generators are initialized at runtime based on your own configuration file, you simply need to add it to your to the classpath.

All generators extend from a base Generator class that extracts the name of the parameter that values are being generated for. For example, in the “IntegerGenerator” above, we create a ScalaCheck Gen object that accepts parameters that define how potential values are generated, in this case integers between a min and max value specified by the typesafe config.

A few things happen after hitting the run button but before the tests actually start. First, the Swagger documentation is retrieved from the service which provides us with important information about what we will be testing such as the endpoints, parameters and expected output. Next, the generators are initialized and seeded. Once the generators are built the tests are ready to start. Each test uses a unique set of parameters as input, which are supplied by the generators. All results are validated by the configured acceptance criteria. Tests that do not match the acceptance criteria output a failure along with the base request and parameter map of the failed request to assist in reproducing the issue.

Wrapping Up

We now run these smoke tests automatically on every deployment as part of our continuous integration efforts which allows deployments to happen faster, easier and with more confidence. The source will soon be available at the RetailMeNot organization page on Github.