Load testing APIs with Go and Vegeta

Web APIs are a ubiquitous part of modern web or app development. We all have to deal with them and often have to build and maintain them, too. But if you operate one, how do you know it’ll stay up when the hordes of the internet send 10x or 1000x the usual traffic to it?

That’s where load testing comes in. The idea is you send a bunch of traffic to your website or API and work out how much it can receive without suffering degradation of service (or whatever your version of the fail whale is).

Depending on the results you can go buy yourself a beer and sit in the sun, or work out what changes you need to make to reach your targets.

How to do load testing

There are a lot of different tools to help with load testing. The way they typically work is you construct an HTTP request, like GET http://www.example.com/some/path and tell the tool to send 1000 requests per second to it for 60 seconds. Usually you’ll get some form of report — typically in plain-text or graphical form which you can interpret.

The tools that work this way are too numerous to list, but some popular ones are Apache JMeter, Apache Bench (ab), Siege, and cloud-based tools like Loader.io.

The problem with basic load testing

In complex systems often there is often some kind of caching involved when processing a HTTP request. For example perhaps you have an API which takes some data, performs an expensive (slow) operation on it, and then returns the result. More often than not you can store the result in a cache, and just return that if the same request is repeated.

This poses a problem for load testing because typically load testing tests the same URL and HTTP body content repeatedly. Effort that is cached either by your API service or other services it depends on will be served faster than effort that is not, and so you’ll see a bias toward faster response times. This renders load testing results largely useless because the test data are not anything like real-world traffic.

To avoid biases like this there are a few options — examples are:

  • Replay production traffic to simulate load
  • Generate test traffic that is more like the real-world

Depending on the sensitivity of your data, infrastructure, and type of work your API does, replaying production traffic may or may not be an option. So in this article I’ll focus on generating test traffic.

To learn how to generate and load test with test traffic we’re going to:

  1. Introduce Vegeta
  2. Install Vegeta and see some basic use cases
  3. Learn how to load test multiple targets
  4. Generate some test traffic for a fantasy web service
  5. Join our generator program with Vegeta

We’ll be writing a small program in Go, so if you want to follow along you should install Go from here if you haven’t already.

Introducing Vegeta

Vegeta is a command-line load testing tool (like ab and siege) written in the Go language. We’re going to use Vegeta because it has a very simple interface, and it supports multiple request targets and request bodies via a targets file or piped from STDIN.

We can make use of Vegeta’s ability to send traffic to multiple targets with multiple HTTP request bodies via a simple command-line interface to easily send a lot of varied, more ‘real world-like’ traffic to an API.

Installing Vegeta

Precompiled executables for Vegeta can be found here. Or, if you’re on a Mac and have homebrew installed, you can use the following command:

$ brew update && brew install vegeta

Basic use

The most basic use of Vegeta is something like the following:

$ echo "GET http://example.com/some/page" | vegeta attack -rate=10 -duration=30s | vegeta report

This will make an HTTP GET request to the URL given at a rate of 10 requests per second for 30 seconds, generate a binary report and then display it via the vegeta report command.

Testing multiple targets

Vegeta can also accept a list of targets, as a line-separated file or from STDIN.

An example (from the Vegata docs at https://github.com/tsenart/vegeta) goes like this:

In targets.txt you have:

POST http://goku:9090/things  
@/path/to/newthing.json
PATCH http://goku:9090/thing/71988591  
@/path/to/thing-71988591.json

At /path/to/newthing.json and /path/to/thing-71988591.json you have two different request bodies in JSON encoding.

The targets.txt file contains targets separated by double-newlines, followed by a path to a file containing a request body. vegeta attack will automatically load the body content from these files for us at runtime.

Then you can invoke Vegeta to make the requests to these two targets with:

$ vegeta attack -rate=10 -duration=30s -targets=targets.txt | vegeta report

or

$ cat targets.txt | vegeta attack -rate=10 -duration=30s | vegeta report

Generating test traffic

For the purposes of this article we’re going to generate API traffic for a fictional API, the Pokémon collectors club sign up API.

The API resides at https://pokemon-collectors-club.io/sign_up.json accepts a JSON payload via HTTP POST:

{
"name": "Addicted, New Zealand",
"email": "login@domain.com",
"password": "***************",
"favorite_pokemon_number": 25
}

Testing this API with identical request body data won’t work because the API validates that the email is valid and unique, the password has sufficient complexity, and the “favorite_pokemon_number” is between 1 and 721, so the first request will create a new user, but each subsequent request will return an error.

To generate test traffic we’re going to write a small Go program. Writing a command line program with options (etc) can be tedious, so to help we’ll use a Go package github.com/urfave/cli. And to generate some pretend details we’ll use github.com/icrowley/fake.

Here’s the annotated source code for the program, which I placed in a file named pokemon.go:

package main
import (  
"encoding/json"
"fmt"
"io/ioutil"
"math/rand"
"os"
"path/filepath"
"strconv"
"time"
"github.com/icrowley/fake"
"github.com/urfave/cli"
)
// Panic if there is an error
func check(err error) {
if err != nil {
panic(err)
}
}
func main() {  
var (
users int
)
  // The Go random number generator source is deterministic, so we need to seed
// it to avoid getting the same output each time
rand.Seed(time.Now().UTC().UnixNano())
  // Configure our command line app
app := cli.NewApp()
app.Name = "Pokemon User Data Generator"
app.Usage = "generate a stream of test data for vegeta. Type 'pokemon help' for details"
  // Add -users flag, which defaults to 5
app.Flags = []cli.Flag{
cli.IntFlag{
Name: "users",
Value: 5,
Usage: "Number of users to simulate",
Destination: &users,
},
}
  // Our app's main action
app.Action = func(c *cli.Context) error {
    // Combine verb and URL to a target for Vegeta
verb := c.Args().Get(0)
url := c.Args().Get(1)
target := fmt.Sprintf("%s %s", verb, url)
    if len(target) > 1 {
      for i := 1; i < users; i++ {
        // Generate request data
name := fake.FullName()
email := fake.EmailAddress()
// Password(atLeast, atMost int, allowUpper, allowNumeric, allowSpecial bool)
password := fake.Password(10, 32, true, true, true)
pokemonNumber := rand.Intn(720) + 1
        // Generate a map of the request body that we'll convert to JSON
bodyMap := map[string]string{
"name": name,
"email": email,
"password": password,
"favorite_pokemon_number": strconv.Itoa(pokemonNumber),
}
        // Convert the map to JSON
body, err := json.Marshal(bodyMap)
check(err)
        // Create a tmp directory to write our JSON files
err = os.MkdirAll("tmp", 0755)
check(err)
        // Use the user's name as the filename
filename := fmt.Sprintf("tmp/%s.json", name)
        // Write the JSON to the file
err = ioutil.WriteFile(filename, body, 0644)
check(err)
        // Get the absolute path to the file
filePath, err := filepath.Abs(filename)
check(err)
        // Print the attack target
fmt.Println(target)
        // Print '@' followed by the absolute path to our JSON file, followed by
// two newlines, which is the delimiter Vegeta uses
fmt.Printf("@%s\n\n", filePath)
}
} else {
// Return an error if we're missing the required command line arguments
return cli.NewExitError("You must specify the target in format 'VERB url'", 1)
}
return nil
}
  app.Run(os.Args)
}

Before we can run the program, we first need to install its dependencies:

$ go get

Then install it (this will create a binary called pokemon in $GOBIN/).

$ go install

Then run:

$ pokemon -users=10 POST https://pokemon-collectors-club.io/sign_up.json

We should get an output something like this:

POST https://pokemon-collectors-club.io/sign_up.json  
@/Users/me/code/pokemon/tmp/Joshua Henry.json
POST https://pokemon-collectors-club.io/sign_up.json  
@/Users/me/code/pokemon/tmp/Mr. Dr. Jimmy Garcia.json
POST https://pokemon-collectors-club.io/sign_up.json  
@/Users/me/code/pokemon/tmp/Jeremy Taylor.json
POST https://pokemon-collectors-club.io/sign_up.json  
@/Users/me/code/pokemon/tmp/Mr. Dr. Kenneth Bailey.json

Joining the dots

You might have worked out that since this is the same format that vegeta attack requires we can pipe this command directly into the vegeta attack command:

$ pokemon -users=10 POST https://pokemon-collectors-club.io/sign_up.json | \
vegeta attack -rate=10 -duration=30s | vegeta report

This will send our generated data for 10 users to the API at a rate of 10 per second for 30 seconds. We could easily vary the number of users we generate in order to get a higher or lower proportion of new vs returning users.

Next steps

In this article I’ve explained why basic load testing practises don’t always work well for APIs, what Vegeta is and how it works, and how to generate test data and send multiple different request bodies to an API.

I’ve kept the small Go program generic so you should be able to adapt the approaches I’ve taken for your own API load testing.

What other tips and tools do you know for load testing APIs? We’d love to hear — let us know in the comments below!



Originally published at thisdata.com on August 4, 2016.