Using go doc for REST APIs

Docs and tests are both communication

Very early on while we were building the API for Capsule, we needed to both test and document the API for client developers.

Tools like Swagger were a great way to document the API, but made it possible for the docs to get out of date. It was possible to integrate Swagger in a way that guaranteed the docs’ correctness (e.g. generate our API from the Swagger schema), but we had written enough of API that we were reluctant to do a major overhaul just for that.

At the same time we wanted to add tests that exercised more of the API end-to-end than the unit tests we currently had. Postman provided a nice way to test and share API requests and responses, but it wasn’t trivial to add it to our automated testing/continuous deployment system.

Finally, while each system would have worked fine independently, we had a feeling that these were actually the same problem, and could be solved quicker and simpler together. So we thought we’d try something a little different.

Testing by example

Examples in Go are functions that are expected to write to stdout. They are executed as part of testing, and their output is compared to an “expected output” comment left by the developer at the bottom of the function. When running go test, if the output and comment match, the test passes. If the output and comment are different, the test fails. And, assuming the example passed, the function and its output show up nicely formatted when running go doc.

As a meta-example, this code:

func ExampleReverse() {
    fmt.Println(stringutil.Reverse("hello"))
    // Output: olleh
}

generates these docs:

Two birds with one stone: the same code acting as both docs and tests

and this test output:

$ go test -v
=== RUN TestReverse
--- PASS: TestReverse (0.00s)
=== RUN: ExampleReverse
--- PASS: ExampleReverse (0.00s)
PASS
ok      github.com/golang/example/stringutil    0.009s

If we could use examples to test and document the API, then this seemed like the perfect solution:

• We wouldn’t need to rewrite any API code
• We didn’t have to integrate 3rd-party libraries, just the standard Go tools
• Our docs would always be up-to-date, because they were also tests
• We wouldn’t have to introduce a separate “API schema” language
• Our docs would be comprehensive, because even tests that were meant to check error conditions would show up by default
• Our tests would be easy to write: they’re just comments

How exactly?

For each request/expected response pair that we wanted to document/test, we created a new example function that makes the request to an httptest.Server and prints the response. The expected output comment is then just the expected HTTP response (we actually printed the HTTP request as well, for clarity). The key was to write the example functions in such a way as to show up legibly for developers unfamiliar Go or go doc.

An example like:

func ExampleDrugEndpoints_Get() {
    DrugEndpoints{}.Get() // Makes and prints the test HTTP request and response.
    // Output:
    // # Request
    // GET /v1/drugs/12–3456–789 HTTP/1.1
    // Capsule-Version: 1.2.3
    //
    // # Response
    // HTTP/1.1 200 OK
    // Access-Control-Allow-Origin: https://www.capsulerx.com
    // Content-Type: application/vnd.api+json
    //
    // {
    //   “data”: {
    //     “id”: “12–3456–789”,
    //     “type”: “drug”,
    //     “attributes”: {
    //       “brandName”: “Wolfsbane Potion”,
    //       “genericName”: “Aconite”
    //     }
    //   },
    //   “included”: []
    // }
}

generates these docs:

We made the examples methods of XxxEndpoint types to nicely group the request/responses in the generated go docs

It was pretty exciting once we got to this point, because we no longer felt like documentation or end-to-end testing was a major pain point. From then on, whenever we found a bug in the API, we could fix and document the behavior in one swoop. Also, this laid the foundation for tools that could, for example, automatically notify client developers when changes to the API were made, or prevent backwards-incompatible API changes altogether.

Quickly, the number of example tests we wrote exploded. In and of itself, this was a good indicator that the system was working, and it was usable. But, it also became tedious to manually edit each example comment when an API changed, especially as API responses became larger. Inspired by visual web app testing tools like PhantomCSS, we decided to automate most of the comment tweaking we encountered.

Visual testing using git

We noticed that the examples failed go test most often because we intentionally added, removed, or renamed fields in the API response. Also, whenever an example did legitimately fail because of a bug, the output from go test was too long to help us pinpoint exactly where the expected and actual responses differed.

Both of these problems were solved by writing a program, fixdocs, whose job was simple: whenever an example fails, replace its current “expected output” comment with whatever its actual output was. We would run go test, pipe its output through fixdocs, and instantly have updated each failing example. Of course, it seems circular to “fix” tests by replacing the expected output with the actual output, but a quick look at git diff would let us find exactly which examples changed and how they changed. If we spotted a bug, we could fix it and try diffing again. With fixdocs, it became much faster to figure out whether the API change had the intended effect, just by scanning the diffs.

git diff tells you exactly what API responses will change

Deploying documentation and the API side-by-side

As a finishing touch, we ran a go doc server alongside our API server which gave the API the ability to serve its own documentation. This was very useful, since the beta API server would serve the beta documentation, and the production API server would serve the production documentation, etc. By simply changing the URL from https://api.beta.capsulecares.com to https://docs.beta.capsulecares.com, you could see the guaranteed-to-be-correct docs for the API you were currently accessing, because those docs were also the tests for that particular API binary.

API explorer

The biggest opportunity we had in mind for this documentation/testing system was the ability to craft and send API requests while browsing the docs. This was also the biggest feature we were missing from Swagger.

It was easy enough to change the example functions to make arbitrary API requests if you were already editing the API code, but the experience was much less fluid than if you could do so directly in the browser.

We did find that go doc supports editing and running example code in the browser, using the Go Playground, which seemed like it could work great as a basic API explorer. And for a while we tried to make this work for our code. Unfortunately, this feature is mostly unsupported. It only seems to work for examples in the Go standard library, and seems like it won’t get love any time soon, which is a shame.

Mock data

In the next post, I’ll discuss how we tried to side-step a lot of problems we usually see when defining test data (e.g. brittleness, duplication), and how we ended up with unexpected benefits, like the ability to use the same test data in unit tests, end-to-end tests, and when running the API locally.