Introducing Silk: Markdown driven API tests

The problem

I write a lot of code to test my API endpoints — especially because I am interested in the impact of everything that runs during the lifecycle of a request; how the route is interpreted, what middleware does to the request, how the business logic behaves, and what kind of response I get. Not to mention that sometimes I care about the impact of request headers, and the contents of response headers.

Documentation also quickly gets out of sync as the APIs evolve.

Meet Silk

Silk (silktest.org) lets you write your API documentation in Markdown, and then execute it as an automated test. If your API changes and your documentation is out-of-sync, the tests will fail.

Silk lets you write Markdown documentation for your APIs, and then execute as an automated test.

We are looking for people to play around with the BETA of Silk, and let us know what you think. Pull Requests and feature ideas are very welcome, and are especially important at such an early phase of a project. Please get involved.

Tutorial

To explain how Silk works, we will write a simple markdown document describing a Hello World API, and then execute it with the `silk` tool.

This is the endpoint we are going to be testing: http://outlearn-hello.appspot.com/hello?name=YOUR_NAME_HERE.

The API we’re going to be testing was built as part of an Outlearn learning path about Google App Engine which is worth checking out if you’re interested in using Google App Engine with Go. If you do, you can run your Silk tests against your own app, rather than the one provided.

First, head over to the project homepage and download the latest release of Silk for your platform.

Gophers can just: go install github.com/matryer/silk

Create a new file called `hello.silk.md` and populate it with the following:

# Hello API
## `GET /hello`
Gets a personalised greeting.
* `?name=Mat` // The name of the person to greet
===
* Status: `404`
```
Hello there
```

This simple file follows the Silk markdown pattern of not only describing a typical endpoint test in a human readable way (see pic below), but also in a way in which the Silk parser will understand.

Top level headings are descriptions, but second level headings (prefixed with ##) describe an HTTP request that will be made. In our case, a GET request to `/hello`. We’ll specify the scheme and host when we run the tests — which is extremely powerful as it lets us run our Silk tests against different endpoints (like localhost, test or live). The bulleted items (prefixed with *) let us specify URL parameters, and underneath the separator ===, assertions about the response. Finally, the code within the ``` backtics describes a body.

Markdown renders nicely to look like real API documentation.
We deliberately got a few things wrong in our Silk document so we can see what failures look like.

Run the file using the silk command in a terminal, passing in the live end-point URL via the -silk.url flag, and the name of the file to run as the first argument:

$ silk -silk.url=”http://outlearn-hello.appspot.com" hello-api.silk.md
running 1 file(s)
body expected:
```
Hello there
```
actual:
```
Hello Mat.
```
— — FAIL: GET /hello
hello-api.silk.md:14 — body doesn’t match
— — FAIL: silk (0.31s)
FAIL

So it seems like we’re expecting “Hello there”, but actually the API is returning “Hello Mat.” Let’s update our documentation to match the API:

# Hello API
## `GET /hello`
Gets a personalised greeting.
* `?name=Mat` // The name of the person to greet
===
* Status: `404`
```
Hello Mat.
```
The body between the ``` back tics has been updated at the bottom

Re-run the tests and notice that we get a different failure:

$ silk -silk.url=”http://outlearn-hello.appspot.com" hello-api.silk.md
running 1 file(s)
Status expected: 404 actual: 200
— — FAIL: GET /hello
hello-api.silk.md:11 — Status doesn’t match
— — FAIL: silk (1.24s)
FAIL

In our test, we indicate that we should get a 404 (Not found) status, but we are getting a 200 OK. Let’s fix that, and add an additional test:

# Hello API
## `GET /hello`
Gets a personalised greeting.
* `?name=Mat` // The name of the person to greet
===
* Status: `200`
* Content-Type: `text/html; charset=utf-8`
```
Hello Mat.
```

Here we have changed the status to 200, and added a check on the `Content-Type` response header.

Run our tests one final time to see them pass:

$ silk -silk.url=”http://outlearn-hello.appspot.com" hello-api.silk.md
running 1 file(s)
PASS

Conclusion

Silk lets you write human readable documentation, and run it as part of your automated test suite.

You can:

Let us know what you think by tweeting me.