Tricking Colleagues into Writing Documentation via Contract Testing
Part of my day job is trying to get folks to write API documentation, so the hordes of new developers joining the company on a weekly basis are not stuck trying guess at contracts by poking around years old code. Convincing API developers to take the time to write specs has involved dangling a myriad of carrots.
One was a continuous integration pipeline to convert their in-repo OpenAPI specs to read-only Postman Mirrors (because who wants to maintain Postman collections manually?!!), and some folks are excited to use JSON Schema to offer client-side validation (now that we’ve solved the JSON Schema <> OpenAPI issue.)
Solving OpenAPI and JSON Schema Divergence
My previous article explained the divergence between OpenAPI and JSON Schema (a.k.a the subset/superset/sideset…
The carrot we’ll be discussing here is contract testing: the art of validating the shape of data provided by the API is what you think it is.
Folks are regularly asking “how can we keep our documentation up to date”, and we have covered a few solutions to that, one of which is using JSON Schema. Unfortunately thinking of it this way leads to responses like “ok greeeeeat, when I have finished spending a week writing up documentation, I get to write a bunch more stuff to ensure its up to date. Yaaaay! 🙄”
Well that’s a fairly backwards way to think about the whole thing. I tell people: API specifications are a really powerful way to get contract testing, and they just so happen to render as documentation too. Once you’ve confirmed your code conforms to the contract, you know your documentation is correct, so it’s pretty much been done for free.
What is Contract Testing
A lot of applications just check that the endpoint doesn’t entirely crap out, which is the most basic sort of unit test:
it 'should return HTTP OK (200)' do
Sure ok, that’s better than literally nothing, but we need to know what fields are there, that those fields have a specific type, and we need to make sure that doesn’t randomly change!
Some folks will add some specific checks to their unit tests, maybe something like include_json:
it "has basic info about user" do
Others will bump this off to BDD-style testing with Cucumber or similar tools, which is essentially doing the exact same thing:
Feature: User APIScenario: Show action
When I visit "/users/1"
Then the JSON response at "first_name" should be "Steve"
And the JSON response at "last_name" should be "Richert"
And the JSON response should have "username"
And the JSON response at "username" should be a string
And the JSON response should have "created_at"
And the JSON response at "created_at" should be a string
And the JSON response should have "updated_at"
And the JSON response at "updated_at" should be a string
.... ugh it goes on and on ...
At this point it is essentially contract testing, but its very laborious, and is a whitelist. If a new field shows up and we’ve not written a bunch of assertions for its type, we have no way of knowing if its correct. Clients might start using it, later it could change, and our test suite would never know.
Something else to consider, is that writing these assertions for each field sucks. You want your unit tests to confirm the shape of the response under all sorts of scenarios, but you do not want to copy these lines over and over again. This leads to folks shoving some assertions into the first unit test, then just crossing their fingers that other scenarios output the correct shape. 🤷♂️
JSON Schema for “DRY” Contract Testing
Don’t Repeat Yourself, and define reusable
schemas/components/user.json files, populated by JSON Schema:
"type": ["string", "null"],
Learn more about the basic JSON Schema keywords over here:
The basics - Understanding JSON Schema 1.0 documentation
When learning any new language, it's often helpful to start with the simplest thing possible. In JSON Schema, an empty…
Armed with a bunch of reusable contracts, the only thing to do is check if those contracts match the code. Ruby users can do this with json_matchers by Thoughtbot.
Put this config wherever you handle your RSpec setup (or check out their alternative approaches):
JsonMatchers.schema_root = "docs/components/schemas"
Then it’s simply a case of adding new tests for that context, or adding the expect line to your existing tests:
it 'should return HTTP OK (200)' do
endit 'should conform to user schema' do
That’s it as far as contract testing goes. Run your tests, tweak the responses and JSON Schema, try and get it to fail. Remove a required field and it’ll moan. Typecast an int to a string and it’ll get sad.
If you need help creating these JSON Schema files, there’s a few hacks to speed you up.
What about Docs?
Often early stage OpenAPI documentation is a lot of paths and maybe some properties, but folks often leave the schemas “until later” (never).
With the schema missing you lose all benefit of API specifications: you have no example values, attribute tables, enums for status fields, types, etc. Luckily the new JSON Schema fill exactly that gap!
You’ll need to watch out for the divergence issue, or use Speccy to convert this JSON Schema & OpenAPI combo to “pure OpenAPI” in some build/CI step, but this works juuuust fine.
Then it’s just a case of turning that OpenAPI into beautiful documentation.
Turning Contracts into Beautiful Documentation
Continuing on the topic of contracts — metadata that describes your data — it makes sense to see what easy early wins…
Give it a try and let me know what you think. I really like this simple, powerful, approach to keeping code up-to-date with your API specs.
Comment with your various language/framework implementations for solving this. Some of my friends have made PHP+PHPunit wrappers, and theres some Node stuff floating around, but I don’t have the links at time of writing.