Ditching the blueprint: towards a code-first approach to API management
At Lenses.io, we like to think that our APIs are not an afterthought, but a foundational aspect of our product. Our REST API constitutes the main mechanism whereby our web-based GUI and CLI client tap into the data and the functionality exposed by Lenses’ backend. At the same time, the same API provides our customers with a means to integrate our product into their very own infrastructure, business processes, and internal dev tools.
Like most software products, as Lenses grows in number of customers and features, we are experiencing some non-trivial challenges in evolving our API and governing its growth:
- We want it to be simple and consistent, while at the same time not breaking backward compatibility with current and possibly previous versions.
- We want it to be as much as possible forward compatible with future versions of third party systems we integrate with (e.g. Kafka, Kafka Connect, Schema Registry, etc.).
- Most importantly, we want it to be well documented, and easy for colleagues and customers alike to discover.
This latter point may not sound as particularly challenging. As different flavours of APIs (e.g. REST, GraphQL, gRPC, etc) have become ubiquitous, there’s no shortage of tools to author and publish web-based API docs, often providing some degree of interactivity (e.g. Swagger UI live editing, REST playgrounds, etc). However, software documentation is only useful to the extent to which it is also accurate and up to date. While it is relatively easy to publish them, keeping the API docs truthful to the current state of a fast evolving software system can be challenging.
This is mostly due to the fact that updating and publishing the docs is often decoupled from the continuous integration process that governs the software production. Over several development cycles, this creates the conditions for the documentation to potentially drift from the actual implementation, as keeping the two in sync is pretty much entirely left to discipline. Through direct experience as API users, designers and maintainers, we have learned that inaccurate documentation can have a significant cost, with severe business consequences ranging from confusion, endless back and forwards among dev teams, bugs, and ultimately loss of confidence in the product.
Throughout the rest of this post I would like to share some learnings gained on our journey towards a code first approach to API docs, as we strive to aggressively automate, leaving as little as possible to human intervention, discipline and good will.
Our legacy API stack
The Lenses API consists of more than a dozen groups of endpoints, powering a broad set of functionality such as accessing data entities (e.g. Kafka topics), enriching third party APIs (e.g. Kafka Connect, Schema Registry) and streaming SQL query results. Most of these endpoints follow the well established REST approach. However, we also have non trivial functionality implemented through streaming mechanisms such as WebSocket and Server Sent Events.
Currently, our API docs are produced through a description language called Blueprint. Blueprint has served us well for a while, providing us with a simple and direct way of documenting API endpoints based on syntax embedded in the ubiquitous markdown format. Its low barrier of entry has allowed engineers across different teams and technical specialisms to collaborate effectively. That said, we found that this approach also suffers from some serious shortcomings:
- While facilitating contributions, its relatively unstructured, markdown-like syntax also made it difficult for us to enforce consistency and completeness. As the linting/parsing performed by this tool is rather weak and permissive, the inclusion of important structural details such parameters and request/response payloads is left to the developers’ diligence and their willingness to master Blueprint’s arcane syntax and conventions.
- Most importantly, the fact that Blueprint is entirely agnostic of our backend’s source code, and completely detached from its build process results in a tedious maintenance burden, whereby updating the API definition adds up to already long list of chores and sign-offs needed before a feature can really be called done.
Towards a standard-based, code-first approach
It feels like the design first approach embraced by Blueprint doesn’t fit well into the reality of our product and workflow. Unlike in microservice architectures where a single API interface is implemented across separate systems (sometimes using different technologies), Lenses is deliberately implemented as a monolithic application, with a unified codebase. So, why not treat such a codebase as the single source of truth to generate documentation from? This approach started to look feasible to us only recently, when we became aware of Tapir: an excellent Scala library that allows us to do so in a way that feels idiomatic to both our programming language and web framework of choice.
In Tapir, API Endpoints are described using the highly expressive Scala type system. These formal endpoint descriptions drive the generation of an OpenAPI spec — a widely adopted format to describe and document REST APIs — as well as the implementation of the actual endpoint logic. This approach gives us strong guarantees that the two are in sync, as the Scala compiler will statically check that at build time.
Of course, like many good things, introducing Tapir comes with some costs, especially when retrofitted into a large, pre-existing code base:
- Until we have ported all the existing API endpoints to Tapir, both the old Blueprint docs and the new one have to coexist. While the migration is still half way through, this duplication of effort can feel a bit daunting
- While before we had engineers from different chapters (backend, frontend and ops) contributing to documenting the API, this maintenance task now falls almost exclusively under the backend chapter’s responsibilities.
- In order to work its magic, Tapir uses some fairly advanced features of Scala (e.g. generic derivation, macros, etc), which can impact developer productivity to some extent (in particular by making the IDE less responsive). As we got more proficient with using the library, we found that a good strategy to mitigate this is to wrap its complex API into a simpler one, designed around our specific use case. This reduces the need for developers to learn about its intricacies as they can just reuse a pattern established throughout the entire codebase.
Overall, we felt like the increased degree of automation and safety introduced by this approach compensates for the above costs. Moreover, managing the API documentation as code opens up exciting opportunities. To mention a few:
- As Tapir automatically derives JSON schemas from scala types, we found that we can now lift and shift code snippets from our unit tests to produce meaningful examples of request/response payloads.
- The ability to abstract common documentation patterns into functions and other code constructs should lead to less duplication, more reuse and hopefully a higher degree of consistency across endpoints.
Beyond docs generation
While we are not done with our migration yet, we are very excited about being able to publish an OpenAPI spec alongside each Lenses build (both internal and public). Beyond generating up-to date documentation, embracing this open standard means that Lenses and its users can benefit from a growing ecosystem of API tools:
- The OpenAPI spec can be imported into the Postman client in just a couple of clicks, providing developers and testers with an excellent tool for exploring and interacting with the API without having to write any code.
- Generating test doubles (or perhaps full-on SDKs …) to be used by other components of our product implemented in a different language (e.g. our TypeScript UI build, or our Golang CLI).
- Detecting breaking changes in the API by diffing new versions with the current one, in a similar way in which Confluent Schema Registry does for Kafka topics’ Schemas.
In this post, we have discussed some of the key challenges of evolving an API driven system, focusing on the problem of keeping API documentation and implementation in sync. While our journey has led us to adopt a code first approach to API automation, for others a design first approach might be the right choice. Whichever of the two approaches your organisation decides to go for, it is worth investing in automation and tools to keep the various software artefacts that your API glues together tightly integrated. API documentation is a great place to start, but it doesn’t have to end there!
In the next posts in this series, we will look at the technicalities of how to implement such an approach across both backend and front-end. Please do subscribe in order to be notified when part two and three are published!
