In my last post — Minimally Invasive API Versioning — I talked about how the Mobile API team at RetailMeNot handles versioning our Clojure APIs using a macro called versioned. This macro alone worked great for over a year, but we needed more.
In this post I will cover how we evolved it to help ease our feature rollout process, and how we gained a super-easy feature flagging system basically for free.
To recap, the versioned macro introduces a convenient syntax for annotating source code with version-specific information — things like “this line was added in version 3” or “this line was removed in version 7.” We call these annotations version-qualifiers. Initially we had several. some were “binary”, like added, removed, only, which let the developer specify code that should or should not exist for a specific application version; and others, like changed, lets the developer specify alternative implementations based on the application version.
We call these qualifiers collectively linear version-qualifiers, because their functionality is based on the linearity of versions: something like “added in version 2”makes no sense without the assumption that V1 precedes V2, which precedes V3, etc.
Like I said, this reasoning got us very far, but it has one major drawback: we typically ship multiple features per version, and when we do encountering this version-specific code would often leave developers wondering, “Why was this change made? Which feature was it a part of?”. Furthermore, say a feature slipped and we needed to move it to a later version, we found it difficult to go back through the code base to identify and edit those places only related to that feature.
The fundamental problem is that when we annotate code with linear version-qualifiers we imbue that code with version-specific information. Stated another way, these kinds of annotations only let a developer look at the code and answer the question, “When did this change ship?”, but this mindset ignores the reality of why we’re making a code change at all: to satisfy the requirements of a feature.
Rather than adding version-specific annotations, we should be adding feature-specific annotations, and then — somewhere else — we should maintain a mapping of which features are enabled in which versions. By doing this, we can accomplish two things. First, we decouple the source code that is sensitive to the app version from the irrelevant particulars about when that code was turned on, and instead we manage that “when” somewhere else entirely. The second benefit is that code that all relates to one feature can be explicitly annotated as such, and so greping the code base for changes related to that feature becomes possible.
We call these feature-specific version-qualifiers “feature-qualifiers.”
This OfferCard function is functionally equivalent to the linear qualifier example above. image will appear starting in V2, and footer will be absent in V3. We achieve this same functionality by using the version-manifest, which defines which features belong to which version. This is something that we could only implicitly handle before when using the linear qualifiers.
As you can imagine, it’s dead simple to turn a feature on or off for any version: simply add or remove it for that version in the manifest. No more scouring the codebase whenever we need to move a feature!
Using feature-qualifiers solved the problem of easily toggling and identifying features in the API, but there was still a niggling problem that we had yet to address.
When we release an app that requires backward incompatible features to be added to the API we define a new content-negotiation version, which only the latest version of the apps will request; however, sometimes we need to code a kill-switch so the apps can revert a feature in the case that something goes dramatically wrong. We call this a “feature flag.”
It may seem like the kill switch could just simply be to make the apps request the previous API version. This works when the only difference between two versions is that flagged feature; however, often these feature flags make up only part of an API version, or perhaps they span multiple versions and so simply reverting to an older API version would have the effect of rolling back many unrelated changes. Rolling back more features than are necessary is simply unacceptable. We needed a way of toggling some features on and off regardless of the current API version.
So for a while we did what any other API team would do. Maybe we’d add a query parameter to an endpoint like “with_feature_X”, and then if the client asked for feature X, we’d litter the code with “if” statements in order to build (or not build) the experience with that feature enabled. But we just finished a whole new feature-oriented method of modifying source code, so surely there was some way to leverage it!
Above, in our version-feature-manifest the feature is either included or not included in a given version, and this dictates how the code for that version ends up being compiled by our versioned macro.
But what if…
What if we could specify that some features are “flaggable” for a version? For example, what if the :offer-card-images feature may either be or not be part of version V2? Maybe what we really have in this case are two different versions of version V2: version V2, and version V2 with offer-card-images:
And that’s exactly how we tackle this problem: we generate a “fake” version for every combination of feature flags for each version listed. In the example above, we will generate V2+offer-card-images for the version V2, and if there was a second feature flag B, we would also generate V2+B, and finally V2+B+offer-card-images.
If the combinatoric potential of this worries you, fear not: All of this happens during compilation, and the runtime overhead of the resulting code is constant for practically any number of versions!
Here is an example of what the generated code ends up looking like:
The last step is to — based on the value of a query parameter or something — upgrade the user’s content negotiated version at runtime from, V2 to V2+offer-card-images, the details of which I won’t bore you with.
In my opinion, this is a great example of how opportunities for reuse tend to present themselves when you can break your layers of abstraction down into their simpler elements. In this case, we decoupled two things: the code changes required to implement a feature, and the version of the API that that feature was a part of. By doing this, we were able to incorporate feature flags into this system and thereby unify the two ways in which API clients can influence the behavior of the API.
Best of all, API developers no longer need to concern themselves with how or why a particular feature is enabled — be it by feature flag or content negotiation—they need only care that it is enabled.
I’m very happy to announce that this library has been recently open sourced, and I would love for you to check it out on github. The project includes the “linear” and “feature” version qualifiers, but not the code to implement a feature flagging system; we implemented that entirely on top of this library.
Thanks for reading!