Minimally Invasive API Versioning

One of the challenges in developing an API for mobile apps is that there is not both a pleasant and reliable way of forcing them to upgrade. This means that in some cases your API needs to support apps that are several years old. The mobile API for RetailMeNot is no exception.

This post walks through one of the ways that we’ve leveraged Clojure to make supporting old API versions, and thus old app versions, significantly easier.

First, I will introduce you to a function that builds a portion of our response.

This is a simple function that builds the meat of an “offer card.” The function is called “OfferCard”; it has three arguments — “image,” “title” and “description” — and it returns a JSON map with its arguments embedded in it at certain locations. The OfferCard is ubiquitous in our app, and many attributes have been added and removed over time to support changes to the mobile apps over time.

Two traditional ways for iterating on a function like this exist: The first way is what I’ll call copy-and-modify. This involves simply copying the source code, picking a new name for the function and then making whatever changes are necessary to fulfill your requirements (and, of course, making sure the caller picks the correct version to call):

You’re probably allergic to copy-pasted code, and rightly so; copy-pasted code, while not always a bad thing, may subtly change over time, and in general just creates another chunk of code that must be maintained.

The alternative is for a new, second function to call the old function, and then mutate its return value to fulfill the requirements. I’ll call this the call-and-mutate method.

This technique also has some tradeoffs. One nice aspect is that modifications to the original OfferCard function will automatically be incorporated in the OfferCard-V2 function, so we don’t have the same problem of having to maintain the same functionality in multiple places as we did in the copy-and-modify approach. The one major drawback is that we’ve created a function that is pretty mysterious: You will have to go on a bit of a wild goose chase to figure out what OfferCard-V2 returns, and you will have to mentally calculate the result by doing all the “assoc-in” (and whatever else) calls in your head. Just imagine this process for OfferCard-V10 or OfferCard-V20! This technique can only support so many chained calls before it becomes completely unintelligible.

What we really want is a method that can combine these two approaches and take the best features from both. As developers, we want a single place to look at in order to figure out what a version of an OfferCard will look like, but we don’t want to have to maintain multiple independent pieces of code that do practically the same thing but with only minor alterations from version to version.

Using Clojure’s powerful meta-programming support we are able to get both of these features.

What would be ideal is if we can simply annotate our code with “version qualifiers,” which would indicate something like “this piece of code was added in version 2” or “this piece of code was removed in version 3.” Once we marked up our code, we could then run a preprocessor over it to do the boring work required for the copy-and-modify or call-and-mutate approach.

This would require us to basically modify the compiler, which is not something that many languages let you do. However, in Clojure, by using macros we can very easily hook into the compilation process in order to implement our version qualifiers! Here is an example of an annotated OfferCard function, the ideal of what we’re shooting for:

A couple of things are going on here: We’ve converted the argument list (image, title, description and footer) into a single map argument. This is similar to Python’s keyword arguments; it allows callers to optionally specify each of the four arguments by name, which is helpful for when they’re generating a version of the OfferCard that doesn’t need one of the keys, like version 3 in this example when we’ve removed the description field.

The second difference is the versioned expression that is wrapping the body of this function. This is our macro, which we use to enhance the Clojure compiler to allow us to write the “removed” and “added” version qualifiers within its scope — without it, these qualifiers would cause compilation errors because Clojure doesn’t recognize them.

Versioned is implemented similar to any other function in Clojure, except that because it can only run at compile time its arguments aren’t (and can’t be) evaluated like they would be at runtime. Instead, the literal source code that the programmer writes is passed in as an argument, and the macro’s job is to return syntactically valid Clojure source code. In this case, source code for the body of the OfferCard function is now passed to versioned so that it can process it into a function that can produce every version of the OfferCard!

All of this allows us to dynamically alter which version of an OfferCard we want this function to return. For example, in version 2 there will be an extra “footer” field; in version 3 the “description” will be removed.

“But what does the generated source code look like?” That is, “what does the versioned macro return?” It turns into something like the following:

It simply compiles into a switch statement! The versioned macro is implemented by essentially applying the copy-and-modify technique to your source code, and stuffing every computed version into a big switch statement. The version qualifiers are simply instructions that tell versioned to include or remove a chunk of the source code when generating one of the switch cases. (If you’re wondering, the-requested-version is a dynamic variable that is set to whatever version the mobile client specifies in its request headers.)

In the examples above I’ve only shown version qualifiers on the keys of maps, but we can also put them in vectors (“[ .. ]”) or into lists (“( … )”), and because Clojure source code is just nested lists, this means we can annotate any arbitrary code.

In this simple example we’ve computed offer-cards by mapping the OfferCard function over our offers, but we’ve also specified that in and after version 2, we also need to add-inventory-positions to those cards. These subtle variations appear all over our codebase, and without a tool like the versioned macro, we would be stuck coming up with an ad-hoc way of handling each minor case. This macro gives us extra syntax we can reach for to solve the problem.

Abstraction is at the heart of much of what we as programmers do, and it is absolutely necessary to build more and more complicated systems with greater ease. This macro demonstrates the power of meta-programming with Clojure, which allows you to build syntactic abstractions. In this case, the copy-and-modify or call-and-mutate ways of solving this problem require a certain amount of boilerplate — that is, a certain amount of syntax — to implement, but our macro lets us encapsulate those details (lovingly referred to as “design patterns” in many contexts) into a syntax that is designed to express that intent.

This relatively simple macro has done quite a bit of heavy lifting to drastically reduce our code footprint and complexity on the Mobile API team here at RetailMeNot. Something like it wouldn’t be possible in many other languages.