API Changes Are Hard

And I would argue that they are hard because of the tools we use and practices we follow. Let’s take a REST approach as an example. It’s pretty popular nowadays. Many people strive to make their API as idiomatic as possible, even though, more often than not, it’s pretty hard to achieve and for good reasons.

Here is an example of typical REST HTTP request:

GET /products/123
Response:
{
  "id": 123,
  "name": "Red Pants",
  "description": "Beautiful red pants!",
  "variants": [
    {
      "sku": "3726-8",
      "urlSlug": "red_pants_m",
      "color": "red",
      "size": 33,
      "sizeInternational": "M"
    },
    {
      "sku": "3726-9",
      "urlSlug": "red_pants_xl",
      "color": "red",
      "size": 37,
      "sizeInternational": "XL"
    }
  ]
}

One thing to notice here is that the size property is deprecated in favor of sizeInternational. As a client I most probably will either use size or sizeInternational, but not both since I either already migrated my client or not yet aware of this change in the API.

Unfortunately, there is no way for the server to know about this. Its task is to give either full product payload or none. So even when all of the clients are already migrated and using sizeInternational property only, the server would not be aware of it. In my opinion this is a big issue with this approach. Not to mention that the server will still send size property to the clients that don’t need it anymore. It may not be a big issue for a single property, but it can become issue for a real-world API, where JSON sub-tree can be deprecated or if the client only needs a very small portion of a server response.

In my opinion we need better tools to deal with these kind of issues. API, just like any software or service, is not static. It evolves over the time and our tools should make it easier for a client and a server to evolve.

I think that there is a big difference between things that make change possible (often with concepts built on top of these things) and things that are designed around the change making it a first-class concept.

One could argue that versioned vendor MIME-types, version query parameter or versioned URI can solve this problem. Indeed, I would agree that this request is an improvement over the previous example:

GET /products/123
Accept: application/vnd.my-org.product.v2+json
Response:
{
  "id": 123,
  "name": "Red Pants",
  "description": "Beautiful red pants!",
  "variants": [
    {
      "sku": "3726-8",
      "urlSlug": "red_pants_m",
      "color": "red",
      "sizeInternational": "M"
    },
    {
      "sku": "3726-9",
      "urlSlug": "red_pants_xl",
      "color": "red",
      "sizeInternational": "XL"
    }
  ]
}

So here we introduced a version on top of our static JSON response. All these approaches definitely make API evolution possible. In practice though, I find it hard to implement and follow. I’m not talking here about big companies with a big budget for design and maintenance of the API. I’m talking more about smaller companies with small teams who have deadlines to follow and a business value to deliver (which is probably not the API itself: API is just a way to deliver the business value).

I often found it to be a huge step for a small team to introduce a versioning into the API. It requires a lot of effort and dedication in a long term. If your have different experience, then I would appreciate if you could share it in the comments.

As soon as you started to version you REST API, there are quite a few non-trivial questions remaining open:

• How long is the lifespan of a version?
• How often do we want to introduce new versions?
• How many versions will/can we support at the same time?
• How can we encourage our users to migrate to a new versions?

In my opinion there is a big trade-off associated with older versions support.

On one hand you want to support as many versions as possible. The main motivation for this is to make you customers happy. It becomes even more important when you go mobile. You don’t really have any control over the version of applications installed on the devices. Some people may update their apps but some may be more conservative and stay with v1.0 for several years. In this case you don’t really want to tell your users what they “supposed” to do or break their experience by removing support of API v1.0 (this of course highly depends of your product and situation, I’m just talking about my personal experience).

On the other hand, every single supported version requires some effort to maintain it (the more versions you support, the harder it becomes to maintain all of them). Also, as far as I saw, there is very little help in this respect from our programming languages, standards, frameworks and tools. More often then not, they were not designed for the change. So everybody generally finds his own way to do the versioning which results in many custom solutions for the same problem.

Speaking about custom solutions… How do we communicate to the client about a change in the API? Somehow we need to tell client that size field is deprecated and there is a better alternative. One obvious place is a hand-written note somewhere in the API documentation. Even better solution can be a custom X-Deprecated header which contains some text with deprecation notice. What should we do if several deprecated fields are present in the response?

Again, you are on your own here. As far as our standards, frameworks and tools are concerned, this problem does not exist.

Versioning is just one possible way to evolve your API. Keeping API backwards compatible is another approach. And to be honest, I would try to keep API backwards-compatible as long as possible unless I really feel necessity to introduce versioning. It’s much easier to maintain, since we only need to maintain one version of codebase. Don’t get me wrong, it still requires a lot of effort to maintain all these deprecated fields and behavior and in long term can become a big maintenance burden, but in my experience it’s easier to implement and follow this approach in practice.

The problem is that current standards and practices do not necessarily help us in this respect. As I described before, the server has no idea which parts of the response are actually useful to a client and the client has no way to tell the server about this. At least no standard or commonly accepted way to do it. So the server has to send all of the deprecated and potentially unused data over the wire in hope that it would be useful to some clients. We also can’t really remove the deprecated fields since we have no idea whether they are still in use and how often they are used.

Fortunately, new standards emerge which can help us a lot in this respect. Recently released GraphQL data query language spec has a lot of potential. Main idea behind it is that a client can ask a server to give it particular data, just like in previous example, but client need to specify exactly which data it needs in form of GraphQL query:

GET /products/123
query ProductInfo {
  name
  variants {
    urlSlug
    sizeInternational
  }
}
Response:
{
  "data": {
    "name": "Red Pants",
    "variants": [
      {
        "urlSlug": "red_pants_m",
        "sizeInternational": "M"
      },
      {
        "urlSlug": "red_pants_xl",
        "sizeInternational": "XL"
      }
    ]
  }
}

The response JSON always reflects the structure of the query. This gives the server knowledge about client needs. On the other hand client now able to tell the server about its needs as well. This also means that client will not get description, color and sku fields simply because it does not need them. They can be changed or even removed in future. But client does not care about these changes to the API and (what is even more important) the server knows about it. Pretty simple, but powerful concept.

There is another GraphQL feature that can be very helpful for keeping your API backwards-compatible and communicate the change to a client: introspection API. In any GraphQL query you can ask for meta-information about the resource. This meta-information includes full introspection of resource types, fields , arguments, etc. Here is an example of an introspection query:

GET /products/123
query ProductIntrospection {
  __type(name: "Variant") {
    name
    fields(includeDeprecated: true) {
      name
      isDeprecated
      deprecationReason
    }
  }
}
Response:
{
  "data": {
    "__type": {
      "name": "Variant",
      "fields": [
        {
          "name": "sku",
          "isDeprecated": false,
          "deprecationReason": null
        },
        {
          "name": "urlSlug",
          "isDeprecated": false,
          "deprecationReason": null
        },
        {
          "name": "color",
          "isDeprecated": false,
          "deprecationReason": null
        },
        {
          "name": "size",
          "isDeprecated": true,
          "deprecationReason": 
            "Please use `sizeInternational` field instead."
        },
        {
          "name": "sizeInternational",
          "isDeprecated": false,
          "deprecationReason": null
        }
      ]
    }
  }
}

Here I get some meta-information about Variant type including the list of all its fields and whether they are deprecated or not. As you can see, the deprecation concept is already built into the language. Now you know that size field is deprecated and what you should use as an alternative.

Also note that I need to explicitly ask for deprecated fields with includeDeprecated argument because by default the result will not contain deprecated fields. This helps new users to avoid using deprecated API.

Both of these features make the “change” a first-class citizen helping API developers to evolve their API. I think it’s a good example of technology designed around the change (at least to some extent).

As a conclusion, I would like to say that the “change” is fundamental to any software and API. By failing to acknowledge this and making it a second-class citizen we are making a big deservice to a development community in general.

We can take some inspiration from the design of our own offices:

from “How Buildings Learn” by Stewart Brand

There is a reason why many office buildings have wiring exposed all over the place. Things like space plan and services change more often than any other parts of the building, so buildings themselves are designed for this change to minimize a maintenance costs.

I hope that in future we will not only brag about beautiful URLs and how RESTful our API is, but also how easy it is to change and evolve it.