Toxic APIs, or Don Quixote 2.0
(This post originally appeared on hubrix.co in March 2017).
In a world of APIs, the “silo problem” is more dangerous than ever
There’s a famous quote from Bjarne Stroustrup, the inventor of C++:
“C makes it easy to shoot yourself in the foot; C++ makes it harder, but when you do it blows your whole leg off.“
I love that quote. I’ve experienced it first-hand coding in C++, but it also illustrates a recurring problem in software architecture. Many well-intentioned and inventive efforts to improve software have led to systems that mitigate minor and medium-sized problems, while creating opportunities for catastrophic problems.
Other efforts, neither inventive nor well-intentioned, make this problem worse. I’m referring to proprietary lock-in, the tendency of any software platform to make choices on behalf of the end-user, developer or system integrator that uses it. Sometimes this is deliberate, other times unwitting: developers work within the conceptual framework they have created, and it can become difficult to summon an outside perspective on that framework. In management consulting, this problem is called “silo mentality” — the inability of a team to think outside its own box.
Increment or Innovate?
The evolution from C to C++ was incremental by design, to leverage the large population of developers familiar with C syntax (and the installed base of C compilers, since the first C++ “compilers” just translated C++ code into C). This made good tactical sense. But when Java came along, in terms of adoption and popularity, it ate C++ for breakfast. Not because it was inherently better or had a friendlier syntax, but because it was object-oriented from birth, with no “legacy” antecedents in a procedural language.
The “silo mentality” can be unconscious or deliberate — in my own experience, usually the former. But in either case, its chilling effect on innovation is equally powerful. Likewise for its effect on integration potential among disparate systems. The combination of incrementalism and silo mentality has catastrophic consequences for software usability, lifespan and ROI.
RESTful APIs can be a powerful defense against “software silos” because they mandate open, documented access to the resources and functionality of any system fitted with such APIs. But adding an API does not magically make a system open, nor does it guarantee harmonious coexistence with other systems. The mindset of the software architect can persist in the definition and enumeration of the defined endpoints. The broader point:
When you use any software, you are buying into the worldview of that software’s architects and developers.
It doesn’t matter if you are aware of that worldview, nor if the creators of that software are aware of it. It is there. It may open some doors but it closes others. An API does not change this fundamental truth. Silo mentality is particularly insidious in APIs because the mere existence of the API dangles the (sometimes illusory) promise of freedom and openness.
Providing a vast number of endpoints will not solve this problem (and can be counter-productive). Using Swagger / OpenAPISpec, while certainly helpful, will not solve it either. To build truly open APIs, we need to change the way we think.
What’s Incrementalism Got To Do With It?
Anyone who sells software will tell you: version fragmentation is the monster under their bed. It is a great and legitimate fear. Just imagine how the people who develop and sell Android feel about this chart:
I call this the Dark Side of the Long Tail. The Long Tail is great when buying non-bestsellers from years ago on Amazon, but it’s dreadful for gaining critical mass on the latest & (we hope) greatest platform. However you slice it, this chart tells us: 81.3% of the Android base is not on Marshmallow.
If you develop apps for Android, you simply hedge your bets, and target “4.2.x or later” — and that gives you a healthy 89.1% of the Android base as a potential audience, at the cost of limiting (most of) your app to “lowest common denominator” capabilities. But if you are the Android team, you sweat these numbers. All those pre-Marshmallow devices are a giant weight, slowing your ability to innovate.
As with silo mentality, this problem gets worse when it comes to APIs. A new release of an API must offer full backward compatibility, version negotiation, or both. If it does not, it breaks every app consuming the API. Compare this to an Android app developer who targets only Marshmallow. That developer has simply reduced the potential market share for the app, but didn’t break anything. API developers don’t have the luxury of that choice.
Every app consuming an API becomes part of the “legacy system” for that API.
This reality leads to a strong inclination to incrementalism among API architects, precisely because the cost of versioning is so much higher than it is for app developers.
At the same time, the very purpose of APIs requires rapid evolution and adaptability. At first glance this seems to leave us with a paradox.
What can we do?
Before describing a proposed solution, here’s a recap of the key problems:
- Silo mentality (willful or accidental) is preventing the so-called “software industry” from becoming an industry in the true sense of the word: an ecosystem where innovation and compatibility are not mutually exclusive
- Every scrap of software, ever, “embeds” the mindset and worldview of its creators in a way that is not always obvious, but is often limiting
- Incrementalism is an unfortunate and dangerous by-product of the way we develop software today
- Silos, implicit mindsets, and incrementalism are particularly toxic to APIs
- The nature of APIs tends to cause their creators to lean towards incrementalism
The English word “quixotic” comes to us from Don Quixote, by Cervantes, the first modern novel (talk about disruptive innovation!). The Merriam-Webster gives us this definition of the term: “foolishly impractical especially in the pursuit of ideals; especially: marked by rash lofty romantic ideas or extravagantly chivalrous action.” This came from the protagonist’s tendency to do battle with windmills.
At the risk of having our own actions dismissed as laughably futile, we at Hubrix intend to do battle with silos (and implicit mindsets), by adopting the following principles in our API designs :
- Zero proprietary interest. We only make APIs, so we have no “platform” to defend (intentionally or otherwise). Our APIs need to deliver value on their own merits, or no one will use them.
- 100% Open Source. All our products will be open source, so that any accidental “silo mentality” or implicit-mindset limitations can be identified and/or corrected by others.
- New version = new API. As with industrial parts, if we change the specifications of an API, it is treated as a new product (or variant) and not as v2.1.3 of “Product X” which bears little resemblance to its v1.0 namesake.
- LTS for APIs. The Long-Term Support practice used for many operating systems and frameworks is, in our view, a categorical imperative and a necessity for RESTful APIs. Designating an API as LTS gives its consumers a reliable time-window for managing the lifecycle of projects that depend on that API.
- “Proactive Agnosticism.” This is a commitment to avoid choices that rightly belong with the API consumer, for example: what datastore technology to use (MySQL vs. Oracle vs. Elastic), or what payload format (JSON vs. XML). This doesn’t mean we abdicate our own best judgement, only that we try to remain wary of forcing choices on developers that are not ours to make.
