On the Law of Conservation of Complexity
In Software, Complexity is Never Destroyed, Only Hidden Behind Abstractions
Wikipedia has a definition of the Law of Conservation of Complexity as it relates to human-computer interaction that states that every application has an inherent amount of complexity that cannot be removed or hidden, only dealt with — either in product development or in user interaction.
The distinction of inherent is important. For any specific architectural design there exists a corresponding level of complexity. You can design a new architecture, which will then change that complexity; but within your fixed solution, there is inherent complexity that cannot be removed.
Wikipedia’s definition also states that complexity cannot be hidden. While I see where this part of the definition comes from, I think this aspect of the law is more nuanced. A valuable lesson that I’ve learned and re-learned throughout my career is this: Complexity is in the eye of the beholder.
A system can appear extraordinarily complex to one, while it appears trivial to another. This difference in perceived complexity stems from myriad sources, including the differing skill and experience levels of the human involved in the interaction and the subset of the system visible to that human. If you’re just looking at the tip of the iceberg, perceived complexity could be very small. If you’re an expert and have been dealing with a particular category of complexity for decades, your version of trivial is not the same as everyone else’s.
In this blog post, I want to address this notion of perceived complexity as it pertains to the Law of Conservation of Complexity, specifically for developer interactions.
First, let’s take a look at some specific human-computer interaction patterns that developers and operations people routinely deal with:
- Consuming an API from a library
- Communicating with someone else’s service(s)
- Installing, configuring, and maintaining a (distributed systems) product
- Troubleshooting any or all of the above
- Teaching others how to do any or all of the above
The last two bullet points are among the most important, yet are often the most neglected of human interactions in our industry.
When we create an API, we’re defining the nature of a human-computer interaction. Humans — people with emotions, time constraints, and tolerance levels — will be using that API. As stated earlier, if the system we’ve built is immeasurably complex, we might still be able to hide the majority of that complexity behind our API.
All too often we end up building APIs that are analogous to exposing a thousand levers, switches, and knobs rather than a “go that way”, “speed up”, or “slow down” interaction. It’s so tantalizingly easy to simply heave our own complexity burden over the wall, foisting it upon our unsuspecting victims (developers), rather than going the extra mile to shield them from it.
The next way in which we have the opportunity to craft a human-computer interaction is through the products we build. I’ll illustrate using Knative here because I like it, I think it has enormous potential: but it also has a steep cognitive burden penalty borne by the developers who use it.
The Knative project builds a layer on top of Kubernetes that provides loose coupling between event sources, event channels, subscriptions, event handlers (services), and routes. The hypothesis is that this decoupling, which enables Kubernetes to scale your workloads to zero and bring them up on demand, will increase developer productivity and potentially produce more cost effective architectures.
The documentation for Knative is written the same way a maintenance manual for a jet engine might be — written for experts. It assumes a certain pre-requisite level of knowledge and high tolerance for implicit context. The complexity here is exposed in the raw, without being hidden or moved.
I won’t go into the details of what is required to fully understand how all of Knative’s moving parts contribute to its features, but should mention that as someone who is a Certified Kubernetes Administrator and builds containerized workloads for a living, I am still overwhelmed by it all.
Installing, configuring, troubleshooting, and developing applications for Knative in service of reducing development complexity can often increase it for those who might not already be the kind of experts knative is targeting.
Realizing that raw Knative might not be the ideal target for a larger audience of service and function developers, Google created Cloud Run, a layer of abstraction on top of Knative, that provides a very simple and straightforward way to interact with the underlying Knative and Kubernetes infrastructure. Google recognized the need and hid this complexity from its target audience.
Not all complexity needs to be hidden. Some of it changes over time. As time passes, some types of complexity mitigators become hardened and more ubiquitous. Like building a 4-level stack of Tetris pieces, that complexity may collapse into a more manageable foundation. Containerization is an excellent example of this phenomenon. It started out as an end goal and now it’s “simply” a building block used in assembling more advanced systems — its complexity has been handled by time, exposure, documentation, and community embrace, but not removed.
I’d like to leave you with the following suggestion: embrace the notion that there is complexity inherent in everything we build. This frees you to spend more time thinking about how much of that complexity should be visible to your users, whether they’re installing your software, consuming your APIs and services, or developing software to run on your product.
Despite our technical leanings, we all have a limited supply of tolerance for cognitive burden and friction-inducing complexity. Let’s try and do the community a favor and acknowledge the elephantine complexity in the room, then shove it in the closet where it belongs.
DISCLOSURE STATEMENT: © 2019 Capital One. Opinions are those of the individual author. Unless noted otherwise in this post, Capital One is not affiliated with, nor endorsed by, any of the companies mentioned. All trademarks and other intellectual property used or displayed are property of their respective owners.
