Antifragile System Design 4: Modularity And Decentralization

The mind’s architecture is modular; the heart’s architecture is not.

— Jerry A. Fodor, “The Modularity of Mind”

Complex systems are rightfully composed of many less complex subsystems (photo by Gerry Roarty on Unsplash)

Let’s begin with a seemingly orthogonal question: Why is it that intricately nuanced and subtly microtonal musical traditions like the Indian raga, the Arabic maqam, or the Javanese gamelan, each undoubtedly much more complicated and texturally deeper than Western “functional” diatonic tonality, nevertheless never brought forth complex symphonic works along the lines of Bach, Brahms, or even Hans Zimmer?

The tentative answer, of course, lies in the c-words: It’s extremely hard to write complex music based on complicated tonalities. Arguably, Richard Wagner may be the only one who halfway succeeded, going some measure beyond European simplicity but nowhere near Eastern depth, while Arnold Schönberg went way over the top and showed that, while experimental music is fun to make, and even more fun to construct absurd speculation about, it’s no fun to listen to for most people; that’s why it has been aptly called highbrow trash.

In the same vein, it’s immensely hard to build harmoniously working complex systems based on complicated parts. Maybe it can be done—nature shows the way—but even in living systems, we have to admit that subsystems have to be somewhat simplistic, specialized, and standardized; at any rate, they have to be magnitudes simpler than the larger system. It’s foolish but admirable in a Quijotian way how modern neuroscience still tries to deduce human thought processes from the analysis of brain cells.

Conversely, if you want to build working complex systems, not to mention antifragile systems, you do well to create them out of simple parts, often called modules, which constitute the system but are less complex than the system itself. Modularity, hence, implies simplicity in the modules—standardized behavior, restricted size and complexity, complete testability, and near-perfect predictability. Just like Western music. Let me rephrase it more succinctly: Never build a complex system out of too complex subsystems. It will shatter and go to pieces. Don’t hire overly brilliant people for government agencies, don’t make cars as smart or smarter than traffic planning (I have the uncanny feeling that we’re heading there), and don’t put 300 master musicians of complicated Eastern vintage into one big orchestra.

I’d even propose that there’s an inverse law at play: The more complex your system, the simpler the components must be. Carl Orff’s Carmina Burana works well with a 200-strong choir and a 100-strong orchestra, but look how primitive the tone material is: Many children’s songs are more complicated than “O Fortuna.”

This is why superfast, decentralized, massively scalable cloud-native databases like YugabyteDB, FoundationDB, and RonDB are indeed just distributed key-value stores. While real-life databases are multidimensional monsters of complexity, a one-dimensional key-value store is the epitome of simplicity: It can’t be simplified without destroying its structure. And it’s precisely this simplicity that makes key-value stores an excellent choice for distributed storage—you have nothing to worry about except managing which key-value pairs are stored where. If you, as a clever system designer, know about the CALM theorem, you can even go one step further and create an add-only store for your application, which implies logical monotonicity and guarantees coordination-free consistency, the stuff dreams are made of.

In software architecture, to stick with the example space I’ve chosen before, things have culinarily evolved from spaghetti via lasagna to ravioli in response to the need to tackle ever more complex problems. What do I mean?

In the olden times, software applications were often simple enough that a single person could understand and implement them. Consequently, all the source code was in one place, line after line, like different-length uncooked spaghetti, hence the name. Neat freaks soon came up with “functions” and “procedures” to structure the code, then “includes” and “namespaces” to split up large files into smaller ones, and then “classes” and “objects” to help developers thinking of software components as digital twins of the things they represented, but come compile-time, it was all dumped together and turned into a cold, hard block of machine code that contained all the input, processing, and output functionalities. In essence, we have a not-so-complex system consisting of one module of equal complexity. It’s one architectural quantum that is deployed wholesale.

Enter the graphical user interface, rather quickly followed by the internet. It became soon clear that dumping presentation, logic, and data handling into the same binary was not ideal, especially as applications got larger and many developers worked on the same project. The lasagna-style layered architectures that followed often came along with many dependencies that couldn’t be “resolved” at compile-time but only at runtime: Third-party DLLs, OS calls, and faraway web APIs that could only be hoped to deliver as promised. Interestingly, they often did, for a time. But, as Toynbee quipped and Taleb echoed, past performance makes us perfectly well-equipped—for the past. Layered architectures helped us separate rights and responsibilities within the development team or even just mitigate the most troublesome architecture blunders in one-person projects, but they still had that unhelpful characteristic of a centralized architecture: Everyone must know just about everything about the whole system in order to meaningfully contribute. That just doesn’t scale well.

The brief but rightfully unenthusiastic hype of service-orientated architectures was superseded by the microservice paradigm, which still holds reign. This is the ravioli architecture I alluded to earlier, where each component is a complete, viable entity unto itself, much like a brain cell, with a clearly defined interface and behavior so constrained and pre-determined that it can be spelled out in a contract. Ideal stuff for complex systems.

Nevertheless, paradoxically it has been discovered that setting up and managing microservice architectures comes with a severe overhead that can hardly be justified for simpler projects. Coding an arcade game? Setting up a single-page web app? Layered architecture may be just right for you. Yet, the simplicity and essential independence of microservices allow not just more complex systems but also distributed, decentralized systems that are spaced out (pun unintended) to enable resilience and, in some cases, like CDNs that deliver data to users from least-distant data center locations, better performance.

Modularity vs Modularization

It’s easy to confuse the two concepts, but they’re as different as food and eating. Modularity, on the one hand, refers to the degree to which a system can be built from distinct subsystems that are smaller and more or less self-contained—the modules. Modularity is a system property. Modularization, on the other hand, is the process of decomposing a system into modules. You increase the number of subsystems and sharpen their boundaries by modularization. Modularization is a methodology.

One thing I‘ll only mention in passing, though: It‘s not enough to modularize ad libitum. You must also ensure a good way of cutting the large system into modules so that nearly all events can be handled by parts of the system without upsetting the rest. Remember, your goal is to find a stable configuration. Again, the great John Gall:

When everything correlates with everything else, things will never settle down.

One way to go about this is by clustering all p-events (events with estimated probability p or higher, as we’ll ignore the rest) into categorical clusters, then creating subsystems so that each can deal with exactly one event category without disturbing other subsystems. If your subsystems are too large for convenience, you cut them into smaller modules, of course, but note that two modules interacting all the time might often be better joined into one. Here’s the task in one sentence: You must de-correlate system behavior. Another quote I lifted from Gall’s funny but slightly eerie “Systemantics” shows by way of a World War II example what happens when you don’t:

… the engine temperature control had a nasty habit of overpowering the automatic pilot, the cabin airflow control seldom saw eye-to-ey with the engine control and the cabin temperature control came up with more side effects than some of our wonder drugs.

Benefits of Modularity

The main benefits have already been hinted at in the context of microservice architectures, but here they are in a bread-and-butter enumeration:

1. Simplicity: Each module is designed to perform a set of closely related functions, making the overall system easier to understand and manage.
2. Maintainability: When a system is modularized, maintenance becomes easier as each module can be tested and debugged independently. If a problem arises in a particular module, it can often be fixed without affecting other parts of the system.
3. Reusability: Modules can be reused in different systems if they are designed well. This reusability significantly saves time and resources in development.
4. Parallel Development: Different teams can work on different modules concurrently, which significantly speeds up the development process.
5. Scalability: Systems can be scaled up by adding new modules or replacing existing ones with more capable ones without affecting the rest of the system. Moreover, heavily stressed modules can be replicated to better balance the load.
6. Interchangeability: If a module becomes obsolete or if a better or more efficient solution becomes available, that module can be replaced without a complete overhaul of the system.
7. Cost Efficiency: Modular systems can be more cost-effective as they allow for incremental upgrades and reuse of modules across different projects.
8. Isolation of Errors: In a modular system, errors are easier to isolate and fix as they are likely to be confined to a specific module.

Downsides of Modularity

It’s one of the ludicrous paradoxes of system design that, more often than not, each promised benefit gives way to its exact opposite:

1. Complexity: The possible interactions of modules increase overhead, testing scope, and complexity of the system, making it harder to understand. Think of the WWII bomber example above.
2. Harder Maintainability: When a system is modularized, maintenance becomes harder as the interactions of modules, especially in the case of module failures and emergent unintended behavior, must be logged, tested, and debugged. If a problem arises, often it can’t be pinned down to a single module.
3. Complicated Reusability: Modules must usually be specifically tailored for the system they belong to, making reuse in practice more expensive than new development.
4. Split-Brain Development: Different teams working on different modules can easily lose sight of each other and the larger system, exhibiting behavior that isn’t beneficial for the system.
5. Constrained Scalability: Most modular systems have bottleneck modules that constrain the maximum load so that no amount of scaling can help. In IT systems, this could be the available bandwidth, the load balancer itself, or even just sensitivity to old-fashioned natural disasters.
6. Untouchability: If a module becomes obsolete or if a better or more efficient solution becomes available, that module often can’t be touched because it’s somehow essential for business continuity, and no one’s sure how to kickstart the system without that zombi module.
7. Cost Explosion: Modular systems can become harrowingly expensive since the modules, the exponentially difficult interactions between the modules, negative network effects, and numerous zombi modules (see 6) must be taken care of.
8. Emergent Errors: In a modular system, errors are often hard to isolate and fix as they are emergent, difficult to reproduce, and unlikely to be confined to a specific module.

Modularity is Necessary for Antifragility

Still, despite being expensive, error-prone, and difficult, a certain degree of modularity is necessary for antifragility. This is mainly due to the need for optionality—whenever a potentially harmful event occurs to a part of the system that isn’t properly modularized so that ensuing errors aren’t isolated and coping mechanisms like restart or failover can’t be used, the whole system is in danger.

In general, the more complex your system, the higher the degree of modularity you need, and the less complex the average complexity of the modules compared to the overall system should be. Antifragile systems improve when exposed to stressors, and modularity is optimally suited for distributing, isolating, and managing stressors.

Modularity is the necessary condition for a system to be deemed organic.

— Taleb

Next Up: Chaos Engineering

As I mentioned in my post on evolutionary design, you can provide variety and wait for trouble to make a selection, but you can also imitate the school of hard knocks and artificially create trouble to ensure module resilience and speed up evolution. This is chaos engineering. We’ll meet complex systems humorist John Gall (again) and dive deeper into chaos, statistics, and reliability engineering.