Communications and Conway’s Law

In my last post, I described what I see as a disruption in the way digital businesses will operate in the future:

Part of that process — and this is what really excites me — is the increasing use of data and APIs to break down communication issues between organizations without requiring the people in those organizations to explicitly communicate with each other. In other words, we can create a communication backchannel that avoids much of the politics, bandwidth constraints and simple inefficiency of human-to-human communication.

I want to take some time to explain in more detail what I mean by that statement, and to explore how this might come to be.

Communication in systems

To start with, it is important to understand how a complex system is affected by the communication patterns between its component parts. A complex system is loosely defined as a large number of agents interacting in dynamic ways producing nonlinear outcomes. Another way of saying “interaction” in this context is communication — the exchange of various forms of information through any number of appropriate mechanisms.

If that last paragraph got a little dense for you, think if it this way. How does our atmosphere get organized to create a hurricane? How does a rainforest sustain its ecosystem when there are so many species of plants, animals and insects involved? How does an economy adapt from a regional focus to a global focus without succumbing to new competition?

The answer is that the different components of these systems are constantly interacting. In the case of the hurricane, these elements are each simply responding to the laws of physics, but in the case of the rainforest and the economy outcomes are being effected by the selection of behaviors based on desired outcomes — the rainforest via natural selection, the economy via human decision making.

Communication — again, the exchange of information — takes place in all of these cases. Without communication, these systems wouldn’t develop and evolve as they do.

Conway’s Law

One fascinating outcome of communication within large distributed software systems integration (even —no, especially — “traditional” IT application environments), is the effect that human communication patterns has on the evolution of how the software applications and subsystems communicate.

This correlation was described succinctly by computer scientist Melvin Conway as thus:

Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.

In other words, software interfaces will be defined in close alignment with the communication patterns that exist amongst the teams building the software.

Makes sense, when you think about it. If team A has a project schedule for the functionality for which it is responsible, and team B has its own schedule, and so on, the only sane way to maintain those schedules is to keep separation between the components being built by each project team.

(I wrote about Conway’s Law specifically with respect to performance data on SOASTA’s blog in March. Worth a read if that interests you.)

Microservices and Conway’s Law

Now, this is where things get really interesting. Imagine breaking down projects into smaller and smaller pieces, with fewer and fewer people involved in each piece. And imagine placing each piece on its own, self-regulated delivery schedule. Individual projects get easier and easier to manage, at the expense of increased communication complexity.

This, of course, is exactly what is happening with so-called “microservice” architectures. It’s a growing development trend, and an architecture being built in to most of the popular cloud platform services today. Given Conway’s Law, shouldn’t microservices either fail to scale due to organizational constraints, or break organizations up into an unmanageable mess?

The answer is two-fold. The first is that organizations that are successful with microservices remove some of the bureaucracy of traditional IT organizations. Netflix, for example, talks about making developers themselves responsible for outcomes, and trusting that they will do the right thing. Others take a variety of Agile and “two-pizza team” approaches to the problem, with oversight from management with specific expertise in coordinating software service delivery.

But the other aspect of why microservices can work in large organizations has to do with the way key information is exchanged between individuals or teams developing services, and individuals or teams consuming them. Great microservices are built with well defined, intuitive (or at least easily learned) interfaces for documentation, support and even the services themselves.

HTTP plays a huge roll here. Some of those interfaces are human user interfaces delivered via web or mobile, but others are REST or RPC APIs that are either self-documenting or have easy to find and interpret documentation. Increasingly, these interfaces include data streams that deliver real time representations of measurements, events and/or alerts. It’s this last type of API that interests me most.

Data and Conway’s Law

Remember, one key phrase in the definition of Conway’s Law is “the communication structures of these organizations”. If you change the way an organization communicates, you change the potential systems structures that organization produces.

By providing information, often in real time, as a service, the concept of smaller teams delivering key capability to a variety of other small teams becomes increasingly viable. The communication structure of the organization gets decoupled from the human network that builds and operates software — and even digital business models.

The implications here are huge. First of all, if you include IT operations systems as services that can stream data, you can decouple ops data from ops personnel. For one thing, this makes it much less risky to explore new correlations within a digital business system. Which makes alignment of operations success to business success much easier to achieve.

There are a myriad of other examples of data streams generated by one business unit that can be consumed and analyzed by another. If the different units of a business can consume those data streams without requiring human negotiation and bureaucracy, digital business will evolve in ways we probably can’t even perceive today.

But wait, there’s more…

While I’m excited about what this composable digital business model might enable, I’m also acutely aware of how new models can open up new social, economic and ethical risks that can cost more than is gained. I will address this much more in future posts, but don’t take my enthusiasm as a belief that data flow and data composition is a silver bullet.

(Alistair Croll spends a lot of time thinking about this. Check out his Twitter stream for more.)

On the other hand, I am excited about how much business data flows start to resemble the problem being addressed by the Internet of Things (IoT) industry today. Real time stream processing, machine learning and pattern recognition, and even data provenance all apply to business data flow as much as they do device data flow. In a sense, we are looking at a future of IT as IoT.

As always, let me know what you think, either via the mentions section below, or on Twitter where I am James Urquhart.