Complexity in a Data Value Chain
Complexity is tough for humans to handle.
In 2007 a HBR paper broke down decision making into a model with four domains “chaotic — complex — complicated — and simple”. While the whole paper on the Cynefin model is worthwhile reading, it had this in particular to say about complexity: “the complex domain is much more prevalent in the business world than most leaders realize — and requires different, often counterintuitive, responses”
For example, there’s been a lot of noise recently about logistics and supply chains.
And most of it is entirely wrong.
Modern supply chains are complex.
Just ask this guy who tried to make a toaster from scratch. TLDR: it cost him nearly $2000USD and took 2 years to build a toaster that would cost £3.99GBP at his local store. It also didn’t work very well… or at all really. All the current political rhetoric about “make everything locally” or “everyone should keep inventory” is basically wrong, or implausible.
A similar problem besets the technology industry.
Companies are often lured by the bright lights and slick pitches of product vendors to find the “right tool” to solve their problem. Only the right tool doesn’t exist, or has shortcoming or only solves part of the problem, or works only in one context, but not in another.
Turns out software systems are complex too. Who knew?
But how to manage that complexity?
This is what the Cyenfin framework has to say on tackling complex problems: Leaders who try to impose order in a complex context will fail, but those who set the stage, step back a bit, allow patterns to emerge, and determine which ones are desirable will succeed.
But how do you “set the stage” to allow patterns to emerge?
Models
This is where “models” are useful.
A model is just a simplified representation of reality that allows people to understand it better. Think of an architects model of a house, or a cartographic map that models terrain.
Modern software systems are simply too complex to understand in their entirety. No one person can keep all of a complex architecture in their head and most struggle to even render it on paper or a whiteboard. The industry is littered with figleaf models like “Software as a Service” or “Event Driven Architecture” — each is an abstraction which illustrates a specific design principle.
So to wrestle with complexity we need to choose the right model.
But not all models are created equal.
The Gartner model for example is a model that is often used in technology strategy but has (rightly) had a lot of derision heaped upon it. It simplifies things to the point of banality — top quadrant good, bottom quadrant bad.
In response to models like Gartner’s, Simon Wardley developed a series of mapping tools in about 2008. Now known as the “Wardley Map” it depicts a value chain (usually an IT value chain but it can be anything) on the vertical access and an “evolution of service” on the horizontal axis.
Goods and services Simon argued tend to evolve from the realm of experimentation and bespoke solutions to more repeatable, reliable products and ultimately commodities. .
Here’s the Wardley Map for making a cup of tea:
And here’s the Wardley Map for an online Photo Service:
[diagrams from Simon’s blog, Creative Commons Attribution-ShareAlike 3.0]
Unlike the Gartner quadrants, the axes on a Wardley Map are continuous thereby avoiding the good-bad dichotomy (and offering a level of nuance not normally present in strategy)
You can do a lot of interesting things with Wardley Maps. You can posit “to be” scenarios and analyse what it might take to get there. You can survey a group to find out the diversity of opinion in the group (or lack of alignment). Just as a tool for building consensus it is fairly powerful.
But one of the most interesting features of Wardley Maps is the ability to identify common components and group them into teams.
[ Wardley Maps Medium, Creative Commons Attribution-ShareAlike 3.0 ]
Simon makes a reasonably strong argument that different components at different levels of evolution require different organisational attitudes. Things that are evolutionary or experimental need “pioneers” who are happy to forge ahead with little structure or guidance. Components that are more mature and reliable (products) require a “settler” or farmer mentality who can “turn the half-baked thing into something useful for a larger audience”. And finally more commodity scale components demand “planners” who can industralise things at a scale that others can’t even comprehend.
The Data Value Chain
Where I work, we do a lot of work on platforms and specifically data platforms. To that end, we’ve developed a Wardley model of what we think is the current industry practice for a good data platform.
But more than just depict a target state, you can use Wardley’s to do easy current/target state comparisons (or gap analysis if you prefer).
And you can do “what-if analysis” on various scenarios.
The ideas for the “ideal state” architecture depicted at the top are based on an analysis of some emerging models in data architecture — notably domain driven data architectures.
Domain Driven Data
Domain driven data architectures are probably best represented in “Data Management at Scale” by Piethein Strengholt.
This book eloquently paints a picture for not only a practical implication of domain driven principles in a data architecture, but also specific practical architectural patterns, including how to implement things like Command-Query-Response (CQRS) pattern on a large scale data platform.
A lot of the domain driven concepts have been bundled up and rebadged by people like Thoughtworks as a “data mesh” — a term which is getting some traction in the industry.
Whatever you call it, the basic concept is to hand responsibility for the data products to the “domains” — effectively to subject matter experts close to the business or end user who can service the needs quickly, but also understand the context of the data. Centralised teams or systems are just too much of a bottleneck to productivity to keep up with demand.
This is achieved by building a reliable and scalable platform which federates some responsibilities and obfuscates technical complexity from the domains.
Organising Your Capability
While that’s fine from an architectural perspective, building platforms is hard.
How do you organise your teams to deliver a successful platform at scale?
Conway’s Law says :
Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.
— Melvin E. Conway
For communication structure, you can broadly read “organisational structure”.
So the architecture of your systems grows to resemble the organisational structure of your business.
In response, someone coined the phrase “inverse Conway maneuvre” to suggest that you can achieve a desired architecture by reorganising your teams to deliver the result you want. While the jury is still out on this one, it’s certainly true that the wrong structure will not deliver a scalable or reliable architecture.
So we need a good organisational model for a (data) platform.
There’s a book for that too.
Matthew Skelton and Manuel Pais’ “Team Topologies” is a model for ‘traditional’ organisations to build an organisational structure along the lines of a modern software company. It optimises organisational structure for the “fast flow” of software changes.
The basic principle of target state for team topologies is depicted here:
- Stream aligned teams move with customers, responding to their needs
- Platform teams build self-service platforms that enable the stream aligned teams to move fast and not break things
- Enabling teams help stream aligned teams adopt platforms and industry leading practices, and act as a conduit of needs back to the platform
- Complicated subsystem teams take the burden of specific undifferentiated heavy lifting from stream aligned teams so they can focus on customers.
The book also offers a lot of helpful advice on interim state models to allow you to transition from a more traditional structure to one based on modern software teams.
It makes a lot of sense in this context, since a modern data platform is just another software platform, and one heavily reliant on good software engineering practices. In a domain driven data architecture, stream aligned teams are domain aligned teams.
So how would you depict the resultant architecture?
Luckily, it turns out that someone has already had a crack at rendering that:
https://www.datamesh-architecture.com/
The clever people at INNOQ have applied team topologies thinking to a data mesh and come up with the diagram above:
- Domain teams are stream aligned teams, producing data products
- They are supported by a team that delivers a self-service platform through critical shared services like a single access layer, automated compliance, monitoring and a data catalogue.
- An enabling team helps the domain teams adopt the platform and improve their data products through sharing patterns and consulting.
- And a federated governance model is delivered by a community approach to policy where the rules are agreed between the domains and the platform team and implemented as automated rules, applied by domain teams to their products.
While this is insightful enough, it is a static model — a target state.
It’s a helpful depiction of where you want to be, but not necessarily how you get there.
Where do you want to go?
How do you get to somewhere you want to go?
Well, usually, you use a map.
And that’s where we can bring Wardley Maps back into the picture.
Using Wardley Maps, domain driven data and Team Topologies we can combine them to produce a unified model for data strategy.
By looking at where the components of your data platform currently sit, and where they should sit, you can determine what team or group should be responsible for them. And by performing an “inverse Conway” you can start to reorganise your way towards a better system.
Conclusion
Models are useful but finding the right model is more useful still.
