Three More Reasons Why Data Mesh Will Fail
No man is free who is not master of himself.
—Epictetus
In the hope that you data people relish a grain of self-referential musing, I’d like to point out that I was somewhat surprised by the tumultuous response generated by the prequel of this post, Six Reasons Why Data Mesh Will Fail. Over the years, I’ve published numerous essays on this platform and elsewhere, under this name and others, all of them forceful and slightly irreverential. But Six Reasons seems to have hit a nerve as if it answered a question not many dared to ask: Is data mesh really as perfect as it looks?
Of course not. In this essay, I’ll continue to stoke the fire and give you three more reasons why data mesh will fail.
1. People Stay The Same
First of all, remember that any IT endeavor can fail in various and kafkaesque ways, but if you take the pains to dissect a number of illustrious wreckages like the Hershey ERP disaster, the 2016 Australian Census joke, and the FBI Virtual Case File wreckage, it becomes clear that, in reality, it always boils down to just three things: Bad project management, bad architecture, and bad product ownership.
Whenever testing is sacrificed to unrealistic deadlines, the scope is creeping like vines on steroids, or project control is either extremely micromanaged or virtually absent, you have bad project management in action.
Whenever quality requirements are unexamined, unrealistic, or vague, when the architecture looks like it has been made by a marketing assistant—or by a committee of pedantic bureaucrats—you have bad architecture in action. I would argue that it’s part of proper project management to verify and validate your architecture: Is the architecture style befitting the nature of the product? And has the architecture been executed well? Nevertheless, a self-respecting architect won’t wait for outside validation; that’s what architecture sparring is for.
Whenever use cases are nebulous, or no clear vision exists, or any sensible alignment with business capabilities and strategies is missing, or no internal or external customers really want the stuff, or when there’s no one who treats the product as their baby, thinking of it first thing in the morning and last thing at night, then you have bad product ownership. Products that are managed but not owned are brittle, weak, and pale from being unloved. Users do feel that.
Alas, people don’t change, at least not on average, and not that much. If you can wreck a data warehouse with bad project management, you can do the same with data mesh. In other words, no amount of ingenious conceptualization protects against the stupidity of people.
2. Data Mesh Is a Lure
I already mentioned elsewhere that data mesh has all the marks of hype; to some data people, it’s blockchain, AI, and quantum computing of the general computing world all rolled into one. As every cognoscente of the human condition will attest, you can only fall in love when you’re suffering from a deep sense of incompleteness and dissatisfaction. The data world is indeed incomplete and dissatisfying; we have makeshift data architectures and historically pullulated data flows with more manual labor involved than outsiders should be allowed to know, so it’s no great surprise that many data engineers are continually on the search for something newer, something better.
But rest assured, wherever you go, whatever you buy; one thing remains the same: You. The hype for data architecture, like any hype, is 100% emotional. Promise riches (cryptocurrency), convenience (AI), and power (quantum computer), and people will lose their heads and their money. As a data expert, however, you can’t afford to run with the flock in wild abandon over the hills of Arcady, lost in delirious ecstasy. Ask yourself: Who benefits? Who is selling what? And know your hype cycle so that you won’t be fooled and can pick what you need and discard the rest. Be the serene clear voice amidst the pink noise of the choir of enthusiasts.
3. Data Mesh Keeps Costing
By cementing the status quo of thinly scattered data and only virtually, ephemerally, feinting the illusion of data integrity through a “self-service platform”, you force potential data consumers to initiate far-reaching data integration projects for every little use case. Imagine a situation where collated data from three non-integrated and geographically distant data assets is required, and you only get some endpoints through your portal—it wouldn’t baffle anyone if a clever data engineer built a little unauthorized data warehouse with a quiet ELT pipeline so that data analysis can be done ad hoc with minimal delay, while the upper deck celebrates the purity and efficiency of the new data mesh with gallons of Moët in blissful oblivion of the reality down below.
Resource constraints are biting uglier as the years go by, free global information trade is far from certain a decade from now, and prices for bandwidth and cloud computing will happily inflate in conjunction with the sickly world economy.
By not only accepting but actively increasing the complexity of already hypercomplex distributed systems, data mesh pours poison into an increasingly unhealthy data landscape. Consider complexity as a form of technical debt, where the amount of manual drudgery, hotfixing, and firefighting is a measure of the interest you pay. As with any debt, there comes a point where debt service eats up all your capital, and there’s nothing left to refactor, invest, or even party. Data mesh architectures address certain challenges at the price of higher complexity, and the curse of complexity has been the downfall of many cultures before.
As usual, I’d like to point out that all is not bad with data mesh. There are good things to be learned, for instance, the vision of data as a product and the idea of bottom-up responsibility. But still, be warned that it’s hype bordering on ideology that isn’t even practical in most situations. Don’t get suckered into it.
