Data Mesh Can’t Win Without a Massive Culture Shift
Data Mesh is a Cultural and Technical Construct. Technically, it works. Culturally, we are failing.
Big Data London 2023 was a huge event, and the buzz this year seemed to be centralized on data mesh. But not upon its successes — more in the context of how data culture, lack of buy-in, and lack of data fluency are impacting its success.
I spent time seeking out fellow practitioners that had been through the school of hard knocks with at least one data mesh implementation. The common thread to what IS working is the data contract and the treatment of data assets as products. The common thread to what ISN’T working is the inability to gain buy in across the company, despite executive sponsorship.
We’ve all read the articles, blogs, and books about how data mesh is the next best thing since sliced bread — yet in actuality, many are struggling or have given up on the idea, touting it as a ‘data mess.’
To set the background, a quick snapshot of my recent experience at a fintech, where I led the company’s first productized deployment of data mesh…
As head of intelligence automation, I was engaged by our Chief Risk Officer to create a solution that would bring together data sets across siloed business teams to create a robust view of the profitability of customers. He didn’t care about the technical solution; he cared about the outcome — trusted, curated data for his analytics team to leverage. The promise of quality, business context, and governed use was also extremely appealing. We had the support of one of the world’s leading technology groups and centralized engineering tooling, but we were free and autonomous to experiment with the solution to deliver on his ask. We chose to take on a data mesh build.
I’m so damn proud of that team — we accomplished so much. We launched our beloved first version of a data contract via open source, gained international recognition for the speed and success of the initial implementation, and delivered our project on time and under 40% under budget. The risk and finance organizations leaned in and supported the data mesh approach. Our data governance partners loved the concept, especially the computational federated governance and the opportunity to integrate with the centralized MDM system. Most importantly, we solved the business problem, and the analyst who used our solutions loved the self-service, intuitive feedback loops, and new visibility from an observability standpoint into data quality and SLAs.
Yet our project was abandoned during a reorganization.
As I wandered the event in London last week, I sought some answers from practitioners who had taken the leap in actually building a data mesh solution.
I asked…
Why do so many data mesh implementations fail?And, the popular answers were (drumroll…)
Lack of change management,
Lack of cultural alignment, and
Slow time to show ROI
It turns out that the fintech I worked with wasn’t the only one to start, stop, and get lost on data mesh.
Industry experts on this sociotechnical revolution are playing musical chairs right now. They are highly sought after to lead the transformation but then quickly abandoned if, in mere months, they do not deliver upon the promised value of computational federated governance, experiencing and leveraging data as a product to gain ROI, self-service with business context for their analysts and new levels of ownership in business domains.
In my team’s case, on the technical side of adoption, our centralized data engineering team was not easy to convince or supportive. Why did we need this extra step beyond the vast data lakes and warehouses they had built? Wouldn’t this cause an increased cost for storage and compute? How would we measure success?
On the business side of adoption, other lines of business outside of our business use had a variety of solutions and BI tools, rogue data sets, and differing processes for compiling analytic data. The resistance to change was strong — “We have data scientists to do this,” “We have a dashboard we leverage.” Yet the analysts and scientists complained that up to 80% of their time was simply spent in finding and cleaning data for analysis.
I was recently asked if one could sell data mesh (and yes, I know there are companies out there selling it), but my answer is no. Or, at least, not yet. If we don’t address both the cultural and change management aspects along with the technical features as we implement, if we don’t implement all four principles simultaneously, we will continue to see failures.
There are some amazing tools and capabilities being built that manage components of the concept, but the vast revolution in a culture that is required to make data mesh stick is still quite a distance away for many, if not most, organizations.
That said, the companies that are succeeding with data mesh will have a huge competitive advantage — a better understanding of the business value and context data, a solid foundation for automation via ML and AI technologies, and an increased level of accuracy and reliability in their outcomes. Enterprises that are able to adapt to cultural shifts, address organizational change, and process engineering, not just the data engineering aspects of this solution, will become tomorrow’s leaders.
To create true and lasting change, we have to revolutionize the way our companies think about data. Every company will be different in where it starts, but at its core, it comes down to embracing a data-driven mindset and ensuring every piece of the organization has accountability to ensure it is adopted. It’s all about culture, adoption, and process — perhaps it’s time to shift our conversation there.
I want to hear from you — have you been a part of a successful data mesh implementation that scaled? Where are you in your journey? How did you address the cultural hurdles to success?
