SSENSE Data Mesh: Going From Vision to Value — Part 1 of 2
“If we have data, let’s look at data. If all we have are opinions, let’s go with mine.” ― Jim Barksdale
At SSENSE, we want to further unleash the value of data in order to increase agility, drive innovation, and improve efficiency. For us, data is the gateway to new opportunities and improving our clients’ experience. It’s limitless in use and the amount generated is increasing at an exponential rate, largely due to the growing number of business models and the percentage of customers subscribing to our platform. While data is abundant and increasing at a rapid rate, we face the challenge of getting value from data at scale. Simply producing or storing it doesn’t automatically generate value. Value is realized by creating a culture and operating model that uses the data to invent on behalf of our customers and partners using actioned insights, analytics, and AI/ML. Getting the optimal value out of data occurs when you transition from data-aware to data-informed to data-driven.
In this article I will touch on the motivation and pressure points to move towards data mesh, the divide between the operational and analytical data world, the inflection point for SSENSE, and the fundamental principles of data mesh.
Data-Aware to Data-Informed to Data-Driven
Conversations about data-driven enterprises often focus on big data tools, and the breakthroughs that have made storing, processing, and analyzing data faster and cheaper. While these are all important, creating a data-driven culture across the organization is essential to go beyond just a few successful data initiatives and islands of excellence limited to certain business areas.
The SSENSE Data Platform is responsible for establishing innovative methods for accessing, governing, and interacting with our organization’s data. We make data/information more discoverable, functionally usable, understandable, trustworthy, interoperable and valuable through advancements in data analytics adoption, data governance and quality. The intent is simple: to collect data and use it intelligently, turning data into actionable insights quickly and cost-effectively.
While the intent is very clear, we also face two pressure points as described by Zhamak in her original article. The architectural and organizational structures of a centralized data platform lead to many challenges and failures mainly because of the following:
- Ubiquitous nature of data and its ability to consume all in one place
- Organizations’ innovation pace and their need to experiment with large numbers of use cases
The centralized model worked for us when we had a simpler domain structure with a smaller number of diverse consumption cases. We started to see cracks as our enterprise domains became richer, with a large number of sources and a diverse set of consumers. Data collection, experimentation, and intelligence were outsourced to a centralized data team. This created a bottleneck for the entire organization which was in dire need of the data. We were playing catchup, either chasing the data pipeline havoc caused by changes in upstream applications and their databases or trying to meet the needs of the domains requiring a data solution asap.
We also faced challenges across the business domains regarding the validity of the data, or their inability to find the data they needed. The lead time and friction to get to the right data made it difficult for domains to dare imagine new experiments. There was a misalignment between producer and consumer needs. Lack of consumer autonomy and lack of data ownership and accountability also contributed to non-optimal outcomes.
Operational Data vs Analytical Data
Many of the technical complexities organizations face today stem from how we have divided data — operational and analytical data: how we have siloed the teams that manage them, how we have proliferated the technology stacks that support them, and how we have integrated them. Data mesh focuses on analytical data. It recognizes the blurry delineation of the two modes of data, introduces a new model of tight integration of the two, and yet respects the clear differences between them. “What is operational data versus analytical data?” This has been a point of confusion for early enthusiasts of data platforms and data mesh. I think it is important to clarify what these terms mean.
Inflection Point
We reached an inflection point and had a choice to make: continue with our existing path and make compromises with our existing centralized model, or take a radical approach, question our beliefs, and adopt a new paradigm. We believe Data Mesh is specifically designed to fit our needs and SSENSE is uniquely positioned to leverage this new mode. We have already adopted a micro-services or domain-oriented architecture, and data mesh is a relatively simple extension. We have built services based on the domain bounded contexts. Now, we will have to apply the same decomposition and modeling to analytical data in each domain.
Data Mesh
Data mesh is what comes after an inflection point, shifting our approach, attitude, and technology toward data. Mathematically, an inflection point is a magic moment at which a curve stops bending one way and starts curving in the other direction. It’s the point where the old picture dissolves, giving way to a new one.
Before we dive into how SSENSE is planning to adopt data mesh to drive growth, it would be worthwhile to understand the basic concepts of data mesh.
What is Data Mesh?
The term data mesh was coined by Zhamak Dehghani in 2019 and is based on four fundamental principles with a very clear vision to address the challenges we have in centralized data platforms.
“Data mesh is a decentralized socio-technical approach to share, access, and manage analytical data in complex and large-scale environments — within or across organizations.” — Zhamak Dehghani
Outcomes
Data mesh sets out to achieve the following outcomes:
- Respond gracefully to change: a business’ essential complexity, volatility, and uncertainty
- Sustain agility in the face of growth
- Increase the ratio of value between data and investment
The Principles
Principle of Domain Ownership
- Business domains should have the autonomy to release and deploy their operational or analytical data products.
- Increasing data business truthfulness by closing the gap between the real origin of the data, and where and when it is used for analytical use cases with the ability to scale with agility.
Principle of Data as a Product
- Conceptually, a mesh is a graph, a network, consisting of nodes and connecting edges. Each node in a data mesh is called a data product as its architectural quantum.
- The single-threaded owner of the data seeks alignment across engineering and product, and treats data as a product instead of an asset.
- Data as a product adheres to a set of usability characteristics:
- Discoverable
- Addressable
- Understandable
- Trustworthy and truthful
- Natively accessible
- Interoperable and composable
- Valuable on its own
- Secure
Principle of the Self-Serve Data Platform
- Simple to use data tools using a common infrastructure framework.
- Diminishes the complexity of data management by introducing common patterns and designs.
- Reduce the cognitive load and total cost of ownership of tools and data of domain teams in managing the end-to-end life cycle of their data products.
Principle of Federated Computational Governance
- Virtualize access to data in a secure and governed way.
- Making it feasible to build in cross-cutting governance requirements such as security, privacy, legal compliance, etc., across a mesh of distributed data products.
Conclusion
The key takeaway here is that data mesh is primarily an organizational change. The responsibilities of analytical data are shifted closer to the business domains. This enables faster data-driven decisions and reduces barriers to data-centric innovations. In part 2 of this series, I am going to specifically talk about the paradigm shift we are making, how we are planning to design a data mesh, the tech stack and the challenges we may face during the course of the journey, and how we could mitigate them.
References
- Dehghani, Zhamak. Data Mesh. O’Reilly Media.
- https://martinfowler.com/articles/data-monolith-to-mesh.html
- https://martinfowler.com/articles/data-mesh-principles.html
- https://www.thoughtworks.com/en-ca/what-we-do/data-and-ai/data-mesh
- https://aws.amazon.com/executive-insights/content/how-do-you-become-a-data-driven-organization/
Editorial reviews by Catherine Heim & Mario Bittencourt
Want to work with us? Click here to see all open positions at SSENSE!
