Demystifying data mesh as a new paradigm in data management
Written by Aparna, Content Creator at Systech Solutions Inc
Data is the critical asset of an organization today. Organizations thrive on data-driven solutions today. Harnessing data to derive business decisions is essential to understand customers better and stay competitive in the market.
Availability and accessibility of data at scale, in an enterprise is essential to accelerate business value outcome. Hence, data management is indispensable to optimize data-driven decisions of an organization. Accessing data across business segments with domain understanding is essential to derive holistic meaningful insights. Data needs to be available, accessible and discoverable in a secured space to interoperate.
The buzzing new architectural principle to get there is with Data Mesh.
What is Data Mesh?
Data Mesh is an ideation to decentralize and democratize data across an enterprise with centralized governance for interoperability.
A domain-driven approach treats data-as-a-product to improve responsiveness within the organization and significantly improves user satisfaction.
Each business function or business line owner in an organization creates, manages and exposes their domain specific data as “data products” in a central repository. This data product is available for rest of the organization to access.
The 4 principles of Data Mesh
- Domain Ownership
- Data as a product
- Self-serve data infrastructure
- Federated computational governance
Each business function or business line owner in an organization creates, manages and exposes their domain specific data.
The domain specific data is available as “data products” which are published data sets, owned by the business function owner. This data product is available for the rest of the organization to access.
Self-serve data infrastructure empowers business users or end users to access cross-domain data independently with reduced data movements in an enterprise.
Distributed ownership between domain owners and central repository enables adherence to governed data interoperability.
Key benefits of Data Mesh
Data availability & accessibility at scale
Data products across business lines co-exist in a data catalog. This makes accessing them with limited data movements within an enterprise possible without expert data team’s involvement.
Legitimacy of data
Domain-data-ownership enforces data reliability for cross-functional analysis. It makes it possible for a finance team to consume marketing data product for their analysis and be able to trust the marketing data as it is from the marketing domain owner.
Improved user responsiveness
Higher degree of collaboration and improved accessibility empowers users to have control over their data environment. This makes them responsive to service requests of the customers being able to access, analyse and operationalize insights from any location.
Time to value
Independence in accessing right data and having control over the data environment is essential to explore and experiment. Incrementally working on data products or fragments makes it easier to foster innovation at scale.
Implementation challenges of Data Mesh
Organizational paradigm shift
To create a distributed domain driven architecture, it is essential to clearly identify and segregate the domains or the logical unit of an enterprise. It is also essential to develop the thought process to treat data as a product.
Make data products discoverable
All the data products available for access should be discoverable with a data catalog. This enables users to know what is available in an enterprise for any analysis.
Make data interoperable
Data from different domains must be able to correlate and bind together in a distributed data architecture. This demands a self-serve interface to facilitate users to create data assets.
Manage data governance
Decentralizing data with centralized governance at a right strike of balance is the key to the mesh architecture. With centralized governance standards and autonomy in accessibility of data assets there is good control in the quality of data management.
Data Mesh with Dopplr
The elementary piece of Dopplr, is the Enterprise Data Catalog. It is a storehouse of enterprise data assets from all of its business segments.
Dopplr provides the business users, a self-serve workspace to create their own data sets in a platform available across the enterprise.
The interface allows domain teams to ingest its operational data and build analytical data models for their analysis is available. Users can also build data products for other cross-domain analysis on demand.
With appropriate governance in place, the data products are available to interoperate for cross-domain analysis that drives an organizations unified goal.
Data Mesh as a solution
Data Mesh is a good approach to enhance the data literacy of an organization. Data fluency combined with right tools and techniques can create a decentralized, self-serve platform that keeps the user responsiveness much faster and results in elated customer relationships.
