Part 3 — How to use the Hybrid Data Fabric & Data Mesh Framework
In my last article we discussed the difference between data fabric and data mesh.
In the nutshell, data fabric is technology focused whereas the data mesh is all about people and processes.
Hybrid Data Fabric & Data Mesh Framework
Most data ecosystems build data products that are aligned to the data strategy. This list of data products helps choose whether to use a fabric or mesh approach. In majority of the scenarios, a combination of both operations data and analytical data products are needed.
A set of data warehouses, data marts, core data at master data layer and reference data set are the essential part of any well-defined data eco system.
As per the best practices and available tool options in the market, we must build these products as a centralized store. This is what a fabric tool or technology in the market is capable of.
But does that mean we will have to compromise on data governance, domain driven data ownership, which enables speed, ease of use and other important mesh principles. The answer lies in your organization’s data strategy.
If it focuses on the “balance between the Technology and people” then you need the hybrid data fabric and data mesh approach.
The hybrid framework design supports the data platform — data products that benefit from both data fabric technology and has data mesh principles. Depending upon the need of the organization's designing the platform one can combine the features to build the hybrid framework.
Let’s see how to apply the hybrid framework to the data platform design.
Apply hybrid Framework to Data Platform design
The diagram shows the design of the fraud detection data product using the hybrid data fabric and data mesh framework. Lets consume the data from the consumption layer upto the source layer( right to left)
- Fraud detection data products at analytics or consumption level is defined by the data product owner.
- Master data for clients who’s claims in the system populates from a multidomain MDM
- Historical data is populated from the data warehouse or mart. The warehouses or marts contain historical data , transactional data that is need for the data science models. This layer also does a lot of aggregation too. So, think of them as a single source of truth. Having a set of centralized data warehouses do help reduce silos.
- For the operational data needs, the operational data store, populates the near real time data.
- All the data needed for these layers is fed by a landing zone which has stored it in a de centralized manner.
- The event bus could be used for collecting real time data and then storing it to feed to the batch based pipelines.
This is how you can use both domain drive ownership, de centralized data principles from mesh and use fabric tools to build centralized data stores which are fed by these domains drives data sets.
The benefits of using the hybrid design are multi fold.
From the risk, people & process perspective , data Governance gets simplified, lineage becomes easier.
The leaner the data flow becomes the faster it can reach to the advance data products.
This design also solves critical challenges that the data mesh principles introduces. We will focus on the challenges and recommended solution in the next article. #hybriddatafabricdatamesh
