Part 2 —Build a data product using Data Fabric and Data Mesh approach to learn the difference

Let’s learn the difference

I have often heard experts all over the world tossing both data fabric and mesh terms without knowing the difference. The objective of this part of the series is to share the differences from a conceptual as well implementation perspective. I will build a same data product using both the approaches to point out the differences.

Let’s learn about the differences before we start mapping it to the data platform design.

Data Fabric

NetApp’s Kurian used the term “data fabric” strategy in 2015, according to the news.

People need data to be protected, secure, integrated, orchestrated and costed, he said.

Its good to share that, Data fabric is listed as one of Gartner’s Top 10 Data and Analytics Trends for 2021

I believe, this concept is highly connected to Fabric Computing from late 1990s

NetApp’s Data Fabric Definition:

A data fabric is an architecture and set of data services that provide consistent capabilities across a choice of endpoints spanning hybrid multicloud environments.

It is a powerful architecture that standardizes data management practices and practicalities across cloud, on premises, and edge devices. Among the many advantages that a data fabric offers, data visibility and insights, data access and control, data protection, and security quickly rise to the top.

While the definition available out are consistent, lets focus on the application of it and learn it practically.

The data fabric approach to build a fraud detection data product is:

- Create a ETL job to source the data, in this case it would be network data and claims data

- Ingest and store the data to a Centralized Repository / location

- Build an API that has logic to join the tables and add rules to this layer

Data Fabric Implementation to build a data product. Pros and cons. (Designed and owned by the Author of this article)

As shown in the figure, the Data fabric approach is largely about building the “Technical integration” layer and enabling a centralized data access. Data fabric tools and technology available in the market promises no code or low code.

This approach doesn’t describe processes, who owns the data, roles, and responsibilities of who builds the assets. Thus, the data governance thread is left out which directly leads to low trust in the data.

Bottom line, the focus of Data fabric approach is on technology.

Bonus Tip: Data Fabric does sound like Data virtualization (DV)? Right? You are wrong. There is a difference and this author points it out.

Additionally, unlike fabric, DV doesn’t store data hence faces challenges in application modernization strategy. Like data fabric, it does have Data Governance challenges.

Data Mesh

I had pointed out in my article the Data Mesh background. Zhamak Dehghani, owns a large credit to talk about the approaches and discussing applications as well. Data Mesh is widely being discussed across the world since 2019. Her book talks in details about the use of a data mesh and its principles.

In summary, Data mesh principles are:

· Domain-oriented, decentralized data ownership and architecture (data is locally owned by the team responsible for collecting and/or consuming the data)

· Data as a product

· Self-service data infrastructure as a platform

· Federated management of computing resources

Lets focus on learning the difference by building the same fraud detection data product using the data mesh approach :

- At the start, define a domain team who is responsible to work towards to build the ETL or (ELT ) job ( with IT team partnership)

- Every domain dataset is stored separately, and owner is defined.

- Copy of this data is used to feed the data product

- The owner of the data product is responsible to write the logic to join these data sets.

Below figure shows the process. It also points out how data ownership is defined from the beginning. The domain owner’s responsibility is stated at every process. This shows how data mesh supports the data governance while building a huge trust in data.

Data Mesh Implementation to build a data product. Pros and cons. (Designed and owned by the Author of this article)

Bottom line, the data mesh is all about the people and processes.

The tools and technologies supporting Data Mesh are still work in progress.

Next article, I will demonstrate the hybrid data fabric and data Mesh approach. Also, its application to the data platform design.