Data Mesh Implementation in A Multi-Cloud Architecture

Photo by Ricardo Gomez Angel

This blog is inspired by Zhamak Dehghani’s article — ‘Data Mesh Principles and Logical Architecture’. If you are new to Data Mesh, please read her article as a precursor to this blog. Zhamak, in her article provides a technology-agnostic, high-level, logical architecture model for Data Mesh and its underpinning principles. This blog tries to move the paradigm forward by explaining how Data Mesh can be implemented in a multi-cloud architecture by using a hypothetical use case in a healthcare setup to manage the current COVID-19 pandemic.

Before we dive into the use case, let us remind ourselves about the 4 core principles of a Data Mesh architecture.

1. “Business domains bounded context is a good candidate for distribution of data ownership.” In simple terms create separate systems and teams, one per business domain. Each Domain must be responsible for both Operational and Analytical Data. The domain’s interface to the rest of the organisation not only includes the operational capabilities but also access to the analytical data that the domain serves. These interfaces can be in the form of Operational APIs and Analytical end points.
2. “Domains Analytical Data plane should be considered as a product. Data mesh implementation should support discoverability, security, explorability, understandability and trustworthiness, for a domain data to be considered a product.”
3. “To support data as a product that domains can autonomously serve or consume, it should encapsulate three structural components required for its function — Code, Data & Metadata and Infrastructure.”
4. “A federated governance model that facilitates decentralization, domain self-sovereignty and interoperability through global standardisation.”

Fig. 1.0

Now that we have reminded ourselves of the core principles, let’s start with a high- level information flow within our use case. To contain the spread of coronavirus, people are encouraged to get tested as soon as they develop any related symptoms. They need to register for the test and then provide a sample for the testing labs to conduct the necessary tests. The registration data and the test results are then combined together along with some data cleansing and sent to a contact and tracing team to encourage the people who have tested positive to come forward and provide information about their recent activities. If the person who has been tested positive (also called a case) informs about other people with whom he or she has met in the recent past (also called a contact), then those people are also contacted and advised to self-isolate.

To implement the above in a Data Mesh, the following domains can be identified:

1. A domain for test registration.
2. A domain for the labs that receive samples and manage test results.
3. A domain that links the test registration details and the test results together and also performs some data cleansing e.g. address validation.
4. A domain that contacts people who have tested positive (a case) and also contact people who have been exposed by the case (contacts).
5. A domain that collects business glossary of terms from all other domains to create a data catalogue and also create a data lineage between the domains, providing a holistic picture.
6. A domain that can do cross functional analysis by analysing data from all other domains to help the Government take more informed decisions.

Fig. 2.0

How do domains communicate with each other in a Data Mesh?

The domain that links the registration data with the test results needs to receive data from these upstream or source domains. This integration can be via any of the exposed methods available in the upstream domains. This can be via a REST API or a message bus. You can use Open Data Protocol (OData) for querying data over the web. You can create APIs that authenticate via a JWT token and then further authorise the caller based on Row Level Security implemented in the Data Product’s database. If the source domain provides a web-hook, then the consuming domain can provide a call-back URL. For messaging type of integrations both the source and consuming domains can adhere to the defined standards for interoperability in Healthcare e.g. HL7- FHIR. For data cleansing, this domain might subscribe to a 3rd party address validation API outside the Data Mesh.

The domain that does the cross functional analysis might need historical data from all the upstream domains. This can be done by using federated query that uses data source connectors and run on a server-less compute to connect and retrieve data from domains running on-premises or hosted on other cloud platforms. Remember, each domain provides good quality operational and analytical data and exposes the associated metadata.

The domain that maintains the business catalogue and data lineage between domains has a two- way communication with all other domains within the Data Mesh. It scans the metadata of each domain via one of its exposed methods, to create a data catalogue and data lineage between the domains and then exposes this along with other information back to all domains in the Data Mesh for their consumption.

Challenges in implementing a Data Mesh

This paradigm shift towards domain-oriented decentralization, brings with it some new challenges. Not much has been defined or discussed about these and have been left to one’s imagination.

1. How do you manage Master Data Management in a Data Mesh? Remember, there can be entity resolution issues between datasets of different domains.
2. What about clients who have already invested huge amounts in creating a monolithic Data Lake and\or Data Warehouse?
3. How do you manage incompatible data between domains and increased cost of operation?

Summary

Data Mesh is not a new technology. It is a paradigm shift in the way we think and re-organise our systems and teams into business domains. On one hand these domains are independent, autonomous and experts of their bounded context as they are closest to the data and on the other hand they interoperate and corelate to form a harmonious eco system called Data Mesh.

This blog is written with the intention of a follow up. The next blog will elaborate other aspects of Data Mesh such as self-serve data infrastructure as a platform and federated computational governance. Stay tuned!

Acknowledgements

I am grateful to Giles Cuthbert, Neil Parker and my other colleagues in the Insights and Data team, Capgemini’s analytics practice, for their invaluable review and feedback.

We’re always on the look out for new talent to bring exciting ideas and ways of thinking. Check out our open roles to see if there could be a match for you.