It's been a couple of weeks since I first heard about Data Mesh. My curious mind can’t stop thinking about what that means. Will it make our data engineers work faster or more complicated? I badly wanted to taste it by diving deep into it.
Around the same time, one of the Instagram foodie bloggers that I follow posted an overloaded beautiful milkshake so colourful for the eyes that my taste buds started lusting for a scrape.
I have finally found that shop that sold this milkshake, in a mall, in the other direction of the city, far away from home.
Sitting at this food court on a lazy Sunday evening — I try to reason my tingles of curiosity about data mesh.
There are at least 50 outlets that sell different varieties of foods, all in a single place. Drawing an analogy to a data platform that we are currently building, where every unit of entity in my organisation has its data hosted on the platform. As all the outlets in the food court are accountable to the mall management, there is a governance framework that's necessary for the data platform.
To understand data mesh, this has been one of the best analogies that I was able to derive after my research.
Some background on the lineage of the term Data Mesh —
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
I was asking Google Bard and Chat GPT how the enterprise data strategies had evolved. Bard came up with some nice elucidation.
Enterprise data strategies have undergone a dramatic transformation over time, driven by technological advancements, changing business demands, and evolving regulatory landscapes. Here’s a summary of key stages in this evolution:
Early days (1980s-1990s):
Focus on data warehousing and reporting. Different departments had their own data stores. Access to data was restricted.
Transition period (2000s-2010s):
Business intelligence (BI). Data integration efforts. Master data management (MDM) solutions. Cloud computing and Big Data.
Modern era (2010s-present):
Data-driven decision making. Advanced analytics and AI. Data governance and compliance. Data democratization. Real-time data. Focus on customer experience and personalization. Sustainability and ethical considerations.
Future of enterprise data strategies:
Hyperautomation. Edge computing.
Data mesh: A decentralized approach to data management that empowers individual teams to manage their own data.
Privacy and security. Ethical considerations.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
This answer was pretty convincing for me to etch the architecture of data mesh in my dumb brain.
The Architecture
Below is a generic and simple architecture to realize a data mesh. It consists of the data platform that hosts the data of different domains and a governance framework that enables the access, management, compliance and security of the data within the domains. An enabling team can enhance the data mesh adaptability.
Domain — This represents a specific business entity or a function of an organisation. Indian cuisine can be an individual business entity in the food court.
Characteristics -
- Business aligned: Each domain is aligned with a specific business function that is responsible for understanding the data’s context and purpose.
- Ownership: A dedicated team is responsible for the data within a domain, including its collection, storage, transformation, access, management, monitoring and security.
- Data products: The domain team develops and exposes data products that provide access to the domain’s data in a consistent and consumable way.
Self-Serve Data Platform —
The data platform team will be responsible for creating a suitable environment to host the data, maintain a catalogue, access management, monitor, and establish policy automation defined by governance principles.
The Platform should enable data sharing, collaboration and data Access. This increases enterprise data agility, improved data literacy, Democratization of data and enhanced data governance.
Federated Governance —
This is an approach that’s collaborative while enforcing centralized control of the data and decentralized autonomy.
A central governance body prescribes the policies and standards for data management, covering the aspects of security, access control and data quality.
For example, the central governance body could prescribe the data retention to be 300 days. The mall governance could prescribe that each outlet could sell vegetarian or non-vegetarian varieties of foods.
The domain teams have the autonomy to implement the centralized policies and standards in a way that best suits their needs and context.
The domain team could set their data retention to 60 days based on their need by acting with autonomy. Similarly, an outlet can just serve vegetarian food and not non-vegetarian food.
I am now complete, chocolates and M&Ms in my milkshake were tasty and fulfilling. Full marks to the Instagram blogger for the recommendation.
I am too glad to clear the haze in my brain about what a data mesh is. I hope I could help you understand too. Get yourself a milkshake this year-end, you deserve a good one.
Am ready to conquer Data Mesh!
