Deconstructing Data Mesh Principles

How the principles of data mesh define the reference architecture and operating model for a new world of data

By Saurabh Kumar and Will Henham

Photo by Alina Grubnyak on Unsplash

Organisations struggling with managing their monolithic data platforms have begun investigating a data mesh architecture and operating model to become more nimble and outcome focussed. However, many are grappling with this new concept of data mesh and how it should be implemented in their organisation.

Data mesh architecture tries to decompose the traditional data monolith by domain ownership and let organisations share data across domains. But is this enough to call your new decentralised platform a data mesh?

This highlights the first principle of data mesh: domain-oriented ownership. However, there are four principles that must work together to address the side effects of this change.

In this article, we’re addressing how these principles are interconnected and work together to provide an ecosystem that can scale out, both as a platform as well as an organisation. As we explore these connections, we’ll develop a conceptual reference architecture and operating model that translates the four principles into a workable future state.

The four principles of data mesh

In her design of the data mesh paradigm, Zhamak Dehghani outlined four key principles for the establishment of a data mesh solution. In our analysis, we found that each principle introduces a set of consequential side effects that are addressed by subsequent principles. The below diagram shows how these relationships offer insight into the mechanisms of a real-world implementation of data mesh.

The four principles of data mesh

Principle 1: Domain-oriented ownership

As your data platform grows — along with the ecosystem producing and consuming data — it suffers from scaling in. This becomes more complex as the diversity of use cases increases, putting pressure on both the platform and the organisation structure. What we really want is to scale out horizontally, using the first principle of data mesh: domain-oriented, decentralised data ownership and architecture.

To decentralise the monolithic data platform, we need to reverse how we think about data, its locality, and ownership. Instead of flowing the data from domains into a centrally owned data lake or platform, domains need to host and serve their domain data sets in an easily consumable way.

Domain oriented decentralised data ownership

In principle, each domain should:

• Have ownership of all data within its domain. This ownership extends to considerations such as access control, data quality modelling, and consumption patterns.
• Be able to develop an end-to-end data solution. This includes considerations for architecture and supporting technologies, such as cloud resources.

The operating model implication of this decomposition is that it requires a set of multi-disciplinary teams that can operate independently of one another. The exact skill sets required for each domain should be determined by business use cases. However, this team would typically need the following roles:

• Product Owner — This person takes accountability of the data product, data access control and Service Level Agreements (SLAs), and the point of collaboration between domain teams
• Data Architect — This team member develops the solution architecture for the domain team.
• Data Engineer — This is the person that develops the pipelines and logic for the data domain team.
• Data Modeller — This team member leads the development of the data model to meet business needs.
• Data Steward — This person performs cataloguing, access control, governance, and metadata management for the data domain team.

Data scientists, software engineers, ML engineers, and data analysts may also be appropriate inclusions depending on the data consumption patterns made available by the domain. Since the capabilities are intended to be shared across data domain teams, this team does not usually have Cloud, DevOps, or security engineers.

The decomposition of a monolithic data platform into data domains is similar to the decomposition of monolithic applications into a microservice architecture. Therefore, it’s appropriate to consider Domain Driven Design (DDD) principles when completing this activity.

This approach is both complex and nuanced, so it’s important not to underestimate how challenging the initial decomposition of the data monolith will be for your organisation. It’s important to consider the effort and complexity of this activity as part of your organisation’s strategy before implementing a data mesh architecture.

Side-effects of this principle

Even after decentralization, data is still siloed within the domain, causing concerns over accessibility and usability of the data. This model leads to duplicated efforts in each domain, including data processing pipeline implementation, technology stack, and tooling.

Principle 2: Data as a product

As mentioned above, the decomposition process raises concerns around accessibility, usability, and siloing of datasets. This is where the second principle of data mesh — data as a product — comes into the picture. Exposing data as a product enables data consumers to easily discover, understand, and securely access data distributed across many domains.

At a minimum, the data product should have the following characteristics:

Below is an illustration of a mesh of interconnected data products, demonstrating how they integrate with other products within the domain teams, as well as across the domains.

Mesh of interconnected data products

Design considerations:

1. A domain team can instantiate its own set of infrastructure which is called a ‘producer’. The output of a producer is called a ‘data product’.
2. The product-relevant data, code, and pipeline are encapsulated within the producers. At the mesh level, we only see cleaned and readily consumable data products.
3. Domain ownership in this model is flexible enough for a domain to own one or more producers and associated data products.
4. The producer publishes the data product via a product interface (integration pattern) defined at the mesh-level. The product interface could be database, table share, API endpoint, pub-sub solution, or a data marketplace, among others.

It’s helpful to categorize data products based on the business need they serve, illustrating the expected outcomes of each product in the mesh. These two categories are:

• Source aligned products
  These products are closely aligned to source applications, with an intent to make operational data available to the mesh with minimal alterations. The data can be stored and served as distributed logs of time-stamped events for any authorised consumer to access.
• Consumer aligned products
  These products are designed to meet a specific set of end-user requirements. The data can easily be regenerated from the source.

Data product development operating model

If we treat data as a product, then the development of a data solution will follow a typical product development cycle. This typically involves the following activities:

Data product development life cycle

As a result, the product owner and domain team will need to consider the following questions when designing and building their data products:

• Who is going to be the customer of this product?
• What business outcome does this product achieve? And how do we measure the success of the product?
• How will people consume the product?
• How much will building the product cost in both time and resources? Is this the right use of these resources?
• Who are our suppliers (in terms of raw data and supporting technology) and what dependencies do we have on them?
• How critical is this product going to be to the organisation? What level of support will we provide going forward?

A robust operating model should address these questions through well-designed processes and ensure that teams have the right capabilities to execute them. However, given the intent to have multiple data domain teams operating independently of one another, teams should develop their own internal processes, governance, and unique culture within the larger operating model framework.

Side-effects of this principle

There is a heavy cost of ownership to build, execute, and maintain a data product. This reduces speed to innovation and lead time for building new data products. What we really want is an abstraction of these common platform capabilities to reduce the overall cost of ownership for the domain teams.

Principle 3: Self-serve data infrastructure as a platform

The data as a product concept raises concerns regarding the cost of ownership for the domain team, leading to the third data mesh principle: self-serve data infrastructure as a platform. The intent of this principle is to develop a set of solution building blocks we’re terming as infrastructure abstractions that can be combined and leveraged to accelerate the build of producers in the mesh.

This approach also helps with standardisation of tools and patterns across data domains.

The key to building the data infrastructure as a platform is to:

1. Keep it domain agnostic.
2. Make sure the platform hides underlying complexities and provides data infrastructure components in a self-service manner.
3. Lower the lead time to create a new data product on the infrastructure.

An illustration of how a mesh can be integrated with a data Infrastructure platform

Platform abstractions can take the form of scalable polyglot storage, federated identify management, or data pipeline orchestration. In essence, any capability that would be shared across producers is in scope for being abstracted.

The abstraction of these platform components gives rise to a data infrastructure platform, offering trusted, secure, and reliable infrastructure abstractions that can be combined to build the producers in the mesh.

Operating model impacts

The benefits of creating platform abstractions in the form of re-usable infrastructure patterns is two-fold: it reduces the burden of developing a data product while avoiding the risk of proliferating complexity in the environment. This ensures the overall mesh remains manageable within the organisation.

All signs point to the establishment of a shared platform team that is responsible for developing these building blocks and owning the overall platform. Much like the data domain teams build data products for the business, the shared platform team should build platform abstractions for the data domain teams. In this way, the data domain teams may be considered the customer of the shared platform team.

Nonetheless, this doesn’t absolve the data domain teams from maintaining their data products. Instead, we would expect to see a shared responsibility model between these teams, with data domain teams responsible for some aspects of the solution (such as data quality, maintaining data models, and ingestion logic) and the shared platform team responsible for other aspects (including infrastructure monitoring, integration platforms, and solution templates).

Operating model impacts of data infrastructure as a platform

Additionally, the role of the shared platform team extends to components that are common to all producers in the mesh. This includes infrastructure monitoring, Identity Access Management (IAM), development pipelines, and shared product interface platforms (such as API interfaces). Having a centralised team develop and own these capabilities helps with standardisation and management of the mesh.

That said, experience with federated models shows that these teams are often poorly resourced and risk becoming a bottleneck for the organisation — the exact issue we’re trying to avoid. If we’re too restrictive and not strategic in our planning, data domain teams may end up waiting for shared platform teams to make a critical service available or go create a solution that is hard to manage and govern.

Therefore, when establishing this function, it’s worth considering the following:

• Developing a shared platform strategy. Identify the MVP platform capabilities and develop these first before scaling the platform to address the rest of the data product team’s requirements.
• Making sure the shared platform team operates with a product development mentality. This will direct focus to developing the right capabilities and ensure the appropriate level of ongoing support.
• Developing an inner sourcing model. This enables data domain teams to develop their own infrastructure abstractions, contributing to a central repository and alleviating some of the pressure on the shared team. This can be incentivised through rewards and recognition programs.
• Developing clear solution guidelines. This removes ambiguity around security, governance, and strategic tooling requirements as early as possible, enabling the shared platform team to focus on getting infrastructure abstractions in the hands of data domain teams.
• Documenting a shared responsibility model. Development and ownership responsibilities of a data product should be clear to both the domain teams and the platform team. This not only helps avoid issues and allows for swift resolutions, but also ensures teams will be resourced with the right skills to perform their function.
• Clearly defining a product release process. When a product is ready to be consumed, it needs to be integrated into the shared interface platforms. To avoid delays, a clear process for executing this integration should be defined, outlining each step, who will execute it, and in what timeframe.

Principle 4: Federated computational governance

With the first principle, the decomposition of data ownership across domains may raise some of the following concerns:

• How can you correlate data across domains? (I.e., join, filter, aggregate, etc.)
• How can you enable interoperability between the polyglot domain datasets?
• How should you enforce standardisations and global governance?
• How do you automate the execution of decisions in the data mesh platform?

This is where the fourth principle of data mesh — federated computational governance — comes into play. It enables the users to gain value from aggregation and correlation of independent data products and helps address the above concerns by establishing a shared data product catalogue.

Federated computational governance in a mesh architecture

A data product catalogue is a meta-data store that captures both technical and non-technical meta-data about our data products and presents it to the organisation. Imagine this to be an online store where members of the organisation can browse available products, evaluating their reliability, level of data quality, refresh frequency, available attributes, data confidentiality, and terms of use. In an ideal world, the user would be able to directly request data access and sign a data access contract, dictating the terms of use and the SLA of the data product.

​​​​​​​Since the data consumer may be a data domain team that wants to use another team’s data product as an input into their product, the relationship between the data domain team and the data product catalogue is a two-way relationship involving both publishing and sourcing data products.

Operating model impacts of federated computational governance

Establishing this solution requires more than simply purchasing data cataloguing software and integrating it with your technical meta-data store — it requires significant strategic planning to answer the following questions:

• What is the definition of a data product and what meta-data must be made available to data product users?
• What does it mean for producers and consumers to enter a contract with one another? And who is responsible for what, as a part of this agreement?
• How will lineage of data (particularly sensitive or licensed data) be tracked and governed to reduce the exposure risk?
• What is the workflow for provisioning data access? And who will execute these steps?
• How will data products be measured to ensure they are compliant with data governance standards?
• How will data products be reviewed and certified to create trust while eliminating duplication and clutter in the data product catalogue?
• How will users be authenticated into the catalogue?

One way of addressing these concerns is with a data mesh data governance function (not to be confused with enterprise data governance teams) that provides strategic direction and takes ownership of the data catalogue, including its configuration, license agreement, design, and ongoing support.

However, there’s always the risk that this team becomes the bottle neck for data product development by being too restrictive or too slow. If we expect one team to manage product meta-data, access control, and provide strategic direction, we’re asking too much. Instead, we must embed data governance and stewardship capabilities into the data domain teams themselves to enable them to execute these activities on a data product level and allow the data governance function to set the standards. ​​​​​​​This requires developing a shared responsibility model between the data domain teams and the data governance team to clarify responsibilities and make sure it’s the right resource mix.

For example, the data governance function would define the meta-data attributes for a data product in the catalogue. This team may also periodically check that these meta-data attributes have been populated and report on the completeness of this data as a KPI. However, the responsibility of actually populating this data would sit with the data domain teams. This means that the data domain team needs someone with a data steward skill set to manage the product-level meta-data and the data governance team needs someone with an auditing skillset to ensure everything is done in alignment with governance standards.

Bringing it together

Working through the four principles of data mesh, it’s easy to see how one principle is required to address the potential risks and implications of another. Therefore, the resulting operating model and solution architecture — if mapped to the four principles — should provide a cohesive approach that addresses many of the issues with a data mesh solution.

Reference architecture

Below is a conceptual reference architecture of the data mesh, demonstrating how the four principles work together to provide an ecosystem that can scale out, both as a platform and an organisation.

Slalom’s data mesh conceptual reference architecture

Operating model

When considering this holistic operating model, we can see how each team’s responsibilities relate to one another, addressing the core principles of data mesh.

Slalom’s data mesh operating model

Conclusion

In summary, data mesh is a complex and evolving topic. In presenting our initial approach, we hope we’ve helped you wrap your head around the concept, and we welcome your thoughts as we add details into this framework.

Slalom is a global consulting firm focused on strategy, technology, and business transformation. Learn more and reach out today.