SSENSE Data Mesh: Going from Vision to Value — Part 2 of 2

In part 1 of this series I touched on the motivation and pressure points of moving towards data mesh, the divide between the operational and analytical data world, the inflection point for SSENSE, and the fundamental principles of data mesh. In this second part, I want to focus on the paradigm shift we are making, how we are planning to design a data mesh, the tech stack and the challenges we may face during the course of our journey, and how we can mitigate them.

Paradigm Shift

Data mesh calls for a fundamental shift in the assumptions, architecture, technical solutions, and social structure of our organizations, in how we manage, use, and own analytical data.

• Organizationally, it shifts from centralized ownership of data by specialists who run the data platform technologies to a decentralized data ownership model pushing ownership and accountability of the data back to the business domains where data is produced or used.
• Architecturally, it shifts from collecting data in monolithic warehouses and lakes to connecting data through a distributed mesh of data products accessed through standardized protocols.
• Technologically, it shifts from technology solutions that treat data as a byproduct of running pipeline code to solutions that treat data and code that maintains it as one lively autonomous unit.
• Operationally, it shifts data governance from a top-down centralized operational model with human interventions to a federated model with computational policies embedded in the nodes on the mesh.
• Principally, it shifts our value system from data as an asset to be collected to data as a product to serve and delight the data users (internally and externally).
• Infrastructurally, it shifts from two sets of fragmented and point-to-point integrated infrastructure services — one for data and analytics and the other for applications and operational systems — to a well-integrated set of infrastructure for both operational and data systems.

Dehghani, Zhamak. Data Mesh (p. 49)

SSENSE Data Marketplace

While data mesh sounds super exciting and cutting edge, we are also acutely aware of the realities of our producers’ and consumers’ ability to adopt. Our first cloud choice is Amazon Web Service (AWS) as most of our upstream domain-oriented microservice architectures are built on AWS cloud. We want to take a more practical approach. Instead of going with a completely federated data governance model, a central data governance model with multiple producers and consumer accounts accessing data products from a single place makes more sense for us. This way, the data platform can still maintain a central data catalog for all data products, with auditing and security maintained in one place. The central data governance account will have resource links to all the data products across the business domain accounts. We are transforming:

FROM

• Use case driven approach that tackles very specific problems using data
• End deliverable, an insight which is difficult to iterate
• Low ”re-usability” across business functions
• Technology oriented, which makes it hard to connect to, prove, and evolve business value to domain oriented data products

TO

• Focus on data value uniqueness in data, quality, access, and cost
• Continuous product release, what’s needed to get it right
• Supporting multiple use cases, which reduces cost and promotes innovation and collaboration
• A product-owner mindset to see the business in the architecture

The Advantages

• All data products are available in one place for querying
• Easier entitlements
• A single database to database resource link is enough (assuming database to database resource links are allowed)
• Single source of data truth and one-way sharing of catalog across various organizations
• Data never leaves business domain accounts
• The business domains still have the choice to define metadata, schema evolution, and manage permissions. The central account does not enforce anything on them.
• Allows centralized auditing
• Allows dev accounts for consumers, which is useful in cases such as model training and serving for mlops

The Challenges

While data mesh promises great new heights, we also don’t see this as a silver bullet. Cultural challenges, outdated governance models, organizational silos, and legacy execution approaches can sometimes stand in the way of realizing this vision. As we embark on this journey we see there could be a few challenges that may come our way:

• Distribution of knowledge and skill sets
• Potential divergence of tech stacks
• Technology solutions may not solve issues created by misaligned business domain incentives
• We may face difficulties in securing and auditing the data
• Product thinking is not for everyone, going from operational database queries to publishing data as a product can be difficult

The Organization Approach

Data-platform and data leaders at SSENSE are keenly aware of these challenges and pushing boundaries to put mitigation plans in place. We have started to embrace an AWS data-driven approach to achieve our goals.

• Culture: continue to challenge our thinking in terms of beliefs, values, and behaviors that create a data-driven culture by engaging, educating, and enabling the stakeholders. The four E’s of data culture:
  - Engage in data-driven decision making
  - Educate everyone
  - Eliminate data blockers
  - Enable frontline action
• Organization: structure and roles that accelerate data-driven outcomes
• Mechanism: processes that enable and scale the effective use of data
• Execution: approach and tools to rapidly unleash the value of data

Conclusion

As we are early in the data mesh journey, we are super excited to be among the early adopters in the data industry. Time and again, SSENSE team members have proven that we have the appetite for adopting emerging technologies. Even though we are early, we have already crossed some key milestones in terms of strategy, structure, people, processes, and technology. Data mesh does not dictate the organizational structure from scratch, it makes some assumptions about the existing organizational structure and builds upon that. We have executive support from top organizational enablers, we have identified the organizational and business domain complexity, we have made a long-term commitment to the transformation, and have put architecture and patterns in place. The SSENSE data platform and domain teams went through PoCs, MVPs on many patterns, and we strongly believe we are well positioned to take this journey.

References

1. Dehghani, Zhamak. Data Mesh (p. 503). O’Reilly Media.
2. https://aws.amazon.com/executive-insights/content/how-do-you-become-a-data-driven-organization/
3. https://docs.aws.amazon.com/wellarchitected/latest/framework/welcome.html

Editorial reviews by Catherine Heim & Mario Bittencourt

Want to work with us? Click here to see all open positions at SSENSE!