How To Shift Your Organization into Data Mesh Mindset?

In spring 2022, the Bizzabo data team began analyzing our existing data architecture model and processes. During the planning of the future data roadmap, we wanted to walk through a reflection process matching it to business needs and scale. We wanted to make sure the processes and tools that we use, set us up for success, allowing us to bring our customers the highest value from data.

We examined the capabilities of our tools, as we use Google BigQuery as our data warehouse, while every data set is transferred with an ETL tool (formerly Airflow and now Hevo) and exposed using Looker embed both internally and externally to our customers. On the Looker side, our product data analysts do the modeling, depending on the value it brings.

On top of enriching our platform with dashboards and reports, we use data to ensure that the new features are optimized, and meet their goals from data refresh frequency to answering urgent business questions.

Our focus was on data architecture, and we narrowed down our requirements to the frequency of data refresh. We thought that the data refresh (real-time, near real-time, and more) was where we should put our effort, and didn’t focus on our data expert team (data engineering team and product data analysts) as a bottleneck in the process, until our CDO & Bizzabo Co-Founder, Boaz Katz, stumbled upon a video presenting #Zhamak Dehghani, elaborating on the four principles of #datamesh in an elegant and down-to-earth manner.

The idea that seemed so reasonable enchanted us. It made us question our monolithic data architecture and made us dig deeper into our existing data culture management. We understood data should be treated as a product, be managed and created by its owners, in order to improve its time-to-market, scalability, and agility allow flexibility and faster data access through a self-service model, and most importantly decentralizes data ownership and distributes it among cross-functional domain teams as traditional data platforms make expert data teams isolated and heavily depended upon — creating a lack of transparency.

What Is Data Mesh?

The #datamesh approach ensures that domains are held accountable for providing their data as products, while also facilitating communication between distributed data.

The data infrastructure provides each domain with the solutions for processing the data, but domains are responsible for ingestion, cleaning, and aggregation of the data to create assets that can be consumed by whoever needs the data.

Every domain is responsible for its ETL pipelines (data transformations), but a common set of capabilities that stores, catalogs, and maintains access controls for raw data is common to all domains. The domain owners can then use the data for their analytics or operational purposes after the data has been served and transformed by the domain.

A thought that led us to examine the methodology was that the software engineering life cycle already implements a decentralized workflow, as we are working in what is known as ‘The Spotify Squad Model’ — a small cross-functional, self-organized team, with end-to-end responsibility, working together towards their long-term mission. On Squads the key driver is autonomy, as it provides employees with a sense of collective ownership. Autonomy is motivating, and motivated people build better products, faster. Autonomy makes decisions to be made locally instead of by managers and committees

So why is the data management culture not aligned?

Is Data Mesh Our Way Forward?

In order to understand if data mesh could be the perfect match for our desired data culture, we embarked on an examination and analysis phase.

In addition to reading books and articles, we have delved deep into #Zhamak Dehghani’s work and ideas. We met Erez Lotan, VP of R&D — Platform at Skai who was very cooperative, as he agreed to meet with us to discuss their experiences with data mesh. Furthermore, we have examined the core capabilities of future technological tools to familiarize ourselves with the ecosystem.

Ultimately, we decided to go for it, as we saw the huge scaling potential in an exponential volume of data.

What Should We Do First?

We have decided to walk through the principles and align it with our culture. The first #datamesh principle is about decentralizing the data’s ownership. Instead of the data engineers and analysts being responsible for the accuracy and trustworthiness of the data, the accountability is on the domain, the houses (Squads) as we call them in Bizzabo.

The good news was that Bizzabo already implemented a domain thinking approach (developing the product in parallel in different houses), so we realized the challenge is relatively not tech-wise, but more the paradigm shift to a data product approach.

Our next step was to identify the most valuable and complementary #dataproduct to start with.

What Is A Data Product?

Any digital product or feature can be considered a “data product” if it uses data to facilitate a value. For example, a dashboard to visualize main KPIs (Bizzabo Home) — this data product is of the type decision support system and the interface to access it is a visualization, another example would be a notification/alert for the organizer (Bizzabo User) — this data product is of decision support component and its interface is our website and mobile app.

How To Pick Your First Data Product?

We examined the current work of the different domains and existing data available to decide which data product to start with. As we wanted to create a data product that would deliver value, but is also complementary (can be combined with other data products and enhanced their qualities), simple, and holds no major calculation and manipulation. We took the time to go over what is available now but is also being developed.

It took us around 3 weeks to decide, we wanted to have a good planning process to avoid rewriting the solution.

What To Expect Next?

Now that we know which data product is first and we have the business knowledge of customers’ requests for data, we can start designing our first data product schema, establish #datacontract (what is a data contract? Wait for post #2 where we explain how we implemented it) and align governance processes in place, to support it.

Want to see how it plays? Stay tuned.