Scout24 Data Landscape Manifesto

Presenting the manifesto
Screenshot from our internal wiki showing the original date. (Thanks Arif for that first version!)

The story behind the manifesto

As a tech team, Scout24 Data Engineering started where most tech teams start: with the tech. We built a modern cloud-based data lake that would scale indefinitely along with the company.

Becoming data-driven requires technical, organisational, and, most importantly, cultural change.

From this disappointment and soul-searching came a vision, not of the technical architecture, but about how the company should view data and how the data team could enable this change of values.

The Scout24 Data Landscape Manifesto

Roles, responsibilities and values for a data-driven company at scale.

Principle #1 (Preamble)

We believe that collecting and analysing data is crucial to understand our business, our customers and the market in order to provide the right services and products.

Principle #2

We therefore believe that everyone in the company must have easy access to the data available and it must be easy to publish data which can be used by others. This requires a solid Data Platform: easy-to-use tools, reliable infrastructure, and simple guidelines for publishing and consuming data in a secure and privacy-aware way.

Principle #3

We believe that exhaustive centralised data management does not allow us to scale to the level of data creation and consumption we aspire as a company, because it creates a bottleneck and introduces accidental, indirect dependencies. Instead, we believe that data autonomy is the only way for data usage to scale across the company. However, for data autonomy to not become data anarchy, there has to be a clear set of basic rules and responsibilities.

Principle #4

We believe that extensive data availability, data discoverability, and data usability are crucial and that — at scale — no one else can ensure this other than the one controlling the source where the data is originally generated.

Principle #5

We believe that the stakeholder of a metric has to be the single owner of that metric and its definition and has to drive its implementation. Without a single source of truth about what a metric means, we risk that multiple diverging and possibly contradicting understandings and implementations develop over time.

Principle #6

We believe that a minimum level of company-wide comparability and reliability of core KPIs is crucial for leading the company into the right direction and can only be achieved with coherent core data. Our executive leadership team is the owner of these core KPIs and the data group represents the executive leadership team in terms of metric ownership.

Principle #7

We believe that transparency is crucial for understanding what the meaning of a metric is. If month-to-month comparability must never break, there is no way to continuously improve metrics and their transparency based on new insights.

The Ultimate Goal

Ultimately, we believe that together these principles will lay the foundation for an inclusive and highly scalable data landscape.


Photo by Stephen Dawson on Unsplash

Two years in: is it a success or failure?

It is now two years since we introduced the manifesto at Scout24 and began using it to guide the development of our data platform. So far the reception has been mixed. Some teams welcomed the autonomy it provides, while other teams objected to the new responsibility.

Acknowledgements

Sincere thanks to the Scout24 Data & Analytics team members involved in the original writing of the manifesto. Although it was a group effort, Arif Wider from ThoughtWorks and Sebastian Herold, now at Zalando Tech, deserve special thanks for their initial contributions.

Scout24 Engineering

All about engineering @Scout24

Sean Gustafson

Written by

Technical Product Manager for the Scout24 Data Platform

Scout24 Engineering

All about engineering @Scout24