Exodus: a self-service data tool strengthening the data culture at SCMP

The data team at the South China Morning Post supports every department with the data analyses and tools they need to make data-informed decisions. Created in 2016, we’re a recent addition to a 116-year-old company, so our first few years were focused on building a robust and scalable data infrastructure to power our initiative to become a data-driven organization. We also built our business intelligence and invested time in training and internal communication to foster data adoption. Our ultimate goal was to give each employee access to the data they needed to drive their decision-making, empowering democratic access to data across the company.

Problem:
As our new data-driven culture quickly permeated throughout the company, we started receiving a large number of ad-hoc data requests from other departments. Departments couldn’t handle their own queries with traditional BI tools (reports and dashboards) because those tools are too rigid for data exploration, so they reached out to the data team with their queries. But the data team was receiving more ad-hoc data requests than we could handle. For each request, data analysts needed to manually extract the data from the data warehouse by creating one-time-use SQL queries. Then the analyst would communicate the results to the stakeholder, resulting in a lengthy email exchange of questions and clarifications. The whole process was too time-consuming to be scalable. We wanted all departments to have access to relevant data, but we didn’t have the capacity to handle their requests. We’d run up against a barrier to further data democratization.

Solution:
To solve this problem, we created Exodus, a self-service data tool that empowers anyone in the company to access the answer to their data queries in real-time. Exodus is:

• Versatile, to support the needs of every department.
• User-friendly, so that the non-tech-savvy can use it.
• Easy to support, maintain, and scale.

Outcome:
Over the past year, Exodus has allowed employees across the company to access the data needed to implement data-informed strategies. Exodus has been used by over 10% of the company to answer 10,000+ requests, performing far beyond our initial expectations of a side project. It has been adopted by all departments across the company, including the newsroom, marketing, product, sales, and executive leadership, providing insights on a diverse range of questions such as:

• What topics from my desk are users most interested in? [Editorial]
• On which platforms do Malaysian readers like to consume content? [by Product]
• Why was there a spike in our traffic 2 years ago? [by Editorial]
• How many loyal users do we have in South East Asia? [by Sales]
• What is the conversion rate of the newsletter widget? [by Product]
• What was the user quality from our latest marketing campaign? [by Marketing]
• When is the best time to post on Facebook to optimize user engagement? [by Editorial]
• Which newsletters have a low open rate? [by Marketing]

How does it work?
The tool is divided into screens, each one connecting to a single table in the data warehouse. The process is executed in 3 steps:

1. On a form, users select the list of dimensions and metrics they want to retrieve and add filters to refine their search.
2. On submission, a SQL builder engine in the back-end generates the corresponding SQL code and executes the query in our data lake via an API call.
3. The results are then transferred to the front-end for consumption.

The output includes:

• a graph facilitating data interpretation;
• the raw data and features to download as CSV or export in Gsheet;
• a permalink to rerun the query, for collaboration or bookmarking;
• the SQL code generated, helping advanced users complete their analysis in the data warehouse

An admin panel makes it easy to add new data sources, dimensions, and metrics, and to manage users.

Screenshot of Exodus data extraction form and results

The technology stack
The first version of the tool was developed with Dash, a framework from Plotly creators to implement web-dashboards in Python. This choice leveraged internal Python expertise and challenged us to learn and explore a promising framework. While this choice was a great way to launch an MVP, it had major feature limitations, a small support community, and felt short on performance.

For the second iteration, we moved to Django, leveraging the Object-Relational Mapper (ORM) to easily create an application database and an admin panel to add data sources and manage users in a few clicks. On the front-end side, we used elegant bootstrap dashboard themes, Chart.js for the graph and Jquery for ajax connections and user interactions.

Key learnings
In retrospective, we identified 5 key elements contributing to the fast adoption of the tool:

• Starting small before scaling
  We started with an MVP including only one data source, basic design and features, and with a technology stack that had slow performance but allowed us to deploy quickly. This allowed us to gain traction from early adopters, who requested features and gave user feedback, which we then used to quickly iterate new versions.
• Every training was adapted to the audience
  Exodus is versatile, so the demo needed to illustrate relevant use cases for each cohort. Training sessions were dedicated to specific teams and prepared in advance with their team head to allow each team to understand how Exodus could help them implement data-driven strategies.
• Focus on the user experience
  Ease-of-use has constantly been the priority; we developed the tool with non-data savvy colleagues in mind. Every training session was an opportunity to observe how users interact with the interface, providing precious insights and ideas for improvement.
• Data informed strategy
  The implementation of a user tracking system enabled us to spot departments under-utilizing Exodus, helping us prioritize training for departments who weren’t using the tool. We also monitored data sources that were not queried often and promoted them with suggested use cases to relevant departments.
• Building a community and leveraging gamification for user retention
  A dedicated Slack channel was initially created to communicate new releases and provide support, but quickly Exodus users started to help each other, sharing tips and best practices. We’ve also gamified the tool, rewarding users who demonstrate high engagement by granting them Power User status, which unlocks beta features and access to new data sources.

Screenshot of Exodus gamification feature

Fun fact: why call it Exodus?
Exodus refers to the book from the Bible narrating the Israelites’ journey from captivity in Egypt to freedom in the promised land. At the South China Morning Post, we’re freeing information from its captivity in technical complexity, allowing users across the company to easily access data and make data-informed decisions.