Developing actionable data insights for editorial teams
In this article, we explain how NZZ, Switzerland’s German-speaking newspaper of record, iteratively developed a dashboard that ranks recently published articles according to a score aligned with our subscription-based business model. The web-based tool empowers online editors to continuously monitor and optimize the distribution of each article across online distribution channels. It provides novel and actionable information and highlights articles that need immediate attention.
Authors: Cristina Kadar (Machine Learning Product Owner & Senior Data Scientist), Dominik Batz (Head Audience Engagement)
Motivation
Writing content — albeit being the most laborious phase — is only one part of a newspaper’s modern production pipeline. A significant portion of an article’s success lies in its online distribution approach. NZZ’s online editors face the daily challenge of monitoring and optimally distributing up to 100 articles across different digital channels such as the front page, push notifications, newsletters, and social media posts. Hence, the objective of this project was to provide a solution for our editors to monitor and optimize, in (near) real-time, the online distribution of recently published articles.
The project aimed to democratize access to unique data from various internal sources, to which editors would otherwise not be exposed to. We set the goal of providing clear and novel, and most importantly, actionable information relevant to the decisions our online editors are faced with hourly. These editors work in the newsroom under strict time constraints, multitasking between different responsibilities and tools, all the while curating the front page and other distribution channels. Our tool was envisioned to reduce complexity and help them monitor the online performance in (near) real-time and in an effective way. Additionally, it should provide them with data-informed, easy-to-implement tips so that each article can reach its maximum potential and thus make the best possible contribution to business growth.
Organizational approach
To meet the objectives outlined above, a highly-paced, cross-functional team with members from the editorial (audience engagement and SEO), data (data science and data engineering), and tech team (front-end development) was set up.
The team operated under the umbrella of the OKR process, iterating over the development process depicted below for several quarters. In an initial phase, we defined the metrics structure and developed the first data pipelines to compute them. Then, we’ve iterated over collecting user stories, developing mock-ups, validating them in workshops with editorial representatives, deploying APIs, and developing the front-end tool. After releasing a new version, each editorial department was visited twice: first, to make sure that new functionality is well understood and properly used; second, to collect feedback for future iterations.
Metrics definition
Many publishers today still rely on very simplistic metrics to measure article performance. The most basic approach is counting the number of article views — a perhaps good-enough metric for publishers that have an ad-based revenue model and optimize for reach, but not suitable for our case. A step further is to look at what happens after a click and measure the dwell time of an article (a.k.a. engaged time), ideally somehow scaled to the length and complexity of the text. This metric sheds light on the quality of the user engagement with the content and is a major step forward. Still, we are missing what happens after the interaction with the article, and how this helps us grow our subscribers base.
Hence, at NZZ we decided on a multi-dimensional metric design. Article performance is measured across 8 different metrics describing attractivity, engagement, and conversion key indicators:
- Attractivity is measured as per article views from anonymous, registered, and subscribed users, respectively;
- Engagement is quantified in terms of scaled dwell time, scroll depth, and recirculation achieved by the article;
- Finally, the conversion dimension is determined by how the article contributes to on-site registrations and subscriptions.
The delivery of every article published recently is quantified across each metric and summarized as a weighted sum in a final article score that uniquely reflects our company’s business goal — namely attracting and retaining paying subscribers in the Swiss and German markets.
Solution
To meet the objectives outlined above, we have developed a web-based application available in two formats:
- a big-screen format for display in the newsroom (see teaser picture);
- a desktop format for individual use (see screen capture below).
The landing page of the tool consists of a dashboard, which presents a list of all recently published articles ranked by their current aggregated article score or by publication date. For each article, we include information on how it performs across the attractivity, engagement, and conversion dimensions — relative to the rest of the articles. Promotional information (such as if and how long the article was featured on the front page) is included.
Discrepancies that need attention (e.g. articles with a high number of clicks but low engagement) are automatically detected and visually highlighted in form of tips. Furthermore, the most popular articles across user groups of particular interest for us (such as readers in targeted markets, women, students, etc.) are awarded badges.
Hence, the dashboard aims at providing a first assessment of the current situation and aids the editors in identifying articles that require immediate intervention. For further investigation and analysis, we have developed a detailed view of each article. This view provides a timeline with information on all historical channel promotions juxtaposed with the article’s historical performance. We feel this is very unique to our tool, offering a wealth of information to learn from on the performance effect of each channel promotion. Finally, the detailed view provides information and tips on how each article can be adjusted for optimal SEO performance (e.g. by setting the dedicated SEO title or linking to other articles) and mobile performance (e.g. by shortening lengthy titles and paragraphs).
Feedback from the editors
The tool has been in use since April 2020 in the newsroom and has received positive feedback from the editorial team:
With the Article Score tool, we can get an overview of the status of our current content and align the planning of our offering accordingly. In exchange with our department heads, the Article Score helps us look for patterns in our reporting and come across topics that call for further in-depth coverage. The application thus gives our readership a voice, which we want to give weight against the background of our journalistic principles.
— Eric Gujer, NZZ Chief Editor
The Article Score dashboard helps me discover content that is of interest to users. It often shows things that were not on my radar. The tool helps us deliver content through the right channels so that new readers and registered users return to us and then subscribe. In addition, the Article Score tool enables our readers to get exactly what they expect from us without having to search for it.
— Christian Steiner, NZZ Online Editor
Outlook
We continue to iteratively gather requirements from the users, improve the design, and grow the Article Score suite of tools. To make the tips more intelligent, we are currently experimenting with causal machine learning techniques for identifying articles that would most profit from a front-page promotion.
The team
Without the highly motivated and hard-working team, this project would not have been possible. Kudos to the core contributors of the first iterations described in this blog:
- Editorial: Tom Schneider (Technology Deputy Editor-in-Chief), Dominik Batz (Head Audience Engagement), Balz Rittmeyer (Digital Designer), Rafael Schwab (Audience Data Analyst), Jonas Holenstein (Product Manager SEO)
- Data: Cristina Kadar (Machine Learning Product Owner & Senior Data Scientist), Paweł Kaczorowski (Senior Data Engineer)
- Technology: Goran Kosutic (Front-end Developer), Christian Stettler (Product Owner)
A shorter version of this text was submitted to the INMA 2021 Global Media Awards and won an Honourable Mention in the Best Data Dashboard category.
If you liked this article, hit the applause button below, share it with your audience, and follow us for more insights.
