Fully Redesigned, Algorithmic-driven Next Reads Section
In this article, I explain how NZZ, Switzerland’s German-speaking newspaper of record, completely revamped the next reads section on the article pages. Using an iterative, experiment-driven, and interdisciplinary approach between the product and data teams, we have introduced a modular and adaptive layout including several algorithmically-driven article feeds. This resulted in very significant uplifts both in the CTR of the section and in the quality of the user engagement with the content afterwards.
Motivation
Fueled by the COVID-19 pandemic and the overwhelming interest in quality journalism, 2020 was a year of unprecedented subscriptions growth for many online newspapers, including Neue Zürcher Zeitung (NZZ), and enabled us to reach 200.000 paying subscribers.
Going into 2021, engaging and retaining our users on the platform became a high priority. To achieve this we looked at areas to improve in the product and core competencies in the company to leverage.
Starting Point
The next reads section on our content pages is a key area to re-engage readers as they finish reading an article. Next to the front page, this high-traffic area is one of the most valuable locations on the product, yet it had a lot of potential to improve one year ago. In the initial state, the section consisted of a long list of related articles manually curated by the author(s) in the Content-Management-System (NZZ uses Livingdocs), as well as an automatic feed consisting of personalized content and advertising recommendations. This mixed feed exhibited a low click-through-rate (CTR) and perceived quality, while also being difficult to adapt to our needs as an externally-provided, 3rd party service.
On the other hand, we had been growing our internal data and machine learning competency for years and had integrated several personalized approaches in different product areas — such as on the dynamic paywall, in the personalized newsletter meineNZZ, or on the Sunday newspaper NZZaS. Still, personalization was not fully leveraged in NZZ’s core product.
Our Goal
We set ourselves the following objectives for improving the entire below-the-article section:
- Substantially increase engagement by offering high quality, relevant content for discovery and a clear and user-friendly design;
- Decrease manual work for this area and free up editorial time;
- Fully leverage 1st party data and data science expertise in-house.
We addressed these objectives in a tight collaboration between the product and data teams and targeted innovations in smart algorithms as well as intuitive UX/UI — as I show in the Results section, both are equally needed to significantly improve the user experience!
From day one, we followed an iterative and data-driven development approach, consistently A/B testing our hypotheses and gathering feedback.
Approach
The section was fully redesigned for web and Android/iOS app with a wide, modular, and adaptive layout, that is easy to scan across all devices.
Next to a shortened version of the editorially curated list and the existing ads, three new dynamic article feeds that leverage a wealth of internal behavioral user data and article metadata have been introduced:
- Topic Feed: an automated feed of articles sourced from the same topic as the article in scope. Hundreds of topics¹ have been created semi-automatically and new articles are assigned to them instantaneously in an internal tool employing natural-language-processing (NLP) techniques;
- Author Feed: an automated feed² surfacing latest articles from the same author — an often requested feature by our readers and journalists;
- User Feed: a fully personalized, high-quality feed of next reads leveraging the historical profile of each user. The recommended articles are selected not only based on their similarity to content the user has consumed in the past (i.e. content-based filtering), but also on other signals such as their popularity and editorial value³. We have adapted the algorithm to different user groups (anonymous vs logged-in users) and markets (Switzerland vs Germany).
We had several iterations and ran several A/B tests so far in this section. We have tested both algorithmic improvements in the feeds (experimenting with the used scores and sorting criteria) and feature changes (evaluating the presence, design, and length of each feed).
Results
We deployed V1 in September 2021. The results reported here are aggregated across our entire article portfolio and all user groups — meaning that for some core article types and user groups the results are even more impressive.
The overall weekly CTR on all content teasers in the section improved on average by a whopping 40%! Furthermore, the results have been consistent across all 4 months past deployment — even after the novelty effect wears off.
We did not only significantly improve recirculation in the area, but also the quality of the engagement afterwards: the completion ratio of the recommendations (i.e. rate of reads completed after the users clicked on their article teasers) increased on average by 63% in the personalized user feed. This result is also very consistent across time.
Some further insights include:
- The editorial list continues to exhibit both high CTR and engagement;
- In the current configuration, the topic feed experiences the highest CTR from all our data feeds, being at par with the manually curated list. On the other hand, the author feed has a better completion rate than the topic feed. Finally, the personalized user feed shows the best downstream engagement from all data feeds.
Next Steps
We will soon deploy V2 of the next reads section based on the lessons learned from the last A/B tests and continue to innovate in 2022. For instance, we plan to create dedicated experiences for different devices, user groups, and article types, as well as introduce new section elements with targeted promotions, such as promoting related editorial newsletters.
At the same time, explore further use cases of engaging and transparent content personalization/automation in the product.
The Team
Without the amazing cross-functional, highly motivated, and hard-working team this project would not have been possible. Kudos to the core contributors:
- Product Management & Design: Maria Leopold (Digital Product Manager), Sabrina Peterer (User Experience Designer), Adrian Wach (UX/UI Designer), Silvia Schiaulini (Product Designer)
- Data: Cristina Kadar (Senior Data Scientist, Data Science & Machine Learning Product Owner), Paweł Kaczorowski (Senior Data Engineer), Miłosz Szymczak (Senior Data Engineer)
- Technology: Christian Stettler (Product Owner), Sebastian Łuckoś (Web Architect), Volodymyr Fertak (Web Developer), Jurica Lepur (Web Developer)
… and the rest of the organization that contributed one way or another: the Audience Engagement team, the QA team, and so on.
Also, special thanks go to Silvia Schiaulini for providing the beautiful visuals in this post.
[1]: More than 80% of our articles have at least one assigned topic, hence a Topic Feed. If you want to explore our portfolio, head to our topics overview page and take a look at our thematic coverage (e.g. the AI topic page).
[2]: The Autor Feed is not part of V1 of the next reads section currently deployed on NZZ, being so far just subject to A/B tests. If you want to have a look at how it works, you can head to our impressum overview page and check an author page (e.g. the one of our social media lead editor).
[3]: The technical details of the User Feed are beyond the scope of this blog. If interested, head to this article where I explain in detail how we designed our personalized article recommendations on NZZaS (the Sunday newspaper). The personalized User Feed in the next-reads section on NZZ (the daily newspaper) follows a very similar approach.
A shorter version of this text was submitted to the INMA 2022 Global Media Awards and won an Honourable Mention in the Best Product Iteration category.
