Introducing: The Email Newsletter Benchmarking Tool

We built a free and open-source tool that allows newsrooms to measure and benchmark the email metrics that matter.

Hi there! My name is Hong Qu. I am the director of technology at the Shorenstein Center.

I joined the team recently and have been working closely with our director, Nicco Mele, and the Center’s business models and information disorder teams on imagining and developing tools that newsrooms need in the digital age. Before joining the Shorenstein Center, I was CTO of Fusion, vice president of product at Upworthy, and a product designer at YouTube.

One exciting initiative the tech team has been working on with the Single Subject News Project is a product called the Email Benchmarking Tool, a new tool that enables newsrooms to understand those email newsletter metrics that are most crucial. In this post, I will introduce why this tool matters, what it does, and what’s next.

Why Email Metrics Matter for Newsrooms

We know that email newsletters are a vital vehicle for audience development, and that’s why we are conducting research — both qualitative and quantitative — to understand the dynamics of operating successful newsletters.

A year ago, the Shorenstein Center published a paper, “Using Data Science Tools for Email Audience Analysis: A Research Guide,” delineating a sophisticated data science framework for analyzing email newsletter performance, including open source code for using Jupyter Notebooks to access, calculate, and visualize data sets that illuminate patterns of user behavior among subscribers.

We believe that “building online audience — and consequently reliable digital revenue — requires creating a repeat ‘habit of news’ with online readers and viewers. The single most reliable digital channel for building a ‘habit of news’ is email.”

But to leverage email as a habit-building, monetizing vehicle, newsrooms need a clear sense of the health and performance of their email list so they can plan strategy, workflow, staffing and activities.

What Is the Email Benchmarking Tool?

We are building out the next incarnation of our original research in the form of the Email Benchmarking Tool as a more accessible service for deriving actionable insights from the analytics. Currently, the tool is available only to MailChimp users.

In its current, minimum viable product (MVP) state, the Email Benchmarking Tool is a site where a user can sign up and enter their MailChimp’s API key, giving us permission to pull the raw data. We then extract and perform calculation on this data to generate six charts that provide insights into the engagement levels of an outlet’s newsletter compared to an aggregate of newsletters from similar organizations. The benchmarking functionality is made possible by the generous contribution of all the users who opt in, thereby allowing their anonymous data to be used to establish the benchmark data points.

What Does It Do?

The Email Benchmarking Tool provides reports that can be used to improve the health, growth, and engagement of any newsroom’s email products.

Specifically, after providing your MailChimp API key, the tool will send you a “report card” of six key metrics that matter: overall list size, list composition, overall open rate, the distribution of your open rates, the percentage of your list that opens 80 percent of the time or more, and the percentage of your list that hasn’t opened in the past year.

The Benchmarking Element

For all of those metrics, with the exception of the distribution of your open rate, the tool gives you both your newsletter’s metric and the average metric for all other users of our tool (“average” referring to the mean across all lists in our data set).

Example of an output from the tool —metric #2 List Composition (see below).

I’ve broken down what each metric means here:

1. Overall List Size: This is a metric traditionally used for email (the total number of users subscribed to your list) and is one that we call a “vanity metric.” Why? Because your overall list size doesn’t tell you much about the quality or value of the users on your list. A better way of thinking about your list involves taking into account the current status of every email address you’ve ever acquired via your list composition (see next), and the percentage inactive (see #6).
2. List Composition: This metric breaks down the total number of unique email addresses from the entire list into percentages of users who are Subscribed (current subscribers), Unsubscribed (subscribers who removed themselves from list or whom the list owner removed), Cleaned (subscribers removed from the list, typically by a service provider after email bounces), and Pending (semi-subscribers stuck in the limbo of double opt in — or, someone who gave their email address but did not hit the confirmation button in their email inbox). By looking at this metric, you can see whether you have an Unsubscribed or Pending problem that needs to be addressed.
3. Overall Open Rate: This metric, another “vanity” one, shows your list open rate. While list and campaign open rates are the traditional ways of looking at your email performance, a better alternative for examining your list open rate is through a distribution of your subscribers’ individual unique open rates (see next).
4. Distribution of List Open Rate: This metric shows the distribution of user unique open rates for current subscribers on your list. For example, you can see what percentage of your list opens zero percent, 10 percent, 20 percent, all the way up to 80 percent of the time. We think newsrooms should focus on retaining, growing, and monetizing that segment of 80 percent-plus openers; on moving users with lower levels of engagement along the engagement funnel; and on removing the users who stay inactive over time.
5. Percentage Open > 80 Percent: This metric shows your most engaged subscribers — those who open between 80 percent and 100 percent of your emails. These subscribers deserve greater scrutiny: what drives their activity? What are a few user personas that capture this audience? It is worth trying to figure out how to move more of your subscribers into this portion of your list.
6. Percentage Inactive in Past Year: This metric shows you the percentage of current subscribers on your list who haven’t opened one of your emails in the past 365 days. Inactive subscribers can cloud your metrics, affect your email’s deliverability, and make it harder to understand your list dynamics. We recommend that newsrooms manually remove those inactive users from their list (after sending them a re-engagement campaign).

It’s our hope that news organizations will embrace and adopt this tool to improve newsletters. We are building free, open source data science metrics for MailChimp newsletters to empower audience development teams to grow their base of habitual readers and supporters. The more users in the system, the more accurate the benchmarking becomes. Hence, we encourage you to sign up to try this service and give us feedback. If you are a developer or designer, we also welcome your contributions to help us build it. The source code is available on Github. Also feel free to reach out to me directly.

In the next post, I’ll describe more about what we’re doing to improve the tool — including adding filters to characterize the newsrooms for a greater level of benchmarking detail,and adding the capability to run the tool through particular tags across mailing lists.

You can read more about newsletters in a forthcoming email newsletter guide, a collaborative project among the Shorenstein Center, The Lenfest Institute for Journalism and Yellow Brim.