Choosing journalism metrics that actually count (and are countable!)

Fluffy goals, unmeasurable metrics, and forgotten impacts, oh my! Here’s how we’re helping newsrooms fix that shit.

By Julia Haslanger and Stephanie Snyder

Newsroom: Our goals for audience engagement? “Regain trust. Listen better. Hear from more people. Become part of daily routines.”

OK, those are all nice things. How do you plan to measure each of those?

Newsroom: Oh. Uhhh…

As Hearken has evolved over the last three years, so has the journalism industry. We’ve gone from helping a handful of newsrooms better listen to their audiences and the public they’re trying to serve, to a roster (that’s still growing) of 140 news organizations.

Why? Audience engagement has moved from a supplemental project to a necessary strategy rooted in building public trust and growing reader / listener / viewer loyalty. And people in newsroom leadership — with ties to editorial and business priorities — understand that.

One major stumbling block we discovered was that news organizations didn’t know how to measure the goals they knew (or heard) they should be setting around better connecting to their audiences.

Starting in spring 2017, we began to centralize the goals our partner newsrooms were stating when they started working with us, and noting any metrics they specified or seemed to be using. As we updated and analyzed that document throughout the year, two themes became clear:

• Newsrooms are not practiced in setting measurable goals. (An example of an unmeasurable goal: “We’d like people to feel that they’re being listened to.”)
• The metrics newsrooms picked often didn’t actually map to the goal they stated. (Example: “Be less of a one-way street” paired with the metric of “page views”.)

Another problem emerged over the course of 2017: As we began working with newsrooms, we asked them for editorial goals and business goals. Rarely did the editorial employees have a sense for how the business team could measure success.

In early 2018, Hearken knew more than 20 news organizations would join us over a short period of time through the Community Listening and Engagement Fund. This was the perfect (and necessary) moment for us to create a streamlined process to help newsrooms set goals, identify metrics, and track them. We set out to create a menu for newsrooms, offering them options for goals and a selection of metrics that directly map to measuring their progress toward that goal.

A preview of what we developed:

The menu has three sections, one for organization-wide goals, one for the journalists’ goals, and one for the business team’s goals.

How we got there

The first step we took was to organize all the goals we’d heard over the past two years, both from our current partners and from people Hearken’s business team talked with during the sales process.

We sorted them into three piles:

• Too big, vague, need nested goals in order to measure (E.g. “Serve people better” or “Build trust”)
• More specific, but still need nested goals in order to measure (E.g. “Create more inclusive journalism where people see themselves reflected”)
• Very specific, measurable (E.g., “Hear from readers in ZIP codes outside our core subscriber base”)

Who else we learned from / who inspired us

After compiling that background, and analyzing it to find a few key problems we hoped to solve, we next looked for guidance from outside our organization. We reached out to Josh Stearns from the Democracy Fund.

We knew Josh would be a great person to give us insight and feedback. In his work handling grants for newsrooms and community engagement projects, he likely encountered similar challenges around setting meaningful goals and measuring success.

(Josh is also a kindred spirit and a long-time champion of Hearken. We thought he might be willing to take a couple hours and share his expertise with us if we showed him that doing so would benefit our company’s future and the futures of our partner newsrooms.) 💚

Josh agreed to help, and we met with him in early March. Some of the discussion items in the 90-minute meeting:

• What metrics are we hearing about in our work?
• What metrics/goals (or types of metrics/goals) do we see as useful?
• What are challenges we hear related to establishing metrics?
• What are challenges we hear related to actually tracking those metrics?
• What metrics should be standard/universal/recommended on a menu?
• What other problems have we identified but are not addressed in our solution?
• Who else in the industry (or similar industries) is working on those problems?

The conversation really helped us focus our efforts. To share a few of the gems from Josh:

• “How can Hearken make the invisible impacts of journalism more visible? There are really important things that will never show up in Google Analytics.”
• “What is a metric that makes people whole? The more questions, the more fullness that will be on display. So much of what we measure is focused on segmentation. Let’s not lose sight of the wholeness of our communities.”
• “News organizations didn’t need to think about goals for a very long time. They used to be able to do it all. Increasingly, they can’t do that anymore. Goals become important in prioritizing and how to use limited resources.”

We also used that conversation as a jumping-off point to draw inspiration from others who have attempted similar resources. Full reading list at the end of this post!

Looking forward: A focus on impact

Part of this process also involved us separating “goals” from “impacts.” Goals are the things you *want* to have happen as a result of doing this work. Impacts are the things that *actually* happen, which may not be tied to any goals or metrics you set.

Now that we had designed a good system for helping newsrooms think about goals, we turned our attention to helping them think about impacts. The first challenge was helping newsrooms understand how it benefits them to track impact at all. Once we drafted some go-to language around that, we began shaping a system for them to track it.

As we thought about impact in the framework of our audience engagement ring (more on that here), we broke potential impacts into three categories:

Civic engagement

Examples:

• People said they were able to make a decision on a ballot measure with your help
• More people vote in local elections
• Increased public knowledge and action around a particular issue as evidenced through community participation in events

Institutional change

Examples:

Newsrooms:

• Respond to common theme of questions by establishing a new beat, special series or event
• Journalists get in the habit of looking to their communities for story ideas and questions, rather than their own experiences or competing media

Local institutions (example: local government / community organizations / other media):

• City officials cite the newsroom’s public-powered work during a council meeting
• Individuals or organizations work to effect change around an issue brought to light by journalism created in response to audience engagement
• Media outlets and organizations work to hold public officials accountable for and motivate them to fix problems raised by the audience through the public-powered journalism process

Audience loyalty and brand awareness

Examples:

• A thank you note from a reader whose life was demonstratively improved by knowing the answer to a question
• People saying they were motivated to join a membership program or subscribe due to your public-powered work
• Newsroom staff members are invited to speak about and promote the organization’s work / mission

Just as newsrooms see an increase in community engagement the more they are transparent about involving the public in the editorial process, our hope is that newsrooms will see an increase in impact the more they highlight the change happening because of their public-powered journalism.

Newsrooms can work their way around the pie: Highlighting an increase in civic engagement can help lead to institutional change → highlighting how your community created institutional change with your guidance can help lead to audience loyalty → highlighting your most loyal readers and how they are civically engaged can create FOMO and lead back to more civic engagement.

Now that we have a framework for communicating about impact, our next step will be to draft and test resources that our newsrooms can use to track their own impact. We’ll share an update down the line about what we learn as we develop that impact tracker for (and with) newsrooms. If you have any experience in this space (or ideas) please let us know! We love learning from the community around us.

Reading list

Here are some of the resources we referenced as we thought about all this (and that we continue to use as we build a structure to track impact):

• Joy Mayer’s 2011 research: Measuring the success of audience engagement efforts
• Metrics for News from American Press Institute
• Melody Kramer’s 2016 Poynter piece: 50 ways to measure your analytics (with apologies to Paul Simon)
• CIR’s Impact Tracker
• What We Talk About When We Talk About Impact: One News Organization’s Approach to Practicing Journalism with a Purpose (Chalkbeat)
• Weekly report emails from Contextly, Switchboard and other startups