Engagement Metrics are Hard to Measure but Maybe That’s a Good Thing

There’s a Columbia Journalism Review piece that started a shit storm in my corner of the journalism world — The Audience Engagement Industry Struggles with Measuring Success. In it, Jacob Nelson reports on his experience being embedded with Hearken, a platform that enables journalists to partner with the public throughout the reporting process.

I recently received my master’s in social journalism from CUNY and became slightly obsessed with the idea of measuring impact. Now a journalist with Spaceship Media, working on our project, The Many, I’ve made engagement journalism a full-time career.

The Many is a closed Facebook group for women of all political stripes, from hard right to hard left and everything in between. It will run at least through the midterms and is a place for conversation about all sorts of political and social issues. Along with my colleague and fellow social j alumna Alyxaundria Sanford, I moderate conversations and report on stories for the group and for local partners based on these discussions. On top of that, I use metrics to gauge the success of our project so far, what the group needs, and what our next steps should be.

We’re a small team that’s deeply involved in the project. We know each of the 271 women (so far, we’re still growing) and understand how each member plays into the group’s dynamics. Still, we don’t want to make decisions without empirical evidence.

“But little empirical data currently exists that can corroborate the belief that the public’s lack of trust in journalism — and journalism’s sometimes-related financial issues — can be solved by an increase in audience engagement,” Nelson wrote.

I’m not going to point out that there’s empirical data out there — Hearken has done a wonderful job of doing that already. However before discussing the industry’s struggle with measuring success, Nelson failed to detail which metrics are being used and how they’re being interpreted. Surely, our metrics at Spaceship and how we read them are going to be different from other newsrooms.

At Spaceship, I can’t look at metrics like shares or scroll depth. Other metrics like number of comments, reactions and reach take on a whole new meaning when your goal is to measure the quality of a conversation instead of, say, how viral an article went on social media. Instead, I’m looking at the relationship between qualitative and quantitative data. For example, I know that the number of comments in a thread isn’t indicative of meaningful interactions between members. Beyond that, I’m interested in metrics that standard analytic tools can’t measure. How many Republicans, Democrats or Independents started a thread? How many of them are actively participating? How many inter-party conversations were there?

We’re partnering with Civil Politics to do textual analysis of conversations in the group to help gain some insight on what these numbers mean. Moreover, once this project is through, we won’t be able to measure the quality of the experience for our participants with hard numbers. Engagement is treating journalism as a service. We can’t measure the impact of our project without recognizing the importance of qualitative measurements. Emphasis on hard numbers alone misses the point of engagement entirely.

But Nelson is right about one thing — the term audience engagement isn’t clear — to us or the public. Before we can discuss how to measure success, we must acknowledge that success varies from newsroom to newsroom or even from journalist to journalist.

For Spaceship, measuring success means putting the group first, listening to their needs and reading between the lines for story ideas before we come up with them ourselves. Because of the nature of this project, there’s a blaring distinction between audience (local newsroom partners) and community (women in The Many). But that’s not the case for everyone.

That engagement is an ambiguous term, as Nelson has claimed, is not necessarily a bad thing. There’s no blanket method for this kind of work. I’ve written before that engagement is best defined by what it’s not. However, we as an industry should take it upon ourselves to swallow Nelson’s criticism and digest it. That is what engagement journalists would do anyway. We can continue to do the work, better define what it is and what it means to us, until engagement journalism is accepted as simply … journalism.

Spaceship Media’s The Many is a closed, moderated Facebook group for women across the country and of all political stripes from hard left, hard right and everything in between. The group will run at least until midterm elections in November and is a place to talk civilly and respectfully about a range of political and social issues. Visit our website to learn more about the project and join the conversation by filling out this form.