3. Are we using the right metrics? Do pageviews get more attention than they deserve? What metric most accurately captures the added value of a media outlet? Must this metric differ when measuring journalistic added value (good done in the world, corruption exposed, etc) as opposed to business added value (revenue potential, engagement, etc)? If pageviews isn’t the optimal measure, which stakeholder loses the most as a result of that imperfection, and thus has the most to gain from better metrics? Is there a way to disincentivize click bait and incentivize investigative or impactful reporting?
Are there metrics POLITICO isn’t currently using that could shed better light for our clients of the value we are providing them and the importance of our content to readers? Are there better ways to systematically store and deliver narrative information on the impact of POLITICO’s journalism? Does the fact that POLITICO’s articles and slideshows are paginated cause more harm by inconveniencing our readers than benefit by increasing pageviews? What could POLITICO gain by adopting Upworthy’s “attention minutes” measure, detailed below and released a few weeks ago?
Nieman Lab — August 17, 2012
Metrics are powerful tools for insight and decision-making. But they are not ends in themselves because they will never exactly represent what is important. That’s why the first step in choosing metrics is to articulate what you want to measure, regardless of whether or not there’s an easy way to measure it. Choosing metrics poorly, or misunderstanding their limitations, can make things worse. Metrics are just proxies for our real goals — sometimes quite poor proxies.
…it might not always be the case that a larger audience is better. For some stories, getting them in front of particular people at particular times might be more important.
If journalism is supposed to inform, then one simple impact metric would ask: Does the audience know the things that are in this story? This is an answerable question. … The point is to define journalistic success based on what the user does, not the publisher.
While newsrooms typically see themselves in the business of story creation, an organization committed to informing, not just publishing, would have to operate somewhat differently. Having an audience means having the ability to direct attention, and an editor might choose to continue to direct attention to something important even its “old news”; if someone doesn’t know it, it’s still new news to them. Journalists will also have to understand how and when people change their beliefs, because information doesn’t necessarily change minds.
Fortunately, not all data is numbers. Do you think that story contributed to better legislation? Write a note explaining why. … Not every effect needs to be expressed in numbers, and a variety of fields are coming to the conclusion that narrative descriptions are equally valuable. … The important thing is to collect this information reliably and systematically, or you won’t be able to make comparisons in the future. … Ideally, a newsroom would have an integrated database connecting each story to both quantitative and qualitative indicators of impact. … With that sort of extensive data set, we stand a chance of figuring out not only what the journalism did, but how best to evaluate it in the future. But nothing so elaborate is necessary to get started. …
Most importantly, we need to keep asking: Why are we doing this? … It’s impossible to evaluate impact if you don’t know what you want to accomplish.
Nieman Lab — February 6, 2014
This morning, Upworthy announced it would be using a new metric — “attention minutes” — as its primary tool to determine how they’re doing. The goal is to blend traditional eyeball-counting metrics with figures that more accurately measure engagement or how much the audience actually likes the content that they’re making. And they’ll be releasing their source code so other publishers can use it.
[Upworthy:] We love thinking this way because it rewards us for sharing content that people really enjoy and find valuable — not just stuff they click on a lot. It may mean that we don’t do quite as well on uniques or pageviews, but that’s a tradeoff we’re happy to make because this is a metric focused on real user satisfaction.
Of course, Upworthy isn’t the only publisher feeling dissatisfied with the mighty pageview and its many successors. … The New York Times [is] working on “pageviews above replacement,” … YouTube’s “Time Watched,” Medium’s “Total Time Reading,” and Chartbeat’s “Average Engaged Time.”
Attention minutes are calculated using an assemblage of what Upworthy calls “signals.” [Upworthy chief executive Eli] Pariser says they started out by asking, “What are the ways you can know if it’s a sad, lonely, ignored tab or if someone is watching intently?”
Some of these signals include the length of time a browser tab has been open, how long a video player has been running, and the movement of the mouse on screen. “This method doesn’t store raw information — it follows a stream of events and every few seconds asks, ‘Is this person still paying attention?’” Pariser said. “If they are, that sends a signal to a data warehouse that says, ‘Chalk up a few more attention seconds for this post.’”
If you measure performance in pageviews, you encourage slideshows. If you measure performance by social shares, you encourage clickbait headlines and giant Like buttons. Finding a metric that lines up with a publisher’s goals is one of the most important things it can do to encourage better work.