Metrics
We get what we measure and we are measuring the wrong things in media. Our old, mass-media metrics of reach and frequency — translated into their digital equivalents: unique users and pageviews — turn out to be profoundly corrupting. They drive media outlets to do anything to bulk up their numbers, prioritizing quantity over quality of service and engagement and motivating some to game the system with dark-hat search-engine optimization and click bait, not to mention click fraud. Now I don’t mean to suggest that we from legacy media were pure as arctic ice. I worked for the New York Daily News. I wrote tabloid headlines to entice readers to buy my newspaper — coin bait rather than click bait. But online, small temptations can turn into systems of corruption at scale. Three illustrations:
About.com was the first media company built for the Google age. It produced the answers to questions commonly asked in Google searches. Thus its content rose up high in Google search, getting it huge traffic from Google as well as highly relevant Google ads and millions of dollars a year in revenue. That was a virtuous circle until Demand Media and other so-called content farms came along to game the system, creating a platform to pay writers with a few spare minutes the minimum possible to make content that was barely good enough but would still rise in search, attracting easy audience and easy advertising. Google ended that gravy train — throwing About.com’s baby out with Demand Media’s dirty bathwater — to protect the quality of its search results.
Buzzfeed and Upworthy perfected the listicle and the grabby headline (You won’t believe what happened next but it will change your life!) to manipulate not Google’s search algorithms but instead us, enticing people to share and “like” their content. But that bubble, too, is set to burst as Chartbeat found that people share before they read — if they ever read at all. There’s a reason that Huffington Post, for example, displays links to share — to Facebook, Twitter, Reddit, StumbleUpon, Tumblr, Google+, and email — right next to its headlines rather than at the bottom of its posts. Sharing a link turns out not to be an indication of engagement — as in, “they liked my story so much they had to share it with their friends.” Sharing pumps up mentions and maybe unique users and pageviews but not attention. So what good is it?
Finally, consider the newspapers that got so good at SEO and social links (not to mention the occasional and unpredictable burst of pageviews that comes from a link by Drudge) that they exploded their traffic but ended up with large proportions of users from outside their markets. That distant traffic is next to worthless — as I mentioned earlier, an in-market user can be worth 20 times what an out-of-market user is worth to a news site that has only in-market advertising sales. But most newspapers I know got caught in an egotistical traffic game: size for the sake of size and bragging rights.
These examples of the corrupting nature of unique users and pageviews make the case to build a new set of metrics upon which to build news and media businesses.
The Demand Media example argues for starting with basic engagement metrics: not how many people glance off a site and then disappear like a meteor shower hitting the atmosphere, but how many people return, how often, and how long they stay and engage. The interactivity of online allows for many more acts and measures of engagement, some of which I listed earlier in my reverse-pay-meter scenario: from joining in a discussion to contributing content to collaborating in the work of a site.
The case of shared and liked links producing little engagement and value augers for switching to attention metrics. Chartbeat’s Tony Haile has argued convincingly that attention is a much more productive measure of media value than traffic. I told how Medium and FT.com plan to charge advertisers for the total attention minutes they get instead of mere impressions. Bonus: TV advertising is also sold on time and advertisers love buying TV, so we can hope that selling time online will help draw dollars away from TV to online news ventures.
And my newspaper example is evidence of the need to shift to relationship metrics: How many people do you know? What do you know about them? (Start with finding out whether they live in your market.) What reasons did you give them to reveal themselves to you? What relevance and value did you return to them for the data they shared? What value were you able to recognize as a result? What do you know about the interrelationships of people inside your community? How can you help them to better connect with each other, sharing knowledge, helping, and transacting with each other? What do you know about their behavior because you have a relationship with them? How does that knowledge drive not only usage, engagement, and loyalty but also revenue? What more do you know about their demographics and buying behavior because you’ve built trusted relationships and how will that increase the value of what you sell to advertisers? If the relationship strategy offers an opportunity to rethink journalism as a service business that builds real and lasting value, we need to measure the things that will drive that strategy.
In the long run, attention, engagement, demographic, behavioral, and relationship metrics will beat mere mass measures. That’s because advertisers are waking up to realize that they, too, must operate by such new measurements. But all the metrics I have listed so far still tend to be mediacentric: How many people read, returned to, spent time with, liked, or interacted with us and our stuff?
There is a richer set of metrics that matter to the mission of journalism — metrics around impact and accomplishment. These metrics must start not with us but with the people we serve. They must measure whether they meet their needs and accomplish their goals. Thus journalism built around impact doesn’t start with producing media’s content; it starts with listening to the public’s wants, needs, and goals so we can measure our success ultimately against whether those goals are achieved. (Therein lies the heart of our rationale for starting a new degree at CUNY in social journalism, a journalism built first on listening to and understanding and then serving a community; more on that in the afterword.)
Impact is terribly difficult to measure. It cannot easily be reduced to a number in a chart. But that is precisely why we should pursue these measures, because they force us to see people as individuals again; they force us to listen. Columbia University scholar James Carey argued that media were corrupted by the desire to reduce people to mere numbers — pro vs. anti, red vs. blue — as exemplified by the public-opinion poll. Polling, he said, “was an attempt to simulate public opinion in order to prevent an authentic public opinion from forming.” His alternative: conversation with the public. “I believe we must begin with the primacy of conversation,” Carey wrote. “It implies social arrangements less hierarchical and more egalitarian than its alternatives.” Yes, we have conversations online now, plenty of conversation in bottomless comments sections under our articles. But those comments are still mediacentric, reacting to the content we produce. No, the proper conversation journalists should have must start with the public. These conversations must start not with us speaking but with us listening. Thus I come to better understand my friend Jay Rosen’s interpretation of Carey’s admonitions to journalists: “The press does not ‘inform’ the public,” Rosen said. “It is ‘the public’ that ought to inform the press. The true subject matter of journalism is the conversation the public is having with itself.”
Now I return to amend the relationship metrics I proposed above: How many people have we met — where possible, face-to-face? How rich and informative were our conversations with them? How well do we understand them and their communities? How well have we heard or discerned their needs and goals? That is the starting point.
Only then, only when we have listened well, can we ask the next questions: What does an individual or a community need to accomplish its goals? How can we help? Is it information you need? How can we help you share what you already know? If what you need is not known, then how can we bring reporting to the task? Is it understanding you need? Then how can we bring explanation and context or education? Is it functionality you need? Do we have the skills to implement or build that? Is it organization you need? Can we help convene a community to deliberate and accomplish its goals?
Only after we have contributed to a community’s efforts to reach its own goals can we begin to think of measuring success — that is, impact. Chalkbeat, a beat business that covers schools in a scattering of cities across America, asks whether its communities have improved their schools. Founder and CEO Elizabeth Green emphasizes that Chalkbeat does not say how that should be done or even how that should be measured (for its users will debate long and hard about, say, the efficacy of testing as a metric of educational quality). But Chalkbeat can listen to whether the community says its schools are better. The investigative reporting enterprise ProPublica asks whether corruption is reduced and thus equality enhanced. A local blog might ask whether neighbors see improvements in the quality of life in their communities.
These questions are not easily answered — if at all — with numbers. The answers are by their nature qualitative and made up of signals that can be gathered in ongoing conversations. That may seem like a quixotic equation that will frustrate an industry and its financial supporters — whether investors or philanthropists — accustomed to running the business according to easy numbers. But the more valuable journalism and the services it provides are to communities, the more valuable those enterprises will be. That is an article of faith that guides Silicon Valley startups: Start with making yourself useful and value will follow. If it’s good enough for them….
You can buy the book or the Kindle at Amazon or buy it directly from OR Books.
