What Makes a Good Metric?
Over the last nine years, I’ve spent a lot of my time thinking about audience, metrics, culture, and journalism. In that period one thing has remained a constant: a broad sentiment that the metrics we have aren’t good enough. A few years ago at the first Newsgeist Europe, I stood up in defense of page views. It still shocks me how surprising some people found that. It shocks me even more that people still instinctively recoil from a metric that does a simple job well.
The other side of this coin is the restless search for a better metric. Many people across the industry instinctively feel that our data isn’t complete without a number built around our higher ambitions or that captures the quality of a piece or publisher in a way that Reach and Time Spent cannot.
This is entirely understandable, not least because there may well be practical applications for such a measure at the industry level. That’s why Frederic Filloux and my brilliant former colleagues, Graham Tackley and Matt McCallister, have spent so much time and effort attempting to capture concepts like Quality and Impact in metric form. It’s essential that this kind of work happens, and it’s great to see Google’s DNI fund backing it. Working with Matt and Graham on impact, I learned a great deal about the fundamental problems you encounter when trying to nail down something this big and subjective in a way that makes sense.
A key thing to question is the belief that, because one thing can be measured, that another thing must be measured. This principle can lead to vast expense, the creation of aggregated metrics that can obscure detail or the use of proxies that can misinform and result in terrible decisions across an organization. Language is often bent out of shape to bridge the abyss between realism and idealism. Beware naming metrics after the initial aspiration rather than the finished product. Think about, for example, what the term “Unique Users” implies.
All metrics have the capacity to go wrong. Decent culture has to be established around any number to avoid warped behavior.