Beyond the cult of numbers
We need a more sophisticated approach to social media metrics for editorial decision-making.
As veteran ABC journalist Chris Uhlmann’s post-G20 rant about the U.S. President’s inability to act as the leader of the free world went viral around the world in early July, social media monitoring alarms would have sounded at ABC News Online: It’s not every day that even Harry Potter author JK Rowling lends a hand in promoting your content.
We know about the global echo to Uhlmann’s comments in good part because monitoring of the social media impact of news articles is now widespread practice in the industry. You don’t need to be BuzzFeed, whose fundamental business model has been built around crafting viral content, to be interested in measuring the return on investment of your journalists’ time; almost all news outlets now engage in such monitoring at least to some extent, drawing on a broad range of in-house and external solutions.
One concern that has been raised in response, of course, is that this will see the skewing of news towards increasingly bland, popular content and away from critical material that engages meaningfully with the substantial issues of the day — a buzzfeedisation of mainstream news, while Buzzfeed itself has begun to invest in producing more serious journalistic coverage.
But a second, even more insidious problem in this context is that while many news outlets are investing good money into their Facebook and Twitter monitoring efforts, their understanding of what those social media impact metrics actually mean remains rudimentary at best.
As BuzzFeed founder Jonah Perett has pointed out, there is no one universally applicable “god metric” that can be used as an online equivalent of TV ratings figures.
In turn, this absence of a significant market pull for better metrics has also led to stagnation in the development of more sophisticated analytics approaches: A 2016 report for the University of Oxford’s Reuters Institute for the Study of Journalism warns that “contemporary forms of analytics are very good at understanding the main ways in which people used digital media in 2010”, but have simply not kept up with the times.
Conversely, though, The Guardian’s director of architecture, Graham Tackley, also notes from personal experience that “when I’ve talked to people about … more [complex] metrics than page views they … really genuinely have no idea about what they actually mean”.
Instead, what has emerged to date often more closely resembles a kind of newsroom cargo cult, in which upward trends in basic audience metrics are celebrated and pursued as ends in themselves, rather than critically interrogated as indicators of broader patterns in audience engagement.
This resembles the mentality of a stockbroker who does not care about the fundamentals of the assets being traded but merely pursues a short-term points gain for its own sake. Or alternatively, we might employ the metaphor of one of the newsroom managers who participated in Edson Tandoc Jr’s 2014 study of the use of news metrics, who “compared using web analytics with getting hooked on drugs. ‘It’s like crack,’ he said, grinning. ‘You can sit here and watch it, popping all night.’ ”
In fairness, all those pretty graphs don’t just serve as addictive eye-candy. Tandoc’s article provides fascinating insight into the efforts of news outlets to maximise the shareability of their content, from A/B-testing headlines and accompanying images to fundamentally changing the balance of their coverage.
For ABC News, for example, the viral success of Uhlmann’s piece might create the temptation to encourage him to get ranty more often.
Yet who is served by such a simplistic pursuit of quantity over quality? What is the benefit to Australian news audiences if more locally produced stories reach a global audience? A much more sophisticated approach to incorporating social media metrics into editorial decision-making is needed here, and is only gradually emerging.
A significant part of the problem here is that many of the metrics on social media engagement, and especially those provided by major commercial social media monitoring services or by platforms like Facebook and Twitter themselves, remain black-box solutions that have never been subject to serious independent scrutiny. Though they are common across the industry, their customers are forced to blindly trust the numbers they produce, without knowing much about the interpretive assumptions baked into these metrics.
Metrics frameworks developed in-house at major news organisations often provide more transparent information to editors and journalists about what they count and how, but for commercial reasons such solutions are rarely shared and standardised across news organisations, and therefore don’t tend to allow for benchmarking across the industry.
Scholarly research may be able to address this dilemma (but I may be a little biased here). Australian academics are among the leaders in international internet research efforts, and some of us have developed some very long-term data sets on audience engagement with the news in leading social media platforms.
At QUT’s Digital Media Research Centre, for instance, we have been tracking the sharing of links to Australian news sites on Twitter since 2012 through the Australian Twitter News Index (ATNIX), resulting in a unique dataset that shows both overall long-term trends in site popularity (with ABC News and The Sydney Morning Herald consistently most shared, and news.com.au gradually catching up) and specific short-term spikes around particular issues and topics (Figure 1, above).
These longitudinal data enable an assessment not just of the absolute number of shares for any one story or site, but more importantly also of whether that number is significantly diverging from the long-term trend for similar content. Arguably, the second of these metrics is a great deal more important than the first. (I publish an overview of these trends on a monthly basis in The Conversation.)
But even such metrics are still insufficiently fine-grained to be truly meaningful. The next step in the development of news analytics must be to more thoroughly incorporate what we know of the demographics or — more appropriately — the post-demographics of social media users. Here, we are interested less in the socioeconomic status or other personal attributes of individuals as they may exist outside of social media platforms, and more in the underlying structures of the platform communities themselves.
For instance, it’s one thing for Uhlmann’s comment piece — and other coverage of the G20 or the U.S. presidency — to be shared widely amongst the usual suspects: those hardcore political junkies on either side of politics who are constantly engaged in Twitter’s #auspol hashtag and equivalent spaces on other social media platforms.
It’s quite another if these articles suddenly also reach social media users who ordinarily couldn’t care less about the political horserace. At that point, the dumpster fire that is the Trump White House has jumped containment lines and turned into a much more significant blaze that may indicate a fundamental shift in public interest and opinion.
Our work at the QUT Digital Media Research Centre has enabled some analysis along these lines already. For one, we’ve developed the first comprehensive map of follower/followee network structures in the Australian Twittersphere, which shows clear clustering tendencies around major topics of shared interest, whether teen culture or sports or politics.
For any given story or news site, this makes it possible to assess where in the overall network this content is being shared — does it pique the interest only of those groups who already have a long-standing interest in the topic, or does it also reach those who ordinarily could not care less? Does it spark just a brief flurry of localised retweeting, or does it go viral across the network?
Such metrics, again, must be read carefully, of course, and not every news organisation may pursue the same goals here. ABC News might ideally seek broad take-up across the entire population, while a specialty news outlet like Crikey may be more interested in deep engagement by a more narrowly defined group of followers. But whatever their specific aims, the incorporation of such more qualitative dimensions into previously largely quantitative metrics has the potential to result in a considerably more fine-grained, useful picture of how social media audiences engage with Australian news content.
Our own work in this field has focused largely on Twitter, in part because access to meaningful data on audience engagement with the news is slightly more straightforward on that platform than on others. But the social media landscape rarely remains static — there is a need to seriously explore the data available for other platforms, ideally through collaboration between researchers, news organisations and platform providers.
As BuzzFeed founder Jonah Peretti, who might count as something of an authority on the matter, has pointed out, there is no one universally applicable “god metric” that can be used as an online equivalent of TV ratings figures. Instead, we must continue to compare and correlate a range of data points that cover different aspects of news engagement by social media audiences.
This research is supported by the Australian Research Council Future Fellowship project “Understanding Intermedia Information Flows in the Australian Online Public Sphere”, and the ARC LIEF project “TrISMA: Tracking Infrastructure for Social Media Analysis.
This piece is from Issue 89 (August 2017) of the Walkley Magazine.
