Fake news outperformed real news on Facebook? Case not proven

Buzzfeed’s data doesn’t support its conclusion

Buzzfeed has published new data that, Craig Silverman says, show that fake news outperformed real news on Facebook in the latter stages of the presidential campaign.

Well, sort of. I’m as anti fake news as anybody, and think Facebook needs to be much more accountable in the way it handles it.

But Buzzfeed’s analysis is limited.

It’s analysed the total shares, reactions and comments of the 20 best-performing “false election stories from hoax sites and hyperpartisan blogs”, compared with the total shares, reactions and comments of the 20 best-performing stories from a selection of major news outlets. There are 8,711,000 of the former compared to 7,367,000 of the latter.

So what’s wrong with this?

The problem is with picking a relatively small number of stories and implying these represent news across Facebook as a whole. In fact, individual stories — no matter how grotesque, nor how widely shared they are — may well be highly unrepresentative of the entire platform, given how much content is shared and how many people use the platform. Even the highest-performing story, the notorious ‘Pope endorses Trump’ piece on Ending the Fed, had less than a million ‘shares reactions and comments’. That’s a lot, by the standards of most mainstream political news stories, but when you’ve got 1.18 billion daily users, it’s not that much.

Do we have any particular reason to think that this sample is unrepresentative? Well, as it happens yes we do, even within Buzzfeed’s analysis. Although the biggest stories in its ‘fake/hoax’ list score more highly than their equivalents in the ‘major news outlets’ list, the numbers drop away more quickly. No 20 on the fake list got 149,000 shares/reactions/comments, compared to 234,000 on the major provider list.

If we took the list further, and got more into the long tail of both segments, we’d find major news stories on average performing better than the fakes. It’s possible, in my view likely, that if you aggregated all shares/reactions/comments on all stories from the chosen major news providers, you’d have a much bigger total than for those from fake sites.

I’m not dismissing the problem here. If millions of people are sharing, liking or commenting on a fictitious story that says the Pope endorses Donald Trump, how many more millions are seeing it, and how many believe it?

But the problem with Buzzfeed’s analysis is, it’s easy for Facebook to bat this kind of weak criticism away — which it has, pointing out to Buzzfeed that “there is a long tail of stories on Facebook. It may seem like the top stories get a lot of traction, but they represent a tiny fraction of the total.” Buzzfeed says that Facebook also said that native video, live content, and image posts from major news outlets saw significant engagement on Facebook — presumably not reflected in its analysis.

If we want to force Facebook to take this problem seriously, we’re going to need better data. Ironically of course, nobody’s better placed to do this analysis — or make the data available to others to do so — than Facebook itself.

EDIT: I swapped messages with Craig Silverman on this, who agreed main news sites have more engagement overall. But pointed out that this was the best that could be done with the data available.