Should publishers use Facebook’s Breaking News Label?

Lukas Weidenholzer
Apr 3, 2019 · 5 min read

In our most recent study, we conduct an analysis of Facebook’s upcoming feature for publishers, the Breaking News Label. After normalising for the popularity of each publisher’s content we find that shares to Facebook with the Breaking News Label get on average 51% more traffic than comparable shares without the label. However, we also observe that publications using the label typically see no change, or even a decrease, in overall traffic, clicks, reshares or engagement because other shares perform more poorly than they normally would when the Breaking News Label is used.

The Breaking News Label is an upcoming feature on Facebook that enables publishers to apply a visual indicator to stories that are considered breaking news. As part of Facebook’s ongoing effort to bolster the effectiveness of quality journalism on the platform, the Breaking News Label is designed to give publishers a tool to clearly differentiate breaking news posts from less time-sensitive editorial content. In March 2018, the feature was announced to be in closed beta, available only to a handful of select publishers.

Image source: Facebook

Publishers can choose for the label to remain active for a period up to six hours. Apart from the initial visual indication, the Breaking News Label also allows publishers to string together updates to an ongoing news story. To prevent overuse, each publisher can only use the Breaking News Label for one share per day, with an additional budget of five labels per month that can be used at any time.

To assess the value of this feature, our data science team has conducted a study into the effect of the Breaking News Label on publishers’ traffic from social media.

Among of the hundreds of publications relying on Echobox daily are several major publications which took part in the Breaking News Label beta. Since March 2018, they have used the Breaking News Label to tag close to 200 shares on their main Facebook Pages as breaking news. In our study, we compare how much traffic was generated by the labelled shares relative to over 8,000 shares that were not labelled.

When choosing the article to apply the Breaking News Label to, publishers will naturally tend towards articles they consider to be of high interest to their audience. This introduces selection bias, since these articles are not equally drawn from the population of all articles, but rather from a subset of high-potential articles. To avoid our results being distorted by this selection bias, we needed to identify shares for which the Breaking News Label has not been used but which are still representative of the kind of high-interest article a publisher might have used the label for.

To do this, we used our AI’s prediction of how well each article was likely to perform on social media, based on a large set of various factors. Combining this prediction and the actual performance after being shared allowed us to measure by how much each share outperformed our initial prediction. Figure 1 visualises this for shares with the Breaking News Label (orange) and those without it (blue). We observe that on average articles with the Breaking News Label outperform our initial predictions by a far greater degree than articles without the label. On average, shares with the Breaking News Label outperform our AI’s initial prediction of the share’s performance by 51%.

Figure 1: Outperformance comparison of shares with Breaking News Label vs. those without.

To test whether this increase could have occurred randomly, we drew 3,000 samples of 200 shares each from all shares without the Breaking News Label and measured how often a random sample would exhibit a comparable difference in performance. The observed variance in the data indicates a probability of less than 2% that the Breaking News Label posts outperformed the other posts by chance. This is strong evidence that individual shares with the label do indeed generate more clicks than they would have otherwise.

However, we do not find any impact on overall traffic. If shares labelled as breaking news benefited without any negative impact on other posts, a medium-sized publisher, with 25 shares to Facebook per day, could theoretically expect a ~4% increase in monthly social traffic by using the full monthly quota of Breaking News Labels. However, the same publishers for which we just showed the significant positive impact of the Breaking News Label on individual share performance actually saw an overall decrease of 2.5% for clicks, similarly for reshares, likes and comments after they started using the Breaking News Label (Figure 2). Additionally, when considering other publishers, we have found no evidence supporting the expectation that the Breaking News Label benefits overall page performance.

Figure 2: Total normalised clicks per day before (blue) and after (orange) the publishers started using the Breaking News Label. The vertical axis has been normalised using the average number of clicks before use of the Breaking News Label.

While the Breaking News Label is clearly effective at boosting the traffic of individual labelled shares, we have found that these performance benefits do not translate to any overall performance increases.

The root cause is most likely that shares for which the Breaking News Label was not used performed relatively worse than they could have. It seems likely that this is at least partly a symptom of an underlying cannibalisation effect. Shares with the Breaking News Label engage viewers’ attention more aggressively and could thereby make other editorial content on the page seem less interesting or relevant in comparison to the flashy breaking shares.

While the Breaking News Label is evidently generating increased viewership for breaking stories, it decreases traffic to other stories and therefore does not raise overall traffic from social media.

Watch a summary of our research here.
www.echobox.com

For more frequent updates, follow @EchoboxHQ on Twitter. For more about Echobox, go to www.echobox.com.

Echobox Insights showcases what we have learned about AI…

Echobox Insights
Lukas Weidenholzer

Written by

Data Scientist at Echobox

Echobox Insights

Echobox Insights showcases what we have learned about AI, social media and journalism in the digital age by building the first artificial intelligence that understands the meaning of content. Find out more about Echobox at www.echobox.com.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store