Since February 2018, YouTube has added notices of funding sources to a handful of state-owned channels so users could “better understand the sources of news content”. YouTube provided no details on desired outcomes from the notices and has not published any study of their impact, but has deferred to these notices when pressed on their role in spreading autocratic messaging. YouTube has not publicized its guidelines for choosing which state-owned channels to label, and no criteria are immediately obvious.
In an attempt to understand the effects of YouTube’s notices (i.e. “labels”), we conducted an analysis on the impact of labeling among Kremlin-owned channels. We found that labeling a channel as government-backed had no statistically significant effect on the channel’s subscriber count over time.
In a previous blog post on the YouTube-Kremlin partnership (see Reuters coverage here), we gathered a list of Kremlin-owned channels based on Wikipedia’s list of Russian media outlets and their ownership, verifying the ownership claims on the outlets’ websites. We noted a channel as “labeled” if any of its videos contained YouTube’s information panel indicating government-funded content, and “unlabeled” otherwise. About 40% of channels in our dataset, all of which were state-owned, were labeled as such. We then visited each channel’s page on Social Blade, a well-recognized social media statistics platform, to obtain historical subscriber counts. Since we were interested in following the longitudinal performance of channels over time, we limited our analysis to the subset of channels that have existed since the beginning of 2018, before the labels were introduced in February (n=45).
Since YouTube has not stated the expected results of its labels, we chose to measure the effect of labeling on subscriber count, which is often viewed as a proxy for a channel’s reach and influence. While labeled channels in our dataset had higher subscriber counts on average, many channels with massive subscriber counts remained unlabeled. The lack of any obvious characteristics distinguishing labeled and unlabeled channels offers a natural experiment for us to explore.
We used the Difference-in-Differences (DID) technique to estimate the causal effect of labeling on subscriber count. DID calculates the treatment effect of a policy (applying state-backed labels) that was implemented at a particular point in time (February 2018) by making use of observational data from before and after the policy change. Specifically, we pulled subscriber counts from January 1, 2018 and January 1, 2019 because some labels were said to have been applied in February of 2018, but others could have trickled in during later months. By comparing average changes in the outcome variable (subscriber count) over time between the treatment group (labeled channels) and control group (unlabeled channels), we can attribute the difference in the changes to the treatment itself.
Since we do not know how YouTube selected state-backed channels to be labeled or not labeled, the DID estimator may suffer from some statistical biases. For example, the treatment and control groups are assumed to experience parallel trends in subscriber count over time in the absence of treatment, i.e. if labeling had never occurred. This assumption would not hold if YouTube selectively labeled channels experiencing high subscription growth rates. Despite these limitations, the DID estimator can still provide insight on the direction and magnitude of the impact of labeling.
The DID estimator did not find a statistically significant treatment effect at the 5% level. Thus, we cannot claim that the state-backed labels had any impact on subscriber count.
We could conduct a more statistically rigorous analysis if YouTube shared more information about its labeling process and goals. What characteristics of a channel determined whether it would receive a state-backed label? What behavioral change did YouTube hope to achieve through this policy change? How does the company define success in this endeavor? YouTube is in a unique position to help preserve democracy, and simply adding state-backed labels is not enough. YouTube must increase transparency into its transparency initiatives to allow the public to make more informed viewership decisions.