Hashtag Burnout? A Control Experiment Investigating How Political Hashtags in Social Media News Articles Shape Reactions to News Coverage Around Social Issues
This post summarizes the CSCW 2019 paper “Hashtag Burnout? A Control Experiment Investigating How Political Hashtags in Social Media News Posts Shape Reactions to News Content” by Eugenia Ha Rim Rho and Melissa Mazmanian.
The purpose of our research is to investigate whether the practice of including political hashtags in social media news posts motivates or deters people from wanting to further engage with related social issues and by large, their impact on civil discourse in online venues through news articles on social media.
Motivation of Our Experiment
Online social movement activists and news organizations assume that trending political hashtags effectively capture the nowness of social issues that people care about. With growing social media presence, news companies increasingly use political hashtags in article headlines and social media news post — a practice aimed to generate new readership through lightweight news consumption by linking a particular story to a broader topic.
Asides from journalists, we know that people who use political hashtags — supporters, activists, politicians, aka hashtag producers — are doing so to denote alignment with an issue, personalize expression of why a particular story is important, and encourage others to engage with the content. However, we do not know how this practice is received and whether or not the general audience (majority of whom are passive content consumers merely exposed to hashtags) engages with hashtagged content in a manner that is aligned with this intent.
In fact, response to political hashtags can be complicated as demonstrated with the events surrounding #MeToo and #BlackLivesMatter. Despite the initial positive enthusiasm, overtime, reports show increasing backlash and polarization against key issues embodied by political hashtags. In this vein, we assume that political hashtags affect how people make sense of and engage with media content. However, we do not know how the presence of political hashtags — signaling that a news story is related to a current social issue — influences the assumptions potential readers make about the social content of an article.
Experiment Design
To address this, we carried out a randomized controlled experiment across 1979 participants, to investigate how the presence vs. absence of political hashtags (particularly the most prevalently used #MeToo & #BlackLivesMatter) in social media news posts (actual articles shared on Facebook by news publishers) shape reactions across a general audience. We launched a survey randomizing the display of news posts to participants who answered questions & commented in reaction to the news post that either contained or excluded the political hashtag (control group) as shown in Figures 1A & 1B.
We also included positive and negative comments as a second factor to mirror closeness to reality of the news posts in the social media ‘wild’ as they often contain affective comments under the article comment threads. Hence, our 2x3 factorial experiment was designed with the following factors and levels:
- Political Hashtag: (1) included, (2) excluded in the news post
- Comments: (1) no comment, (2) positive comment, (3) negative comment
Key Findings
Below, we illustrate a few snippets of our findings from the experiment. For full details, please refer to the paper. Key findings from our work show that:
1) When hashtags are included in news posts:
- people perceive the news topic to be significantly less socially important.
- people are less motivated to know more about social issues related to the news content.
Our regression output (Table 2) below demonstrates that the presence of political hashtags in news posts negatively affects people’s perception that the news topic is socially important (a .048 standard deviation decrease in perception of social importance of news topic).
Our control variables also significantly contribute to a decrease in perception of social importance, showing that men (compared to women) and those who are politically conservative (compared to those identified as politically liberal) find the news topic less socially important. Comment valence is also a significant predictor. Compared to those shown news posts appended with negative comments, participants who saw news posts with positive comments towards the news content found the news topic to be significantly more socially important.
Similarly, when political hashtags are included in news posts, people are significantly less motivated to know more about related social issues (a .053 standard deviation decrease in motivation to know more about social issues related to the news post). The control variables — gender, political view, and comment valence — also have a significant impact. Men, politically conservative individuals, and those who were shown negative comments significantly want to know less about social issues related to the news post compared to their counterparts who are female, politically liberal, and from the positive-comment group.
2) It’s not just the conservatives, but also the politically moderate who have strong partisan reactions to news posts with political hashtags.
In order to see how political hashtags shape partisan reaction towards news content across the political spectrum, we used kernel density estimate (KDE) to visualize the distribution of responses from those who identified as “Extremely Liberal”, “Moderate or Middle of the Road”, and “Extremely Conservative” (Figure 2).
The distribution of responses among those who identify as extremely liberal for both hashtag and control groups (Fig.2, left) is left-skewed, meaning in general, these folks tend to regard social media news posts about gender & racial issues less partisan regardless of hashtag presence. By contrast, for the extremely conservative participants (Fig.2, right), the distribution of responses is more right-skewed. However, the distribution for the hashtag group (blue) peaks higher and is even more right-skewed than the non-hashtag group (red), implying that hashtag presence makes extremely conservative people perceive the news as more partisan.
While these are somewhat expected results, what is really interesting is the people in the middle (Fig.2, center). Among those who identify as politically moderate, those who saw news posts with hashtags (blue) perceive news posts to be more partisan than those shown news posts without hashtags (red) as illustrated in the shaded gap between the blue and red distributions (for independent t-test results, see paper).
3) In fact, among the politically moderate respondents (who identify as neither liberal nor conservative), those shown news posts WITH political hashtags react in their comments with
- More suspicion towards the content credibility of the news post
- Greater focus on the politics of the hashtag
- Less attention directed towards the social topic portrayed in the news
Figures 3A and 3B, are identical news posts — one version containing the hashtag (left) and one without (right). The table below are comments written by politically moderate respondents in reaction to the news posts shown in respective versions.
As shown, there are obvious differences in discourse patterns between the hashtag and control groups in their commenting behavior. Take comments from the hashtag group for example:
“This is a load of crap on a number of levels. When broadly defined, Pluto was a planet too. The numbers on this topic have been inflated before and it appears they have again.”
“I’m sorry — 43%of men?!?!?! Now that’s a load of shit. Especially 17% of “unwelcomed sexual touching”. This article reads “FAKE NEWS”.”
Compare these with comments from the control group:
“My first reaction is that these numbers are really high. Most women are sexually harassed, and almost half of men are sexually harassed. That’s remarkable and sad. Looks like a big social problem.”
“I find the information listed in the article to be non-biased. Frankly these numbers seem low to me.”
“I like that they are showing the statistics of the percentage of men that experience sexual harassment as well and not just the women.”
Respondents from both groups are reacting to the same details shown in the news post; however, the interpretation as well as the language used to interpret the content are vastly different between the two groups.
Comments from the hashtag group refer to the news article statistics as “load of crap” or “load of shit” while reactions from the control group are much more nuanced and subdued: “My first reaction is that these numbers are really high” or “Frankly, these numbers seem low to me”.
Furthermore, the linguistic style of comments from the hashtag group are more emotionally intense, marking extreme reactions towards the news content (e.g., “43%of men?!?!?!”). Even those who seem to be in favor of the hashtag movement use aggressive language to convey support of the movement and refer to those against it as “You idiots”, claiming, “there’s a reason why [#MeToo] f****ng exists, dimwits!!”.
**For a more deep-dive interpretation of these comments, please take a look at our paper.**
4) Finally, if you’re on Facebook several times per day, you’ll find news content with political hashtags more partisan compared to those who use Facebook less often.
Assuming those who use Facebook more frequently are likely to have a higher chance of being exposed to these social media hashtags, we compared responses around perception of partisan bias towards news posts across different levels of Facebook use — between those who were high (use Fb several times per day) versus low (less than once per week) intensity users. We visualize this difference in Figure 4, which shows an interesting difference between the low and high-intensity users in terms of perception of partisan bias towards hashtagged news posts.
Among those who use Facebook less than once per week (Fig.4, left), the distribution of responses between the hashtag and the non-hashtag groups are similar, meaning partisan reaction towards the news content is not really affected by the presence of political hashtags. However, for those who use Facebook several times per day (Fig.4, right), the distribution of responses for participants who were shown hashtags (blue) is more skewed towards the right, in other words, perceive the news post as more partisan, compared to the control group (red). For results on statistical significance of difference, see paper.
In fact, repeated exposure to political hashtags can lead to a sense of saturation and the false drumming up of interest in hashtag movements. From the perspective of a participant who uses Facebook several times a day, such interest is perceived as forced and overstated:
“Honestly, social media is generating something akin to false positivity, or inflated enthusiasm. I am so entirely sick of this NPR American bullshit. I am very liberal; I am not racist or anything but it’s just too much. If I see another post on the Internet about someone being a “Strong Woman”, I’m going to lose my fucking mind. No one is benefiting from this. We are not really more “informed” as a society now that the internet is here… less information and more of the same opinions on the same side of the line…As an intellectual I cannot cosign this mentality” (Female, 35, Liberal).
If you want more details on our work or if you find this research helpful, please consider further reading & citing below. For any questions, please feel to reach out to the authors.
Eugenia Ha Rim Rho and Melissa Mazmanian. Hashtag Burnout? A Control Experiment Investigating How Political Hashtags in Social Media News Posts Shape Reactions to News Content. 2019. CSCW. ACM, Austin, TX. 25 pages.
Eugenia is a PhD candidate at the Informatics Department of University of California, Irvine and recipient of the Roberta Lamb Fellowship.
