A Framework of Severity for Harmful Content Online
This blog post summarizes the paper “A Framework of Severity for Harmful Content Online,” which offers a theoretical lens for researchers, policymakers, and moderators to assess the relative severity of harm of many different types of content online. This paper will be presented at the 24th ACM Conference on Computer-Supported Cooperative Work and Social Computing, a top venue for social computing scholarship. It will also be published in the journal Proceedings of the ACM (PACM). A free PDF version is located here.
CONTENT WARNING: This paper contains sensitive and potentially triggering content, including discussions of death, sexual assault, emotional trauma, self-injury, eating disorders, and animal abuse.
In 2014, a blog post that attacked and made false claims about game developer Zoë Quinn started a series of controversies and events later famously known as Gamergate. Prior research, as well as journalistic writings, broadly characterized Gamergate as a harassment campaign that induced vast emotional harm. But the harm caused by Gamergate was much more complex than “harassment,” and the people it impacted also extended beyond Zoë Quinn: multiple women who came to Quinn’s defense received prolonged rape and death threats. However, in stark contrast, Nathan Grayson, the reporter alleged to have exchanged favorable game reviews for a sexual relationship with Quinn, only received accusations of journalistic misconduct. These incidents clearly show that the harm in Gamergate was complex: it involved multiple kinds of harm, in different levels of severity for different people.
As demonstrated by the example above, alongside the benefits of online communities has been the proliferation of harmful content and interactions, ranging from hate speech to violent imagery to destructive misinformation. Content moderation is currently the predominant method for addressing harm on online platforms. Moderation can occur in three ways, often in tandem: volunteer moderation, paid commercial moderation, and automatic algorithmic moderation.
While moderation is currently the solution to mitigating harmful content and interactions online, ideally keeping most of that content away from general users, there are difficulties in enacting appropriate harm mitigation at scale and in ways all parties agree with. Given issues of prioritization and labor have been persistent in online moderation, how can we better understand the relationships of different harms to better triage?
To improve moderation practices, one possible approach is to consider the severity of different harms. Within social computing, there are a few existing frameworks that define relationships of harm, but none have considered the severity of that harm. Therefore, it remains unclear how to operationalize the moderation of different forms of harms. Further, it is difficult to discern when harm is most impactful and when.
To understand the relationships between the wide variety of harms that occur online, we conducted an interview study with two groups of participants: “experts” and general population. First, we interviewed experts who work in content moderation, platform policy, and occupations focused on harm assessment and mitigation, such as law. Second, we interviewed general population participants who use social media regularly. General population users offer a broader insight on what it is like to actually interact with the platform interfaces where harm occurs, and are the “end users” harm mitigation practices are meant to protect. We adopted constructivist grounded theory methods in conducting and analyzing our research. The figure shows the constant comparative analysis we did throughout the study.
We probed participants during interviews on their perspectives on severity in the context of their card-sorting decisions. The table shows the 20 content categories we asked participants to rank and discuss. Our goal was not to understand how specific instances in the card sorting activity were differently ranked; rather, we present a framework that describes the thinking behind severity rankings.
Through our analysis, we identified four Types of Harm (physical, emotional, relational, and financial) and eight Dimensions along which the severity of harm can be understood (perspectives, intent, agency, experience, scale, urgency, vulnerability, sphere). We present our findings in the form of a framework for researchers and practitioners, so that they can determine the severity of online harms — whether some harms are worse than others and what makes them worse. Our framework provides a tool to address the severity of differing harms as a set of complex, contextual, and overlapping factors.
Going back to GamerGate as an example, the high scale of harassment with so many actors across so many platforms and offline could be considered more severe than if it had been a low scale event of very few actors. There were also numerous types of harm and dimensions at play. For example, targets had low agency to stop attacks, attacks occurred using multiple mediums, and many actors had high intent to harm.
Deciding on the right approach to reducing harm in online communities is not a trivial undertaking, and we do not assert that there is a single correct solution. Gamergate, an example of a complex set of different harms that affect different people at different levels, shows that simply recognizing that a behavior is harmful is not enough — it is also important to understand how harmful the behavior is.
Therefore, we assert: the only wrong way to address harm is to not consider severity at all. After all, it clearly would be wrong to equate accusations of journalistic misconduct with death threats. When harms are co-occurring, particularly on large-scale social media platforms, understanding the relationships between them is crucial to deeper analyses of experiences of harm, prioritization pipelines for moderators, and more informed engagement with the perceptions of end-users on harm mitigation practices.
Morgan Klaus Scheuerman, Jialun Aaron Jiang, Casey Fiesler, and Jed R. Brubaker. 2021. A Framework of Severity for Harmful Content Online. Proc. ACM Hum.-Comput. Interact. 5, CSCW2, Article 368 (October 2021), 33 pages. https://doi.org/10.1145/3479512
