Human-Machine Collaboration for Content Regulation: The Case of Reddit Automoderator

This blog post summarizes a TOCHI paper about the use of automated tools for content moderation on the social media website Reddit that will be presented at the 22nd ACM Conference on Computer-Supported Cooperative Work and Social Computing in Austin, Texas.

Moderating content on social media websites involves making trade-offs between the goals of achieving high efficiency and sustaining low costs. It is possible that moderation systems can considerably prevent spam, harassment and other forms of abuse on a large community if enough expert human moderators are available to carefully review each post. But that may drive up the costs of moderation to unacceptable levels. One potential solution to minimize such costs is to automate content moderation.

Recognizing this, some scholars have begun developing automated (often, machine learning based) solutions to automate certain aspects of content moderation. Many platforms are also increasingly deploying tools that automate the process of identifying problematic material and taking appropriate moderation decisions. There is optimism in the industry that AI and machine learning tools can eventually replace the thousands of human workers who are currently involved in making moderation decisions, either voluntarily or as paid workers. But do these automated tools work in practice?

In this research, we investigate this question in the context of the social media website Reddit. We ask: How are automated tools used to help enact content moderation on Reddit? How does using these tools affect the sociotechnical process of moderating Reddit content? What are the benefits and challenges of using these tools? We participated as Reddit moderators for over a year, and conducted interviews with 16 moderators to understand the use of automated moderation tools.

While preparing for this study, we found that Reddit has an open-access API (Application Programming Interface) that allows bot developers to build, deploy and test automated regulation solutions at the community level. Access to this API has encouraged the creation and implementation of a variety of creative automated solutions that address the unique regulation requirements of different Reddit communities. We focused on one of the most popular automated tools, called Automoderator (or Automod), that is now offered to all Reddit moderators. Automod allows moderators to configure syntactic rules in YAML format so that these rules make moderation decisions based on the configured criteria. We show that Automod not only reduces the time-consuming work and emotional labor required of human moderators by removing large volumes of inappropriate content, it also serves an educational role for end-users by providing explanations for content removals.

An Automod configuration snippet written in YAML format. This rule labels all comments posted by users having less than 1 comment karma score as spam and removes those comments, assigning ‘Low Karma’ as the removal reason.

Despite the many benefits of using Automod, its use also presents certain challenges. It requires moderators to develop new skills like configuring Automod rules and conduct additional activities like defending against deliberate avoidance of Automod filters and correcting false positives. Since these new tasks are not trivial, the use of Automod creates new challenges of training and coordination among moderators. Our findings also reveal the deficiencies of Automod in making decisions that require it to be attuned to the sensitivities in cultural context or to the differences in linguistic cues.

A comment posted by Automod in response to its removal of a submission on r/photoshopbattles. Automod on r/photoshopbattles has been configured to automatically detect submissions that do not begin with ‘PsBattle:’, remove those submissions, and post this message as reason for removal.

Building on our case study of Reddit Automod, we provide insights into the challenges that community managers can expect to face as they adopt novel automated solutions to help regulate the postings of their users. For example, they may have a reduced level of control over how the regulation system works — as moderators reduce the number of posts that they manually review and delegate to automated tools, these tools may make mistakes that could have been avoided owing to their limitations of evaluating the contextual details. Moreover, moderators may not be able to understand the reasons behind some actions taken by automated tools. Another possible challenge is that moderators may have to make decisions about the levels of transparency they show in the operation of automated tools — if they are too transparent about how these tools are configured, these tools may be exploited by bad actors.

In addition to these challenges, we also highlight how the use of automated tools may affect how moderators design community guidelines. We found that Reddit moderators sometimes create posting guidelines that play to the strengths of Automod so as to make the work of moderation easier. For example, guidelines like “describe the image you’re posting” provide additional material for Automod to catch. However, complying with such guidelines may increase the amount of work that end-users have to perform. Therefore, using automated tools may affect not only the moderators but also the other stakeholders in content regulation systems.

We also highlight that an increased reliance on automated moderation tools can contribute to situations where content moderation may seem unfair. Since automated tools can’t always consider the context of a post, they may consistently censor individuals with certain viewpoints, and increase online polarization. On the other hand, these tools may catch and remove posts that are problematic only at the surface level but allow proliferation of bigoted viewpoints that are subtle and avoid automatic detection at deeper levels of meaning.

Usually, how content regulation occurs on social media platforms remains a trade secret and is not revealed publicly. In this research, we provide details of how Reddit moderators distribute the work of content regulation between human workers and automated tools. Our comprehensive description of Reddit regulation provides an important reference point for how human-machine mixed initiative regulation systems can be designed and deployed on other platforms.

For more details about our methods, findings and design suggestions, please check out our full paper that was published in ACM Transactions of Computer-Human Interaction (TOCHI). For questions and comments about the work, please drop an email to Shagun Jhaver at sjhaver3 [at] gatech [dot] edu. Citation:

Shagun Jhaver, Iris Birman, Eric Gilbert, and Amy Bruckman. 2019. Human-Machine Collaboration for Content Regulation: The Case of Reddit Automoderator. ACM Trans. Comput.-Hum. Interact. 26, 5, Article 31 (July 2019), 35 pages. DOI: https://doi.org/10.1145/3338243