Helping Authors Understand Toxicity, One Comment at a Time
Our team at Jigsaw uses artificial intelligence to spot toxicity online, and part of our work focuses on how to make that information more useful to the platforms and publishers that need it to host better conversations. Sometimes that means helping platforms moderate conversations more effectively, but we’ve also been exploring how we can help the users — the people actually writing the comment or post — better understand the impact of their words.
We all understand how toxicity online makes the internet a less pleasant place. But the truth is, many toxic comments are not the work of professional trolls or even people deliberately trying to derail a conversation. Independent research also points to how some people regret posting toxic comments in hindsight. A study we did with Wikipedia in 2018 suggested that a significant portion of toxicity comes from people who do not have a history of posting offensive content.
If a significant portion of toxic content was just people having a bad day or a moment of tactlessness, we wanted to know if there was a way for us to harness the power of Perspective to provide real-time feedback to people as they were writing the comment — just a little nudge for people to consider framing their comments in a less harmful way. Would that extra moment of consideration make any difference?
Several of our partners using Perspective API added what we call “authorship feedback” into their comment systems, and we worked together to study how this feature affects conversations on their platforms. The idea is simple: they use Perspective’s machine learning models to spot potentially toxic contributions and the platform provides a signal to the author right as they’re writing the comment. (This required carefully crafting the feedback message: eg. less-than-encouraging messages can have completely the opposite effect.) By suggesting to the user that their comment might violate community guidelines, they have an extra few seconds to consider adjusting their language.
Here’s what we learned from those studies.
