Image for post
Image for post

Our team at Jigsaw uses artificial intelligence to spot toxicity online, and part of our work focuses on how to make that information more useful to the platforms and publishers that need it to host better conversations. Sometimes that means helping platforms moderate conversations more effectively, but we’ve also been exploring how we can help the users — the people actually writing the comment or post — better understand the impact of their words.

We all understand how toxicity online makes the internet a less pleasant place. But the truth is, many toxic comments are not the work of professional trolls or even people deliberately trying to derail a conversation. Independent research also points to how some people regret posting toxic comments in hindsight. A study we did with Wikipedia in 2018 suggested that a significant portion of toxicity comes from people who do not have a history of posting offensive content. …


Image for post
Image for post

By: Jared Cohen

Jigsaw’s work requires forecasting the most urgent threats facing the internet, and wherever we traveled these past years — from Macedonia to Eastern Ukraine to the Philippines to Kenya and the United States — we observed an evolution in how disinformation was being used to manipulate elections, wage war, and disrupt civil society. By disinformation we mean more than fake news. Disinformation today entails sophisticated, targeted influence campaigns, often launched by governments, with the goal of influencing societal, economic, and military events around the world. …


By: Daniel Borkan, Jeff Sorensen, Lucy Vasserman

In April we launched a Kaggle competition where we challenged the competitors to build a model that recognizes toxicity and minimizes unintended bias with respect to mentions of identities. For the competition, we released the largest known public dataset of comments with toxicity labels and identity labels for measuring unintended bias. We share datasets as a way to encourage and enable research that benefits the entire industry, and this data has already sparked some exciting research. To continue that momentum, today we are expanding this dataset by releasing the individual annotations from human raters.


Image for post
Image for post

Online gaming sites are one of the fastest-growing sectors of social media — the industry generates upwards of 300 billion by some estimates — but with that explosive growth comes issues with toxicity and online harassment. That’s the exact problem FACEIT, the leading independent competitive gaming platform for online multiplayer PvP gamers, wanted to solve.

In addition to creating a positive and immersive gaming experience for their more than 15 million users, FACEIT also wanted to incorporate innovative technology to enhance the work of the human moderators, while encouraging new ways for the community to engage with each other that was free of harassment without stripping the community of its personality. …


Image for post
Image for post

By: Sameer Syed

Local newspapers have the important role of helping people know about and act on local issues. Most readers consume their local news online, but still expect news providers to show a genuine connection to the community, like reporters personally engaging with the area or showing an understanding of the community’s history.

The Southeast Missourian serves an active group of readers who care about what’s going on in the community — and they aren’t shy about sharing their opinions. Readers post tens of thousands of online comments each year and almost every reader (86%) discusses articles they read online with friends, family, and community members. This digital townhall builds a sense of community, encourages neighbors to get to know each other and find out how local affairs will affect their daily lives. To keep the comments free of toxicity and harassment, the Southeast Missourian spent the last year experimenting with Perspective API, Jigsaw’s machine learning-powered moderation tool, and updating its own moderation policy. …


Image for post
Image for post

Taringa!, the second-largest social media platform in Latin America, is no stranger to comments. The platform moderates more than 150,000 comments each month and they’ve spent years creating a system that ensures every comment and piece of content contributes to a positive user experience. They came across Perspective in early 2019 and wanted to test how machine learning could enhance their moderation process, while keeping the user experience seamless and positive.


Image for post
Image for post

By: Patricia Georgiou, Marie Pellat, and Daniel Borkan

When we launched Perspective in 2017 our goal was to improve conversations online at scale. Since then we’ve added the ability to detect toxic comments in Spanish, partnering with organizations like El Pais, increased the accuracy of the English-language model with help from partners like The New York Times, Change a View, and Wikipedia, and built Tune, a Chrome extension using Perspective’s language models, to give readers control over the comments they see.

Today we’re excited to continue growing Perspective’s and Tune’s language capabilities. Perspective is now available in French for comment moderation, starting with Le Monde, one of the French-speaking world’s most trusted news sources, and Tune is available for French-speakers to customize how much toxicity they want to see in comments. …


Image for post
Image for post

Since we launched Perspective, our technology that uses machine learning to spot abusive language, we’ve experimented with new ways to leverage this technology. Perspective’s most common applications fit into two categories: helping community managers find and respond to toxic comments, or helping authors improve their contributions when their posts might violate community guidelines. Both of these use-cases are important, but neither directly empowers the largest part of the online community — the readers.

Most of us spend more time reading online comments than writing or moderating them. As we read, a single toxic post can make us give up on a discussion completely and miss out on reading valuable thoughts buried underneath the shouting. Toxicity also has a chilling effect on conversations, making people less likely to join discussions online if they fear their contribution will be drowned out by louder, meaner voices. …


Image for post
Image for post

Transparency is one of the five core values we set out for Perspective and the Conversation AI research initiative when we launched. We’ve honored that value by including a demo of our model on our website, and publishing research papers, datasets, and open source models. Today we’re excited to take another step to increase transparency by sharing our first version of a Model Card for Perspective API’s TOXICITY model.

What are Model Cards?

The concept of a Model Card is being introduced this week at the Fairness, Accountability, and Transparency* Conference in Model Cards for Model Reporting by M. Mitchell et al, a collaborative paper between researchers across Google and outside the company, including Jigsaw’s Conversation AI team. Model Cards are short documents that go alongside publicly available machine learning models to share information that those impacted by the model should know, such as evaluation results, intended usage, and insight into model training processes. Building on the growing research on algorithmic fairness and unintended bias in machine learning systems, the paper recommends that evaluations should be disaggregated across different demographic groups and communities, meaning that performance metrics are computed and published independently for different identity-based subsets of the evaluation data. Publishing per-identity evaluation enables developers to understand how model scores and performance might vary between identity groups, allowing them to make informed decisions about how, where, and whether to best apply the model.. …


Image for post
Image for post

Cyberattacks against democratic institutions are on the rise and have steadily increased in intensity over the past few years. With citizens across Europe heading to the polls in May, defending these organizations from digital attacks has become a pressing concern.

Today we’re announcing the expansion of Project Shield to European political organizations — extending free protection to campaigns and candidates ahead of the EU parliamentary elections in May 2019. Now both European political organizations will have the same access to Google’s defense technology that previously was only available to news organizations and human rights groups.

Project Shield was launched in 2016 to protect independent news and human rights organizations from DDoS attacks, but in the wake of major election breaches and targeted political cyber attacks, Project Shield can now be used to give political groups the same defense. …

About

Jigsaw

A unit at Google that uses technology to make the world safer.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store