How Automated Tools Discriminate Against Black Language
Platforms can have the power to moderate not just content, but language itself
--
One of my favorite aspects of social media is coming across amazing work by activists, creatives, and academics. I get especially excited to see work by fellow women of color, whose perspectives are often left out of mainstream media and activism. So naturally, when I discover that posts by women of color are being filtered out of my feed, I am skeptical and upset but not surprised.
This recently happened as I was using Gobo, a social media aggregator and filtering platform created by my colleagues at the MIT Center for Civic Media. Gobo was created to address the lack of knowledge and control that people have over how their social media feeds are filtered. It aims to give control back to the user by allowing users to adjust how their feeds are filtered along six categories: politics, seriousness, rudeness, gender, brands, and obscurity. When the user adjusts the filters for each of these categories, posts will either get filtered in or out of their feed, and Gobo will tell them why.
I recently joined the Gobo team to understand how effective these filters are and how they could be more useful. As it stands, Gobo is less a product and more of a provocation for understanding how filtering works on social media platforms. Gobo is far from perfect, and all six filters demonstrate some unexpected or unwanted behaviors. For example, the filter for brands doesn’t differentiate between posts from corporations and posts from nonprofit organizations. The gender filter fails to take nonbinary folks into account. And the seriousness filter mislabels posts about harassment as being “not serious.”
The rudeness filter is what stood out to me the most — not necessarily because it performed worse than the other filters, but because of whose posts were being filtered out. When I set my feed to show me posts that Gobo deemed “less rude,” I noticed that many of the posts from women of color disappeared. Posts that were filtered out and marked as “very rude” included ones like these:
