Semantics: The hidden struggle of automated content moderation

State of the web

The digital age has come in hard and fast. Americans now spend an average of 11 hours a day interacting with digital media.¹ Often it’s hard to even remember life without it.

But think back to the early days of the web. The wild west of auto-playing Nsync songs, flashy cursor trails, and questionable web color schemes. I’m going to make the case that while websites are less painful on the eyes and ears these days, we live in more of a wild west of web content than we ever have.

Hiding behind complexity

As digital technologies get more and more complex, they also reach higher and higher levels of abstraction. That abstraction has coincided with fewer and fewer regulations as legislators often lack understanding and struggle to grapple with many of the nuances of these complicated technologies. Look no further than the Facebook congressional hearings for some prime examples of this.

This has left the field wide open for content companies to operate largely without restriction, often left to police themselves. In that freedom, many have grown in their scope and reach to colossal, global scales. Several of these companies are now a daily feature in billions of lives.²

“Technology isn’t eroding the fabric of society, it has become the fabric of society” — Tristan Harris.³

Yet that scale, paired with the lack of regulation and oversight has meant that despite the direct impact these platforms have on the lives of their users, there is little required accountability.

Many of these content platforms are often used, both voluntarily and involuntarily, as testing grounds for psychological experiments, without any of the social impact consideration psychological experiments are typically bound to.⁴

Voluntary user testing at that scale garners concern, but could also provide a massive social asset with that unprecedented scope now available to researchers. Still, a larger risk can come from the psychological impact bad actors can create utilizing that same access.

Such a powerful tool then needs to be transparent and well maintained. There, we run into a pressing technological challenge; content moderation.

There’s a lot of posts.

We’ll start with a case study, Facebook.

With their rapid growth and massive global reach recently clocked at 2.6 billion monthly active users. Facebook often sits at the forefront of the technical problems troubling social media companies.

Every 60 seconds on Facebook there are: 317,000 status updates; 400 new users; 147,000 photos uploaded; and 54,000 shared links.⁵

That’s a whole load of memes, latte art, dog pictures, and questionable political posts.

Out of that quagmire of varied content arises the online ethos which seems to largely shape our social fabric. More and more, people turn to these platforms as their primary source of news. In the US 55% of adults now get their news from social media either “often” or “sometimes”.⁶

Wading through that sea of content.

So, given that as we mentioned, this volume of content and it’s respective reach is of large importance to social psychology. How can we ensure that content remains clean and honestly representative?

There’s no conceivable way to look through all those posts by hand without practically having an employee per user. What are some technologies that help comb all those posts down, so we can focus on ones that might need review under the platform’s Terms of Service?

For the sake of brevity we’ll keep to a top-level abstraction, but this basic review will hopefully help jump into the discussion on where these moderation technologies are used, and where they can fall short.

Digital Hash Technology

Digital hash technology is the practice of taking an image or video and converting it down into a numerical value specific to that image or collection of keyframes.

This can then be stored and compared against other content uploaded to the site to check for copyright infringement or already known prohibited content.

Pattern Recognition

You’ve probably seen a few instances of image pattern recognition systems floating around the web. Take for example the popularly despised reCaptcha. Is that grey blob a traffic light? I don’t know.

Often these machine learning systems are set up with an initial network of categorizing reference content that allows the machine learning system to form a collection of patterns. It can then look for content that matches those patterns with a specific percentage certainty. For example, whether an image or an element of the image is of a car, a dog, or a person. Or perhaps when it comes to social content, a post who’s image violates a particular term of service.

Natural Language Processing

For the text body of posts, a different machine learning process, NLP allows for analysis of text posts on a variety of dimensions such as tone or content. Similar to image recognition, the system is often built off of models based on training data.

There are several different implementations, but for content moderation something like sentiment analysis may be employed to analyze the tone of a post. That can then be used in conjunction with other implementations to try flag racism or hate speech.

Cool, but I’m still seeing some weird stuff in my feed.

Unfortunately as I’m sure you might have discerned just from reading the title, automated content moderation just isn’t quite there yet in many places.

These technologies have lots of potentials, and undoubtedly help ease the burden of moderators. However, they struggle with a few fundamental issues, most pressingly semantics and contextual meaning.

Semantics here refers to the study of meaning as it relates to a particular phrase or word. Context is a piece of that puzzle. How does the location or structure of that sentence or that word influence its meaning?

Consider the phrase:

“I saw him on the hill with a telescope”

Even this simple short sentence has a variety of interpretations.

Did you view him on the hill using a telescope?

Was he on top of the hill holding a telescope?

Did there just happen to also be a telescope on the hill?

That’s a bit contrived, but it demonstrates how fuzzy things can get. When we start to drift into determining whether certain posts contain, say, hate speech; the nuances of semantic meaning become even more important and far less defined.

We have a tough enough time determining what people mean when speaking to them face to face, and there we are afforded the luxury of facial expression, tone of voice, and other social cues. Technologies such as NLP often lack the contextual resources to be able to make an informed adjudication on a piece of content.

What happens when it goes wrong?

This lack of contextual understanding has created difficult situations on both sides of the content moderation debate. In 2016, Facebook removed a famous Pulitzer winning photo of the Vietnam War posted by Norway’s Prime Minister Erna Solberg because it contained the image of a naked 9-year-old girl fleeing after a napalm attack.⁷

The Prime Minister had posted the photo in response to it having been removed on many other notable pages. In a lengthy memorandum, the Prime Minister struck out at what she considered “An overstep of authority”, stating “I don’t think you seriously considered what you were doing before removing this post”.

In this case, the digital hash system found something which would have nearly perfectly fit one of its similarity rules for restricted content. It did its job exactly as intended, but it hit a nuanced exception to the rules. There likely wasn’t a malicious premeditated attempt from Facebook to infringe and restrict free speech, but rather a wide range of technical edge cases which aren’t yet accounted for.

These are the cases where human content moderators are still needed; however, even when something is flagged for closer manual review, there may still be trouble.

Doing it the hard way comes at a cost.

Often the response passed to Facebook is just to handle all content moderation manually. Unfortunately, as we’ve seen, that approach is a sheer logistical impossibility. With their moderation force of 15,000, it translates to a ratio of 21,000 posts per moderator per minute to keep on top of.⁸

While there will always be an edge case where manual moderation is necessary, even at its best, manual content moderation is not a clean solution. As a costly expense for companies to maintain at such a volume, content moderators are often stretched thin and overworked. They’re generally told to strictly follow stringent content guidelines, outlined in a company policy which is constantly shifting.⁹

And this doesn’t even take into account the mental cost to these workers. Content moderators, coming face to face with all the worst posts that come across these platforms, are at risk of developing cases of anxiety and PTSD.⁹

Between a rock and a hard place.

While it might seem like this post sets out to demonize content companies such as Facebook, that certainly isn’t the intention. While I do believe they hold a high level of social responsibility, and much more could be done to bridge the gap to where we currently stand, they also find themselves stuck at a tough crossroads.

Too much moderation is seen as an infringement of free speech and an overstepping of boundaries from a private company.

Inaction, however, allows the misuse of the platform for nefarious means, often to the public detriment. As it stands Facebook’s response to the conundrum has largely been reactive rather than proactive.

Following the backlash surrounding Russian interference in the 2016 US Presidential election, Facebook went on a spree of finding and deleting fake accounts. They have continued that push and now also publish a transparency report on their moderation efforts in the wake of the congressional hearing. On average they have deleted 1.65 billion fake accounts per quarter, demonstrating the sheer scale they are facing.¹⁰

A global company, facing a global problem.

That response and push towards transparency are very encouraging, yet most of it centers around western markets where the outcry has been the loudest.

There remain issues where comparatively little foreign language moderation is present, if available at all. Their community standards are only available in 41 languages¹¹. Yet in contrast to that moderation disparity, only 11% of its user base is located in the US & Canada¹². Aware of this freedom, bad actors in less regulated regions are capitalizing on the digital free reign for nefarious means.

In 2018 top military officials of Myanmar, posing under fake accounts with over a million followers, utilized the platform to spread propaganda targeting and marginalizing members of the country’s Muslim Rohingya minority group.

The techniques utilized were popular ones, very similar to the ones employed by Russian actors. Fake pages and accounts flooded the web with incendiary posts at peak viewing times, exploiting volume to dominate online forums. These posts, often disguised as divisive opinions spread on group pages, were often almost indistinguishable from posts by real users. Military accounts who moderated these groups as fake users also silenced and deleted posts critical of their policies.

This incident was credited with inciting violence and persecution of the group creating a massive forced migration.¹³

Where that leaves us.

It’s a strange world, where suddenly private tech companies are being asked to define the boundaries of free speech for the world.

This joins the list of many pressing issues that have fallen on our laps with the rapid expansion of technology. As we’ve seen, it’s a complicated space, and there aren’t easy answers. However collectively we can take some actionable steps towards better systems.

We must ensure that transparent auditing of content moderation processes are at the forefront of government agendas, in turn ensuring it becomes a forefront priority for the main market players. Systems with such a large impact on the general public need to be accessible and transparent to that public.

We also need to ensure we keep working towards technical solutions despite the complexity. I’m excited to see what the future holds for this space. Breakthroughs could be invaluable in reducing the human cost, but in the meantime workers on the frontlines of content moderation must be working in more supportive, sustainable environments.

I think we have seen some recent silver linings. In response to the recent Covid-19 outbreak, many content companies stepped up their efforts to quell disinformation surrounding the pandemic by visibly displaying warnings near posts advocating misinformation.¹⁴

Regardless of whether more could be done, it has served as a growing point of media attention on the subject of disinformation. Hopefully it opens up the discourse on how social platforms can be used against us.

When we begin to align the interests of these companies and the public good, real valuable solutions can be put forth.

Citations

1. Time Flies: U.S. Adults Now Spend Nearly Half a Day Interacting with Media. (2018, July 31). Retrieved May 3, 2020, from https://www.nielsen.com/us/en/insights/article/2018/time-flies-us-adults-now-spend-nearly-half-a-day-interacting-with-media/
2. GlobalWebIndex. (n.d.). Audience Insight Tools, Digital Analytics & Consumer Trends. Retrieved May 4, 2020, from https://www.globalwebindex.com/
3. Harris, Tristan, and Aza Raskin. (30 Jan. 2020) “Episode 13: Mr. Harris Goes to Washington.” Your Undivided Attention Podcast.
4. Goel, V. (2014, June 30). Facebook Tinkers With Users’ Emotions in News Feed Experiment, Stirring Outcry. Retrieved May 08, 2020, from https://www.nytimes.com/2014/06/30/technology/facebook-tinkers-with-users-emotions-in-news-feed-experiment-stirring-outcry.html
5. Aslam, S. (2020, April 22). Facebook by the Numbers: Stats, Demographics & Fun Facts. Retrieved from https://www.omnicoreagency.com/facebook-statistics/
6. Suciu, P. (2020). More Americans Are Getting Their News From Social Media. Retrieved 8 May 2020, from https://www.forbes.com/sites/petersuciu/2019/10/11/more-americans-are-getting-their-news-from-social-media/#57785da13e17
7. Facebook reinstates Vietnam photo after outcry over censorship. (2020). Retrieved 8 May 2020, from https://www.reuters.com/article/us-facebook-norway-primeminister-idUSKCN11F194
8. Thomas, Z. (2020, March 18). Facebook content moderators paid to work from home. Retrieved May 2, 2020, from https://www.bbc.com/news/technology-51954968
9. Newton, C. (2019, February 25). The Trauma Floor. Retrieved April 29, 2020, from https://www.theverge.com/2019/2/25/18229714/cognizant-facebook-content-moderator-interviews-trauma-working-conditions-arizona
10. Facebook Transparency Report | Community Standards. (2020). Retrieved 8 May 2020, from https://transparency.facebook.com/community-standards-enforcement#fake-accounts
11. Fick, M., & Dave, P. (2020). Facebook’s flood of languages leave it struggling to monitor content. Retrieved 8 May 2020, from https://www.reuters.com/article/us-facebook-languages-insight/facebooks-flood-of-languages-leave-it-struggling-to-monitor-content-idUSKCN1RZ0DW
12. Facebook Inc. (2020). 2020 Q1 Earnings Report. Facebook. Retrieved from https://s21.q4cdn.com/399680738/files/doc_financials/2020/q1/Q1-2020-FB-Earnings-Presentation.pdf
13. Mozur, P. (2018). A Genocide Incited on Facebook, With Posts From Myanmar’s Military. Retrieved 8 May 2020, from https://www.nytimes.com/2018/10/15/technology/myanmar-facebook-genocide.html
14. Rosen, G. (2020, May 7). An Update on Our Work to Keep People Informed and Limit Misinformation About COVID-19. Retrieved April 29, 2020, from https://about.fb.com/news/2020/04/covid-19-misinfo-update/