On May 14, 2022, 18-year-old Peyton Gendron walked into a grocery store in Buffalo and murdered 10 people. While Gendron acted alone, he took steps to ensure his actions would reach a vast online community, distributing links to a live stream, his online diary, and his 180-page manifesto across the internet. Twitch, the platform where the shooting was initially live streamed, acted swiftly, shutting off the feed in less than two minutes. But within seconds the video was reuploaded to less moderated platforms, where some recordings remain to this day. By spreading this footage so widely, including in like-minded extremist communities, thousands of others were encouraged to mimic Gendron in violent copycat attacks.
Copycat attacks are a well known but preventable phenomenon; they are made possible when detailed information is readily available about how an attack was executed. Gendron was a copycat himself, describing how the violent white supremacist attack live streamed from Christchurch New Zealand four years earlier had served as a model for him. Attackers fuel copycats by ensuring their live stream goes viral across a broad ecosystem of platforms, beyond the ability of any one platform to remove it.
Platforms have long removed users and content, like extremist attack videos, that violate their policies with the intent of keeping users safe. Algorithms to identify this content rapidly have gotten sophisticated, and global consortiums like the Global Internet Forum to Counter Terrorism (GIFCT) have been developed to share information, but extremists persist online — in large part because they, like their live streams, are spread redundantly across platforms. The growing interconnectedness of mainstream and alternative platforms has created substantial challenges to curbing hate and violent extremism online and underscored the need to develop enforcement tools that can be used across the digital ecosystem. Original research below illustrates how this ecosystem of alternative, less-moderated online platforms — from file-hosting sites to chat apps — has become more expansive and integrated with mainstream platforms in recent years. Taking an ecosystem-level lens to removing extremism is vital to preventing the next Buffalo shooter.
Given that these less moderated platforms are more readily accessible and interconnected, the question we at Jigsaw kept returning to was: do removals (aka deplatforming) on major platforms help to combat hate online?
To help answer this complex question, we deployed a variety of research approaches to triangulate at an answer. We partnered with scholars at George Washington University to conduct two studies on how the online hate ecosystem reacts to offline events and large-scale removals by platforms. We also worked with anthropologists at ReD Associates to interview people who had experienced the removal of their account or posts on a social media platform. Lastly we conducted a series of 25 case studies on influencers who had their accounts removed from a major social media platform in the last two years for violating platform hate policies.
Taken together, these studies reveal six insights about how hate and violent extremism online are evolving, offering valuable clues for how moderation approaches must evolve in turn to make the internet safer.
1. Offline events activate online hate in unexpected ways.
Coverage of real-world extremist attacks has increasingly highlighted the online lives of the perpetrators. The posts, live streams, and social networks of violent extremists have become central to public attempts to make sense of senseless atrocities. At Jigsaw, we wanted to know if there are patterns in these online hate posts that might help us better anticipate offline attacks, or, alternatively, if an understanding of online reactions to offline events could help platforms better anticipate and moderate subsequent spikes in online hate. Looking beyond extremist attacks to events like elections or high-profile assassinations, we wanted to understand the relationship between offline events and online hate.
As the Guardian notes, prior research has found connections between offline events and subsequent spikes in hateful posts, but these studies have typically focused on only a single, moderated platform, like Facebook or Twitter. To understand the connection between offline events and online hate, it is essential to look across the online ecosystem holistically. To do this, researchers at Jigsaw partnered with scholars at George Washington University to collect over 59 million posts from over 1200 public communities on six platforms, including three that have and enforce (to varying degrees) policies against hate speech — Facebook, VKontakte, and Instagram — , and three that are less-moderated — Gab, Telegram, and 4Chan. We then developed machine-learning models to assess the type of hate in each post, with the goal of seeing how different types of hate fluctuate after real world events, like extremist attacks. More details on the methodology can be found in the study, published in Plos One.
We found that offline events can trigger substantial spikes in hateful speech, including against unrelated groups, and that elevated levels of hateful activity can continue long after the discussion of the event itself subsides. In other words, hate can be “activated” by an event like the 2020 US election or the murder of George Floyd, and stay at elevated levels for weeks or months after.
The machine learning models we built allowed us to detect what types of hate speech were being posted, identifying with 92–98% accuracy whether a post contained hate that was anti-semitic, or targeted individuals based on their race, gender, religion, ethnicity or immigration status, or gender identity or sexual orientation-based. We found there was a 250 percent increase in race-related hate posts following the murder of George Floyd. Notably, this increase in race-related hate lasted throughout the remainder of 2020 and, as of the end of data collection in January 2021, had not dipped to previous levels.
We found elevated levels of hate after events for all types of hate, not just race-related hate. In the case of the George Floyd murder, the volume of posts increased for every type of hate we measured, including seemingly unrelated forms of hate like religious and antisemitic speech. It appears that once an event “activates” hate communities online, the frequency of their posts increases regardless of how directly related the event is. Our data is not able to offer a clear explanation for why this happens, or a causal link from events to the hate posts. But these findings illustrate a complex relationship between offline events and online hate. These early findings, however, offer some insight into how platforms might better anticipate platform moderation needs in the wake of major offline events.
2. Removing extremist movements can have a boomerang effect.
Content moderation actions by platforms occasionally go far beyond the removal of specific content or individual accounts to broad removals of dangerous groups and movements. In the summer of 2020, Meta removed hundreds of accounts and pages associated with the far-right Boogaloo movement from Facebook and Instagram following allegations members of the movement had used the platform to plan the murder of a federal agent. But the effect of these policies to remove movements en masse is poorly understood. Do mass enforcement actions effectively remove extremist movements, and, if so, for how long?
In a follow-up study, Jigsaw and researchers at George Washington University analyzed text-based posts on six platforms to identify content associated with four extremist movements. The study, currently a working paper under review, developed novel classifiers that determined the likelihood a post was from one of four prevalent extremist movements, including the Boogaloo movement.
We found a dramatic drop in Boogaloo-related content on public pages on the platform immediately following Facebook’s announcement.
It’s impossible to quantify the full impact of Meta’s Boogaloo crackdown on Facebook — let alone on the wider online ecosystem — due to the large number of private groups from which we were unable to collect data. Researchers at Meta helped to shed some light on this question by publishing findings on six significant deplatforming efforts that targeted hate movement leaders and their networks. Their findings reinforce what we saw post Boogaloo removal: “the disruptions successfully created friction in the ability of the organization to organize its closest audience” and reduced consumption and engagement with hate for both movement followers and adjacent networks.
But we know that extremists operate across multiple platforms concurrently, so we also analyzed public posts on 4chan and four other platforms where the Boogaloo movement was also popular. To our surprise, we did not see a significant subsequent increase in Boogaloo posts after Facebook’s removals. This suggests that if there is indeed a “migration” effect to a new online home after mass removals, it was not to other public spaces in this case. Instead, it seems likely that a significant portion of the Boogaloo movement went private or left the platform following this policy change by Meta.
The movement of extremist groups to private channels and groups has mixed, poorly understood implications. It is harder for law enforcement and researchers to keep tabs on movements that interact primarily in private communities, but going private also hampers the movement’s ability to recruit new members from mainstream audiences. Going private can further splinter movements into smaller groups, with mixed implications for engaging in future violence.
While Meta’s broad ban on the Boogaloo movement was initially successful in terms of reducing public Boogaloo content, its impact appears to be short-lived. By the end of the study period, 18 months after Meta’s policy change, the quantity of Boogaloo-related content on public Facebook groups, as identified by our model, had nearly returned to its pre-enforcement level. This reemergence was not the result of the same accounts or groups simply returning, however. The Boogaloo content that reemerged had evolved and came from new accounts.
We took a deeper, qualitative look at these new posts that the model clearly indicated were Boogaloo-supporting content. There had been a marked evolution in the language used, but the underlying messages and content formats were similar, suggesting the extremist actors were intentionally using more coded language to avoid detection. For example, discussion of the second civil war and the word “Boogaloo” almost completely disappeared from the posts we collected on Facebook, having evolved into the near-rhyme “big igloo” and then “the luau,” before simply being replaced by the igloo emoji. The specific content of memes also shifted after the policy change, but the visual aesthetics, including the use of Hawaiian shirts, red laser eyes, and crude photoshops of a smirking Shiba Inu Doge, remained consistent. While our model was trained exclusively on text, and thus unable to analyze images, it was still able to detect these Boogaloo-related memes based on surrounding text. Our findings highlight the importance of highly nuanced models to detect the strategic circumvention of platform policies by nefarious actors.
The boomerang effect we witnessed here with Boogaloo content requires further study across other platforms and movements, but this case suggests the importance for platforms to continuously adapt their enforcement efforts long after they develop policies against extremist movements.
3. Extremists and hate spreaders engage in “platform arbitrage” to defray costs of being removed.
As platforms have drafted and enforced stricter policies against hate and violent extremism, actors intent on spreading this content have deployed a series of strategies to make it less costly if they lose their access to a single large platform. These strategies exploit the discrepancies between platforms’ moderation practices, moving content and communities from high to low moderation spaces in a dynamic we refer to as “platform arbitrage.”
One strategy has proven key to enabling this arbitrage practice and connecting these communities across platforms: the “follow me” link. This is when users prominently place links in their profiles or post descriptions that encourage following their parallel profiles on less moderated platforms. These links often come with explicit calls to “follow me” or to
get the full story” by clicking through to their profile on less popular platforms.
A related strategy that facilitates platform arbitrage is posting “clips” or short segments of content on mainstream platforms in order to drive viewers to the full version — video, podcast, or other content type- hosted on less-moderated platforms. This practice allows extremist and hate influencers to stay on and recruit followers from major platforms without tempering their statements. The use of clips evidences a nuanced understanding by many extremist actors of what dogwhistles and allusions to hateful content the mainstream platforms will allow in order to attract and redirect interested audiences elsewhere.
When major figures from extremist movements are removed from platforms, and are thus no longer able to redirect traffic themselves, they often rely on their supporters. These accounts, sometimes called “minions,” can play a key role in connecting mainstream and fringe platforms, often porting, or “re-platforming,” extremist content from niche corners of the internet to the most commonly used sites. Minions re-uploading removed content, sharing “follow me” links, and posting clips can enable platform arbitrage of hateful and extremist content to continue long after the original content creator is removed.
This arbitrage is possible because creators and their followers increasingly use a mix of mainstream and alternative platforms simultaneously. A 2022 Google study found that participants in the US used an average of five different online platforms daily, often including niche or alt-tech platforms. Most participants surveyed spent most of their time on the most moderated platforms. But, curiosity, maturing understanding of tech affordances, and specific, niche community interests lead some users to less-moderated sites.
A number of meaningful frictions and trade-offs have thus far prevented these less-moderated platforms from reaching a large number of users, with the exception of Telegram with over 700 million monthly active users. Most alternative platforms are unknown to the majority of internet users, and many of the communities on them are closed, requiring would-be users to have a direct link to the group. Other significant sources of friction to these alternative platforms include but are not limited to poor user experience, subpar algorithmic recommendations, difficulty discerning coded insider language, and a lack of network effects.
But online users are not stuck choosing either these lower quality experiences on alternative platforms or moderated, user friendly experiences on mainstream social media. Increasingly, the reality is that people exist simultaneously on both, with content and whole communities moving seamlessly between platforms. The Buffalo shooter took advantage of this reality with accounts across numerous mainstream and alt tech platforms, enabling him to share his ideas widely beyond any single platform’s removal.
As the ecosystem of online platforms grows more interconnected, the “follow me” pattern once driven primarily by hate actors is only likely to become a weaker signal of extremism, but remain a key enabler of platform arbitrage.
4. Circumvention strategies to evade removal vary by generation and geography.
We found that these strategies to circumvent removal by mainstream platforms varied more by age and geography than by the type of content individuals created. In the US, where a thriving ecosystem of alternative platforms has cropped up in recent years, many creators have multi-homed, spreading their presence across multiple platforms including those with little moderation. This not only lowers the costs of posting hateful or extremist content, it encourages creators to send followers to their alternative platform accounts where they are less likely to face moderation or demonetization. In India, however, where the alt-tech ecosystem is far less mature, people removed from mainstream social media platforms have fewer viable options to reach an audience. Influencers there have primarily turned to messaging apps like WhatsApp to regain social connections after a removal.
To better understand the impact of removals on larger influencers, Jigsaw conducted a separate series of case studies on 25 popular misinformation and hate creators in 2022. We found creators who faced sanctions for their content — including removal from at least one major online platform — exhibited a range of circumvention strategies, but with generational variation.
Almost every misinformation and hate creator we looked at relied on email newsletters to maintain contact with their audience in the event they were removed from mainstream platforms. Younger creators, especially millennials and members of Gen Z however, also employed much more sophisticated approaches to reach their audiences and secure a revenue stream. For example, many younger creators used ephemeral live streams that can be harder to moderate. They also engaged their audiences through realtime, monetizable chat features, which in some cases allow viewers to donate or pay creators to have their comments featured. Many have also branched into fundraising from fans via cryptocurrencies, all of which are strategies to offset the impact of demonetization and/or removal on mainstream platforms.
The most sophisticated creators even set up their own streaming channels or used open broadcasting software (OBS) to embed monetized chats and simultaneously stream across multiple platforms. In both this of creators and our ethnographic study of average users who faced removals, we found many proactively created and promoted alternative accounts on the same platform, providing immediate backups in the event one account is removed.
5. A Backfire Effect? Immersion in less moderated spaces may increase toxicity and boomerang back to moderated spaces.
In interviews with 34 people in India and the US who had their posts or accounts removed, we heard that the experience often felt disorienting and frustrating. These were not influencers with large followings to maintain, but instead casual users who turned to social media primarily for community and entertainment. Their experiences of having a post or account removed for hate or misinformation were catalysts that often dramatically changed their perceptions and subsequent online behaviors.
In the wake of a removal, including of a favorite creator, many people we met turned to alternative platforms with less moderation. Some were lured by curiosity to understand ‘what was really so bad’ about a creator’s removed content, while others left the mainstream platform(s) out of frustration with what they perceived as unfair treatment. They wound up on alternative platforms through the “follow me” ecosystem links posted in their social networks (described in point 3). Once on these less moderated sites, people reported being exposed to more extreme ideas. Journalist Justin Ling describes how imageboards, in particular 4chan’s “/pol/” board, are toxic breeding grounds for extremism, by design. The Buffalo shooter credited 4chan in his manifesto for teaching him “the truth”, the racist belief that “the White race is dying out.”
For many, these immersive forays onto alternative platforms didn’t last long. Beyond the numerous frictions alternative platforms pose, described above, our participants also described a desire for something the smaller platforms couldn’t offer: network effects. People wanted to be online where the public debates were unfolding:
“I missed the interaction on [Twitter]. It was the arena for big debates.”
“After three weeks I got the itch back. I just gave in to their rules I guess.”
Almost all participants we met ultimately returned to mainstream platforms within just a few weeks after removal.
But when they did, they behaved differently. For many people we met, they counterintuitively began posting more extreme and toxic content, despite the risk of subsequently being removed, again. For many, their posts shifted away from discussing local issues or entertainment to address more contentious national political topics or culture wars. For some, this was accompanied by the use of more coded hateful terms. This suggests that the experience of being removed and subsequently immersing in less-moderated forums can result in a boomerang of coded, hateful content back to mainstream platforms. Researchers at the University of Southern California also found this effect in a study on Reddit, where users who joined a hateful subreddit subsequently posted more hateful messages in other communities, capturing this “radicalizing” effect of one community onto others that our study participants described.
There are limitations to what can be drawn from such small, non-representative samples, but there is evidence of these patterns at scale too. A larger-scale study with data from 10,006 users by researchers at four universities similarly found that “users who get banned on Twitter or Reddit exhibit an increased level of activity and toxicity on Gab, although the audience they potentially reach decreases.” Our participants also noted that moving to smaller platforms meant a decrease in reach, but this only served as a deterrent to creators committed to building their brand. This ability to either reach large audiences with moderated content or small audiences with hateful content exemplifies the trade-off that experts have described as “freedom of speech vs freedom of reach”.
6. Small platforms are being exploited by violent extremists for hosting, discussing, and sharing hateful content.
There is great diversity in the ecosystem of alternative platforms, and not all of them are low-moderation havens for extremists by design. Some simply lack capacity or resources to prevent extremism from spreading on their platforms. Bad actors are increasingly exploiting these smaller file hosting and content sharing platforms knowing they may have less stringent policies and fewer content moderators to address violations. Tech Against Terrorism (TAT), in a report with the Global Internet Forum to Counter Terrorism’s (GIFCT), studied this pattern of exploitation across GIFCT member platforms, large and small. TAT found that prevalence of content from the most extreme organized hate and terrorist groups varied dramatically. On one mainstream platform views of policy-violating extremist content amounted to just .06 percent of all views, while on smaller platforms that number was as high as 10 to 50 percent of all views.
In user interviews, Jigsaw researchers found that many of these smaller platforms do not want to serve as a repository for violent or illegal content, but they often lack the resources and tools to identify and remove it, and to keep it from cropping back up again. Some of the platforms most heavily exploited by violent extremists have just a handful of staff members — in some cases only one. Yet in the most extreme cases, like a livestream of an ongoing attack, mere seconds can matter when taking action. Pending regulations in a number of markets pose further imperatives, demanding that illegal content be removed within a few hours. At the moment, however, most of these small platforms have little more to rely on than user reports of hateful or violent content — a noisy signal. Respecting the users of their platforms often requires nuanced, time consuming evaluation of the content and its context.
The violent extremist manifestos, images, and videos included as hashes in GIFCT’s and TAT’s databases contain some of the most noxious content that can be found online, but it is also just the tip of a much larger iceberg.
The fast pace of evolution in how violent extremists use online spaces is likely to accelerate with the introduction of generative AI tools. As platforms look to slow the spread of hateful content online, technology can and should play a central role in helping moderators quickly identify content that may be illegal or in violation of their own standards.
