Discovering early signals of doxing and calls to harassment online
Attack strategies used by online harassers have evolved over time, including their scale and ability to cause offline harm. Coordinated harassment — when others are encouraged to use intimidation, humiliation, or threats to purposely amplify an attack — can take many forms, and negatively impact internet users across the political and demographic spectrum. These types of attacks are typically precipitated by what is called a “call to harass.” As part of our ongoing research, Jigsaw partnered with NYU to see if we could use online calls to harassment (CTH) as a way to measure and understand harassment ecosystems.
An early warning system — potentially built on the open source models we’ve created here — could help platforms and targets of harassment secure their information and end harassment sooner, saving time, money, mental health, and physical security, not to mention improving the services online platforms provide.
Our methodology: A model built to spot the start of coordinated harassment.
Online harassment can take many forms, including “doxing” (or doxxing) — the intentional release of personal information. Lower effort forms of harassment include “mass flagging” — where a group of harassers wrongly flag content to be removed and/or get accounts banned, or “hashtag hijacking” — where individuals usurp existing hashtags for malicious purposes. While many platforms have explicit policies prohibiting malicious activity towards other users, some users still seek ways to encourage harassment and are constantly probing for gaps in enforcement.
Online harassment can have serious implications, resulting in reputational damage, mental health harms, financial and job losses, as well as threats to physical safety. And no one online is safe from being targeted — politicians, activists, journalists, and other public figures across parties and geographies have been harassed into silence online.
To better understand the scale of doxing and other forms of coordinated harassment, Jigsaw collaborated with a team of security and privacy researchers at NYU. Our approach was novel, focusing on the coordination of harassment itself by detecting inciting language rather than individual attack types. This language can include examples such as “[name] must be harassed, get her phone number and address,” “Post FB & Twitter accounts so we can spam him with hate,” and “Let’s mass report his twitter and youtube… [sic].”
We began by building two separate pipelines to identify CTH, the incitement for others to pile on precipitating the actual harassment, and doxes. We manually coded an initial set of positive and negative examples of doxes, resulting in 1,227 positive and 10,387 negative examples. “Negative” examples included instances that might read as potential doxes but in fact were not, such as a tweet encouraging constituents to call their Senators’ offices or an online invitation to a party with someone’s home address. For a detailed explanation of how we derived models from this dataset, see the methodology section of our paper.
Our findings: A rapidly evolving harassment landscape.
We used both manual analysis and machine learning models to review tens of thousands of posts and identify the relative prevalence of different kinds of online harassment, including doxing, across five types of platforms. The platforms included text from imageboards, group chat platforms, blogs, and cut-and-paste text storage sites. Our results showed that out of 14,679 posts, approximately 9% of CTH threads contained a dox, meaning the vast majority of incitements to harass did not involve a dox. Meanwhile, 18% of dox threads contained a CTH, which means the other 82% of doxes revealed personal information, but did not explicitly call for anyone else to take action. Part of the challenge in tackling online harassment is the number of different forms of harassment — doxing, mass flagging, toxic speech — that often co-occur in the same conversation. However, by capturing and analyzing multiple types of harassment, the models we created with NYU helped us detect a few trends occurring in the rapidly shifting space.
- Harassment reporting tools are often abused to harass.
While platform reporting systems (for example, the ability to flag or report a user or piece of content) are one of the key approaches used to counter harassment and abuse, our analysis demonstrates that their manipulation is a method of coordinated harassment. Over 50% of the CTH we annotated (3,206 instances) included calls to report the targets to a platform, with 24% used to report targets to other public or private entities such as law enforcement or employers.
2. Doxes can proliferate on platforms without policies against them.
While high profile incidents of doxing tend to be headline-grabbers, across our four data sets, mass reporting was actually far more prevalent — appearing in the largest share of CTH (51% of the total).
Meanwhile, doxing differs significantly by platform, including what type of personally identifiable information (PII) is shared. Interestingly, the only platform in our dataset with a policy against doxing also had the lowest doxing prevalence (see page seven of our study for details). It is worth further exploration to understand whether there is a positive correlation between moderation policies and lower levels of harassment.
3. Some forms of personal information are more likely to be doxed than others.
Doxes often contain multiple types of personal information and the types of information shared varies by platform. For instance, the doxes in paste documents we analyzed contained more types of PII, resembling long dossiers, relative to doxes with one or two types of PII posted on image boards.
Addresses and phone numbers were the most common forms of PII shared, followed by Facebook profiles. We also found that street addresses, phone numbers, and email addresses co-occurred with all other types of personal information less than half the time, suggesting that most doxing focuses on harassing targets online. Automated dox detection systems should therefore be built to account for various PII, reinforcing the value of leveraging classifiers to understand the scale and evolution of online harms like harassment.
Moving forward: Opportunities to stop cross-platform harassment in its tracks.
Harassment trends evolve quickly, and are hard to study across platforms. To help, we are sharing our findings together with NYU and making our open-source Pytorch and Tensorflow models available on GitHub for four of the platform types we focused on, along with instructions on how to use them. We believe this can help lay the groundwork for future work to potentially detect a more comprehensive array of CTH, provide a more accurate assessment of the CTH ecosystem, better discern when CTH are acted upon, how attackers coordinate, and the tools they use to do so.
Longitudinal analysis of CTH could provide insights into new attack types, CTH in new online communities, and whether offline trends and events influence these online communities.
Finally, future research could focus on the role of gendered harassment in a more robust way. An automated analysis of posts by gender found that there was a higher chance of females being targeted specifically for reputation harm, while our manual analysis found that many threats against female targets contained threats of leaking non-consensual explicit imagery (otherwise known as “revenge porn”). Due to the limitations of pronoun-based gender identification and the goals of this study, we were unable to measure the impact of online harassment on female targets, but it warrants further study.
Whether we like it or not, the methods, reasons, and platforms users will exploit to harass others will continue to evolve. Still, we are excited about the potential in the models we have created to understand the building momentum of online harassment, make it more challenging to post incitements to harass online users and build better security tools to protect users.
by Beth Goldberg, Research Program Manager at Jigsaw
