Stopping Twitter abuse: a machine learning approach to identifying misogyny online
Violence, abuse and aggression towards women in online spaces is a sadly intractable problem across all digital platforms.
Associate Professor Richi Nayak, Professor Nicolas Suzor and research fellow Md Bashar from QUT have developed a sophisticated and accurate algorithm that can detect these posts on Twitter, cutting through the noise of millions of tweets to identify misogynistic content.
While currently the onus is on the user to report abuse they receive, the researchers hope that this machine learning solution can be adopted to automatically identify and report this content.
Automating this process can reduce the emotional and cognitive load on users and moderators, and influence platform-level policy to protect women and other user groups online.
Creating the algorithm
Teaching a machine to understand natural language is one of the more complicated ends of data science: language changes and evolves constantly, and much of meaning depends on context and tone.
“We developed a text mining system where the algorithm learns the language as it goes, first by developing a base-level understanding then augmenting that knowledge with both tweet-specific and abusive language,” said Nayak.
“We implemented a deep learning algorithm called Long Short-Term Memory with Transfer Learning, which means that the machine could look back at its previous understanding of terminology and change the model as it goes, learning and developing its contextual and semantic understanding over time.”
While the system started with a base dictionary and built its vocabulary from there, context and intent had to be carefully monitored by the research team to ensure that the algorithm could differentiate between abuse, sarcasm and friendly use of aggressive terminology.
“Take the phrase ‘get back to the kitchen’ as an example — devoid of context, that’s just standard language,” said Nayak.
“But when it’s seen with the understanding of what constitutes abusive or misogynistic language, that can be identified as a misogynistic tweet.
“Teaching a machine to differentiate context, without the help of tone and through text alone, was key to this project’s success, and we were very happy when our algorithm identified ‘go back to the kitchen’ as misogynistic — it demonstrated that the context learning works.”
The model identifies misogynistic content with 75% accuracy, outperforming other methods that investigate similar aspects of social media language.
“Other methods based on word distribution or occurrence patterns will identify abusive or misogynistic terminology, but the presence of a word by itself doesn’t necessarily correlate with intent,” said Nayak.
“The semantic understanding that our model developed through the transfer learning approach is what makes it so unique.”
Building the data set
The key to understanding context is in labelling the data correctly to begin with.
To achieve this, the research team scraped a dataset of 1M tweets then refined these by searching for tweets that contained one of three abusive keywords, resulting in a collection of 5K tweets that were identified as containing misogynistic terminology.
These tweets were then categorised as misogynistic or not based on context and intent, and were input to the machine learning classifier, which used these labelled samples to begin to build its classification model.
The model can correctly identify spelling errors by looking at the distance between the misspelled word and a similar correct word, and categorising it as misogynistic if it’s close enough.
“Sadly, there’s no shortage of misogynistic data out there to work with, but labelling the data was quite labour-intensive,” said Nayak.
Human confirmation of correct categorisation was needed to calibrate the algorithm’s output, but eventually the classifier could run unassisted.
Nayak and her team initially struggled to share the data and their findings with the research community due to its distasteful content.
“Usually we’re so proud to show off the work that we do, but we found this really difficult to talk about, and the data itself is so unpleasant to look at — but this just makes it clearer that the work is needed.”
A machine learning solution for a legal problem
The cross-disciplinary project began when Nayak started a collaboration with Professor Nicolas Suzor from QUT’s Law School and Digital Media Research Centre, whose work focuses on the governance of internet and social networks.
“Dr Suzor is very interested in monitoring social media and creating enforceable policy for managing abuse online, and our team was able to bring a refined machine learning solution to the problem,” Nayak said.
“This research could translate into platform-level policy that would see Twitter, for example, remove any tweets identified by the algorithm as misogynistic.”
This modelling could be expanded upon and used in other contexts in the future, such as identifying racism, homophobia, or abuse toward people with disabilities.
“Our end goal is to take the model to social media platforms and trial it in place,” said Nayak.
“If we can make identifying and removing this content easier, that can help create a safer online space for all users.”
More information
Explore more research at QUT’s Science and Engineering Faculty
Read the paper in Knowledge and Information Systems
Contact Associate Professor Richi Nayak
Visit the QUT Centre for Data Science
