Identifying political bot/troll social media activity using machine learning

I’ve been working with machine learning techniques to identify several varieties of “troll” Twitter accounts that spread propaganda and false information. This material is largely of Russian origin and targeted at right-wing American voters. I organized the Twitter accounts into one of three categories:

1. Automated bots that mostly spam hashtags and retweet links to Kremlin-aligned articles.

2. Accounts operated by professional social media trolling operations, mostly in Russia.

3. Americans, mostly alt-right adherents, who post and retweet the same material as the bots and professional trolls.

I’ve developed a simple website Make Adverbs Great Again demonstrating this technique. You can enter a Twitter username and receive a score indicating how similar the system thinks the account in question is to known troll accounts.

Want Ctrl Alt Right Delete in your inbox? Subscribe here.

The particular machine learning technique used here is a type of neural network called a multilayer perceptron (MLP). For the technically inclined, I wrote this in Python and used Scikit’s implementation of the MLP neural network.

This type of system is “trained” using a set of known examples. For this project, I identified roughly 3000 safe and 1000 troll accounts to use as the training set. The safe accounts were drawn from a variety of sources — some are political, some promotional, some purely social. The troll accounts were identified by searching Twitter for specific hashtag associated with known false stories (#PizzaGate, #SethRich, #SickHillary, etc), and selecting a cross-section of accounts that included both bots and human users. All the sample troll accounts used were strongly focused on propagating these stories, and additionally had tweeted multiple links to state-sponsored Russian media sites such as RT or Sputnik.

There’s a bit of an element of chance to the training process, and the accuracy of the resulting networks varies somewhat. I trained 100,000 neural networks and kept the best ten in terms of accuracy. The resulting networks correctly classify accounts with 95% accuracy, with a larger share of the errors being false negatives (troll accounts that were missed) than false positives (accounts incorrectly classified as trolls). Each individual network makes a yes or no decision as to whether a given account is a troll, and the score shown on the website is the number of the networks (out of ten) that voted “yes”. It’s worth noting that it’s possible for all ten to be wrong, and even the 10/10 scores are occasionally false positives.

In terms of future work, there are a number of additional avenues worth exploring. There are several ways of improving the accuracy further, ranging from improving the training data to using more complex classification techniques. The more interesting avenue for me from a research perspective, however, has been to use the classifier to identify large numbers of potential troll accounts for additional study. I’ve used it as a source of data for several bulk analysis projects. This pie chart showing the breakdown of Wikileaks links shared by bots and trolls, for example, was produced using a sample set of over 14000 accounts identified in this way. Additionally, these techniques could likely be generalized and adapted to other social media platforms, such as Facebook or Reddit.

Want Ctrl Alt Right Delete in your inbox? Subscribe here.

If you’re interested in working with this yourself, you can download the source code for the classifier here. You’ll need to sign up for a Twitter API key (free) and provide your own sets of example “safe” and “troll” accounts in order to make use of it.

The above is an excerpt from Ctrl Alt Right Delete, a weekly newsletter devoted to understanding how the right operates online and developing strategies and tactics to fight back.

Want Ctrl Alt Right Delete in your inbox? Subscribe here.