Donald Trump Detector
We’ve all been there. You’ve got a collection of tweets from an unknown author, and you’re wondering — “Are these Tweets written by Donald Trump?”
Our forebears would have just shrugged their shoulders and given up, surrendering the authorship of these tweets as an ineffable mystery. Luckily for you and I though, we exist in the same day and age as Keras — a deep learning API so simple that even one such as yours truly can start to make use of it.
In this post I will investigate: Can a dense neural network determine if Donald Trump authored a tweet?
An important caveat should be applied to the phrase “Dense neural network”. My experience with machine learning, deep learning, and neural networks is mostly just skimming the Keras docs while working on this project and googling error messages. I had to google the phrase “Dense neural network” to see if it applied to what I’m doing while writing this post, and maybe it does. Point is: This is a description of my investigation, not a machine learning tutorial.
This technical schematic explains my understanding of machine learning and neural networks together.
Basically, arrays of numbers go in, and arrays of numbers come out. When the right arrays of numbers come out, the magic box teaches itself to keep doing more of whatever it just did, and when the wrong numbers come out it learns to do less. The powerful part about this is you get to decide what the arrays of numbers mean and whether or not they are right or wrong.
Lets convert my problem, “Are these tweets written by the President?” into a form that can be satisfied by the magic box. i.e. I want my tweets to be arrays of numbers and my answer (Yes or No) to be an array of numbers. The answer is easy, I want a 1 dimensional array of 1 element, where 0 means “No” and 1 means “Yes”. The tweets are a bit harder.
Stylometry is the process of identifying the author of a text based on characteristics of the text. You can think of it like fingerprinting, but instead of looking at identifying features in friction ridges and whorls we find identifying features in characteristics of text.
What I want to do here is a kind of stylometry leveraging a dense neural network. I’m using the phrase “a kind of stylometry” because all I will be doing is determining whether a text was or was not written by Donald Trump, I won’t be picking out who the author is other than saying “True” or “False” to the question “Is the author Donald Trump?”.
Whenever doing something new, or hard, it’s a good idea to check if smart people have already done the thing. Surprisingly, I didn’t find any Donald Trump Detectors already in existence, but I did find the “International Association of Forensic Linguistics”, and, from a list of papers presented at a recent conference there, I found several interesting research papers that gave me some clues about how I could proceed.
From a paper on “Gender, genre, and writing style” I learned that it’s actually quite possible to infer characteristics of the author based on their writing style. This paper claims an 80% accuracy at predicting male or female authors based on attributes of their writing. The main patterns they found predictive were patterns in sequences of parts of speech, pronoun usage, and articles like the word “The”. Guess which gender uses pronouns more? That’s right, you sexist, it’s women.
A paper on “Automatically profiling the author of an anonymous text” reiterated the prior claims that pronouns and patterns of parts of speech are useful signals for identifying author gender. This paper also highlighted that “Function words” are highly informative for profiling, and pointed out that contractions written with apostrophes included instead of omitted signal greater age of the writer. There is also information here about how spelling idiosyncrasies can be used to help profile the author, but I didn’t wind up using that.
Reviewing this research did a couple useful things for me. First, it showed that it is possible to get some signal from characteristics of text. Second, it gave me ideas for what features I should be looking for (pronouns, parts of speech, apostrophes, etc). These characteristics, combined with a list of interesting features from wikipedia’s stylometry page, plus a few I came up with, let me write the first part of my Donald Trump Detector.
I wrote a function to take a string of text and convert it into a “vector” (i.e. a list) of numbers. This function has many sub functions, each of which add one more number to a growing “result” list. Example: the first element of the list of numbers is the count of characters in the text, the second the number of words, and so on. I added counts for character frequency, mean and standard deviation of word length, sentence length, number of contractions, and frequency count for a short list of “function” words — i.e. words that signal a grammatical relationship (e.g. “do” or “is”).
I also added more complex measurements, like frequency counts for parts of speech, which can be determined by the python Natural Language Toolkit module, the Flesch Kincaid readability score and other “readability” scores, and sentiment analysis using NLTK’s “Vader” sentiment analyzer to break text into how positive, negative, and neutral the text was, and added those scores onto my growing list.
Once I had this function, which turns a text into a vector of 171 numbers, I needed a neural network. Through the process of tinkering, trial and error, and changing things at random, I came up with this architecture:
As I understand it, the Dense layers are there to figure out relationships between the different input numbers, and the “Dropout” layers are there to discard data at random, which should prevent the model from a type of overfitting where the model pays too much attention to specific and non-general features in the training data. The final layer outputs a single element, using a sigmoid activation function, to output a number between 0 and 1. I interpret 0 as meaning “Not written by Donald Trump” and 1 to be “Written by Donald Trump”.
With a model and an interpretation defined the next step is getting some data to train the model. Luckily, Twitter is a good source of tweets.
I first downloaded all of Trump’s tweets and all of my own tweets and spent some time experimenting with different model architectures, different loss functions, numbers of training epochs and so on. The most significant configurations I found were setting the “validation_split” in the fit function of the model to 0.2 and using an “EarlyStopping” callback to monitor validation loss.
The validation split configuration sets aside a portion of the data, in my case 20%, to be used for, well, validation. This means that we train the model on only 80% of the data and keep the other 20% set aside for validation.
Each epoch the model trains it reports on how well it is doing against the training data and the validation data. If you don’t have validation data set aside your model can overfit i.e. it can essentially memorize the training data and get very good or perfect scores on it. Keeping the validation data set aside means that you see how well the model is doing on data it hasn’t trained with. If your model has great performance on the training data and poor performance on the validation data, that means you’re overfitting.
The early stopping call back is related. It detects when you are starting to overfit by monitoring your performance on the validation data and tells your model to stop training when you stop improving on the validation data.
My naive impulse was that the more I trained my model the better it would get. In early versions of my project I hit 100% accuracy on the training data— amazing right? Well, as soon as I figured out I should also be using validation data, I discovered that I had really just trained the model enough that it memorized every tweet in the training data, and was able to identify all of the training data and none of the validation data.
With the validation split and early stopping in place, I soon got to what I feel is a pretty good accuracy result — 80% accuracy. In other words, if I put in 100 Donald Trump tweets into my magic box, then about 80 of them would come back labelled “Trump” and if I put in 100 of my own tweets, about 80% of them would come back labelled “Not Trump”. Does this mean I have an 80% accurate Donald Trump Detector?
Well, because I’m a bit of a dumb dumb, I at first thought that is what that meant. I tested it on some of my tweets and some of Trump’s tweets, and it seemed to be mostly working. Exciting! I then tested it on some random tweets I grabbed off my twitter timeline and found that my model was very happy to call these Trump tweets as well. See: To the same extent that my model was a Trump detector, it was also a “Not-Me” detector. Another way of thinking about it is that the model had learned to predict Trump tweets, but only in a universe where Trump was one half of all tweeters.
The correction for this was grabbing more data. More tweets from many more people, and continuing to train. At first, this also ran into a significant problem. With 100 non-Trumps and only 1 Trump, the model could instantly become 99% accurate by simply always guessing “Not Trump”. The model did, in fact, do this, until I learned that there is a “class_weight= ‘auto’” configuration you can pass to the fit function. This configuration means that the model will weight the different classes in inverse proportion to their frequency — meaning, the “Trump” class was very rare, only 1 out of 101 inputs were labelled Trump, and so Trump tweets were weighted much more heavily when updating the model, to balance rarity.
This fairly quickly trained to a point where my performance against the validation set stopped improving. My original intention was that an output of 0.5 or greater would be the model claiming the input was a Trump tweet, and less than 0.5 would be the model claiming the input was not a Trump tweet. What I found was that the model literally never predicted Trump by this definition.
I decided to change how I think about the model’s output. Instead of taking a 0.5 or higher to predict a “Trump” label, I just looked at all the outputs for all Trump tweets and for all non-Trump tweets. What I found was that non-Trump tweets had an average output value of 0.0147, compared to the average output value of Trump’s tweets which was 0.2069.
In my new conception, the output is a “T-Score”, simply representing how likely a tweet is to be authored by Donald Trump. The higher the score, the more likely the author is Trump. With this model, you really couldn’t look at one tweet and ever walk away with a very high confidence that the author was Trump. However, you could look at a collection of tweets written by the same author and realize that you were seeing really unusually and consistently high T-Scores and then conclude that the author was, probably, Donald Trump.
At least, that was my theory. I trained my model on 100 accounts plus Trump. To test my model and my theory I stepped things up by an order of magnitude and looked at 1,000 twitter accounts. I picked 994 accounts at random, then I threw in the accounts of Trump, his children, the “POTUS” twitter account, and Mike Pence.
My protocol for picking accounts at random was to start with Joe Rogan — who is associated with a wide range of figures, add 20 people he follows to my set of random accounts and then pick a random member of my set of random accounts, take 20 of their followers, and repeat until I had 1,000 accounts.
Then, for each account, I download all their tweets and pass them into the model for T-Scores. For any account with more than five tweets I find an average T-Score per tweet for the account. I then rank all accounts by average T-Score normalized so that the account with the max average-T-Score is 1 and subsequent accounts have an average-T-Score that is their proportion of the max.
Prior to evaluating my results I set four quantitative criteria for my test.
- Trump has the highest T-Score of all tested accounts.
- The “POTUS” account is in the top 10 T-Scores.
- Trump-like figures are represented in the top 10 T-Scores.
- Trump’s (non-normalized average) T-Score is at least 3 standard deviations higher than the mean average T-Score.
Of these criteria, my results showed my model met all four.
- Trump’s average T-Score was the highest of all accounts. With the final model and all of Trump’s tweets, Trump’s average T-Score was 0.163.
- The POTUS account, also, at least nominally, run by Trump was in third place with a score of 0.068.
- The top 10 included “trumphop” (an account which retweets old Trump tweets), Mike Pence, Kayleigh McEnany (Trump’s Press Secretary), Ryan A Fournier (Founder of Students for Trump), Hogan Gidley (Deputy Press Secretary), and, the Christian conservative pro-Trump account “American Family Association”. In other words, 6 of the top 10 accounts were Trump-like figures. Random sampling of my list of accounts suggests this is higher than would be expected by random chance.
- The standard deviation for T-Scores was 0.009 and the mean T-Score was 0.015. Trump’s score of 0.163 was 16.4 standard deviations higher than the mean.
- (Bonus) Donald Trump Jr was in 29th place, Ivanka 34th, Eric 38. All three children were in the top 40 — i.e. 95+ percentile.
My model didn’t come together exactly as I thought it would. That is, it never really predicts that a tweet was authored by Trump by producing an output of 0.5 or greater. The more I think about this though, the more sense it seems to make to me — the model “grew up”, as it were, in a universe with 101 tweeters, and it seems quite reasonable to me that no pattern in the text characteristics I used in this program could ever make you 50+% confident that a given tweet was written by a specific 1 out of those 101 tweeters. Trump has an idiosyncratic tweeting style, but it’s not impossible for other people to write tweets with similar characteristics, and Trump’s tweets aren’t all the same either.
The fact that Trump’s twitter account came out with the highest T-Score out of the 1,000 I used in my final test did not surprise me. The model was basically trained on Trump’s tweets and from experimenting with the set of 100 tweeters I knew it would score Trump much more highly than any random person.
What did surprise me about my final test was that it successfully rated other Trump like figures so highly. The POTUS account getting third place, seems like strong evidence to me that the system is doing something productive or something like what I intend it to. The fact that 6out of the top 10 T-Scores were Trump related also seems like a very positive sign for me.
What I’d like to do next with this project is run an additional test, scaled up another order of magnitude, testing with 10,000 twitter accounts, and use my same 4 criteria, to see if the model continues to perform well. I can improve the test by taking a random 100 account sample of my 10,000 twitter accounts and counting what number of them are Trump-related. This would let me say whether 6/10 Trump related accounts in the top 10 was a good or bad result.
In order to go up another order of magnitude I’ll first need to optimize my tweet to vector code. It took about 24 hours and 4 GB of RAM to process the 1,952,821 tweets used in my final test. I don’t want to wait 10 days to see the results of the next test. Luckily, I think I can easily make this multi-threaded to utilize all of my cores and keep things on disk instead of entirely in memory. This would let me run the 10x test in approximately the same amount of time as the original test.
I could also see putting together a website that utilized this tool. People could go to the website, enter a twitter name, and get a T-Score result for the twitter name.
I have a few other ideas for interesting things to do with this model. The best idea is to make it more general, so it doesn’t detect Donald Trump, but rather anyone. If this worked out, the tool could be used to identify people with sockpuppet accounts, because their sockpuppets should have similar “T-Scores” to their real accounts.
I’m more than happy to share my code, the model, or the data, if anybody is interested. I don’t really expect anybody to read this post, certainly not all of it, though, so I’m not going to go to the trouble of figuring out where and how to upload those things unless someone asks for it. If you’re interested, you can see the 1,000 account T-Score rankings here.
Thanks!
P.S.
A few open mysteries —
- Why does the “trumphop” account, which reposts Trump tweets, not have a much higher, or perfect, T-Score, since it is reposting Trump? I think this has to do with the fact that retweets, when read by the API, always begin with a phrase like “RT @realDonaldTrump:” which may give them some different characteristics than original Trump.
- Why are there are only 795 accounts in my list of 1,000? Some accounts are excluded due to having fewer than 5 tweets. There also seems to have been an issue with my tweet downloading code that didn’t get tweets for accounts even though I know they have more than 5 tweets (I suspect a token expired in my downloader script, and a random number of accounts actually didn’t get downloaded).
- Does T-Score predict anything about political alignment? It was interesting to me that conservative/Trumpian figures seemed to be towards the top. (Although this is not always the case — e.g. Gingrich is towards the very bottom).
