Beijing, sometime in 2014 — I can’t remember the month or the date, but it doesn’t matter. I do remember that it was a Wednesday morning in the office. We opened up Youku (a Chinese Youtube clone) and were delighted to see that our company’s promotional ‘viral’ video was indeed going viral. We had gained tens of thousands of views overnight! The money we’d paid to a digital marketing agency seemed well spent. Except something was a bit off: Why no comments, why just views? We sent our VP of marketing off to ask the agency what was going on.
Thursday morning. Behold! Our problem was solved. Now we had hundreds of comments and tens of thousands more views. Our video was top-ranked on one of Youku’s category pages. Except… why were all the comments so generic and short? They were all one-word phrases like, “Cool!” “Great!” “Good watch! (sorry for the Chinglish) “Interesting!” They could have been typed by anyone, on any video, and they would have been equally (ir)relevant. Further investigation in the analytics dashboard revealed that the views themselves were all coming in blocks of several thousand simultaneously, repeated every few hours.
Later that day, the marketing agency admitted quite freely that we were not paying for real people to watch and comment on the video. They weren’t working with partners or influencers on other platforms to push the video. They just assigned bots to do the work. This was honest manipulation. Honest fakes. They were a little embarrassed that we hadn’t realised this. We were embarrassed, too. It was the first we’d heard of this happening on Youku. We knew it happened on Weibo (China’s “Twitter clone”) because several years before, two social media interns had explained that if we wanted followers we could just buy them — in blocks of about 10,000 — for very low prices, from Weibo itself!
Fake accounts on social media are created for a variety of purposes. In that instance, it was to boost our video to make it trend — so that real people would be more likely to see it. Although we pulled the plug when we discovered the ‘deception,’ many companies don’t mind. Once cheating is the system, you have to play a bit dirty to get ahead. That’s just the way it is.
On social media, fake views, likes, and retweets are used to amplify specific stories or hashtags for political, disruptive, or commercial purposes. Sometimes they’re used to swamp out other stories. Still other times, they’re used to boost a candidate or narrative. As in China, falsely amplified content can start to generate real engagements as it climbs up the trending rankings and real people begin to see and engage with it. Most nefariously, fake personas are used to get inside an audience and then steer it towards a desired political or social outcome.
In Beijing, we spotted the issue by looking at behavioural clues and characteristics of the bot accounts: as a group, their activity played out on unnatural schedules (all viewing the video in short windows of time). Initially, not one ‘person’ commented, despite tens of thousands of these ‘people’ watching the video. And then when they did comment, their comments were generic and not relevant to the video in question — they were equally applicable to any video at all. They commented and left; none stayed around to reply to other comments or interact with other viewers’ comments .Another interesting aspect was that their profiles all featured cartoon profile pics, which was, it must be said, not unusual in China at the time.
By 2016, Twitter, Facebook, and Instagram were awash with fake accounts manipulating trends, pushing stories, amplifying content and hashtags, and riling people up en masse to distort the US presidential election, the UK Brexit referendum (see here) and, over the following years, numerous other democratic events.
Since then, much work has been done to identify fake accounts, even as the technology and abilities of those who create and control the fakes improves. Here, I thought I’d lay out a simple checklist of tells as a “101” guide to rooting out the fakes in your life.
Some of the tips below grow out of the lessons we learned in China. Others are new and particular to Twitter, which remains blocked in China. We’ll run through some tells, and then provide you with some resources/recommendations for follows and further investigation.
First up, let’s look at the account profile itself…
Analysing a suspicious Profile
1. Profile Picture (“Catfish 101”)
Download a picture (e.g., profile picture, or social feed picture) from a suspicious account, and run it through a reverse Google image search, a search function that searches for other instances of an image. To do this, click the camera icon, then “search by image” then “upload image.” Reverse image searching allows you to quickly determine if the picture has been “borrowed” from another source and therefore does not belong to the account in question. The hosts of MTV’s Catfish use this technique nearly every episode, although Catfish fake accounts are usually not very sophisticated. When you are dealing with nation-state or commercial influence networks, they may be more difficult to uncover this way.
Some accounts may “mirror” the image to defeat this simple technique — or change the colours, crop it, rotate it, etc. If you are running a reverse image search, it may be worth flipping the image to try and rumble them.
Not having a profile picture can also be a big giveaway, and (less so) having a non-face one, especially if it is an individual’s account and not a company one. Having said that, my Medium profile pic is currently a tree, but still.
2. AI-generated faces
As the “arms race” between fakes and investigators continues, AI plays an increasingly important role. Recently The Daily Beast unearthed an entire influence network that had successfully gotten fake Twitter personas (some equipped with AI-generated profile pics and some with profile pics stolen from other accounts) invited to write pieces for various publications:
Right-Wing Media Outlets Duped by a Middle East Propaganda Campaign
If you want a hot take about the Middle East, Raphael Badani is your man. As a Newsmax " Insider" columnist, he has…
Adam Rawnsley published a good thread of analysis here:
This week, a different fake account — again with a Deepfake, AI-generated profile pic — was discovered to be attacking activists who had called out infamous surveillance firm NSO:
Deepfake used to attack activist couple shows new disinformation frontier
WASHINGTON (Reuters) - Oliver Taylor, a student at England's University of Birmingham, is a twenty-something with brown…
AI systems are now capable of generating completely “new” faces. Currently, many of these AI systems remain imperfect, and there are some obvious giveaways or tells, even for a human eye. This is especially true for the weaker systems. However, it’s worth noting that they are getting much better, as the example of “Oliver Taylor” in the Reuters link above showed.
Here are some common tells that are useful for lower-quality attempts:
a) Weaker Deepfakes have weird teeth. AI is often bad at doing teeth accurately. Check for too many of certain types of teeth (especially top front incisors between the canines).Check for weird digital pixels (artifacts, or noise) around the edges of the teeth. Check for teeth as a set being out of rotational alignment with other facial features (not a guarantee, but a clue to dig deeper). For example:
b) Weaker Deepfakes have funny ears. AI can be bad at ears too, especially skin folds and consistency with earrings and other ear features. Not that everyone always wears matching earrings, but this can be a clue! Are the ears level, compensating for any angle in the camera’s perspective or the model’s posture?
c) Weaker Deepfakes have messy hair. AI often leaves digital artefacts around the edges of hair etc. Little pixel mess smudges are a giveaway. This is because the boundary between the background and the “person” is apparently tricky to get right.
d) Weaker Deepfakes don’t sweat the small stuff. AI also sometimes makes mistakes around the corners of eyes and with eye-glasses.
If you’d like to try out your abilities, you can test yourself here: http://www.whichfaceisreal.com/index.php These AI fakes are not the best around, but it’s a fun exercise!
3. Handles or screen names
Automated accounts are often set up en masse, so there is no time (or sophistication) to choose a “personal handle.” So you will often see things like @Sue973651179 , which no real (normal) person would accept as their handle.
Previously there was also a trend for influence accounts to have many flag emojis or other symbols in the screen names, but this seems to have spread to the real population now too. Many hashtags in a screen name on Twitter can suggest a bot — e.g., #MAGA #KAG #RESIST, etc.
4. Profile Information
If an account has no profile information, it’s more likely to be a bot, as opposed to a “sock-puppet” or fully “staffed” influence account. Certain influence (as opposed to amplification) groups also try and take on “archetypal” personalities that can fit into well established tribes; thus making infiltration into the social group easier and more likely to be successful. For instance, a influence account might claim to be “a USMC veteran.”
Account set-up date is also a clue, as very new accounts are more likely to be fake / bots. This is especially trueif they have no profile information, and even more so if they have appeared suddenly around the same time as big issue on which they are exclusively focused. Recently, China’s diplomats commenting on the Covid-19 crisis were getting a lot of love from Twitter accounts that were all set up in March and April 2020. Darwinian natural selection on Twitter means that automated accounts do sometimes get culled by the company and reported after time. This doesn’t always happen, but it is more and more likely over time.
Location used to be very easy to fake, and it is still possible with VPNs. Occasionally, those operating a fake persona forget to put on their VPNs and the location may change dramatically over the course of a day! As I mentioned in a previous piece (here), very lazy influence pushers may switch account focus rather than setting up a new batch. A great example of this were those “All-American Accounts” that supported Trump but then suddenly switched to tweeting in French to support Le Pen two years later.
Many of these profile clues become telling when combined with the activity of the account in question. Behavioural tells, like the ones we picked up in the Chinese marketing example mentioned above, can be spotted in both individual accounts and group behaviour.
5. General Activities
Botnets of social media accounts are set up for particular purposes. Whereas most real people tend to do a bit of several things (cat pictures, pro-Trump news, fishing), automated accounts tend to be narrowly focused, at least at any one time. On Twitter, amplification accounts, whether bots or human-operated networks, tend to do huge amounts of re-tweeting, which is easier and quicker to automate or perform quickly, rather than offering their own opinions or responses. At a slightly higher level, tweeting out the exact same phrases as other accounts in the network (to make a hashtag trend, or to push out a message all at once over different areas) can be a clue. If you search for a phrase that an account has used, and find that it is being replicated verbatim by multiple other accounts, you have found a campaign, and perhaps an automated one.
6. Time online
Automated bots can be set to operate at any time, but even then there are some silly tells. For example, Russian-operated networks often used to amplify messaging during office hours in St Petersburg — unusual times for Americans in Wisconsin to be online!. This was possibly because they needed human operators to lay the seeds, and only then used the bot accounts to amplify the themes. Fake accounts run by real humans were even more likely to be on “office time.” The IRA operations in St Petersburg were a bit obvious for this. We can presume that the Russian networks have now learned that this is a good way to spot bots and influence operations, so they might have transitioned their staff to doing shift work. Most normal “American” people aren’t online all day, and they certainly don’t suddenly stop their online activity at 5:30 p.m. St Petersburg time (or even at 5:30 p.m. in their claimed home time zone. A significant number of real people stay online in the evenings.
7. Weird focus
We probably all remember when American white nationalists in Charlottesville chanted, “Russia is our Friend!” However, if an account that is supposed to be some all-American guy from Kentucky, for example, is obsessively pushing and ampliying RT, Sputnik, or other Russian content and messaging, this is kind of a big red flag! Russia, PRC, Israel, Saudi Arabia, and other countries all seem to be running influence campaigns. So, there you go. After all, it’d be a bit weird if a real person from the US was obsessed with one of these countries’ media ecosystems, wouldn’t it?
8. Followers and following
Patterns of who an account follows and who follows them can also offer clues. Real people don’t tend to follow accounts that simply retweet without adding anything new. And they’re probably not going to follow small accounts that retweet all the time and don’t have lots of followers. Real people tend to follow accounts that are interesting, engaging, and have a good mix of content. Amplification networks sometimes depend on all of their individual accounts being intermingled and following other members of the network, which helps content radiate out to all the accounts and eventually even reach real people. Weird ratios between numbers of followers and followees can also be a red flag.
9. More retweets than likes
Another simple ratio-related tell can be discovered by looking at how accounts have interacted with a particular tweet. It would be unlikely for a popular tweet to get say 1000 “retweets” for every one “like” from real people. Most tweets have more likes than retweets because real people don’t tend to retweet something without liking it, whereas they are much more likely to like something without retweeting it. The ratio between the two on Twitter will not be 1:1, but it shouldn’t be that skewed in favour of retweets. If you see a Tweet that has a big disparity in the wrong direction, it could be a sign that an amplification network is retweeting it. You can then go and view some of the accounts responsible for the retweeting… Pandora’s box!
Of course, it’s worth remembering that users can run paid campaigns on Twitter to garner engagements, likes, or video views. So if a Tweet has been pay-promoted in the past, then this ratio may be affected for that reason too.
10. Meme heavy
This has changed a bit, as more and more real people share memes, but influence operations strongly focused on spreading picture memes in 2015-2016, especially on Facebook.
11. Trolling, whataboutisms, disingenious debate
This method was used in website comment sections by Chinese government-funded trolls, known as the Wu Mao Dang or fifty-cent party. With the rise of social media, this kind of behaviour has gone fully mainstream, but back in the day, real, genuine people didn’t usually behave so obstinately in discussions.
12. Odd winks and nods
I have seen several Twitter accounts (some of which I reported and were shut down) that would make snide and coy remarks along the lines of, “Maybe I’m a Russian bot!” This echoes the macho, Putin-style ‘semi-admit it with a wink and a smirk, but still deny it’ style. “No these aren’t our troops in Crimea, who knows who they are!” “Who knows if some patriotic Russians decided to hack the DNC on their own without state direction?”
Influence networks tend to deploy at the same time and interact with each other. A Twitter account worth following to learn more is Conspirador Norteño. This user delivers exposures and public information about influence and bot networks discovered on Twitter:
They really explain things well.
14. Language Skills
There’s no rule that people should speak English, French, German, or Polish well if they live in those countries, but if someone claiming to be from a heartland area is using odd idioms or unusual grammar, it may be a small red-flag worth investigating. Translation is a fine art, and using the grammar of the target language rather than your native language is tough unless you are a high-level user of the latter. As anyone who’s ever learned a foreign language knows, keeping up with online linguistic trends (slang, acronyms, idioms) is not easy, even if the troll farms are hiring from language universities.
Of course, you often need to combine a few of these methods to really get down to the truth. Some bot accounts are easy to spot from just two or three tells. Others are more sophisticated and may be hard to spot even with detailed analysis. AI and more sophisticated algorithms used for account set-up and scheduling are changing the game — but if you see some red flags, dig deeper.
Of course, not all influence accounts are automated, and not all platforms have the same tells. It’s sort of like an arms race, and presumably the smartest “bad guys” learn what to do better next time.
When you’re debating an issue on social media, knowing who you are engaging with is not vital, but it can be useful to know. It is also satisfying to receive a Twitter “Update on the account you reported” and find out that it was indeed part of an influence network. Happy Hunting!
The Atlantic Council’s Digital Forensics Lab https://www.atlanticcouncil.org/programs/digital-forensic-research-lab/ and https://www.digitalsherlocks.org/
Graphika occasionally publishes cutting edge analysis and reports of their activities— eg https://graphika.com/reports/exposing-secondary-infektion/ , or a report into a Fake Face Swarm https://graphika.com/reports/operationffs-fake-face-swarm/or their report on a network attributed to Roger Stone and his associates https://graphika.com/reports/facebooks-roger-stone-takedown/
https://thispersondoesnotexist.com Reload the page to see some pretty high quality AI-generated people.