How the Factitious news game helps people learn to detect fake news
The American University Game Lab released its Factitious news game on July 3rd, 2017. With little promotion and only personal appeals to our social networks, we hoped for the best … maybe 10,000 plays of the web app, if we were lucky.
But the game went viral beyond our wildest dreams: 69,032 game plays* on the first day–251,450 in the month of July. The numbers continue to soar– more than 730,000 game plays (and counting).
Our analysis of more than 2 gigabytes of data has revealed how people can use our game to sharpen their skills for spotting fake news.
- When people viewed the source of an article, they spotted fake articles better than without the source info.
- Players who took the time to read more of the article spotted fake news more often.
- Some fake online media providers fooled our players by providing fake urls like the “TheMississippiHerald.com.” The stories were fake and the sources were too.
We recently released a new version, Factitious 2018!, which presents six new game levels, one every Monday morning, from October 1st through to the mid-term elections with high score lists to show who can spot fake news the best!
This article tells the story of how the Factitious game was designed and developed and how the game is revealing important insights into how people consume online news.
Before you read on, it might be helpful to first play Factitious so you can get a sense of what more than 540,000 players have already experienced.
It started with a simple but insightful question …
What if we could build a game to see if people could tell whether an online story was real or fake?
Maggie Farley pitched the initial concept to me during a break in an early workshop of the AU Games and Journalism JoLT program. Maggie is an award-winning journalist and former foreign correspondent for the L.A. Times and I am an award-winning game designer and former PBS documentary filmmaker, so I immediately realized the potential.
In the feverish next month after our first excited conversations about the possibilities in early February of 2015, Maggie and I created and tested two rounds of paper prototypes that showed great promise with the concept. We also built an educational prototype with Chas Brown and Patrick McEvoy.
Founding AU Game Lab Director Lindsay Grace then started a push to create a public mobile phone version and in the spring of 2017, recent AU Game Lab Grads and JoLT fellows Joyce Rice, Kelli Dunlap, Cherisse Datu, and Lindsay joined the team to produce the Factitious smartphone web app.
At 7am on the morning of July 3rd, 2017 we posted the game on our cloud server and hoped for the best. It was the middle of the summer so there was very little promotion but everyone on the team did what we could by spreading the word on our social networks. We didn’t make any guesses about how many games would be played but if you had asked, I might have said, “little promotion, low web presence, a handful of social networks … 5,000 in the first week, maybe 10,000 total?”
Wait … what? How much traffic’s coming in?
We blew through 10K at 12pm on the first day and it kept on rising. When the dust settled at the end of July 4th, the game had been played more than 104,000 times!
After the total traffic leveled off in August, we figured that was the end of it. But for the next 10 months, people kept playing the game, a lot. By the end of the first year, players had viewed and rated nearly 8 million articles (7,793,271)! We suspect that teachers may be using the game in their classes (which we’ll discuss in the next Medium article … Factitious: Students and Teachers).
Helping People Spot Fake News Better
The game included a key educational feature — a hint that revealed the source of the article. We wanted to encourage people to slow down and think about where the article was coming from.
In our data analysis, we could clearly see that the hints helped players identify real and fake stories better. Only about half of the time did players click the Show Source button but when they viewed the source, they were right more often than people who didn’t have that information (74% success rate vs. 61%). These two articles, one fake, one real, show how the hints can make a big difference.
It seems most players knew that The Onion website presents satirical articles just for fun, so the article was fake. And in a similar way, when players learned that the “almost too weird to be true” article about the cockroach came from CNN, they assumed the article was true, and it was.
But with some of the articles, the opposite happened. After viewing the hint, players scored worse!?! The hints were supposed to improve player’s scores. What happened with these articles?
Maggie Farley did the detective work on this and her summary captured our best guesses about these strange results.
The Mississippi Herald sounds real but is not. The article fooled Fox News and others.
The Daily Beast is legit, but not well known.
Overall, six of the articles presented to players had hints that made players score worse for having read them. The graph below plots the difference in success rate for players who did not view a hint with players who viewed the hint.
These results align with other research by Vanessa Otero at Ad Fontes Media who surveyed and analyzed news media organizations by their journalistic rigor and political alignment. Her chart shows the possible impact of a url’s familiarity and/or taking a political stance might have on perceptions of news media organizations.
Why Is Factitious So Popular?
It’s hard to say exactly what leads to success but there were probably five key factors.
When we released Factitious in July 2017, the news media was abuzz with articles and analysis of fake news. Several organizations created online activities and games that challenged people to identify fake articles (Politifact, DROG in the Netherlands, among others). Our game fed a desire to learn more about this controversial issue.
Kelli’s playful gestural interface–swipe right for real, left for fake–created a engaging user experience. Testimonials to her design were delivered by other fake news games that quickly copied her feature. Having personally tested the game with more than two dozen people, I’ve seen the delighted smiles or surprised looks as people pull across the screen to reveal their success or failure.
The streamlined onboarding let players start a game with just one click (“quick start”) and get engaged by the tinderesque swiping. The effectively simple game design–two choices: real or fake–enticed players to try another article, and another, and another. Players could choose to get more info about the article by scrolling down or clicking the Show Source button, if they wanted, but those actions were not required.
Co-designer Maggie Farley worked with the News Literacy Project (she’s the Chair the NLP’s Board of Advisors) to promote the game to their participating schools and classrooms. This likely stimulated part of the huge traffic we’ve seen during the school year.
And the “finely tuned” collection of “appropriately difficult” articles–not too easy and not too hard–made the game challenging without being insulting, (even while it was subtly teaching players how to spot fake news). We pre-tested the articles and tracked how well testers could spot fake from real articles. We then rejected articles with some simple rules:
- If nearly all of the testers were successful in separating real from fake … then that article was “too easy” and we tossed it.
- If half or less of the testers failed to separate fake from real … then that article was “too hard or confusing,” and we flushed that article too.
We wanted articles that were in the sweet spot between “too easy” and “too hard.”
One of the most effective ways to get players engaged in a game is to present challenging decisions or situations. Most people enjoy dramatically close basketball or baseball games–especially the players who will win or lose. Creating “appropriately difficult” challenges is also a great way to keep players playing your game.
What’s Next (updated 11/22/18)
Factitious 2018! The Factitious “campaign,” which ran from 10/1/18 through 11/11/18, garnered more than 60,000 new game plays, 85% of which occurred during school hours (M-F, 8am-4pm). We will be making some modifications in the coming weeks to align the game to the educational needs of teachers.
The Hidden Audience of the Factitious News Game Medium article! I just posted an article that describes how teachers have given Factitious a second life as a vital educational app. More 410,000 games have been played during school hours, which is more than our viral surge in July 2017. Thousands of teachers around the U.S. have integrated the game into their courses. The article profiles two teachers and how they’ve adapted the game to their media literacy and writing lesson plans.
Factitious: Classroom Edition. We’re currently designing a potential version tailored for high school teachers and college professors. The Factitious team will collect over 300 new articles and then organize our article collection by reading level and topic area. Teachers will be able to select the most appropriate set of articles for their students. We’ve launched the Factitious: Classroom Edition crowdfunding campaign to support this classroom extension.
We’d love to hear from any teachers who have used Factitious in their classrooms. Please email me at factitious2018@gmail.com. Thanks.
** Note on updated stats –After the initial version of this article was published, we did a “deep dive” into the game logs and analyzed at how many articles each player played in the game. We have redefined “game plays” to only count for players who completed at least one round of the game (5 articles). This new stat is about 75% of the previous stat.
