How “News Literacy” Gets Web Misinformation Wrong

Mike Caulfield
Apr 4, 2017 · 16 min read

I have a simple web literacy model. When confronted with a dubious claim:

  • Check for previous fact-checking work
  • Go upstream to the source
  • Read laterally

That’s it. There’s a couple admonitions in there to check your emotions and think recursively, but these three things — check previous work, go upstream, read laterally — are the core process.

We call these things moves. They are generally usable intermediate goals for the fact-checker, often executed in sequence: if one stops panning out, then you go onto the next one.

The reason we present these in sequence in this way is we don’t just want to get students to the truth — we want to get them there as quickly as possible. The three-step process comes from the experience of seeing both myself and others get pulled into a lot of wasteful work — fact-checking claims that have already been extensively fact-checked, investigating meaningless intermediate sources, and wasting time analyzing things from a site that later turns out to be a known hoax site or conspiracy theory site.

To give an example, here’s a story from Daily Kos:

And here’s what students will say, when confronted with this after years of “close reading” training:

  • Who is this Hunter guy?
  • Hunter is a pseudonym, which is bad. How do we know who he really is? Suspicious!
  • What is this Daily Kos site?
  • Who owns Daily Kos? Liberals? Really?
  • There’s a lot of comments which is good.
  • The spelling and punctuation on this page is good, which makes it credible.
  • The site looks well designed.
  • The site is very orange.
  • There’s anti-Trump language on the page which makes it not credible and slanted.
  • The picture here isn’t of the Russians, it doesn’t match, which is fishy.

They might even go to Hunter’s about page and find that the most recent story he has recommended has, well, a very anti-Trump spin on it:

They can spend hours on this, going to the site’s about page, reading up on Hunter, looking at the past stories Hunter wrote. And in my experience, students, when they do this, are under the impression that this time and depth spent here shows real care and thought.

Except it doesn’t. Because if your real goal is to find out if this is true, none of this matters.

What matters is this:

What you see above, in the first paragraph of the story, is a link to the Wall Street Journal, the source of the claim. This “Hunter” might be a Democrat with a pseudonym invoking an 80s police procedural series, but he follows good, honest web practice. He sources his fact using something called “hypertext”. It’s a technology we use to connect related documents together on the web.

And once we see that — a way to get closer to the actual source of the fact, all those questions about who Hunter is and what his motives are and how well he spells things on this very orange looking site don’t matter, because — for the purposes of fact-checking — we don’t give a crap. We’re going to move up and put our focus on the Wall Street Journal, not Daily Kos.

(Disclosure — I used to write a bit on Daily Kos, I know certain front-pagers there, and yes I know that Hunter’s name is not really a reference to the uniquely forgettable Fred Dryer television series).

Once we get to the Wall Street Journal, we’re not done. We want to make sure that the Wall Street Journal’s report on this is not coming from somewhere else as well. But when we get to the Wall Street Journal we find this is original reporting from the Journal itself:

So going upstream comes to an end for us, and we move on to our next strategy — reading laterally about the site. Now in this case, we all might skip that — it is the Wall Street Journal we have here — but the truth is that students might not know whether to trust the WSJ. So we execute a trusty domain search: ‘’, which tells Google to get all the pages that are talking about that aren’t from that site itself:

And when we do that we see that there is a Wikipedia page on this site that will let us know that the WSJ is the largest newspaper in America by circulation and has won 39 Pulitzer prizes.

We do note, looking at the WSJ article, that Hunter has tweaked the headline here a bit. The WSJ says that Trump Jr. was likely paid $50,000, whereas Hunter’s headline is more strident about that claim. But apart from that the story checks out.

Do we trust this WSJ article 100%? No, of course not. But we trust it enough. It’s tweetworthy. And after we’ve confirmed that fact we can go back down to the Daily Kos page and see if that article by Hunter has any useful additional analysis. Over time, if you keep fact-checking Hunter’s stories, and they keep checking out, you might start considering him a reliable tertiary source.

If you use this process, you’ll notice a couple of things. The first one is that it’s pretty quick — the sort of thing that you can execute in the 90 seconds before you decide to retweet something.

But there’s another piece here too — rather than the fuzzy analysis of a single story from a single source you have here a series of clearly defined subgoals with defined exit points: check for previous work until there is no more previous work, get as close to the original as you can until you can get no closer, and read laterally until you understand the source. These goals are executed in an order that resolves easy questions quickly and hard questions efficiently.

That’s important, because if you can’t get it down to something quick and directed then students just think endlessly about what’s in front of them. Or worse, they give up. They need intermediate goals, not checklists.

Fact-Checking the Mailman

Recently the Digital Polarization Initiative has been getting a lot of press, and as a result a lot of people have been sending me alternative approaches to fake news.

Most aren’t good. I’ve already talked about the reasons why CRAAP is ineffective. I’ve been more hesitant to talk about a popular program from the News Literacy Project called Checkology, which is less obviously bad. But in past days I’ve seen more and more people talking about how Checkology might be a solution to our current problem.

Unfortunately, news literacy isn’t the big problem here. Web literacy is. And the Checkology curriculum doesn’t really address this.

As an example, here’s the Checkology checklist:

Now, there’s some good things in here. I think their starting point — check your emotional reaction — is quite good, and it’s similar to some advice I use myself. Thinking about editorial standards is good. Reverse image search is a helpful and cool tool. Looking for reports from other sources is good.

But if you include subquestions, there are twenty-three steps to Checkology’s list and they are all going to give me conflicting information of relatively minor importance. What if there are no spelling errors but there is also no current date? What if the about page says the site is a premier news source, but it has no links back to original sources? What if it cites a variety of things but doesn’t hyperlink?

Even more disturbingly, this approach to fact-checking keeps me on the original page for ages. What if I get all the way through the quarter of an hour that the first twenty-two questions take only to find out on question twenty-four that Snopes has looked at this and it’s complete trash?

This isn’t hypothetical. Given the current reaction time of Snopes to much of the viral stuff on the web you could probably give Student A this long list and Student B a piece of paper that says “Check Snopes First” and the Snopes-user would outperform the other student every time.

And even if there is no Snopes article on the particular issue you are looking at, what good is it going to do you to look this deeply at the article in front of you if it is not the source. Consider our Hunter article:

Let’s answer the questions using Checkology:

Ok. So now we’ve spent ten to fifteen minutes on this article, looking for dates and email addresses and contact emails. Now what? I have no idea. The site appears on a list of fake news sites and doesn’t have a contact page. But it does have a date on the story and Politifact and Snopes don’t have stories on it. The image for the article is an old image (fake!).

And conversely, the article links to the Wall Street Journal as the source of the claim.


Are you starting to get the feeling we just spent a whole lot of time on a checklist that we are about to crumple up and through in the trash?

To put this in perspective, you got a dubious letter and just spent 20 minutes fact-checking the mailman. And then you actually opened the letter and found it was a signed letter from your Mom.

“Ah,” you say, “but the mailman is a Republican!”

How does this make any sense?

Staying On the Page

If you want to read how badly this fails, you can look at some of the stories about the program as it is used in the classroom. Here’s a snippet about some folks using Checkology:

This drives me nuts. It worked out this time, of course, because the story is false. But relying on your intuition like this, based on no real knowledge other than how a claim looks, is not what we should be encouraging here.

Worse, you see the biggest failing here — in this curriculum based around asking questions of a text, the student is not actually doing anything other than asking questions. They are looking at a text and seeing what feelings come from it after asking the questions.

Here’s another student on the same flu story:

I’m not attacking the student here — they are doing exactly what the curriculum told them to: looking at a page and asking questions about it.But you can see here that we just had a student use comments on an article to fact-check an article. Comments!

Comments can be useful, of course. When the trail has gone cold tracing a story to its source, often it’s a comment from someone that points the way to the original story. Sometimes a person points to an article on Snopes or Politifact.

But to get to the truth quickly, comments are usually the worst place to look. At this point, almost every anti-Trump story online has someone under it calling it “fake news”. What do you do with that? How does it help?

Again, this is not what a web literate person does when they hear that the flu vaccine may be bad. A web literate person finds the original source of the claim and then asks the web what it knows about the source. All this other stuff is mostly beside the point.

More than Fiction, Less than Fact

Which brings me to my second (third? tenth?) pet peeve here: there’s a muddling here of the issues of claim, story, and source.

Take that claim on Facebook that over a dozen deaths were caused by the flu vaccine and the CDC banned it. “Fiction,” said the student.

And it’s true that the source she was reading and the story that she was reading were misinformation. But is the story complete fiction? Let’s do a search:

I’m guessing the student read the Natural News story down towards the bottom — Natural News is one of the big suppliers of anti-vaxx propaganda on Facebook. But the story here is not cut from whole cloth. Just reading the two blurbs at the top of the search results I get a pretty good idea what happened. The Wall Street Journal reports that on December 1 Italy suspended, pending an investigation, the use of two batches of the flu vaccine. This was apparently due to 12 people dying shortly after receiving it.

On December 3rd, the BBC reported that Italy had completed its investigation and cleared the vaccine as safe. A bit of domain knowledge tells me that what probably happened is what often happens with these things — flu vaccine is administered to a population that is relatively old and has a higher chance of dying due to any cause. Eventually those sort of probabilities produce a bunch of correlations with no causation.

By the way, this ability to read a page of results and form a hypothesis about the shape of a story based on a quick scan of all the information there — dates, URLS, blurbs, directory structure — that’s what mastery looks like, and that’s what you want your employees and citizens to be able to do, not count spelling errors.

So is this vaccine story “fiction”? I suppose so. It’s not true that the vaccine killed these people, and the CDC certainly didn’t cancel the vaccine. If we were doing a Snopes ruuling on this I’d go with a straight up “False” as the ruling.

But I’d also note there was a brief panic over a series of what we now know to be unrelated deaths, followed by an investigation that ruled the vaccines safe.

You are not going to get that if you stare at a page looking for markers that the story is true or false. You are only going to get that if you follow the claim upstream.

The Checkology list declares that students should “use the questions below to assess the likelihood that a piece of information is fake news.” In that instruction you have a dangerous conflation of source and claim, which is only furthered by confusing questions like “Does the example have a date on it?”

News as source, and news as claim. It’s an epistemological hole that we put our students in, and to help them out of it we hand them a shovel.

The Ephemeral Stream

How do programs like the News Literacy Project’s Checkology get these issues wrong? The intentions are good, clearly. And there is a ton of talent working on it that’s had a lot of time to get it right:

Fundamentally, these efforts miss because what’s needed is not an understanding of news but of the web.

As just one example, the twenty-plus questions that students are asked to ask a document seem to assume that

  1. Sources are scarce and we must absolutely figure out this source instead of ditching it for a better one.
  2. Asking the web what it knows about a source is a last resort, after reading the about page, counting spelling errors, tallying punctuation, and figuring whether an author’s email address looks a bit fishy.

The web is not print, or a walled garden of digital content subscription. Information on the web is abundant. And yet the strategies we see here all telegraph scarcity, as if the website you are looking at was a book you were handed in the middle of a desert to decipher.

The approach also does not come to terms with the Stream — that constant flow of reshareable information we confront each morning in Twitter or Facebook. You don’t have fifteen minutes to go through a checklist in the Stream. You have 90 seconds. And your question isn’t “Should I subscribe to Natural News?” — your question is “Did a dozen people die of flu vaccine?” Whether news folks want to admit it or not, the stream tends to erode brand relationships with providers in favor of a stream of undifferentiated headlines.

Above all, the World Wide Web is a web, and the way to establish authority and truth on the web is to use the web-like properties of it. Those include looking for and following relevant links as well as learning how to navigate tools like Google which use that web to index and rank relevant pages. And they include smaller things as well, like understanding the ways in which platforms like Twitter and Facebook signal authority and identity.

In short, we need a web literacy that starts with the web and the tools it provides to track a claim to ground. As we can see from the confusing and confused reactions of students in the Checkology program, that’s not happening now, and “news literacy” isn’t going to fix that.

If you’re interested in alternative, web-native approaches to news literacy, you can try my new, completely free and open-source book Web Literacy for Student Fact-Checkers.

You should also read Sam Wineburg and Sarah McGrew’s Why Students Can’t Google Their Way to Truth, and the results of their Stanford study which showed that the major deficits of students with regard to news analysis were issues of web literacy and use.

Mike Caulfield

Written by

Teaches web literacy and other things. Recent book: Web Literacy for Student Fact-Checkers.