Photo: BIPGraffiti

From a fact checker: how to sort through news and fake news

When I started fact checking professionally, I had a lot of fun (really, more like a growing nauseating fear) starting to realize how poorly many of the resources floating around the internet were fact checked. My worst fears were confirmed when I went to a fact checking workshop and the speaker warned us that most books “are not fact checked unless the author decides to independently hire someone to check it” or the publishing house explicitly makes it clear that they fact check. (Silent screams)

How can you fact check what you are reading:

1. Always follow the footnotes until you hit a researcher, data set, or person whose livelihood/reputation depends on their consistently strong research. They have a stake in the game and it’s a problem for them as much as you if they lie to you or miss something important. Sometimes you will not reach this point because sometimes new sources “estimate” figures on a trend to help their audience understand their point. You may then find sources citing each other non-stop in a weird circle that I call a “data hole.” It means there is no source. Proceed with extreme caution and look for a better source.
2. When you hit the original data source, think about what the limits of their research may be. If they ask a specific question, it doesn’t mean the data answers an entirely different one. Look at their sample. Look at what they controlled and tested for. The answer is limited to these things and the degrees of uncertainty the author discusses with you. PAY ATTENTION TO THE UNCERTAINTY. If these things are confusing to you, find someone who works with data (and is trained to do so…) and they can help you think through the source you just found.
3. There are still some magazines that prioritize having good fact checkers. The New Yorker is famous for its amazing fact checkers who get to the bottom of every story. The Atlantic and Scientific American are also good about it.
4. Personal internet blogs are the absolute worst. IF they are your entry point to a topic, follow those footnotes. FOLLOW THEM. Do not stop here, this is not going to be factually sound and it will look terrible if you cite it to me or anyone else in research land unless this is a professional researcher sharing notes from their personal lab/research work.
5. Newspapers are under really tough deadlines and this means their reporters are often forced to fact check themselves. This means their audiences sometimes fact check them and their corrections end up in the comments and finally in the piece itself. I was psyched to see the NyTimes Fact Checking team out during debates this election cycle, and that PolitFact won the Pulitzer was a huge deal. You can check out their analysis on different things to see how fact checkers break down these facts and statements for accuracy, offering a score at the end.
6. Human sources are complicated because there is always a degree to which we cannot fully comprehend every detail of every situation, which means we miss things, sometimes important things. Empirical social scientists like surveys and polls and things because they hope this will help shave off some of the biases… but this doesn’t always work (as we well know from Election 2016). Here are some good primers on how to read that data from the NYTimes: on Polling, on polling interpretation and prediction, on survey errors.

Perhaps worth noting, before this I was a full time wary social scientist. I think there are cool things to learn from our field, but there is a whole lot of noise and error and overstatement of results. I encourage you to proceed with caution, engage with those footnotes, and always ask authors for their degrees of uncertainty. IF they cannot tell you what they worry about within their “findings,” run. Hope this helps!