A surprising look at the AP “Fact Check”

I’ve been researching self-described fact-checking outfits as part of a broad, long-term project to preserve and strengthen our basic grasp of reality and facts — and our ability to distinguish facts from opinions and complex judgments. There are some serious epistemological issues that need to be sorted out if we’re going to have a civilization with reliable fact-checking. Also, political ideology and partisanship is looking like a huge threat to our shared grasp of reality, and we need a lot more — and more rigorous — social science research in this area.

For my first installment, I want to highlight the performance of AP Fact Check. On November 30, I went through the fact-checks on their homepage, one by one (they were in descending chronological order). Here I’ll walk through the first seven fact-checks I encountered. I only include the standalone fact-check articles, which focused on one specific claim. I skipped the feature called NOT REAL NEWS: A look at what didn’t happen this week, which was an aggregation of quick hits that didn’t offer any links to sources. I also skipped the feature called Science Says, a phrase which, as a scientist, horrifies me.

AP Fact Check is remarkable, and in fact unique among American fact-checkers, in several respects:

1. 94 percent of the time AP targets a conservative or Republican claim (almost always Trump). This holds true across all standalone fact-check articles from November 1, 2017 through January, 18, 2018. This pattern flagrantly violates the Poynter Institute’s code of principles. Poynter has certified AP Fact Check as abiding by its principles.
2. AP always argues against the conservative/Republican claim. Often they’ll claim it’s false, but other times they’ll just sort of peter out in a hostile op-ed without actually conducting a fact-check.
3. AP generally does not provide any sources or links for their claims. In fact, they don’t even provide links to the claim or quote they are fact-checking. Importantly, this violates Google’s policy for fact-checkers that it promotes at the top of search results, yet inexplicably Google still promotes AP. It also violates Poynter’s transparency of sources principle, yet Poynter certified them.

Every one of these things is quite remarkable on its own, compared to other fact-checkers. Politifact, the Washington Post (Glenn Kessler), and even FactCheck.org will sometimes evaluate things leftists say, sometimes rule that leftists are wrong, and sometimes rule that a conservative said something true. And they normally provide sources and links to backup their claims. They’re all partisan and unreliable, but they seem to understand that you cannot do what AP does and call yourself a fact-checker.

Every fact-check below starts with AP’s exact title in bold, and those titles are also links to the respective AP article. Let’s begin…

AP FACT CHECK: Bank sector not in ruins, despite Trump beef

TRUMP’s tweet: “The Consumer Financial Protection Bureau, or CFPB, has been a total disaster as run by the previous Administrations pick. Financial Institutions have been devastated and unable to properly serve the public. We will bring it back to life!”

THE FACTS: The sector is far from devastated.

Federally insured commercial banks and savings institutions reported more than 5 percent growth in the third quarter from a year earlier. Of more than 5,700 institutions reporting, more than two-thirds (67 percent) had year-over-year growth in quarterly earnings. The proportion of unprofitable banks fell. Quarterly net earnings also were up in the second quarter.

This is odd. Trump was talking about CFPB’s impact on financial institutions — presumably those institutions that have tangled with it (see below). AP responds with various aggregate statistics for the banking sector. They’ve gone off on a tangent that has nothing to do with Trump’s claim. It’s as though they didn’t even Google it, and might not have any idea what Trump was talking about. As a result, there is no fact-check here — AP’s points don’t collide with Trump’s claim at all.

The Consumer Financial Protection Bureau (CFPB) has been accused of corruption, abuse of power, and of extorting money from the financial institutions it investigates. That last issue arises because apparently the CFPB — unlike other regulatory agencies — funds itself from the fines it imposes, creating an apparent conflict of interest with respect to the resolution of its investigations. Ronald Rubin writes:

Targets were almost certain to write a check, especially if they were accused of subjective “unfair, deceptive, or abusive acts or practices.” Even the size of the checks didn’t depend on actual wrongdoing — during investigations, Enforcement demanded targets’ financial statements to calculate the maximum fines they could afford to pay.

Rubin’s essay appears in the first page of Google results (for the term CFPB, without any other terms), as does this account by a Democratic staffer who helped create the agency. (For a different perspective, see Gillian White’s account of how the CFPB has helped victims of absurd charges from some institutions.)

The CFPB was slammed by a federal appeals court: ruling here (PDF), and Thaya Brook Knight at Cato has written about it here.

Trump’s statement presumably had something to do with all the complaints against the CFPB, though a credible fact-checker would find out exactly what Trump was referring to just in case it was something else. AP Fact Check didn’t address his statement, or the well-documented accounts of the CFPB’s behavior. As a result, their implicit claim that Trump’s statement was false is itself false. The fact-checker was wrong.

Moreover, AP provided no sources or links for any of the irrelevant claims they made. For example, they said: “Of more than 5,700 institutions reporting, more than two-thirds (67 percent) had year-over-year growth in quarterly earnings.” Setting aside its irrelevance here, that’s the kind of claim that we really need sources for. Where did they get these numbers? Incredibly, they don’t say. They rattle off lots of statistics without even hinting at the source. I’m not aware of any other fact-checking outfit in the world that makes claims like this without any trace of a source. AP is truly remarkable.

There is another issue. Google recently annointed itself the Ministry of Truth. It’s promoting purported fact-checks at the top of search results, highlighting the conclusion of the fact-check so that readers don’t even have to open the article. If we google “Trump CFPB fact check” (without quotes), this is what we find:

This is the false AP fact check, republished by CBS News. Readers are misled into thinking Trump’s claim was false. The conclusion that “The sector is far from devastated” implies that Trump said it was devastated, which he did not. The title also implies that he said the sector was “in ruins”, which he did not. In fact, he said nothing about the banking sector.

Here Google is deceiving the public by promoting a fake fact check. Moreover, they’re promoting this fact check even though it violates their stated policy:

Analysis must be traceable and transparent about sources and methods, with citations and references to primary sources.

As noted above, AP didn’t cite or identify any sources at all. There are no references or links anywhere in their article. This is true of almost all of their “fact-checks”.

AP FACT CHECK: Trump oversells results of ‘tremendous’ trip

TRUMP, summing up a trip he took in May: “In Brussels, I urged our NATO allies to do more to strengthen our crucial alliance and set the stage for significant increases in member contributions. Billions and billions of dollars are pouring in because of that initiative. NATO, believe me, is very happy with Donald Trump and what I did.”

THE FACTS: No, billions are not pouring into the organization because of Trump. It’s not even an issue of financing NATO as an organization. The issue is how much countries in the alliance spend on their own military budgets. That appears to be creeping up, but even so it is hard to find Trump’s fingerprints on what is happening.

AP takes Trump to be referring to contributions to the NATO organization. However, his concern — as AP notes later — was about NATO member nations who were not spending as much on national defense as they had committed to (2 percent of GDP). AP does not a perform a fact check in any case — they don’t tell us whether these countries increased their spending to the tune of billions (and billions) of dollars. That’s precisely the information we need to evaluate the truth of Trump’s claim, but AP never provides it. In the remainder of their op-ed, they state:

A campaign to increase military spending began in 2014, before Trump was president, when NATO members agreed to stop cutting their military budgets and set a goal of moving “toward” spending 2 percent of their gross domestic product on their own defense by 2024. It was not a commitment and Trump had nothing to do with it. Only five of the 29 members — the U.S., Britain, Estonia, Greece and Poland — meet or exceed 2 percent. NATO Secretary-General Jens Stoltenberg said he expects Romania to reach the target this year, with Lithuania and Latvia likely to reach it next year.

Trump has put pressure on members with lagging spending to do more. It’s not clear that pressure has made them “very happy with Donald Trump.” He frequently made such claims about NATO in his first months in office.

Troublingly, they don’t provide any sources or links for any of their claims. They just make these sweeping claims and apparently expect readers to take their word for it. This is a rich example, so let’s work through it sentence by sentence:

A campaign to increase military spending began in 2014, before Trump was president, when NATO members agreed to stop cutting their military budgets and set a goal of moving “toward” spending 2 percent of their gross domestic product on their own defense by 2024.

Without sources it’s hard to evaluate AP’s claim, but more importantly it’s irrelevant since it doesn’t contradict Trump’s claim. At issue is whether Trump, in 2017, got NATO members to increase their defense spending by billions of dollars. It doesn’t matter whether they also made an earlier commitment — countries (and people) can make more than one commitment in a three-year span. (It’s also unclear why “toward” is in quotes.)

It was not a commitment and Trump had nothing to do with it.

It’s unclear why AP is saying it wasn’t a commitment — again, no sources and no detail. The fact that Trump had nothing to do with a 2014 agreement has no bearing on his involvement in a 2017 agreement. AP seems to be struggling with basic logical reasoning here — with how logic works — which is alarming for a fact-checker.

Only five of the 29 members — the U.S., Britain, Estonia, Greece and Poland — meet or exceed 2 percent.

This would appear to be the essence of Trump’s complaint. It’s unclear why AP thinks this sentence belongs in an argument against Trump. Again, there’s a strange deficit in reasoning ability here.

NATO Secretary-General Jens Stoltenberg said he expects Romania to reach the target this year, with Lithuania and Latvia likely to reach it next year.

This is not relevant to the fact-check.

Trump has put pressure on members with lagging spending to do more. It’s not clear that pressure has made them “very happy with Donald Trump.”

If AP thinks it’s not clear, they probably shouldn’t insinuate that it’s false. If they want to fact-check these sorts of claims (NATO is happy with Trump), they should do so. Since they didn’t bother to fact-check it, by rights they should keep their opinions to themselves. Note also that a statement like “NATO is happy with Donald Trump” isn’t a good candidate for fact-checking, for at least two reasons: 1) It’s vague because “NATO” is vague — there will be many ways to define and assess what “NATO” thinks. Poll the heads of state of NATO member nations? Ask the NATO Secretary General whether he’s happy with Trump? Poll NATO staffers and military officers? Poll the citizens of member nations? 2) Some of these methods are likely to be impossible, such as getting a bunch of heads of state or staffers to tell a fact-checker whether they’re happy with Trump’s effect on members’ defense spending.

Competent fact-checkers will need to understand what sorts of claims can be cleanly evaluated, and why.

He frequently made such claims about NATO in his first months in office.

This is not relevant — it doesn’t logically bear on the truth of Trump’s claim.

This op-ed was remarkably unprofessional for a self-styled fact-checker. They didn’t attempt any fact-checking, and thus withheld the numbers that would allow readers to evaluate Trump’s claim.

I looked into it, and it appears that Trump told the truth (at least about the Carl Sagan money — the billions and billions). The Washington Post reported on June 28, 2017:

NATO allies of the United States plan to boost their defense spending by 4.3 percent this year, Secretary General Jens Stoltenberg said Wednesday, a response in part to intense pressure from President Trump that the nations invest more in their militaries.

The Post further reports that the new spending total by non-US members will be $295 billion. Now we can calculate the increase in dollar terms:

1. 1.043 × OriginalSpending = 295 billion
2. OriginalSpending = 295 billion / 1.043 = 282.84 billion
3. 295 billion − 282.84 billion = 12.16 billion

The non-US members have increased their spending by $12.16 billion. This would appear to validate Trump’s claim (I assume we’re going to treat “billions and billions” as equivalent to “billions”. If we want to be pedantic, we could for example stipulate that 3 billion qualifies as “billions”, and so 3 billion + 3 billion, or 6 billion, would seem to qualify as Carl Sagan money. And that’s only half the $12.16 billion total here.)

AP’s conduct here was shameful. They deceived their readers. They failed to perform a simple fact check, and as a result spread misinformation far and wide. They also don’t seem to have adequate training in logical reasoning, which is alarming in its own right.

AP FACT CHECK: Trump tells a tale about Air Force One

President Donald Trump told a fanciful little tale Tuesday about Air Force One being denied landing rights in the Philippines during a trip by his predecessor because bilateral relations were so bad. That didn’t happen.

Before boarding the plane in Manila to come home, Trump bragged to the press that he’s pushed relations with the Philippines to new heights.

“And as you know, we were having a lot of problems with the Philippines,” he said. “The relationship with the past administration was horrible, to use a nice word. I would say horrible is putting it mildly. You know what happened. Many of you were there, and you never got to land. The plane came close but it didn’t land.”

That prompted a lot of head-scratching.

President Barack Obama last visited the Philippines in November 2015, arriving in Manila after an overnight flight from Turkey. There were no problems with landing the plane. Obama used the visit to announce the United States was transferring two ships to the Philippine Navy.

This is bizarre. Trump said “Many of you were there, and you never got to land. The plane came close but it didn’t land.”

AP said “President Donald Trump told a fanciful little tale Tuesday about Air Force One being denied landing rights in the Philippines during a trip by his predecessor because bilateral relations were so bad.”

Where did they get this bit about Air Force One being denied landing rights? Trump didn’t say anything about Air Force One being denied landing rights. That’s a strangely specific thing for AP to inject in their write-up given that it’s not in the quote.

We can’t proceed here because AP is (again) referring to a claim that was never made, much like their bizarre write-up in response to Trump’s statement about the CFPB. This is very confusing. It’s possible they didn’t include the whole quote, and that at some point Trump said something about being denied landing rights. If that’s the case, they need to find that quote.

Which brings us to another disturbing fact — they don’t provide any links to this quote or interview. As usual, AP fails to provide any sources or links whatsoever.

I don’t know what Trump was referring to. He was presumably talking to the media, so “you never got to land” could refer to a decision by Obama as much as the Philippines. Someone would have to follow up with him to know what he was referring to specifically.

AP FACT CHECK: Trump on veterans’ health care, economy

Veterans Day prompted President Donald Trump and his administration to take stock of what’s been done to fix health care for those in uniform. They claimed more progress than has been made.

That tendency to overreach extended to trade and the economy as Trump visited Japan, South Korea, Japan and then Vietnam, where he told U.S. veterans of the Vietnam War that the Department of Veterans Affairs has made “amazing” strides and already “is a whole new place.”

His remarks and a White House account of progress at the VA did not acknowledge old problems that persist. For example, a key effort to improve waiting times by revamping the VA’s electronic medical record system may not be completed for eight more years — when Trump will be out of office.

There’s no fact-check here. This piece starts off as an op-ed, where they take vague pre-emptive shots at Trump before they tell us what he said.

Except that they don’t tell us what he said. Rather they first quote a single word — “amazing” — and then the phrase “a whole new place”.

That’s it. The rest is AP paraphrasing, but given their misconduct in the other fact-checks, I assume most people are not going to trust AP’s paraphrasing. What makes this even more ludicrous is that, as ever, they don’t provide any links or sources, making it impossible for the reader to know what Trump actually said beyond the six words AP gives us.

In any case, they don’t fact-check his alleged claim that the VA has made amazing strides or is a whole new place. Notably, this is not valid fact-check material — these are his opinions and judgments. We can’t adjudicate these sorts of opinions and judgments as true or false, especially figurative expressions like “a whole new place”. They’re not matters of fact — someone’s opinion that the earth is flat can be evaluated because it’s a matter of fact, but that an organization has made “amazing” progress is a subjective judgment. The AP staff seems to lack the necessary training to perform fact-checking.

AP also complains that Trump did not acknowledge “old problems that persist”. However, this complaint is outside the bounds of a fact-check — no one is obligated to talk about whatever AP thinks they should talk about, much less agree with whatever AP thinks is a problem. When humans talk about something, they tend to talk about that thing, not other things. And when humans are excited about some positive development in an organization, they don’t necessarily take time to mention every remaining problem they think the organization has. What people talk about is up to them, not fact-checkers.

Washington Post didn’t pay Roy Moore accusers

This is mostly about a mysterious voicemail left by a person claiming to be a Washington Post reporter, and offering money for dirt on Alabama Senator Roy Moore. There was also a short-lived Twitter post suggesting the Post was offering money for such dirt.

AP claimed that the Post didn’t make these offers, because the Post denied it. (AP’s headline is a bit off — this was about offers to pay, not claims of actual payment.) I’m not sure that we should generally take someone’s denial as proof that an accusation is false, but in this case I’d concur that it’s reasonable to believe the Post. The reason is that the evidence behind the accusations was so sketchy that we’d have to dismiss it. In the case of the voicemail (here), the caller said his email address was albernstein@washingtonpost.com — there is no Al Bernstein at the Post, and that email address bounces because it doesn’t exist. I don’t know what the point is of pretending to be a reporter only to leave a fake email address and name — even as a false flag operation, it was pretty flimsy.

(They may have used the name Bernstein to exploit people’s vague awareness of a Bernstein associated with the Washington Post because of the legendary Woodward and Bernstein, but I don’t know how much that would buy them given that they didn’t leave a working email address.)

The Twitter account that posted unverified claims of the Post offering money for dirt on Moore was apparently deleted shortly afterward. So there was nothing substantial to this story, and I can’t find any fault with AP’s fact-check here.

AP FACT CHECK: Anti-Muslim videos misrepresent what happened

This article refers to three videos Donald Trump posted on Twitter. It features three headings: FALSE, APPARENTLY TRUE, and MISSING PIECES.

Under FALSE, AP discusses a video of a man beating up another young man on crutches, apparently in the Netherlands.

However, the video is authentic — as AP notes, the Dutch embassy reported that the perpetrator was sentenced for the assault. The original tweet accompanying the video says “Muslim migrant beats up Dutch boy on crutches!” AP claims that the “migrant” part is false, and they apparently think this makes the video as such “FALSE”.

However the evidence against the “migrant” claim is the Dutch embassy saying that the perpetrator was born and raised in the Netherlands. There seems to be a lack of attention to detail here. That he was born and raised in the Netherlands tells us he’s not an immigrant. It doesn’t tell us whether he’s a migrant — a migrant is not necessarily an immigrant, but can simply be someone who travels for work, often around seasons (see Merriam-Webster and Cambridge Dictionary). I have no idea whether this young man is a migrant, and AP doesn’t tell us.

UPDATE (August 25, 2018): I got feedback that in Europe the word migrant is often used as synonymous with immigrant, so my paragraph above is probably bogus. Medium doesn’t have a strikethrough feature, so I’m just leaving the paragraph there for the sake of historicity and integrity.

Also, I think the question of whether the young man is Muslim may be murkier than I thought. The Dutch embassy didn’t comment on that aspect of the video’s description, which implied to me that it was accurate. When people make claims A and B, and the relevant authorities release a statement saying A is false, but make no comment on B, that almost always means B is true. I can’t find any official statement on the whether the young man was Muslim, and a Google search only provides leftist and MSM sources, none of whom have obtained an official statement. So the question of whether the young man was or is Muslim is unresolved. My original report continues below…

In any case, the video is authentic, and no one disputes that the perpetrator is Muslim or that he beat up a Dutch boy on crutches. Every word of the Tweet is true with the possible exception of the word “migrant”. I would find it strange to call such a video categorically “false” if only the migrant claim is false, but I expect this to be a matter of opinion. If we switched to a proportional assessment of truth, we would have to say it was mostly true, since four of the five elements of the sentence are true, and the video is authentic.

NOTE: My matter-of-fact discussion of these videos should not be taken as endorsement of posting or retweeting them. Fact-checking requires a certain discipline and a focus on the facts and claims at issue — my views concerning the wisdom or decency of posting these videos is out of scope.

Under “APPARENTLY TRUE”, AP discusses a video titled “Muslim Destroys a Statue of Virgin Mary!”

The video is authentic, and features a man destroying a statue of the Virgin Mary while speaking in Arabic. AP notes that the Middle East Research Institute identified the man as Sheikk Omar Raghba, a Syrian. There is nothing contested here — the description of the video is true.

AP seems unwilling to simply report that the video and its description are true. Instead, it claims that the “circumstances are not verified”. It’s unclear what they mean by circumstances, or what else they need. They close by saying: “Also well known: Anti-Muslim extremists in the U.S. and other countries of the West have torched mosques.”

That statement was surprising — it’s not relevant to the fact check, and illustrates how unprofessional, undisciplined, and partisan AP is. They seem unwilling to ever report that something a conservative said or posted is simply true. In such cases, they always add sauce to the fact check, typically in the form of a leftist talking point — in this case, the attempt to draw moral equivalence between Muslim extremism and anti-Muslim extremism.

Under “MISSING PIECES”, AP discusses a video described as “Islamist mob pushes a teenage boy off roof and beats him to death!”

The video is authentic — as in the Dutch case, the perpetrators here were prosecuted and sentenced for their crime. (They were sentenced to death.)

The video takes place in Alexandria, Egypt. Nothing here is disputed, at least not by AP. Strangely, AP closes by complaining that “The wider context of those killings was ignored — the fact that both sides had blood on their hands. It was a time of violent protests by pro-Morsi protesters and a violent military crackdown. More than 1,000 people died in the military’s dispersal of Islamic protests.”

A video of an event can only be what it is — a video of that event. This video is less than two minutes long. AP’s complaint that the video or its filmers ignored a “wider context” seems to demand that the video be a documentary that shares AP’s opinions about the conflict (e.g. that “both sides had blood on their hands”). But it’s simply a short video of one event — it’s unclear how it can be anything else. There’s nothing its filmers can do about that, and no one is obligated to be Ken Burns. In any case, AP’s complaints and opinions are beyond the scope of a fact check.

It’s unclear how statements like “It was a time of violent protests by pro-Morsi protesters” and “More than 1,000 people died” are relevant to the authenticity of the video or the truth of its description. Saying that it was a time of violent protests by pro-Morsi protesters doesn’t conflict with the video, since the video shows pro-Morsi protesters killing someone.

Saying that more than 1,000 people died seems especially irrelevant — what implications would any particular death toll have for the authenticity and truth of the video? What if 100 people had died? Or 5,000? Or three? Would any of those numbers tell us anything about the video?

As usual AP cites no sources and provides no links for any of the videos or the claims AP makes. Note that Twitter has since censored the videos, even though they were authentic, and banned the British group that originally posted them.

AP FACT CHECK: China-US business deals are largely gloss

There’s no fact-check here. It’s unclear what AP wants to do in this case. This is about Trump’s visit to China and the trade deal he announced there, but there is no Trump quote. There is a quote from Chinese President Xi Jinping, but it’s unclear whether or not AP thinks that it’s fact-checking it. Their write-up starts with:

A trade and investment package announced during President Donald Trump’s visit to China is more about the art of diplomacy than the art of the deal.

The package, said to be worth more than $250 billion, puts a symbolic gloss on fundamentally strained economic relations between the U.S. and China.

AP is starting with opinions and commentary — this is an op-ed. I don’t know what else to say about it. There may be some useful information in it, but as usual there are no links so the reader is trapped in another AP op-ed without any easy way to validate its claims.

Conclusion

The performance of AP Fact Check was so disorienting that I sat on it for more than a month. In six of seven fact-checks, AP’s claims were false or mere opinions that we cannot evaluate. In at least two cases, they seemingly failed to read the claim they were checking and went off on tangents that had nothing to do with the claim. In cases where simple data would resolve the fact-check, like the NATO member defense spending claim, AP chose not to provide readers with the data.

As I noted in the beginning, AP seems to only target conservatives, and consistently rules against them. I’m not aware of another fact-checker that behaves this way. I think it’s clear that AP Fact Check is a partisan scam, not an actual fact-checking operation. Even if they had the integrity to perform fact-checking, it appears that they lack the training in logic and a basic grasp of the epistemic issues that are fundamental to the enterprise: what facts are, as opposed to opinions and judgments; what sorts of claims can be evaluated as true/false; and what sorts of claims can be feasibly fact-checked from a practical standpoint.

Google’s involvement here bears special attention. They haven’t done the basic epistemological work that they would need to do to be reliable facilitators and disseminators of fact-checks. It doesn’t seem to have to occurred to them that a self-identified fact-checker can be unreliable, incompetent, or corruptly partisan. As a result, Google is disseminating fake fact-checks. In other words, right now Google is an agent of misinformation and fake news. This is an enormous problem, a problem so large that it intimidates me a bit. I can’t overstate how dangerous it is for a civilization to have monopolies like Google filtering and distorting reality for hundreds of millions of people. There’s really no upper bound to how much damage Google and other tech monopolies could do to our society by behaving this way.

Reality is sacred. Facts are sacred. Or they damn well need to become sacred right quick if we intend to have a working, peaceful civilization.

Something seems to be going very wrong at Google. Last week Eric Lieberman reported that Google’s fact-checking dissemination effort is exclusively targeting conservative outlets. What’s more, Google falsely attributed claims to conservative media like the Daily Caller — claims they never made — and then presented “fact-check” results by leftist outfits purporting to debunk the conservative claim that no conservative actually made. That’s jaw-dropping. The good news there is that Google has now corrected the false fact-checks that Lieberman reported. (The Federalist has more on the scandal.)

To be fair, Google’s role in all this might be the result of algorithms, not corruption. Google has an algorithmic bias. They think they can solve almost any problem with the right algorithm, deep learning, what they amusingly call “AI”. What I think Google needs to understand is that not all work can be done by algorithms and code, at least not in 2018. Sometimes you need qualified humans using their brains. To know if a fact-check is correct or not, we need brains, epistemology, actual intelligence. Google needs to develop a credible epistemic framework for fact-checking before it tells millions of people that a fact-check is really a fact-check. We need clear and defensible rules and definitions around what facts are, and what they aren’t.

Google also needs to come to terms with political bias and how it is shaping fact-checking right now. I’ve already evaluated many other fact-checkers, and it appears that leftists were quick to seize upon the perceived authority and propaganda value of fact-checking — every one of this first generation of American fact-checkers is a partisan leftist operation. (The UK has better, less partisan, outfits.) Like Google, they haven’t bothered to lay out an appropriate epistemic framework for fact-checking, nor do they seem to have any training, so they end up being unreliable. Absurdly, they all only hire leftists, which is like waving a banner saying that they’re not even trying to do this right. Google should definitely be alert to whether a purported fact-checker only hires partisans of one camp. In any case, it’s a mess, and I’ll be detailing this mess in future installments.

UPDATE 2018–01–25: I cleaned up the three numbered points in the intro. Instead of saying “AP always focuses on conservatives, with one exception”, I now just give the percentage as 94 percent of the time, which generously counts two times they sort of evaluated a leftist/Dem claim sandwiched in a fact-check of a conservative. I also added the Poynter principles and AP’s non-compliance.

UPDATE 2018–09–03: On June 27, 2018, I posted an update saying AP had silently modified some of its fact checks after I debunked them. I was wrong about that. The Internet Wayback Machine contradicts my memory, and I think in any contest between my memory and the Wayback Machine, we have to go with the Wayback Machine. I was also wrong in claiming that the AP Fact Check never mentioned the CFPB (in the body of their fact check, called The Facts) — they mention it as “the bureau”. Thanks to Brandon Shollenberger for catching both errors.

The core of that section stands. AP never fact-checked Trump’s claim — they just go off on a tangent about the banking sector, which was republished by CBS, and presented as an actual fact check by Google.

José Duarte has a PhD in Social Psychology and is preparing to launch the Valid Science Center to advance rigorous, useful, and likely-to-be-true social science. The Center will also propose a nonpartisan epistemic framework for fact-checking, and publish a comprehensive almanac evaluating every single fact-check performed by all the major American fact-checkers.

You can reach Joe at jose.duarte@asu.edu. Twitter here. Website here. My favorite journal article is here.