Fact check this, PolitiFact

Challenging PolitiFact’s illusion of standards

About a week ago, I published to Medium an article looking at the definition of fact-checking put forward by PolitiFact creator and Duke professor Bill Adair.

Adair said it isn’t fact-checking to fact check just part of the political spectrum. I suggested Adair’s argument fails the test of coherence.

On June 28, 2016, Adair’s successor as editor of PolitiFact, Angie Drobnic Holan, repeated Adair’s incoherent rule for fact checkers and proposed a framework of common principles for fact-checking.

Holan’s framework, in “What do fact-checkers around the world have in common?” illustrates the degree to which the fact checkers at PolitiFact fail to register self-awareness. The following section headings track with the main points of Holan’s rules for fact checkers.

Transparency

Holan says fact checkers should champion transparency. She said it should be clear to readers why the fact checkers rated a statement as they did.

Unfortunately for Holan, PolitiFact has never bothered to reveal any objective means of distinguishing between its “False” rating and its “Pants on Fire” rating. The only difference, going by PolitiFact’s definitions, comes from “Pants on Fire” statements counting as “ridiculous” in addition to false. With no objective and defined measure of “ridiculous,” how is any reader supposed to understand why a rating received a “Pants on Fire” rating instead of a “False” rating? PolitiFact’s other “Truth-O-Meter” ratings share similar problems with subjectivity.

PolitiFact’s “Truth-O-Meter,” as it was designed, places the transparency Holan advocates out of PolitiFact’s reach.

About as objective and scientific as a Ouija board

Fairness

Holan says “We use the same standard for every fact check.”

The PolitiFact Bias website, which I help edit, features many examples countering Holan’s claim.

A classic example from PolitiFact Bias compared extremely similar claims from the Florida Democratic Party and Josh Mandel, an Ohio Republican running for the U.S. Senate. Both charged a member of the opposite party with raising his own pay while serving in Congress.

PolitiFact Florida found it true that Republican Bill McCollum had raised his own pay. PolitiFact Ohio found it “False” that Democrat Sherrod Brown had raised his own pay, arguing that members of congress could not raise their own pay.

The circumstances for McCollum and Brown were closely parallel. PolitiFact used a different standard for one than it did for the other, and has never transparently addressed the discrepancy.

More recently, PolitiFact has flaunted its unfairness by allowing definitions for words like “voucher” and “assault rifle” ample flexibility to support one political party over the other while consistently denying that “Ponzi” financing legitimately describes Social Security — even though professional journals contain abundant examples of experts doing just that.

If PolitiFact has the kind of transparency Holan claims, one could easily find a principled reason why PolitiFact will not recognize the use of “Ponzi scheme” that appears in reputable journals. But nobody will find such a reason, for PolitiFact is unfair and lacks the transparency Holan claims it has.

Thoroughness

Holan says PolitiFact shows its thoroughness by using “the same rigor for every fact check.”

One would think that if PolitiFact Florida used the same rigor as PolitiFact Ohio on claims about congressmen raising their own pay that both, not just one, would find that congressmen can’t raise their own pay.

Jeff D., co-editor at PolitiFact Bias, found a more recent example. After PolitiFact instituted a policy of evaluating the underlying argument behind claims involving numbers, PolitiFact planted a “Mostly False” on Republican presidential nominee Mitt Romney for saying the recession had hit women the hardest under Obama. PolitiFact said Romney was right by the numbers, but his underlying argument was false. When Bill Clinton later said the economy performs better under Democrat presidents, PolitiFact found no good evidence that policy differences accounted for the numbers, but declined to dock Clinton for using a false underlying argument. Clinton received a “True” rating from PolitiFact.

Where’s the rigor PolitiFact brought to the Romney fact check?

Independence

Holan mentions a number of features contributing to PolitiFact’s independence, but I’m restricting my criticisms to two of them.

Holan says PolitiFact does not take positions on the issues it fact checks. How does that show independence? I do not see how it shows independence, and Holan does not explain. Invoking respected media expert Jay Rosen, I would say that PolitiFact does take positions on issues it fact checks. But PolitiFact fastidiously avoids allowing preferences to be made public to help preserve the appearance of nonpartisanship. Does secrecy contribute to independence?

Holan also says “We take measures to avoid any undue influence from funders, fact-checking supporters or anyone else.” We wonder how PolitiFact avoids feeling the pressure for-profit journalism puts on its staff to generate clicks from its predominantly left-leaning audience.

Call it a test of transparency.

Accountability

After patting PolitiFact on the back for holding politicians accountable, Holan tells her readers that PolitiFact gets its accountability from the public.

(W)e fact-checkers are accountable to our readers and the public. We promptly correct errors and address legitimate reader concerns.

A May 18, 2016 fact check from PolitiFact Missouri makes Holan’s claim the stuff of comedy.

PolitiFact Missouri fact checked a claim from Democratic gubernatorial candidate Chris Koster about the gender pay gap. That fact check remains filled with errors despite a detailed public critique and vigorous efforts to bring the problems to the attention of those responsible (including Holan).

Doubtless Holan would like for people to believe PolitiFact Missouri has not fixed the item because the concerns about it are not legitimate. That notion cannot withstand scrutiny. PolitiFact has not corrected its misquotation of Koster. Nor has it corrected its butchery of the source it once used to claim a discrimination-based gender pay gap as high as 18 percent. Nor has PolitiFact posted a correction notice letting readers know it changed a key figure in the fact check from $7.5 billion to $1.7 billion. Neither figure is right.

PolitiFact Missouri’s example by itself destroys any credibility Holan might have in claiming PolitiFact is held to account by the public.

A good set of aspirational standards

There is nothing wrong with Holan’s set of standards in terms of aspiration. A fact checker should be transparent, fair, thorough, independent, and accountable.

But PolitiFact has had years to work on pursuing high journalistic standards and succeeds only in looking hypocritical when it publishes its standards.

Physicians, heal thyselves.