Journalism and the machines: what ClaimBuster told me about Australian politics
Visitors to Australia’s Federal Parliament are often surprised by the robust verbal confrontation between the government and the opposition technically known as questions without notice, more commonly as question time (QT).
A theatrical highpoint of every sitting day, QT is part intellectual cage fight, part kindergarten spat — and all psychological warfare. Political journalists watch the hour-long QT as drought-stricken farmers view the clouds. They look for signs, they read the climate.
But what if you were interested in facts? Would QT in the House of Representatives satisfy such an urge? My guess is the knee-jerk reaction of most Australian voters would be, no way (or a less polite variation thereof); politicians are to facts as oligarchs are to taxation: uncommon bedfellows.
But to give them (politicians not oligarchs) the benefit of the doubt, I decided to search QT for facts using ClaimBuster, a machine-learning tool developed by computer scientist Chengkai Li and his team at the University of Texas, Arlington and in collaboration with journalism experts and computer scientists from Duke, Stanford and Google Research.
Its algorithm searches sentences for key words and structures that are commonly found in factual statements. ClaimBuster is a step along the way to fully automated fact-checking, the Holy Grail of computational journalism. It provides an intriguing glimpse into the role machines, bots and artificial intelligence will play in journalism’s future.
(If, like me, you are starting to grapple with that issue, it is certainly worth a look at Jeff Jarvis’s recent Medium post and an academic article by the University of Texas’s Naeemul Hassan and others including Li and Bill Adair, the founder of PolitiFact.)
ClaimBuster is not fully automated fact-checking. It doesn’t rate the truth, truthiness or otherwise of a political statement. It is not the grail. It uses natural language processing to rate whether a sentence is check-worthy on a scale of 0 to 1.0, with the higher number being most worthy.
I’d urge you to try it yourself. It’s cool.
So back to the QT of March 17, the most recent in the Federal political calendar: just what did ClaimBuster find?
Apologies for breaking the flow here but a bit of background about politics is need for non-Australians: 1) there is an election coming, probably within a few months; 2) the Liberals (a party nominally of the right) are in power, led by Prime Minister (PM) Malcolm Turnbull; 3) Labor (a party nominally of the left), led by Bill Shorten, is in opposition. Other key players include the Treasurer Scott Morrison and his Labor opponent Chris Bowen.
QT follows a regular pattern: the opposition asks the PM or a minister a question and he/she answers, then the government’s backbench asks a minister or the PM a question and he/she replies. The opposition hopes to expose and embarrass the government; questions from the government’s side are designed to give ministers or the PM a chance to show off — or beat up on the opposition. Occasionally, one of the parliament’s handful of independents or minor party members get to ask a question.
There were 17 questions asked on March 17, eight by Labor, one by an independent, and the rest by the Liberal backbench. I cut and pasted the transcript of the eight Labor questions and the government’s answers into ClaimBuster and recorded the number of sentences with a score of .60 or above. That seemed a reasonable bar to set in the search for facts. Or so it seemed.
Of the 270 sentences analyzed, just 15 or 5.6 per cent came back with a check-worthy score above .60. Labor questions recorded only four of the plus .60s. The government’s responses, including the ironic use by Morrison of a previous Labor minister’s statement, were the rest.
Here is a selection of what ClaimBuster found, with the check-worthy rating at the beginning of each sentence:
Turnbull to Shorten:
0.66 The government spends over $20 billion a year on Medicare.
Turnbull to Labor’s deputy leader Tanya Plibersek:
0.91 Honourable members should recall that the previous Labor government locked in funding for the dental NPA [national partnership agreement] at the relevant levels of $69 million in 2012–13, $155 million in 2013–14 and $119 million in 2014–15.
Bowen to Morrison:
0.69 Doesn’t this mean that the only action on income taxes taken by this Liberal government since the last election has been to increase the amount of income tax paid by Australians?
Bowen to Morrison:
0.67 Despite all the talk since you became Treasurer; private sector wages growth is the lowest it has been since records began, while living standards have fallen for a record seventh consecutive quarter.
Morrison to Bowen
0.83 In fact, they would propose $60 billion of additional expenditure over and above what is currently in the budget and forward estimates — and to pay for that they propose just $1 billion in savings and $7 billion in extra taxes.
And here are some of the lowest ClaimBuster scores:
Turnbull on a former colleague:
0.24 He is a great Australian.
0.20 He was a great minister.
Morrison to Bowen:
0.34 The member for McMahon [Bowen], coming up here in his audition for Vogue magazine, wants to actually talk down the economy.
And this from Speaker Tony Smith, whose job it is to control parliamentary debate:
0.15 All members are equal in this House.
Unsurprisingly, ClaimBuster found some of the most check-worthy statements in the exchanges between ministers and their own backbench or when ministers used those questions, known as Dorothy Dixers, to attack Labor. Here, for instance, is Turnbull extolling the virtues of government policy:
0.71 So every lever of policy that we can pull is being directed at that goal: a $1.1 billion innovation and science agenda; a defence white paper with a massive investment in innovation, industry and jobs in Australia; and free-trade agreements, as I noted, with the growing economies — particularly the largest economy in East Asia, China.
It’s clear that ClaimBuster’s algorithm appreciates numbers — a staple of human fact-checkers too — and doesn’t care much for bland assertion. What makes a great Australian?
Much of what ClaimBuster flags could benefit from a full fact-check, some of it not. Bowen and Morrison both make claims worthy of checking; Turnbull’s statements, in this small example, are either lacking in grit or are bogged down in detail, though his statement on dental funding might on a quiet day be worth a check.
But that’s not the point.
Overall, ClaimBuster strikes me as a pretty useful guide for journalists and those members of the public who wish to spend time using an algorithm to help find facts. It is not so much a watchdog as a bloodhound, always sniffing about for an interesting scent. And that’s the key word: always.
ClaimBuster doesn’t sleep and can be used to search out check-worthy claims at all hours of the night and day. It helps journalist and others to keep an eye out for lurking facts when they are off doing other things, such as sleeping or reporting on other matters.
That’s certainly how Adair sees it: “Chengkai and his team have built a marvelous tool that can help fact-checkers with one of their biggest challenges — finding factual claims,” he says. “ClaimBuster is like a reporter who never gets tired of watching the political debate. It can ingest massive amounts of political discourse and identify the claims that should be fact-checked.”
Occasionally it throws up a curveball.
Immigration minister Peter Dutton attacking Bowen:
0.74 Not only did the member for McMahon [Bowen] preside over the worst period in immigration history in this country — where people drowned at sea and thousands of kids went into detention — let us go to what he did next: he also opposed our moves to tighten the visa character test, which we have now presided over and which saw people kicked out of this country.
ClaimBuster would have been totally ignorant of the political hot button it presses by giving such a statement a high score. And, as a fact-checker and knowing the basic facts of the story (people did drown at sea; kids were sent to detention), I might be tempted to excuse that statement from further checking.
The phrase “worst period” piqued my interest. Perhaps this indicates the cleverness of ClaimBuster’s algorithm: it can detect a factual claim within a sentence that has other stuff in it. “So not only is ClaimBuster a tireless bloodhound, it also smells stuff that the editors don’t,” says Adair.
With ClaimBuster pricking my conscience, I just might have due cause to review or double check my judgment. Was it really the worst period in immigration history? Can you fact-check that? It’s all good grist to the journalistic mill, prompted by an algorithm.
This is where the machines and the bots will really help out: as a value/bias free (notwithstanding, the actions of hackers) mirror on the world we journalists set about creating. Or so we think.
This is a trend very much worth thinking and talking about — and acting upon. I have recently added artificial intelligence to the list of potential services on offer from my project, GoClevr.media, at the City University of New York (CUNY) Graduate School of Journalism.
With a Federal election looming in Australia, I can certainly see a role for ClaimBuster and its successors in keeping voters informed and politicians honest. I’d be interested in talking to potential media partners in such an exercise.
Peter Fray is an entrepreneurial journalism fellow at CUNY. He is professor of journalism practice at University of Technology Sydney, the founder of PolitiFact Australia and the former editor-in-chief of the Sydney Morning Herald.
Thanks to Bill Adair and Chengkai Li for their contributions to this article.
