Auditing: The Little Indicator that Could
The MIT Election Data and Science Lab manages the Elections Performance Index (EPI), an objective, nonpartisan assessment tool evaluating U.S. election administration. Following the recent update to the index with data from 2016, we are dedicating a series of posts to exploring the EPI’s underpinnings.
Our post today was penned by Charles Stewart III, who is the founding director of the MIT Election Data and Science Lab.
Of the seventeen indicators that make up the Elections Performance Index, post-election auditing is the one that’s tailor-made for elections geeks. The term “auditing” is drawn straight out of the most buttoned-down profession, and it suggests a no-nonsense approach to making sure that everything was done according to Hoyle. Election audits aren’t identical to financial audits. However, the idea behind election audits is similar — to give assurance to voters and candidates that the election returns accurately depict the outcome of an election.
The post-election auditing indicator in the EPI is pretty simple and basic. It records whether a state requires a post-election audit of the vote total, either through statute or administrative directive. It is not based on the type of audit performed — risk-limiting vs. fixed-percentage, for instance — nor is it based on the outcome of the audits. All it asks is: “audits, yes or no?”
But, that might change. It is now widely recognized that risk-limiting audits are the preferred way to test the accuracy of the outcome. Efforts are also under way to extend auditing to other parts of the election system. As a consequence, one shouldn’t be surprised to see the details of this indicator change for the 2020, or even 2018, EPI.
Why audit elections?
The idea behind auditing election results is an old one. As early as 1965, the California election code required that the accuracy of an automated vote count be verified using a public process of manually tallying the votes in 1% of precincts.
Greater attention began to be paid to post-election audit requirements in the immediate aftermath of the 2000 presidential election. Two events spurred this interest. The first was the poor performance of the punch-card voting machines in Florida during the 2000 election. The second was growing distrust of electronic voting machines that had no paper record of the individual ballots cast. In the former case, audits were seen as a way to identify machines that might be failing. In the latter case, advocacy in favor of paper “back-ups” of electronic voting systems, called “voter-verifiable paper audit trails,” would be effective only if the paper records were actually compared in a systematic way to the electronic tallies.
By the time the EPI was developed around 2010, a robust field of post-election auditing was beginning to take shape. More and more states were writing post-election audit requirements into their codes. Scientists and engineers were beginning to develop practical procedures that would allow election officials to apply increasingly sophisticated sampling techniques to post-election audits. A very helpful summary of the state of thinking about post-election audits in the late 2000s was published by the Brennan Center in 2007.
Auditing has become more common
The EPI can be used to show the growth in the utilization of post-election audits over the past decade. As the accompanying figure shows, the number of states with an auditing requirement has grown from 23 in 2008 to 34, plus the District of Columbia, in 2016. As I have written in a recent retrospective essay on the EPI, the rise in the adoption of post-election auditing over the past decade is one of the most important reasons why the average overall EPI score has increased during this period.
Focusing only on the rise in post-election auditing over the past decade misses the variability of the auditing programs that the states have adopted. California, for instance, still maintains its 1% precinct audit, although recent legislation opens the door for other methods. South Carolina, which uses electronic voting machines without a voter-verifiable paper audit trail (VVPAT), checks to see if the various electronic reports generated by the voting machines are consistent with each other. Colorado has begun implementing risk-limiting audits, and other states have passed legislation to mandate them in the future. A challenge for the future of the EPI is to reflect the variety and sophistication of auditing requirements as they evolve in the states.
The new kid in town: risk-limiting audits
The most typical form of post-election auditing is called the “fixed-percentage audit.” Like the name implies, it involves selecting a fixed percentage of the ballots cast (or precincts), hand-counting the ballots, and then comparing the hand-count to the automated count.
With “risk-limiting audits,” officials select a randomly chosen set of ballots and hand-count them carefully. Then, the officials check to see whether the winner reflected in the sample of ballots is consistent with the winner from the unofficial tally of all the ballots.
An important difference between risk-limiting audits (RLAs) and fixed-percentage audits is that the number of ballots chosen to audit varies under RLAs. If the unofficial tally shows the winner prevailing by a large margin, fewer ballots are sampled, compared to a situation where the unofficial tally shows a narrow margin.
The term risk-limiting refers to one important feature of RLAs. With an RLA, the election jurisdiction has to decide the confidence they wish to have that the automated vote-count is the correct one. Said another way, the jurisdiction has to decide how much risk they are willing to take that the audit will incorrectly conclude that the automated vote-count is correct. If a state sets a risk limit at 10%, for instance, then if the automated count declares the wrong winner — either because of fraud or error — the sample of ballots will fail to catch the error 10% of the time, but will catch it 90% of the time. If the risk limit is set at 1%, there is only a 1% chance that the audit will fail to catch an erroneous result.
There is a trade-off in setting the risk limit and the number of ballots that must be randomly chosen and hand-counted. The smaller the risk accepted, the more ballots that need to be hand-counted. At the extreme, if a state accepts 0% risk, then it would hand-count all ballots, which is what a recount is.
The mechanics of an RLA are such that if enough discrepancies are revealed in the counting of the sampled ballots, one of two things can happen. First, a larger sample can be drawn, to increase the confidence in the audited results. Second, all ballots can be recounted.
The idea of RLAs has been developed by statisticians working for many years. A well-titled introduction to RLAs, “A Gentle Introduction to Risk-limiting Audits,” is a good place to start in learning more about them.
Colorado passed a law in 2009 requiring risk-limiting audits, implementing it in 2017. New Mexico has had a type of adjustable-percentage audit that functions like a hybrid between fixed-percentage and risk-limiting audits. At least five other states — California, Colorado, Ohio, Rhode Island, and Virginia — have passed laws moving those states in a direction that favors RLAs.
An ad hoc committee of the National Academies of Science, Engineering, and Medicine recently released a report, entitled Securing the Vote, which called for all states to implement RLAs within the next ten years.
Under most circumstances, RLAs have the advantage of requiring the inspection of a smaller number of ballots in an audit than fixed-percentage audits. RLAs don’t solve all auditing problems, however. For instance, the small number of ballots reviewed make it hard to uncover certain types of fraud, unless the discrepancies are very large. For that reason, I would argue that RLAs are a powerful tool, but should not be the only post-election auditing tool in the election administrator’s belt.
Wait: There’s more!
Vote-counting is a critical task in election administration, but it is not the only critical task. While post-election audits have gotten the most attention within the topic of assuring the quality of election administration, other problems can arise in election administration that undermine the integrity of outcomes. Voters can be given wrong ballots. Candidates can be missing from ballots. Voter registration rolls can be incorrect. The wrong addresses can be assigned to election districts. The list could go on.
Elections are a detail-rich activity, and efforts have been made to develop methods of auditing all those details. For instance, Michigan runs “procedural audits,” to ensure that local jurisdictions are following good procedural practices before, during, and after Election Day. Other states run logic and accuracy tests on ballot counters after the election, to check whether a “test deck” of ballots is properly tallied. New methods are being developed by academics to help assess whether election districts were drawn correctly, while academics and private organizations have worked on ways to ensure that voter registration records are correct.
The National Conference of State Legislatures has a very useful Web page that summarizes what states are up to on the broader election auditing front.
Keeping up with the world of auditing through the EPI
As I noted at the outset, the post-election auditing indicator in the EPI is very simple, reflecting the world of election auditing around 2010. Over the past decade, this world has become much more complex. On just the topic of post-election ballot audits, the consensus is emerging that RLAs are the gold-standard for providing assurance that an election’s outcomes were correct. Shouldn’t the EPI reflect which states are using RLAs and which aren’t? Should the EPI reflect more thoroughly the full range of election auditing procedures that states mandate?
The EPI has served a useful purpose in identifying the states that are committed to post-election ballot audits through their laws and regulations. To stay relevant, the EPI now needs to dig deeper into what states are actually doing in those audits, and identify which have the most thorough programs of checking that the details were right.
