Beware of exit polls…
Recently, I have been wanting to expand my knowledge of probability and statistics. After browsing the internet for good learning resources, I finally decided to start reading “Introduction to Probability”, a book by Jessica Hwang and Joseph K. Blitzstein that originated from a series of lectures on probability and statistics taught at Harvard University. If you have an interest in this subject, I absolutely recommend this book. It is extremely well written, thorough, and thought provoking. Another major bonus point is that it is freely available as an online script at http://probabilitybook.net. The book also contains hundreds of practice exercises, many of which have full solutions available here. This is extremely important because probability and statistics — much like all quantitative subjects — are best absorbed through a learning-by-doing process.
While solving exercises, one of them (2.11) caught my attention because it takes a seemingly innocent and mundane problem to reveal surprising and somewhat counterintuitive results (actually this is a recurring experience in probability theory!). It considers the probability of predicting the correct winner of an election based on the results of exit polls. This problem is very timely this year, as around half of the world will embark in some type of election in 2024. Let’s take a look at the problem setting.
An exit poll is a survey taken of voters just after they come out of the voting booth. If you are into politics, or even if you simply have tuned in on an election night, you know that as soon as enough voters have cast their ballots, every news channel begins to release exit polls trying to predict before their competitors who will be the projected winner. This is of course done before all the official votes are tallied, and can be notoriously inaccurate if the survey participants are not sampled correctly. Let me give you some recent examples:
- Indian General Elections (2004): In the 2004 Indian general elections, exit polls indicated a landslide victory for the ruling Bharatiya Janata Party (BJP)-led National Democratic Alliance (NDA). However, when the official results were announced, it was the Indian National Congress party and its allies who emerged victorious, forming the United Progressive Alliance (UPA) government.
- United States Presidential Election (2004): In the 2004 presidential election between incumbent President George W. Bush and Democratic challenger John Kerry, exit polls initially indicated a tight race in several key states, leading to speculation of a possible Kerry victory. However, as the official results were tallied, Bush won comfortably, exceeding the expectations set by the exit polls. Some experts labelled these exit polls as “the most inaccurate of any in the past five presidential elections”.
- United Kingdom General Election (2015): In the UK’s 2015 general election, exit polls suggested a hung parliament, with the Conservative Party falling short of a majority. However, when the actual results came in, the Conservatives won a clear majority of seats, defying the predictions of the exit polls. This discrepancy surprised many political analysts and highlighted the limitations of exit polling.
The main issue in all three of these cases was that the sample of people who were asked and agreed to participate in the survey were not representative enough of the overall population of voters. This is exactly a situation that the exercise tackles.
Consider an election with two candidates, which we shall call Candidate A and Candidate B. Upon leaving the voting booth, each eligible voter is invited to participate in an exit poll and asked whom they voted for. Some voters accept, but some others refuse. The probability that a person accepts to participate in the exit poll is 70% if they voted A, but only 30% if they voted B. There might be many reasons for this. Candidate B might be more controversial and people are not comfortable revealing they voted for them, or maybe the B electorate is simply more reserved. In the exit poll, 60% of the respondents claim to have voted for A (and we assume that they are all honest). This would suggest a comfortable victory for A. What is the actual probability of a victory for A?
What appealed to me the most in this problem was how surprising the results are given the extreme simplicity in obtaining the solution. The only two tools needed to understand and solve the problem are the law of total probability and Bayes’ rule. Let’s review them briefly before we attempt a solution.
The law of total probability (LOTP)
Probability theory is at its core nothing but a set of rules on how to assign uncertainty to some falsifiable events. Examples of events from the problem above are “a voter will vote for candidate A” or “a voter will participate in an exit poll”. Any event can be represented as a subset of the total sample space S, i.e. the union of all the possible events taking place in the given framework, and probabilities are numbers constrained between 0 and 1 that quantify our level of (un)certainty about each event. In essence, then, probability theory is just set theory with a structured way of assigning real numbers (probabilities) to each set.
We typically choose to normalize the total sample space such that its probability is P(S)=1. Furthermore, we have one more axiom in order to deal with mixing events: the probability of a union of events is the sum of the probabilities for each event. For the simplest case of just two events, we can thus write
P(A∪B) = P(A)+P(B)
The law of total probability is a formula derived from this axiom. It rests on the realization that any event in the sample space can be decomposed into a union of disjoint events. The figure below visualizes this statement.
In particular, if we restrict ourselves again on two events A and B, we can write
A = (A ∩ B) ∪ (A ∩ B^c)
where B^c defines the complement of B, i.e. everything that is not in B. This statement is visualized in the next figure.
Now, by applying the axiom for the probability of unions, we obtain
P(A) = P(A ∩ B) + P(A ∩ B^c)
This form can be made even more insightful by using conditional probabilities, which is what we are after in this problem since we are dealing with the probability that an event occurs (e.g. a person votes for Candidate A), given that we know another even has occurred (e.g. a person is willing to participate in the exit poll).
Conditional probabilities
Conditional probabilities are written as P(A|B), which can be read as the probability that event A happens, given that we know event B has happened. They are the foundation of most of probability theory because they allow us to constantly update our knowledge about events, making them dynamical quantities. The definition of conditional probabilty is
P(A|B) = P(A ∩ B)/P(B)
which makes a lot of sense intuitively: we consider the event where both A and B happen, and normalize its probability with the probability that B happens, since we know that B has already happened with probability 1. Effectively, we reduce the sample space to B only, and look for the size that A occupies in this reduced space. The second figure above can again help you visualize this.
Now, notice that the intersection event appears in the definition of conditional probability? That means we can solve for it, and insert it in the formula for P(A) above (and we can do the same for the event B^c). This yields
P(A) = P(A|B)P(B) + P(A|B^c) P(B^c)
which is the well-known formula for the law of total probability for two events. What it says is quite profound: if we know P(A|B), P(A|B^c), and P(B), we can reconstruct P(A) (note that the probability of the complement P(B^c) is nothing but 1 — P(B) by using again the axiom for the union of events applied to the entire sample space written as S=B∪B^c). But there is more! Enter Bayes’ rule.
Bayes’ rule
Since the intersection event is symmetric, meaning A ∩ B = B ∩ A, we can choose on which event to condition to get it. We thus have the following equality
P(A|B)P(B) = P(A ∩ B) = P(B|A)P(A)
Solve for P(A) (or equivalently for P(B)) and you get another way of determining the probability of an event by knowing conditional probabilities with a different event:
P(A) = P(A|B)P(B)/P(B|A)
This is another rewriting of a formula known as Bayes’ rule, which is usually stated as
P(A|B) = P(B|A)P(A)/P(B)
Bayes’ rule occurs over and over again in probability and statistics and is fundamental to simplify problems by employing conditioning. It’s an extremely powerful mathematical tool that allows us to switch between different types of information. For example, we might want to know the probability that a person develops a disease (event A) given some risk factor (event B), i.e. P(A|B). However, we typically have direct access only to the probability P(B|A) because we first identify the disease, and then reconstruct potential risk factors through patient anamnesis. If we know P(A) and P(B) (i.e. by running statistics on populations), Bayes’ rule can give us an estimate for the potential of developing the disease in the first place.
The solution
After having refreshed our knowledge of the LOTP and Bayes’ rule, we now have all the tools needed to solve the exit poll problem!
Let us first define all the quantities of interest. Let us call A the event that a person votes for Candidate A, and W the event that a person is willing to take part in the exit poll. Then we can assign the following probabilities based on the problem statement:
- P(W|A) = 0.7
- P(W|A^c)=0.3
- P(A|W) = 0.6
We are after P(A). By using Bayes’ law, we can rewrite P(A) as:
P(A) = P(A|W)P(W)/P(W|A) = 0.6/0.7*P(W) ~ 0.86 P(W)
On the other hand, the LOTP gives us
P(W) = P(W|A)P(A) + P(W|A^c)P(A^c) = 0.7P(A) + 0.3(1 − P(A))
by diving the entire event space in the complementary events A and A^c and using the fact that P(A^c) = 1-P(A). Now, if we insert P(W) in the first equation for P(A), we get
P(A) = 0.86 ( 0.7P(A) + 0.3(1 − P(A)) )
Solving for P(A), we obtain P(A)~39%. This is quite a surprise! The probability of Candidate A actually winning is lower than 50%, even when it looked like they were the clear winner from the exit polls!
I hope this brief but insightful exercise will raise your awareness for the limitations of exit polls and also show you the beauty and power of thinking conditionally. On the next election night, take the exit poll frenzy with a grain of salt and wait until all the votes are officially counted before acknoweldging the projected winner.
