Where You Live Affects What Your COVID-19 Test Means: A Visual Interpretation
An important disclaimer before you read this: in this post, I discuss COVID-19 testing, with topics including false positives, false negatives, and the interpretation of testing results and their consequences. This post does not constitute medical advice — please consult your doctor when making medical decisions for yourself or others. This information is also not meant to contradict social policy or public health policy that may be in place, but rather to help people understand those policies more fully and help everyone navigate these times in an informed manner.
Today, a myriad of states, municipalities, and private companies are launching wide-scale studies to understand the prevalence of COVID-19 in their respective populations. While accuracy has been highlighted as a serious concern in the case of antibody testing, it turns out that under the right conditions even high accuracy tests can produce almost meaningless results.
This was the subject of my previous post, where I discussed the false positive paradox and how proper interpretation of COVID-19 testing can be wildly counterintuitive. Here we’ll revisit these concepts with visualizations that illustrate the paradox more clearly. (A huge thanks to Yael Yungster and Jeff Mekler for designing and creating the visualizations).
A Hypothetical Scenario
It’s a Saturday morning in Ohio, and Robin is heading to a drive-thru clinic to get tested for COVID-19. The city of Cleveland recently decided to test 10,000 of its inhabitants in order to understand the current prevalence of COVID-19, and Robin was randomly selected. Much to her surprise and dismay, and despite not displaying any symptoms, she tests positive. She’s instructed to stay home and isolate herself for the next 14 days.
Let’s assume the study uses a high quality test that has only a 1% false positive rate, meaning that non-infected people have only a 1 in 100 chance of testing positive.
What’s the chance that Robin actually has COVID-19?
It may seem logical to think, “well it’s 99%, of course.” But this is in fact wrong. Very wrong. In this scenario, Robin’s actual chance of having the disease may be as low as 17%.
If this seems surprising, you’re not alone. As is often the case with probability questions, human intuition can lead us wildly astray. Let’s dive deeper to understand how this is possible.
Tests for COVID-19, no matter how accurate, are not infallible. Like all medical tests, a fraction of sick patients will test negative (i.e. a false negative), and a fraction of healthy patients will test positive (i.e. a false positive).
In order to estimate Robin’s probability of infection, we’ll make the following assumptions:
- The test has only a 1% false positive rate
- The test has a 30% false negative rate — consistent with previous reporting
- Cleveland’s infection rate is 0.3%. (As of April 23, data indicates about 0.1% of the population is currently confirmed to be COVID-positive, but the true infection rate is likely much higher. This chosen rate is largely meant to be illustrative.)
What turns out to be the confounding factor here is not the error rates of Robin’s test themselves, but rather how they compare with the prevalence of COVID-19 in Cleveland.
Visually Interpreting Robin’s Positive Test
Let’s represent the population (hypothetically) being testing in Cleveland by the following grid.
Each box in the grid represents an equal number of people participating in the testing. There are 1000 boxes total, so 1 box = 10 people.
Below, we indicate in red the boxes that correspond to the expected number of infected individuals in the study. Three boxes denote the infected participants, corresponding to 0.3% of the population (or 30 people). The remaining 997 gray boxes correspond to healthy, non-infected participants.
Of the healthy population in gray, we expect 1% to result in false positives. This corresponds to approximately 10 squares, indicated in orange.
Lastly, we also account for false negatives at a rate of 30%. Of the three red boxes representing actually infected individuals, we expect approximately one to correspond to patients erroneously testing negative.
We can now consider the population that has tested positive in the study in totality. By comparing the number of true positives (those who are actually infected, shown in red) and false positives (healthy individuals, in orange), we can tackle the question of what a positive test actually means for Robin.
The takeaway is striking. As the boxes suggest, we only expect 2 out of every 12 people who test positive for COVID-19 in this study to actually be infected. That means that even though Robin tested positive for COVID, she only has about a 17% chance of actually being infected.
What exactly is going on here? The phenomenon we’re seeing here is referred to as a false positive paradox, and it arises when the false positive rate for a test (1% in this case) is higher than the base rate of the condition being tested for in the population (0.3%).
This scenario underscores the vital importance of interpretation of test results with proper statistical methods. Moreover, it suggests another highly counterintuitive reality: how you interpret a positive test depends on where you live.
Consider an identical experiment run on 10,000 people in New York City using an identical test and testing procedures, only now assuming a base rate of infection of 2% in the population.
Because of the much higher base infection rate in New York, many more people participating will be infected with COVID-19 compared to the Cleveland study. On the other hand, because the false positive rate is unchanged, roughly the same number of false positives will exist.
Here you can start to see a significant difference in the two populations. Whereas in Cleveland, false positives were much more common than true positives, in New York that is no longer the case. Even when removing false negatives, the number of infected individuals testing positive for COVID-19 (shown in solid red) outnumber the non-infected population that is falsely testing positive (in orange).
Each of the two studies has roughly the same number of false positives, represented by 10 boxes in each case. While Robin’s positive test implied she only had a 17% chance of having COVID-19 in Cleveland, were she instead living in New York, the same test would imply a 59% chance of being infected. If instead of 2%, we assumed New York’s base rate were 5%, her chance of being infected would be almost 80%.
(Try this interactive tool yourself to understand how prevalence and test accuracy affect the interpretation of a positive test result.)
The fact that identical tests in two different cities can have completely different interpretations is a confusing concept regardless of whether you’re a patient, a doctor, or a policymaker. And while our example involves testing for the COVID virus, the same exact principles apply to antibody testing as well. Anytime the false positive rate for an antibody test exceeds the prevalence of antibodies in a population, the false positive paradox can rear its head.
The consequences for misinterpreting a positive test in either case are significant. An individual who is overly certain they’ve built immunity may naively ignore best practices like frequent hand-washing, not touching their face, and participating in social distancing. Such behavior may not only be a danger to their personal health, but to those around them as well. If that person lives in a city with a very low infection rate, they could easily have less than a 50% chance of having COVID antibodies even with a positive test result. (And what’s more, little is understood about the level and duration of immunity that COVID antibodies actually provide).
The Need for Well Understood Testing
An added complication here is that false positive rates for a given test can be difficult to pin down. While a test may boast a certain accuracy estimate, that estimate always carries a level of uncertainty. In other words, rather than being one precise value, it is more fair to say that the false positive rate for a test lives within a range of values. How wide a range? That depends on how thoroughly the test has been validated against known negative samples. The more thorough the validation, the narrower the range will be.
A test with a reported 0.5% false positive rate might really imply that the false positive rate lies somewhere between 0.1% and 2.0%. Why does this matter, you might ask? Well, if the upper end of the range eclipses the prevalence in the population being studied, a false positive paradoxical situation can ensue.
This seems to be exactly what happened in a Stanford study posted to medRxiv last week which concluded that 50–85 times as many residents of Santa Clara County, California had been infected by COVID-19 compared to the number of confirmed cases there. While the estimated false positive rate for the test (0.5%) was much lower than the number of participants who tested positive (1.5%), the researchers appear to have improperly accounted for the uncertainty in the test accuracy. Because the upper end of the range of possible false positive values was 1.9%, the authors shouldn’t have ruled out the possibility that all participants testing positive were simply false positives. While there are likely other conclusions to be drawn from the study, the authors’ main initial finding appears to be flawed.
So what does this mean for widespread antibody testing? Are such studies doomed to lead to widespread false positive paradoxes? The short answer is that such studies have to be done with a careful understanding of Bayesian statistics, using antibody tests that are well understood. When testing a population with low COVID prevalence, researchers will have to use tests with extremely low false positive rates — and with very low uncertainty in those false positive rates — to ensure meaningful results. And of course, peer review is essential to weed out erroneous conclusions.
On top of that, patients participating in antibody studies will have to be careful about drawing strong conclusions about their personal results. Depending on where they live and the accuracy of their test, the meaning of a single positive test may be tenuous. This underscores the critical role of local public health officials in guiding policy and informing citizens — including both patients and doctors.
Despite the obstacles, population studies will ultimately continue to get better so long as we continue to invest in developing accurate tests with well understood false positive rates. Understanding virus and virus-antibody prevalence is vitally important to epidemiologists modeling the course of the pandemic, policymakers looking to make data-driven choices, and ultimately our overall success in relieving suffering during this pandemic.
That said, we’ll all be well served to think scientifically in the mean time. When a new population study claims that “X% of our population tested positive”, we shouldn’t mistake this for the actual prevalence in the population. The first questions asked should be 1) what was the false positive rate of the test, and 2) what range of possible prevalence values are implied. This is what any epidemiologist would do, and what any informed citizen should do as well.
Thanks again to Yael Yungster and Jeff Mekler for designing and creating the visualizations which beautifully illustrate the false positive paradox. Be sure to try out this interactive tool to understand how prevalence and test accuracy affect the interpretation of a positive test result.
