Headlines about the Barnstable County SARS-CoV-2 outbreak suggest the vaccine isn’t working. The reports provide no evidence that’s true.

The recent headlines around the SARS-CoV-2 outbreak in Barnstable County, Massachusetts, are causing confusion. The writers take a factual piece of information (75% of those who were infected were vaccinated) and elevate it to headline status. The problem is, this fact is misleading and the headlines are really only interesting if misinterpreted it to mean that the vaccines are not working.

Here’s the problem: you can’t use a line or two from a headline (or anywhere else) to determine if the vaccines are or are not effective. In fact, if you actually read beyond the headlines, or if you read the CDC report which documents the Barnstable outbreak, you see that they acknowledge and emphasize that their data cannot be used to assess how well the vaccines are working. But in this age of Tweets and TikToks, many only skim the headline, draw conclusions, and then share on social media. As such, people who generate those headlines need to be mindful rather than misleading. This is why Dr. Fauci and others have to emphasize publicly and repeatedly that the vaccines work despite infections in the vaccinated. But if you were an unvaccinated person considering the vaccine, would you be more hesitant if you read a headline saying 75% of those infected were vaccinated? I suspect I would be.

I’ve written before about why you can’t use information only from those who got sick in an outbreak to assess vaccine effectiveness, but here’s another quick illustration. Consider two extremes. Imagine vaccines don’t exist. In the case of an outbreak, pretend 500 people get sick. We could say that 0% of people who got sick were vaccinated and that would be factually true, since there was no vaccine. Now imagine that 100% of a population is vaccinated. In the case of an outbreak (which is possible since no vaccine is 100% effective), 50 people get sick. We could say that 100% of people who got sick were vaccinated, and that also would be factually true. However, that statement does not mean that the vaccine doesn’t work. The point here is that focusing only on the percent of infected people who were vaccinated misses the fact that the overwhelming majority (read: a super higher percentage) of people who were vaccinated did not get sick.

Let’s take another example which shows even more starkly why these headlines are misleading, and potentially dangerous. Imagine I conducted a study of death in motor vehicle accidents, and I wanted to know if seatbelts were protective. (The data below are completely made up and only offered to illustrate the point.) The data are presented in what epidemiologists call a 2x2 table as it cross tabulates data by the factor we want to determine the protectiveness of (seat belt use, yes/no) and the outcome (death yes/no). Now suppose we had data on 100,000 motor vehicle accidents that looked like this:

From the table, we see that the chances of dying in a motor vehicle accident are much less if the person is wearing a seat belt than if they are not. In fact, 10 times less (10% without, 1% with a seat belt). Leaving aside ways in which bias can creep into observational studies like this, if I looked at this data, I’d conclude that wearing a seat belt is a smart idea and I’d buckle up.

But imagine a reporter looked at this same data and wrote the headline, “65% of those who die in a motor vehicle accident were wearing a seat belt.” This is a factually true statement — there were 950 deaths among those wearing a seat belt out of 1450 total deaths — but it is also misleading and invites misinterpretation. The headline ignores the huge denominator among seat belt wearers (95,000) and the relatively smaller denominator among the non-seat belt wearers (5000), and the fact that people wearing a seat belt were 10 times more likely to survive than those not wearing one. Hence, a quick glance at this headline could be misinterpreted to mean seat belts are harmful rather than protective.

Once you saw the full data, you’d probably be pretty frustrated about the misleading headline. The headline could be changed to “65% of those who die in a motor vehicle accident were wearing a seat belt, but experts emphasize that seat belts are still effective” but chances are that you might not believe those experts or even remember the second part of the headline.

Reporting only the percentage of people vaccinated among those who were infected in an outbreak follows a similar pattern. Just replace “death” with “infected with SARS-CoV-2” and “seat belt” with “vaccinated” in the example above.

In the Barnstable case, to know if the vaccine is protecting people, we need to know the following: Among every person who was in Barnstable County during the outbreak, how many were vaccinated and how many were not (or perhaps we limit to a smaller geographic area where the outbreak occurred). We could then calculate the percent of people infected in the vaccinated group and the percent of people infected in the unvaccinated group and compare them to see how well the vaccine was or was not working. Even more usefully, we could calculate the same percentages for hospitalizations and deaths, which would offer a real insight into the protectiveness of the vaccines.

So why aren’t the headlines reporting this information, which could provide an accurate measure of how protective vaccines are against infections and, more importantly, severe disease and death? Simple. Because that data is harder to find and takes time and resources to collect and careful study to analyze. We’d need to estimate the size of the whole population, including the number of visitors in the area at the time of outbreak, find them and ask them if and when they got the vaccine. That’s hard to do quickly.

Now, there is some useful information for the public in the Barnstable County outbreak headlines — namely that a) there were very few hospitalizations in the vaccinated group and b) even among those who are vaccinated it is still possible to get infected and therefore are likely able to spread infection and that the delta variant is highly infectious. As such, we should continue to take precautions like masking in indoor public spaces. But if that was the main message, then the headline would likely reflect that. And they don’t. So when you see a headline about the percent of an infected group that were vaccinated, wait for the full story before drawing any conclusions about whether or not the vaccine works.