Let’s Visualize State-by-State Shutdown Effectiveness on COVID-19
Many are wondering when we should begin to loosen social distancing measures and which ones should we loosen first?
Parents want to let their kids play on park swings and others are looking for a break from sheltering at home. However, none of us want to be responsible for unnecessary deaths and we all want to do our part, so these can be difficult questions to think through and perhaps even uncomfortable to ask.
How effective are State wide shutdowns versus more measured approaches? That is what I will analyze in this post by looking at the data. As a spoiler, there is not much evidence that shutting down, or at least shutting down faster, has had a significant impact on total deaths. How can this be? Let’s walk through the numbers. You can also download my data and analysis spreadsheet here.
First we need a framework to help us think through these choices.
Tomas Pueyo gave a nice framework for how to think about this by laying out the potential effects that different social distancing measures can have on the spread of the virus. The rate of viral spread can be defined by R. If R =2 then for every person who gets the virus, that person will infect 2 more. If we can get R to below 1 that means for each person who gets the virus, they will transmit it to less than one other person. That would be a great position to be in to control the spread.
Below is a summary of possible measures that can reduce R as proposed by Tomas Pueyo in The Hammer and the Dance.
Least to most disruptive/damaging social distancing measures
At the top are the least damaging measures such as testing, hand washing, and checking our body temperature. After that we start to get more draconian such as travel restrictions, closing of sporting events, schools, and most businesses.
To make wise decisions, we need to better understand how effective each measure is in terms of both the damage it causes and benefit it provides.
Statewide Stay-at-Home measures were put in place as an emergency measure
When most statewide stay-at-home measure were put in place during mid to late March, there was a growing fear that we would “end up like Italy” and that we were on the precipice of bodies lining streets across the United States if we didn’t take action. People felt a real sense of panic about an impeding disaster and pulled the emergency brake lever to just stop nearly everything in its tracks.
We had a lot less data back then, so a study of those March decisions can be evaluated at some point in the future. What’s more important is that now that we have some data, and actually a lot more data, what have we learned from taking these measures?
Statewide closures obviously worked, right? It’s not so simple.
Many people point to NY as evidence that statewide closures work. While NY has certainly mitigated unchecked growth of the virus, we can’t yet say for sure how much of that was from cancelling events, or washing more hands, or wearing masks, or staying at home, etc. We may have “common sense” intuition, but analyzing data is usually a superior approach.
The only real way to know how effective shutting down has been is by comparing two similar situations in which one shut down and one did not. Since we don’t have a bizarro parallel universe of New York that didn’t shut down, we can at least compare the actions of all the different States and see what outcomes the different decisions had.
Let’s compare speed to shut down to expected death rates
So one way to visualize effectiveness of shut downs is to compare how quickly States shut down to the total number of currently expected deaths, normalized to population size of course.
According to Tomas Pueyo:
In this theoretical model that resembles loosely Hubei, waiting one more day creates 40% more cases! — Flattening the Curve
That certainly sounds scary! One day adds 40% more cases? With a situation like that, we should all shut down as quickly as possible.
Let’s see what actually happened…
I graphed the projected total deaths (past and future) using the IHME model versus how long it took a State to shut down from the time it started seeing people dying (when it reached 0.5 deaths per million).
One would expect that the faster a State shut down, the less deaths it would incur, but that’s surprisingly not what we find. There is virtually zero correlation between speed of shut down and expected death totals.
Now there are a lot of States out there with small populations, so one may argue that they may not paint a complete picture. So I went ahead and re-ran the numbers but this time limiting the graph to only the 15 most populous US States:
We again find no meaningful correlation between speed of State wide shutdown versus projected total deaths. How can this be? Well, there may be much more important factors influencing the spread of the virus, including:
- Voluntary behavioral change of residents such as washing hands, staying away from large groups, not touching one’s face, etc.
- Municipal level shut downs such as what happened in Texas. It may not make sense to treat a Dallas the same as a 200 person town.
- More measured closures such as restaurants but not all businesses
- Natural slow of viral spread, perhaps related to weather
- Different health profiles of State residents
Ok, but 0.5 deaths per million to start the clock may be causing noise. What if we use a higher threshold?
Using a lower death number as the relative marker to start the clock allows a bigger window of time and therefore a better range of data to find patterns. However, 0.5 deaths per million is also noisier since the numbers are so low. If we 10x that and graph the same speed-to-closing based on when the State reached 5 deaths per million and compare that to expected deaths, we get a correlation with an R Squared of 0.156.
This is a bit of a stronger correlation than the prior analysis, but is still extremely weak to explain the differences we are seeing in death totals among the States. If one was to take this data at face value, we would have to consider that there may be much important forces at play than a State shutting down even once an infection reaches a meaningful level of penetration.
It’s important to remember that this doesn’t graph the relative date of Stay-At-Home orders. It graphs the relative date of Stay-At-Home orders compared to when the state reached meaningful deaths, defined as 5 deaths per million. This latter comparison is more insightful as to the effects of closing early vs later, or not closing at all as we’ll discuss later in international examples.
This is not to say that shutting down does not reduce the spread of the virus, nor does it say that perhaps the psychological effect of how serious a State projects the problem may also change people’s behavior.
What we can say for now is that how quickly a State shuts down has not meaningfully correlated with the total expected deaths.
State shutdowns do slow down the spread but speed to shut down appears weakly correlated with outcomes
It would be reasonable to assume that State wide shut downs almost certainly does slow down the spread of the virus to some unknown extent. What we don’t know is how effective a State wide shut down is compared to other measures taken such as washing hands, wearing masks, avoiding large events, or letting multiplicities make their own decisions with local restrictions. The above charts are one data point that would at least indicate a very weak correlation between speed of shutting down and saving lives.
Statewide shutdowns costs us trillions of dollars, deprives children of full education, and ruins many lives; could washing hands, wearing masks, and avoiding large events be potentially more effective while a lot less damaging?
A lot is still unknown, but these are exactly the kinds of questions we need to begin to answer so we know when and what social distancing measures we should loosen first, and it should not be taboo to challenge the wisdom of the road we are on.
Speed to closing schools also doesn’t meaningfully correlate with reduction in deaths
Below is a similar analysis but instead of looking at a shutdown date I compared the time lag (or time advance) between shutting down all schools and when a State started recording deaths, defined as reaching 0.5 deaths per million.
Based on the R squared correlation, we would conclude that there is extremely weak (5%) correlation between speed of school shut down to the projected death totals that a State will experience.
This is not to say that shutting down schools or States is the wrong answer, but it is intended to begin a data driven conversation on what measures are worth taking and when.
What about other countries?
Nearly all countries have followed the shut down approach. Some such as South Korea and Singapore were able to implement extremely strong early testing and contact tracing, and for at least a while avoided shutdowns.
Challenging the shutdown dogma has become taboo. “What, do you want millions to die?” might be a response that one gets, but we again need to look at data, which so far experts have continued to be wrong about over and over again.
Let’s take a look for example at Sweden.
If Sweden were a US State it would be one of our 10 most populous, so it’s not exactly a small sample size. Sweden has enacted some of the measures in the hierarchy of social distancing, but it has not gone as far as a complete shut down.
Let me bring back the framework from Tomas Pueyo, because it is such a valuable conceptual reference:
Sweden has taken the less punitive steps at the top of the chart, but has refused to lock down their country. Their playgrounds and schools are still open, as are restaurants and bars.
Sweden’s actions are about encouraging and recommending, not compulsion. Two days after Spain imposed a nationwide lockdown on March 14, Swedish authorities were encouraging people to wash hands and stay at home if sick. On March 24, new rules were introduced to avoid crowding at restaurants. But they very much stayed open.
So did many primary and secondary schools. Gatherings of up to 50 people are still permitted. —CNN, April 10, 2020
So how is Sweden actually doing?
Sweden is a great comparison because their deaths started happening at nearly the exact same time as the United States, so it’s more straightforward to compare.
What we find is that both countries saw a quick rise of daily deaths starting around March 20th, peaking around April 10th and then plateauing, and in Sweden’s case then dropping very quickly — at least so far — more time will tell the rest of the story.
Sweden and the United States are clearly very different countries, with different cultures and different geographies. The United States itself is quite diverse in its internal response to COVID, as we saw earlier in this article.
A key takeaway may be that it is reasonable to challenge the conventional wisdom that shut downs are the leading (or even meaningful) contributor to saving lives when compared to other measures such as washing hands, cancelling large events, measuring one’s temperature, and staying home when exposed to somebody who was sick.
Where to from here?
I started this article by saying that we must ask which social distancing measures we should lift first and when. Our shutdowns do not come for free, with nearly a million people losing their jobs daily, tens of millions of children receiving little to no education, not to mention the unknown mental health damage that we may be unwinding for years.
Given the data we see across the States and with countries like Sweden, it should not be taboo to question the narrative that wholesale shutdowns were, or at the very least still are, the right approach.
We should all rely a little more on data, and less on fear, when considering the next steps for our country.
