Coronavirus Shutdown Effectiveness — Visualized (Part 2)

Comparing 50 US States, Spain, Italy, France, UK and Sweden

Times Square stands mostly empty as much of the city is void of cars and pedestrians. Getty Images.

4/26/20 Update: After Medium temporarily censored this post, the analysis was featured in the Wall Street Journal. (link)

This is a follow up to my April 12th Let’s Visualize State-By-State Shutdown Effectiveness article. Although it’s only been a week since that post, a lot has changed and we have a lot more data.

The main element I am analyzing here is the effectiveness, or at least the correlation, between how quickly a State ordered Stay-At-Home, or business closures, and the COVID-19 deaths it accumulated in the 3 weeks after presence of coronavirus. This is important because it can help inform how quickly we lift lock downs now as well as how quickly we should consider enacting them in the future.

Intuition may tell you that the faster a State orders a shut down, the fewer deaths it would see, but that is surprisingly not reflected in the correlation analysis.

A lack of correlation between speed to shutdown and deaths implies we may have overestimated the need to shut down and underestimated other social distancing measures.

The updates to the analysis are:

1. Now using reported deaths instead of IHME death projections.
2. Added other western countries for comparison; Italy, Spain, France, United Kingdom, and Sweden, in addition to the 50 US States.
3. Time is normalized from when 1 death per million residents is reached

You can jump down to the results if you like, but first I’ll explain the methodology.

Normalizing to population is important to get an honest picture

There has been some poor analysis out there failing to normalize basic data such as population size. Below is a chart from one of the world’s most elite consulting firms in a report meant for industry leaders. In the chart they claim that national trajectories “diverge based on demographics and measures taken” and then you see the United States running wildly at a higher rate than all the other countries. Seems bad for the US, right?

But this is nearly worthless because it doesn’t normalize to population size. Since the US has 5.5x more people than Italy, then of course you would expect it to have more cases. Everything has to be normalized to population size to make sense.

The slide above claims the United States had “the steepest rate of growth” but that’s primarily because it has the largest population of all the countries listed.

When normalized for population, the United States is much more similar to other western nations. Below are the same countries normalized to population size, which gives a completely different picture.

I also added Sweden in there because Sweden has not shut down and the world is watching to see how that strategy will play out. It’s therefore important to include Sweden as a reference point to maintain intellectual honesty in all such comparisons.

The problem with the above graph however is that each country started picking up cases at slightly different times. So while the steepness of the curves can be compared for rates, the final outcomes cannot. For that you also have to normalize time not based on a calendar date but based on a milestone in cases or deaths.

Normalizing to a starting milestone — 1 Death per Million

To get all the data on the same timeframe, I’ve normalized the starting milestone to be when a State reached 1 death per million residents.

This is intended to mark a moment in time in which we can say with a high degree of confidence that the virus was definitively present throughout a State.

One death today means that at least 100 people had it ~16 days prior, which means that as it grows exponentially during those 16 days, there would be many hundreds if not thousands of infected at the milestone date, per million.

Furthermore, I am tracking 21 days of mortality data because that would capture at least some meaningful effects of a shutdown in the mortality cycle. This data did not exist a week ago and each week that goes by we can get a better and better understanding with more data.

Now let’s look at the results

There’s a lot going on in one graph, so let’s first walk through it

First, here is a general illustration of the chart you’re about to see…

The vertical Y-axis represents the State’s total accumulated Covid deaths 21 days since the start milestone; that’s the 1 death per million start of the clock.

The horizontal X-axis reflects the number of days a State took to shut down relative to the same milestone of 1 death per million. Speedy States are to the far left, slow States are to the far right. So for example, NY passed 1 death per million on March 20th but didn’t shut down until March 22nd, so it goes on the “2 days” column on the X-axis for speed to shutdown because it shut down 2 days after the mortality milestone.

On the other hand, Delaware reached 1 death per million on March 26th, but it shut down on March 24th, so it gets a “-2 days” on the x-axis for shutting down two days before reaching the mortality milestone.

NY therefore ordered its shut down 4 calendar days before Delaware, but since we are normalizing the shutdown to a mortality presence mark, we can see that Delaware actually shut down 4 days before NY relative to the mortality milestone of 1 death per million. While this is a bit complicated, it’s key to getting to the heart of the true correlation.

I hope that makes sense. Ok, let’s keep going. Here are the results:

In places where business closure were issued before or instead of stay-at-home, date of business closures were used.

So what do we see?

1. There is very little correlation between speed of shutdown and total death after 21 days. In fact, there is a slight negative correlation of -0.05 R², but that is basically an insignificant correlation.
2. You have some States that shut down well after others States (relative to reaching the 1 death per million mark), but they are all over the map on outcomes. This is another way of saying there is little correlation.
3. Spain, Italy, UK, and France are in a similar category to some of the harder hit US States but not nearly as bad as NY. Sweden had fewer deaths in the first 21 days compared to the UK, Italy, and Spain, even though it has kept its schools, malls, and bars open. More on this later.

Some of the smaller States get a little murky, for example Oklahoma closed down non-essential businesses on April 1st but hasn’t actually issued a Stay-at-Home order according to covid19.healthdata.org. So here is the same graph but without the smaller US states which remained open. The results are similar, and in fact the correlation becomes even weaker with an R² of 1%.

In places where business closure were issued before or instead of stay-at-home, date of business closures were used.

We need to be careful not to infer causality, but what can we conclude?

There is no general correlation between how fast a State shut down and how many people died in the first 3 weeks following an early mortality milestone.

So what factors may explain the different outcomes?

If speed to shutdown doesn’t cause such a large difference in outcomes, what is it?

Social distancing does work, but we need to get smarter on exactly which social distancing actually contributes the most benefit.

I would next want to look at and control for the following 8 variables:

1. Population density
2. Extent of public transportation use
3. High transmission events such as conferences and sporting events
4. Amount of travel from to/from coronavirus hot spots
5. Level of voluntary social distancing behavior
6. Hygiene practices such as washing hands, cleaning, masks, etc.
7. Health and age demographics
8. Standards used to classify COVID vs. non-COVID deaths

The above variables, and others, are critical for us to better assess risk factors for a pandemic as well as taking measured steps for its mitigation.

Coronavirus transmits through close contact. Nobody should be doubting whether social distancing works as a general concept. Given the extraordinary economic and mental damage that shut downs cause, we ought to more carefully consider whether shutdowns are effective compared to all the other measures we can take.

Bonus: Population density seemed the most interesting so I ran some numbers on that…

X and Y axis on a logarithmic scale. Source: covidtracking.com

The correlation R² here is 0.36 on a linear best fit, and 0.41 on an exponential best fit, which is more meaningful than random but not extremely strong. A better chart for population density would probably be to compare metro areas rather than States. Nonetheless, we do see that population density is a correlating factor to increased deaths, but that should not be surprising for a virus that is transmitted person to person and benefits from close contact.

The above is on a logarithmic scale otherwise most states get bunched up in the corner, but it’s still interesting to look at it on a linear scale:

Source: covidtracking.com

There seems to be two patterns that emerge. There’s the France, Italy, UK, CT/MA/NJ story, and then there’s the Sweden, LA/MI, Spain, and NY story.

Although it’s not fair to group them all in the same category because Sweden and France are at 77 and 69 deaths per million after 3 weeks, while NJ and NY are 2–5 times higher.

It’s also important to consider that Sweden never shut down while all the other places I just listed did, which again reinforces that legal shut downs may only play a small role in the slowing down of the spread when compared to other factors such limiting travel to/from hot spots, degree of public transportation use, washing hands, cancelling high transmission events such as conferences, and whether people are staying home when sick or exposed.

So while speed to shut down does not show meaningful correlation, population density has some correlation and we should study variables 1 through 8 listed above to see which ones have the biggest impact.

Conclusion: We need to better understand the cost/benefit of different kinds of social distancing

This is important as our nation begins a marathon of mitigation against coronavirus, and in this struggle we need to better understand the benefits and the trade offs of all the measures we can take. Allowing the purchase of jewelry and lottery tickets but not gardening supplies is an example of how misguided some of our government decisions have been.

Next time somebody seems to panic and wants a shut down, we should remember that there is likely a long list of less costly measures that may be even more effective before we jump to the most drastic extreme.