Dead Wrong (aka The Numbers Behind The Headlines)

This is part of the Class of COVID-19 series. To read more, click here.

February 2020. Everyone is talking about South Korea. Parasite has just swept the field at the Oscars. Trump is characteristically dismissive on the matter.

March 2020. Everyone is talking about South Korea. COVID-19 has just swept the world. Trump is characteristically dismissive on the matter.

In both cases, South Korea could be forgiven for not really giving a fuck. Not only did they become the stand-out success story of the Oscars, but they have since become the stand-out success story of the current pandemic.

NEON, the film studio behind the subtitled Parasite, respond to Trump’s insults

The US, just three weeks after Trump promised “…packed churches all over the country” instead spent Easter weekend surpassing Italy as the global leader in deaths from COVID-19 (it does not take a stable genius to figure that the two might be linked).

South Korea, meanwhile, having learnt hard lessons from the SARS and MERS pandemics, implemented a swift and decisive response to its first few cases. As a result, it was able to not only flatten the curve, but also to secure one of the world’s lowest case fatality rates.

But what do we mean when we talk about fatality rate? Italy has consistently made headlines for hitting close to 13%. The UK is not far behind. Iran and Turkey, the two least developed countries in the top 10 for confirmed cases, have case fatality rates of 6.2% and 2.1% respectively. The US is down at 4%. Are outcomes really that much worse in the UK?

COVID-19 does not affect everyone equally, and there are plenty of social and demographic reasons that contribute to the difference in mortality rates. However, as with any piece of analysis, we must first understand what we are measuring, before we look into the why.

Credit: Der Spiegel

What you think the headline means vs what it actually means

When people talk about mortality rates, generally what they care about is:

“Of everyone who gets this disease, what proportion will die from it?”

What the newspapers are reporting on, however, is:

“Of everyone who has tested positive for the disease, what proportion have died?”

There are three fundamental differences:

• What does it mean to get the disease?
• When are they going to die?
• What even is death?

If a tree develops a persistent cough in the forest

As covered in last week’s post, not all reporting is created equal. South Korea has tested widely: in doing so, it has identified cases in people who are asymptomatic or who have only mild symptoms. These people are inherently unlikely to die from the virus. They have therefore increased the denominator — people who have tested positive for the disease — without increasing the numerator — people who die from the disease.

Drive-through testing in Daegu, South Korea

In Italy and the UK, testing has largely been restricted to people who are presenting more severely. This has meant a smaller number of confirmed cases, but these confirmed cases are more likely to die. It is highly likely that there are a number of undiagnosed cases who will go on to make full recoveries, which are not represented.

By only looking at everyone who has tested positive for the disease — the case fatality rate — rather than everyone who has the disease — the infection fatality rate — large variances can appear between countries, even when they have the same success rate in treating the virus.

No Time to Die

One apparently concerning trend is that most countries are showing a steady increase in the case fatality rate. Are healthcare systems around the world collectively falling apart? Are we managing to go backwards in how we are treating the virus?

The underlying issue — once again — comes down to how we are measuring.

COVID-19, when it kills, does not do so instantly. A study by the Chinese National Health Commission showed that, in fatal cases of the disease, the median time from first symptom to death was 14 days. The case fatality rate looks at the ratio between everyone who has been confirmed as having the virus, and everyone who has died. It does not take into account those who have been confirmed as having the virus, but who have not yet died.

Suppose you have a disease that kills 1 in 10 people that it infects. Most would consider the mortality rate to be 10%.

Now suppose that this disease takes 5 days to kill. If 100 people are infected on Day 1, then the mortality rate for the first 4 days will be reported as 0%: 100 cases have been confirmed, 0 have died. It is not until Day 5–100 cases have been confirmed, and 10 have died — that the mortality rate jumps up to 10%.

A more insightful metric might be:

“Of all confirmed cases who started showing symptoms more than X days ago, how many have died?”

This is not a perfect metric. Not everyone who will die from the disease will do so at the same point after infection. Not everyone will become a confirmed case at the same point after infection. However, it still gives a more realistic view of what is happening.

By looking at the ratio of current total deaths to confirmed cases as at 7 days ago, you get a steadier mortality rate — rather than one which has increased 6x in a month

Definition of dead

Even when it comes to the ultimate full stop, countries are not in agreement.

Italy and South Korea count everyone who has died who has also tested positive for the disease, regardless of whether COVID-19 caused the death. This may be vastly inflating the mortality rates: the Italian Ministry for Health, reported that only 12% of death certificates showed that death was directly due to coronavirus — the remaining 88% had at least one pre-morbidity.

The UK goes even further: anyone who has even had COVID-19 mentioned on their death certificate — whether they were tested, or simply a suspected case — is included. At a time when coronavirus is on everyone’s mind, it is easy to see how doctors are more likely to attribute anyone with COVID-like symptoms to the disease. When you are a heavily PPE-d hammer, everything is a nail.

This phenomenon happened in the swine flu epidemic in 2009: during the epidemic, case fatality rates were estimated to be between 0.1%-5%. Post-fact analysis of medical notes to ascertain cause of death revealed that the actual figure was 0.02%.

On the other hand, poor testing can lead to an under-reporting. In the US, for instance — rapidly overtaking Italy as the country you really don’t want to be in right now — there is an increasing number of complaints from people whose relatives have died from suspected coronavirus, who were unable to be tested. When you don’t have enough testing kits to test the living, you’re not going to waste any on the dead. In this instance, it is likely that the confirmed deaths due to COVID-19 has been significantly understated. Many countries — the UK included — also only count deaths that occur in hospitals in their daily reporting: deaths in care homes are often added in at a later date. These deaths typically increase the total count by over 50%.

Know thy metric

Now, more than ever, it is important to understand the metrics behind the headlines. At a time when it is near impossible to avoid the news, being able to understand what is actually being measured — rather than just what you expect is being measured — will help to make sense of why the numbers are what they are. Knowledge is power, but right now it is also security, reassurance, and some small semblance of control.

Photo by Jon Tyson on Unsplash

This is part of the Class of COVID-19 series. To read more, click here.
For other data distractions, visit @thecolourofdata.