Coronavirus: two easy actions to launch now

It is now proven that we are not looking at the right numbers. We urgently need to take two easy actions that will reduce uncertainty and help close the knowledge gap.

We don’t really know how many people have been infected by Covid-19. We don’t really know how many people died. In a previous article, Coronavirus: the key numbers we must find out, I explained why official pandemic metrics are massively off-track, in all countries, and proposed two concrete actions to help fix this:

1. A call for governments to open-source their raw mortality data, in real-time, with maximum geographic and demographic granularity.
2. Large-scale randomized self-diagnosis surveys, to gain unbiased insights on the true dynamics of the pandemic and the local effectiveness of confinement measures.

Right now our core knowledge of the Covid-19 pandemic is distorted by our inability to test all patients and account for all deaths. As a consequence:

• We are underestimating the pandemic. This is becoming a major public trust issue for democracies. Governments must act quickly or the political cost will be very high.
• Our clinical knowledge is biased. By design, it is based on the patients that doctors have been able to follow. But many patients never reach hospitals and die at home. What if they have different profiles, different demographics, different risk factors, different symptoms? We wouldn’t even notice.
• We are navigating in the dark. Lockdown measures are deployed. They look necessary, but their social and economic costs are abysmal. Will they be sufficient? It’s not even clear. When will they end? Nobody knows. What will come after lockdown? We have no idea.

The two proposed actions won’t solve all problems but they will bring much-needed clarity. They are easy to deploy. Their cost is minimal. They will not defocus scarce medical resources. In other words, these actions are no-brainers.

The underlying ideas are neither new nor eccentric. At the core, this is basic science and common sense, and many people are coming up with similar ideas. Comparable initiatives have emerged in the past few days, some independently, others in reaction to my article. Scientists and engineers are launching open-source and open-science collaborative projects. A growing number of informal groups and non-profit organizations are calling for governments to be more transparent and launch smarter data collection strategies.

I am certain that people will come up with ideas that are smarter, better, deeper.

But we cannot afford to wait. We are facing the biggest health crisis in a century and, at the same time, the biggest economic crisis in a century. The level of uncertainty on the core metrics of the pandemic is creating the perfect conditions for conspiracy theories, unhealthy politicization and bad public decision-making.

There is no reason to delay basic practical actions that can bring critically needed clarity.

In the past few days, new data and new analysis have emerged that further strengthen the case for the two proposed actions. This is what this article is about. It also complements my initial post by discussing some practical implementation aspects.

1. Early insights from raw mortality data

On March 26, Claudio Cancelli and Luca Foresti published an analysis of death records from several municipalities in Lombardy. Cancelli is the mayor of Nembro, a city with 11,500 inhabitants. Here is the analysis for Nembro:

Blue: expected deaths based on prior years. Green: official Covid-19 deaths. Red: actual 2020 death record.

Without Covid-19, you’d expect to see 35 deaths in Nembro in this period (based on 2015–2019 averages.) Adding the 31 deaths officially attributed to Covid-19, you’d expect a total of 66. Yet Nembro’s death records shows that 158 people died in the period, an excess of 125 deaths compared to prior years.

The real death toll in Nembro is 4x bigger than the official count.

The raw data isn’t publicly available to cross-check the computations. But the described methodology is sound. If confirmed, the numbers compiled by Claudio Cancelli and Luca Foresti will hold valuable lessons:

1. The excess mortality represents 1% of Nembro’s population. If every resident of Nembro had been infected, the Case Fatality Rate (ratio of deaths to cases) would be 1%. However epidemics flame out once a big enough fraction of the population has acquired immunity (herd immunity effect) and not everyone was infected. While some excess deaths could be indirect side-effects (disruption of health services, lockdown), the shape of the curve suggests that most deaths are caused by the virus itself. In short, the CFR in Nembro is credibly above 1%, possibly around 1.5%.
2. Where the pandemic is uncontrolled, it quickly becomes the leading cause of death. As explained in my initial article, the incrementality approach (assessing the death toll by comparing raw death counts with prior years) can only work if the signal-to-noise ratio is good. It was predicted to be the case in Lombardy: these numbers bring a striking validation. Data confirms that incrementality is the simplest and most robust method for measuring the pandemic net death toll in the regions with high virus circulation.
3. In Lombardy, the vast majority of deaths are below the radar. The 4x factor measured by Cancelli and Foresti is shockingly high. We will need more data points to see whether Lombardy is an outlier or the pattern repeats elsewhere. The issue is bigger that just a shocking error in official numbers: it could point to massive biases in our clinical knowledge. Why would 3 out of 4 deaths occur outside hospitals? Because of healthcare collapse? Or because the symptoms are different, possibly more abrupt? We don’t know.

This new data strengthens the case for governments to open-source their raw mortality data, in real-time, with maximum geographic and demographic granularity. This is made necessary by:

• A public trust emergency. No-one trusts the official counts. People are left with counting the number of obituary pages in local newspapers. Governments and municipalities must step up now or they risk losing credibility.
• A public health emergency. We don’t know what we don’t know. If there is a substantial gap between officials death counts and real death tolls, we need to understand where this excess death toll lies. What are the demographics? How is is distributed geographically? Is it associated with the confinement or the virus itself? Are we missing something about the disease? Is there a gap in our healthcare response? The raw data will require sophisticated analysis, but thousands of scientists are ready to help. Open-source projects have already been launched to collaboratively study the data, as soon as it is released.

Not all countries are equally equipped to gather and share raw data on mortality. In some places it might be available at national level, in others at local level only. It may be digitized or paper-based. Regulatory and privacy issues might put restrictions on what can be publicly shared. There are some challenges, but this shouldn’t be an excuse for not acting.

Maximal transparency is urgently required. It will help authorities rebuild much-needed public trust.

France is an example of a country where raw death records are centralized and made available to the general public with maximal granularity: at individual level, with full name, place and date of birth, place and date of death.

This dataset is a key asset that will help scientists across the globe better understand the pandemic. There is only one catch: data is released on a monthly basis, the next release being due in 10–15 days.

In response to the pandemic, INSEE (the French statistical agency that maintains the dataset) did start publishing weekly summaries from March 27th. Here is what their data shows for Haut-Rhin, the most impacted department in France:

As it was to be expected, the signal is sharp. From March 16th, observed deaths have more than doubled compared to prior years: the pandemic has become the leading cause of death in Haut-Rhin. Is the increase bigger than official Covid-19 death counts? Probably, but it is hard to robustly estimate the gap with so few data points. We have to wait for the next weekly update.

[Apr 6 EDIT: the new weekly update released after my post appeared shows that the increase is at least 2x bigger than the number of officially recorded Covid-19 deaths. The majority of deaths are officially below the radar. Who are we missing, in which demographics, and are we missing them because of lack of hospital resources or because they have atypical symptoms? These are urgent public health questions that further strengthen the case presented here.]

INSEE should be thanked for adjusting to the crisis, but it would be great if they could go one step further and release the incremental lines of their master database on a day-to-day basis (the same cadence that governments view as relevant for communicating their official death counts.) Their weekly update is a summary with no demographic data. What if the gap between the official death count and the observed death is concentrated in specific age groups, for whom we would need to adjust our healthcare response? We could find this in the data. We could find something else that we still have no idea about.

Do we really want to wait until mid-April to find out that we’ve missed something important? Of course not.

Every day matters. The data will require complex analysis, as the pandemic itself is causing unique population shifts (urban residents confining in the countryside, patients transferred between regions to balance health-care resources.) But making it fully public will spark a collaborative science effort and unlock analytical resources several orders of magnitude more powerful than the internal capabilities of government agencies. It will make the whole world faster and smarter.

Raw mortality records are the only place to search for reliable answers on the true death toll of the pandemic.

The current approach of reporting deaths attributed to Covid-19 should continue, but the observed excess of deaths should also be communicated to the public.

Any discrepancy between the two counts should be investigated, as it will hold answers for better understanding the pandemic and its side-effects, improving the public response, and fixing the diagnosis and attribution guidelines.

2. Science is officially in the dark

One of the most prominent epidemiology teams working on Covid-19 is led by Neil Ferguson at Imperial College London. Their publications are trusted by governments across the globe as a key source of policy advice.

In their latest report published this week, the authors note that “case data are highly unrepresentative of the incidence of infections due to underreporting as well as systematic and country-specific changes in testing.” Therefore they don’t use case data in their models.

But you can’t build epidemiological models without data. So the authors work under the assumption that official death counts are “of sufficient fidelity to model.”

As we have seen, this is problematic: official death counts are incomplete and biased. But even assuming that these figures were reliable, that would still be a big problem. Mortality is a trailing indicator. There is the incubation period. Then the disease typically starts with mild symptoms. It can take a week or more before symptoms get serious. Then some patients fight for their life for weeks and weeks.

The scientific approach trusted by government to inform their policy is: wait for three weeks and count the bodies.

This health crisis is also a science crisis and a data crisis.

To explain how serious this is, let me translate into plain English the reports’ findings on the effectiveness of lockdown measures.

A key parameter of a pandemic is its propagation coefficient Rt, which measures how many secondary infections result from a single infected patient. The coefficient depends not just on the virus, but also on social aspects such as lifestyle and urbanism.

In Italy, at the outbreak of the epidemic, the report estimates that R0 (the initial value of Rt) was somewhere between 3 and 4. As it is the scientific norm, the estimate comes with a confidence interval, as displayed in the graph below.

Does lockdown really work in Italy? Well, this leading study says that we just don’t know.

With 95% confidence, the initial value was between 3.1 and 4.2, as indicated by the green bands. Most likely, it was between 3.4 and 3.8, as indicated by the dark green 50% confidence subinterval. (Disclaimer: I’m just reading from the graph so my numbers could be slightly off.)

An R0 value around 3.5 is pretty high. It means that, on average, each contaminated person infects 3.5 other people. This explains why the initial spread of the virus was so explosive.

Left unchecked, an epidemic with R0=3.5 will infect more than 70% of the population before it flames out due to herd immunity. In Italy, this means 40 million infections and, assuming a Case Fatality Rate at 1%, 400,000 deaths. But 5% of the 40 million patients would require intensive care, which cannot possibly be delivered on this scale. CFR could jump to 5%, which means 2 million deaths. Just in Italy.

This basic math explains why all government leaders, even the most delusional ones, inevitably come to the realization that drastic action is needed.

On this aspect the report is both conclusive and compelling. It demonstrates beyond doubt that the existing confinement measures succeeded at massively slowing down the progression of the pandemic in Europe. It even includes an estimate for the number of lives saved so far — about 60,000 at this point.

But the report fails to answer the most important question: is lockdown successful at containing the pandemic, or is it just slowing it down? This may sound like a small nuance. It is not.

Have another look at the above graph. The 95% confidence interval for the present value of Rt in Italy ranges from 0.35 to 2.15.

Note the Rt=1 black line that crosses the central dark green region. This black line is very important. The future will be completely different depending on whether Rt>1 or Rt<1.

• If Rt<1, then the worse is over. The infection peak has passed and soon we will see a reduction in daily death counts. Depending on the exact value of Rt, the decline could be fast or slow. But any value below 1 means that the situation is under control. After the decline, the question will be: by how much can we relax the confinement measures to stay below 1 until we find a vaccine or a cure?
• But if Rt>1, the pandemic has slowed but it is not contained. It will get worse, and worse, and worse, day after day, week after week, despite the confinement, until herd immunity is reached or harsher confinement measures are implemented. With Rt=1.6, for example, Italian’s healthcare system would still collapse, with an outcome quite similar to that of Rt=3.5, except that it would happen more slowly and you’d only get 1 million deaths instead of 2 millions. Of course this would call for much harsher confinement measures to be implemented ASAP.

So, is lockdown successful at truly containing the pandemic in Italy? Well, what the report says is that there are 30% chances that it is successful and 70% chances that it isn’t successful. Seriously?

Italy is just an example. There is not a single country in Europe for which the report is able to conclusively assert that the current lockdown is successful at containing the pandemic.

This level of uncertainty is unacceptable.

Not after weeks of lockdown.

The world urgently needs clearer answers.

[Apr 6 EDIT: in the past few days, death counts have gone down in Italy, Spain and France. This is a positive development: lockdown might be working. My point in this post isn’t that lockdown isn’t working or that we cannot eventually find out: it is that leading epidemiologists are incapable of accurately predicting this and we have to wait for several weeks to find out.]

I am not saying that the authors didn’t do their modelling job properly. It is very possible that, with the available data, it was impossible to do better. But in that case the core scientific question should have been:

Which new data should we urgently gather so that the models can finally give conclusive answers?

This is where large-scale randomized self-diagnosis surveys come into play. The idea isn’t a silver bullet, but it is a very simple and very practical way of gathering near-time insights on the true value of Rt, city by city and day after day.

It is cheap, easy to deploy and easy to scale, even in countries with limited resources.

If we want to envision a life beyond lockdown, we urgently need to start building a global knowledge base that will help sort out the actual impact of each individual containment measure in each individual geographic and cultural context. Until we have the capability to test everyone everywhere on a regular basis, I don’t see any other way of gathering unbiased estimates of Rt with sufficient temporal and geographic resolution.

The idea has gained some traction, especially among mathematicians and data-scientists. However it can only be deployed with the adequate level of government support. Some groups have started the necessary advocacy effort, others are working on real-life experiments.

Let me give a bit more details on how the approach works.

First, “randomized” really is the most important word. Achieving proper randomization is the most important success factor in the approach. The purpose of randomization is to eliminate bias.

Many online self-diagnosis tools already exist. These tools are extremely useful but they serve a totally different purpose. Their goal is to help with patient diagnosis and triage, to relieve doctors and call-centers from some of the load. But the data collected by these tools isn’t randomized: only people with symptoms and who are aware of these tools will use them. Because of these two massive selection biases, data from online self-diagnosis tools cannot be use to track the dynamics of the pandemic. If a self-diagnosis website is discussed on TV, it will see a spike in traffic and a spike in detected cases. This will not mean that there is a spike in infections.

By the way, this is the same exact reason why data from actual testing (not self-diagnosis) is biased and cannot be used to accurately model the pandemic.

In practice, here is how a decent level of randomization can be achieved. Pick a massive database, for example that of a large mobile telecom operator, and randomly draw 100,000 people out of it. Push them a simplified self-diagnosis survey by text message, that they can answer by texting back. Repeat every day, each time with a different sample, and log the results. In a nutshell, this is all you have to do. (The survey shouldn’t be nominative and all collected data should be anonymized.)

The goal is NOT to accurately diagnose individual patients. The survey has to be very clear about that: it has no medical significance and patients who are concerned about their symptoms should be redirected to the right resources and services. The survey will NOT provide a direct count of the number of infected people. But it will collect unbiased proxies of the pandemic progression across space and time.

This nuance is essential as it influences questionnaire design. Because online self-diagnosis tools aim at delivering accurate diagnosis at individual level, they typically ask many questions. This is OK because people who use these tools are worried about their symptoms and willing to spend some time sharing them. By contrast, randomized surveys will engage many people without any symptoms and who aren’t interested in responding. The response rate is an important success factor as it will impact the quality of the randomization. If you ask too many questions (or questions that are hard to answer) the response rate will drop, which will create a bias (willingness to answer boring questions isn’t equally distributed across the population.) Randomized surveys should aim at maximizing response rate at the expense of individual accuracy. In particular, the questionnaire should be really short and simple.

This is why we should consider ourselves lucky that, besides the typical flu-like symptoms, many patients with Covid-19 experience anosmia — the scientific name for losing the ability to smell — a symptom that is weird, memorable and specific. As explained in my initial article, not all patients with Covid-19 will experience anosmia, but where the pandemic is active it is likely that the majority of patients with a recently declared anosmia are infected by Covid-19.

At this point we do not know the prevalence of self-diagnosed anosmia among patients infected with Covid-19. We do not know how it varies across demographics and cultural backgrounds, and if it depends on virus strains. But there are so many things we don’t know, starting with the actual number of people infected with the virus, that this uncertainty shouldn’t stop us. We have to start somewhere.

In its simplest expression, a randomized survey could consist of a single question: “In the recent weeks, have you experienced any loss or reduction of your ability to smell? Please reply: 1 if you are currently experiencing this, 2 if this happened but you have now recovered your smell, 3 if you haven’t noticed any change.”

Follow-up questions about other symptoms could then be addressed, but their relevance and usefulness will decrease once response rates drop too much.

How do we analyze the survey from the data? First, under the assumption that the prevalence of anosmia among Covid-19 patient is constant, we can use the survey to directly estimate Rt, city by city, week after week. This is the most obvious benefit of the survey and it does not need any sophisticated analysis.

Once we’re able to do randomized testing of the population, we will be able to calibrate the survey: we will know that if x% of respondents replied that they currently experience anosmia, it means that y% of the population have recently been infected by Covid-19. Randomized testing is the gold standard to track the pandemic. But it requires two things that are not readily available: 1/ the willingness to divert some of the existing testing capability toward randomly selected people with no symptoms, 2/ a practical way of physically testing a large cross-section of the population while it is supposed to be under lockdown. Randomized surveys are cheap uncalibrated alternatives to randomized testing.

Measuring Rt is the trillion dollar question and randomized surveys can provide unique insights.

It may even be possible to self-calibrate randomized surveys in the absence of randomized testing. As the pandemic progresses in highly-impacted regions, the herd immunity factor will start kicking in and, for the same value of R0, the rate of new infections will start to drop. By using multiple data points and looking at the shape of the curve, we may be able to recover a good approximation of the missing calibration factor.

This is why it is so important to start data collection without delay. If we had more properly recorded the trajectory of the pandemic metrics in Lombardy, where herd immunity effects have started kicking in, the whole world would be better equipped to find the right answers.

The saddest thought I have about our data collection gap is this: if we have failed to properly record how fast the pandemic was progressing before lockdowns started, does it mean that we have already lost our memory of what life was before the lockdown? Will we have to relearn everything from scratch, the hard way? Let us start gathering data from the places that are still open, before it is too late.

Our ignorance, in itself, is a key factor of this crisis

We are facing the biggest health crisis in a century and, at the same time, the biggest economic crisis in a century.

Tens of thousands of people have died, not just because of the virus, but because governments haven’t been able to anticipate and contain the crisis early on.

Many more will die, not just because of the virus, but because resources will continue to be misallocated and supply chains will continue to break at unexpected points.

Dozens of millions of people have lost their jobs, not just because the economy has stopped, but because business leaders have no idea when it will start again.

The virus is dangerous but the scale of the crisis isn’t about the virus itself.

It is about our repeated inability to make the right decisions, because we don’t understand, because we don’t know.

It is about our ignorance.

Until we figure out the basic parameters of this pandemic, until we understand the impact of our protective actions, there is simply no way out.