The needles in the haystack

Introduction

I am interested in the testing for Covid-19. This is the part of the Covid-19 complexity that is similar to problems I know very well. I am an observational astronomer with twenty years of research experience. In all my work, I am dealing with small samples which are extracted from much larger samples. Finding these small samples, keeping track of the way they have been selected, and understanding their limitations is essential in every analysis I am doing. This is the needles in the haystack problem. The needles: stars with particular properties or people with specific illnesses. The haystack: a large population of stars or people. The structure of the problem is similar.

A lot of competent people analyse Covid-19 case numbers for various countries and regions and publish them every day. I like the plots that my colleague Andreas Burkert prepares, but that is more an aesthetic preference. All these plots show confirmed infections. The only way to confirm an infection is with a test. Different countries may have very different strategies for selecting the people who are tested. That alone could explain some of the differences between countries. Testing is only one aspect of figuring out how the infection is spreading, in addition to that other strategies are used which are more indirect, but nonetheless well established. You can’t fight something without knowing where it is.

Available data

I don’t have access to any data that others do not have. I’m focusing on the UK and I am using the official numbers published every day by the Department of Health and Social Care. The information is very limited. I do not know how the people who are tested have been selected exactly (the ‘selection function’, to give it a name). I can find out the geographical location of the confirmed cases, but not for the tests. I do not know the age distribution or any other characteristic of the people who are tested. I do not know what fraction of tests goes wrong and gives false positive or false negatives.

Some general rules

When dealing with this level of uncertainty and with a rapidly changing situation, it seems important to me to focus on information that is meaningful and to filter out the noise, as much as possible. That means, among other things: 1) Don’t pay too much attention to daily numbers which will be affected by all kinds of fluctuations. Each fluke outlier causes 24 hours of worrying. Instead, focus on trends that continue over several days. 2) Focus on official numbers, unless there are obvious reasons to doubt those. 3) Limit speculations to a minimum and leave it to people who are trained to speculate in this specific field.

The figure

In red the number of tests done per day which scattered around 1500 for the first 10 days of March, then increased significantly over three days. What the dropoff on the last day signifies is not clear. In yellow the number of tests per million people, which is currently at 600. This is a cumulative number that will grow over time. It can easily be compared with other countries. It is a a lot lower than South Korea (about 5000 on March 13th), lower than in Italy (1400 on March 12th), but probably broadly comparable to other European countries. The blue line is the number of confirmed cases in the UK, for reference. The full dataset with some additional columns lives in a Google Spreadsheet.

Three strategies to find the needles

In this type of problem, you are left with imperfect strategies. The ideal way of finding the needles is to go through the entire haystack, which means: testing every single person. This is entirely impractical — no country will come close to that. The next best thing is to test a random subsample, ideally one that has characteristics similar to the general population, and then to extrapolate to the general population. Even that is impractical to do on a large scale right now.

For comparison: Polling before elections is such an attempt to extrapolate from random samples. A thorough poll usually runs over several days and samples thousands of people. Asking one voter in a poll takes minutes. The outcomes are usually stable from one day to the next. From previous elections there are many ways to figure out how to construct the sample and how to extrapolate. For Covid-19, we do not have that kind of information. The situation is changing every day. The geography of the infections is not the same from one day to the next. A test takes several hours to process, and requires specific equipment that is not available everywhere. Currently the test capacity in the UK is at least 5000 per day (as evidenced by the number on March 14th). Expanding to 10000 per day seems to be the plan (at least it was on March 11th). Not enough for random sampling.

What remains is the third best option: Limiting the sample drastically to the people who have a much higher chance of being infected, i.e. those with some defined set of symptoms or those who have been in close contact with people with symptoms. This necessarily means not getting a full picture, but there is no other way.

Ideally one would like to expand the tests gradually, away from the most likely sources of infection. But this or any type of random sampling would take tests away from people who show symptoms and need them urgently. Even administering tests needs resources that are perhaps more urgently needed elsewhere. Ramping up testing will have consequences that are difficult to assess from the outside. There is no free lunch. More on the difficulties of expanding tests in this illuminating Twitter thread.

In essence, at this point you are not probing the whole haystack anymore, but small clumps of hay that can be suspected, based on some evidence, to have most of the needles in it.

An excursion to brown dwarfs

This section is mostly a detour for distraction, and can be skipped without loss of information. Distractions are important. Brown dwarfs are failed stars. I am trying to figure out how they form. When we search for baby brown dwarfs, we have to make hard choices because the ‘test’ is expensive and takes a lot of time on a big telescope. So, we only look in certain places. We only test objects with certain colours. And that’s still too many, so, we only test some of those. I lose sleep at every filtering step and worry that I might miss the ones that I really should not miss. Every conclusion about the numbers, the spatial distribution, or the characteristics of the brown dwarfs depends on knowing what I throw away. The worrying takes the form of additional tests, simulations, and modeling. But it’s the only way. It happens in every subfield in astronomy I am familiar with. I hope it happens as well as part of the Covid-19 strategies. End of distraction.

The selection function

Who are the people who are currently being tested? It is certainly not a representative sample. It will necessarily be a very small part of the haystack. The latest information from Saturday, March 14th: Tests are administered to people admitted to hospitals with respiratory illnesses and to people in residential or care settings where an outbreak has occurred. If that’s the case, all asymptomatic cases and all mild cases will not be registered, that means, the majority. Since late February the NHS is operating some drive-through testing facilities for people who may carry the virus. Since people in hospitals don’t drive, this is probably unrelated to hospital admissions.

Right now only a few percent of the people who are tested are positive for Covid-19, cumulatively 3.4%, and 9.2% for the testing done on March 15th. That was the highest number by far. In absolute numbers, it is a few hundred per day so far. Many of the the remaining people will have some other kind of respiratory illness with flu-like symptoms, or perhaps have been directly in contact with confirmed infections.

How many people actually have flu-like symptoms in a normal year, without Covid-19? According to the official flu surveys, the number of people who go to see their GP with flu-like symptoms is, in a normal year, in early March, about 5–10 per 100000 per week, which translates to about 500–1000 per day for the whole population of the UK. Only a small fraction of them would have been admitted to a hospital. This number varies per year, but judged by the numbers from January, rates in 2020 are lower than in previous years. For comparison, several thousand people have been tested for Covid-19 every day over the past week. So, the number of tests is high compared to the number of people who in a normal season would have gone to the doctor with flu-like symptoms.

That means: Perhaps the range of people who are tested is broader than the public statements indicate. Maybe the scope of the testing covers a significant number of asymptomatic people. It seems that there are not enough people with serious respiratory illnesses for that many tests. Not yet at least. Enough speculation.

Outlook

A lot of uncertainty, but a few things do seem clear: The testing capacity is ramping up, but this will take time. The number of tests per capita is now in the range comparable to other European countries, as far as we know, although the UK is behind the infection curve by several days compared to mainland Europe. Obviously it would have been better to ramp it earlier and be in a better position now, but there we are. We have conflicting information about the selection function, and more clarity on this front would be helpful. The number of confirmed Covid-19 cases is soon going to exceed significantly the number or ‘normal’ flu-like illnesses. That means the detection rate in tests will go up over the next week. The selection function may become easier. Most available tests will have to be used for people with clear symptoms. Let’s hope, however, that this will remain a needles in the haystack problem.