On COVID-19 Serological Testing

13 min readApr 29, 2020

(content warning: blood; I have been advised to change the names of the manufacturers. If you contact me, I can provide the original names.)

The last time I went into the office was February 25th. The COVID-19 pandemic was just beginning to unfold in the United States. I had a business trip to King County, Washington (Seattle) that week. While we were in Seattle, the first cases of COVID-19 were being announced in the region. Sometime after returning home, on Saturday, we learned that the outbreak in Seattle had already claimed lives.

This whole experience worried me, and it didn’t feel like anyone was going to save us. I’m not a biologist. I’m a computer person. I like solving problems. I decided sometime by the 3rd week of March that I was going to try to figure out what was going on with us. It didn’t seem like anyone else was going to, within a reasonable amount of time. I knew I was going into this with a strong chance of developing a case of DKE¹.

My partner and I have risk factors. This left us in a state of heightened paranoia. Every cough, or ache could be attributed to COVID-19. We were told not to worry by many — friends, the media, and others. Even though this was the case, my original manager told me to prioritize the health of myself, and my partner. We took this opportunity to begin the process of isolating ourselves on March 3rd, 13 days before the shelter in place warning was put into effect in San Francisco.

My partner and I were pretty sure we had COVID-19. She exhibited the symptoms starting early March, and I began to exhibit them mid-March. We experienced what we would now recognize as hallmarks of the illness — cough, fever, low SpO2, and GI problems. With the addition of my time in King County, I was worried I had brought it to our home, and put both of us at risk. To make things worse, I later learned that someone who had been in my proximity during the business trip had later developed COVID-19.

Unfortunately, since we’re not famous, nor are we a high enough risk category, we couldn’t get PCR based testing. Our doctors informed us that it was unlikely that PCR testing would become available for cases like ours for some time. Not only were we both sick and worried, we wanted to know if this was it. The uncertainty was driving us nuts. Part of us hoped that this was it. That we had been infected, and were immune because we didn’t think reinfection was possible². Based on the data³, it seemed like our viral load would become undetectable by the end of the month. We were put in a place where even if we had immunity to this virus, we wouldn’t know.

Testing

There are two types of tests for SARS-Cov-2 / COVID-19 that I know about.

The first one, that everyone knows about, is PCR. In PCR, one takes a sample, and replicates the RNA of the virus. It doesn’t require much RNA to start with, and that makes it incredibly sensitive. The downside is that there is a narrow window in which it can detect the virus. The detection window is quite limited because at some point your body fights the virus off, and the RNA levels drop to 0. These tests, at the smallest scale, can be run on a ~$600 machine called the MiniPCR⁴, with the addition of a few other pieces of equipment and consumables. Even if we were to obtain the necessary equipment, I would be utterly confused as to the mechanism to run the assay.

The other type of test is a serological antibody test. This looks for the antibodies that are produced by the body’s reaction to SARS-Cov-2. The antibodies take some time to make, so there’s lag between the time that infection occurs, and the test shows up positive. Because these tests listen for the echoes of the infection it remained positive long after symptoms have resolved.

These serological tests come in different forms, the simplest of which is a lateral flow assay¹⁷ with antigens that bond to specific SARS-Cov-2 antibodies that are present in the sample. These are simple tests that could reasonably be performed by non-expert personnel⁵. The FDA has issued guidance allowing these tests to be deployed to healthcare workers and has relaxed the regulations around them (but not completely).

Acquisition

I wanted an antibody test. I knew that our window was going to close sometime in April, where our IgG levels would drop below a detectable concentration. We needed these quickly.

So, where did I turn? Alibaba. At the time, Chinese biotech firms had begun to develop these lateral flow assays and were making them available on Alibaba. This was before the FDA gave guidance to DHS to begin seizing immunological tests. These tests only spent a short while on Alibaba before manufacturers (presumably on Alibaba’s will) removed all mention of them. Our orders were in for a handful of different vendors, each with a lead time that ranged from 1 to 15 days, knowing that our antibody levels could decline below the detectable level before they arrived.

I needed a backup plan, as our time was going to run out. Fortunately, there were well put together lists of other vendors that made tests. Even with this list, most of these companies were large enterprises, and I had no idea how to buy these things. The one thing that kept popping up alongside the phone numbers and email addresses was WeChat. I didn’t have a WeChat until a month ago. A friend who spent time in China gave me the hookup, and I was in line to buy a handful of tests from BR and SLE.

The tests from SLE arrived 3 days later via courier. On the other hand the tests from BR were reported to be in Erlanger, Kentucky. They had been confiscated by the courier and handed over to CBP. In order to retrieve them from CBP, I would have to go through a lengthy process of dealing with HSI⁶ (ICE). Given my background, this seemed like a legal and bureaucratic nightmare.

As a backup plan, I put myself in line to acquire tests from domestic companies, LBP, BRT, and GH.

The results

We acquired lancets, alcohol pads, and a sharps container. As the tests began to arrive, we began to take them. We wanted to take multiple tests from different vendors, as given their origin, we had no idea which ones worked and which ones didn’t — little did I know that this would turn into an interesting problem. Test after test showed the same thing. Negative.

One of many serological tests we took. This indicated that antibodies were not present.

Sensitivity & Specificity

We knew the likelihood of a false result was high with these tests. They could give us a false positive due to the lack of “specificity,” or a false negative due to the lack of “sensitivity.” They advertised numbers like 80% sensitive, and 97% specific. This lead me to believe we had a 20% chance of a false negative, and a 3% change of a false positive. Some test providers tested with known antibodies of other common illnesses, giving me somewhat of a better idea about the specificity problems associated with the test. This left us with the chance of a false negative.

We have tested this kit with positive specimens of parainfluenza virus antibody, influenza A virus antibody, influenza B virus antibody, Chlamydia pneumoniae antibody, Mycoplasma pneumoniae antibody, adenovirus antibody, respiratory syncytial virus antibody, hepatitis B surface antibody, type C Hepatitis virus antibody, Treponema pallidum antibody, human immunodeficiency virus antibody, EB virus antibody, measles virus antibody, cytomegalovirus antibody, enterovirus 71 antibody, mumps virus antibody, HKU1 virus antibody, OC43 virus antibody, NL63 virus antibody, 229E virus antibody, it does not have cross reaction with the above antibodies.
Said the salesperson from “SLE”

Since the body makes antibodies for many different viruses throughout our lifetime (including other types of coronaviruses), manufacturers must try to ensure that their test only returns a positive result for a specific antibody. This idea is considered a test’s “specificity,” which is the probability that a test will correctly identify a person without disease as a negative result.

The lay person may think of sensitivity in terms of “thresholds.” or at what concentration of antibodies will this test turn positive. Instead, the biostatistics interpretation of this number refers to the probability that
when a disease is present (or in this case, antibodies to the
disease), the test will be positive. To determine this, manufacturers take serum that tested positive via RT-PCR and perform their assay on the same sample. Of those samples, the number that returns positive is used to calculate the sensitivity. This calculation approach does not take into account the antibodies in the samples. What I see as particularly problematic, is that the samples are not tested at various titers. Often, when manufacturers have more time, they will stratify their tests by age, and days since the onset of symptoms. Lastly, sometimes they document their tests performance against quantitative measurement. In this rush to produce tests, we lack that information.

IgG levels in serum after infection are not constant. They begin to rise days after infection, and slowly drop after infection⁷. We’re lucky in some ways, because the SARS-Cov-2 IgG antibodies seem to stick around for a while.

The last thing to bring to the surface is the 4 types of results that feed into the above described numbers:

FP: False positive (This means the test lacks specificity, and is giving a positive result for someone who does not have antibodies)
FN: False Negative (This means the tests lacks sensitivity)
TP / TN: True Negative / True Positive (These are good!)

We also care somewhat about accuracy (N-FP-FN) / N, where N is the number of samples that tested positive via PCR.

In my opinion, false positives, at worst, can give someone an artificial sense of security, and lead to them making high-risk decisions. False negatives, on the other hand, lead to us underestimating the number of infected individuals. At the individual level, this matters if one believes that they cannot be reinfected by SARS-Cov-2. At the macro-level, these being incorrect can result in us massively misestimating hospitalization and mortality rates. On top of that, we can further refine the number of TPs, by testing those who test positive via these rudimentary mechanisms with another mechanism, like PRNT or ELISA⁸. Lateral flow assays could act as a screening test. A positive result could be used to funnel people into more sophisticated tests that are more expensive and time consuming, but may have better accuracy.

The Problem

These tests may not be accurate enough to be useful⁹. One specific example is the manufacturer “Zhejiang Orient Gene Biotech Co.” They report the sensitivity of the IgM test at 87.9% with a specificity of 100%. For their IgG test, they report a sensitivity of 97.2% with a specificity of 100%. Academia has tested these kits¹⁰, and has discovered very different numbers.

The results revealed a sensitivity of 69% and 93.1% for IgM and IgG, respectively, based solely on PCR-positivity due to the absence of a serological gold standard. The assay specificities were shown to be 100% for IgM and 99.2% for IgG.

The specificity (FP count) is fairly good. On the other hand, the sensitivity is somewhat problematic.

Why Is This Dissonance Bad?

The FDA has changed their approach to how they are allowing these tests into the United States. The folks at the FDA were incredibly nice and patient with me and explained their new “notification” process. In fact, they responded to me within tens of minutes, as I emailed them into the late evening, PST. They require that a manufacturer shares the Cross-reactivity/Analytical Specificity, Class Specificity, and Clinical Agreement with the FDA. Once that information is shared, the FDA approves it, or asks for more information. It is up to the manufacturer to follow “good practices,” as laid out in the FDA’s guidance¹¹. In addition, the manufacturer has to pay a $5,236 to the FDA to register their facility.

The reason the FDA shifted their strategy to a far more liberal one is because they wanted testing to come online quickly. My fear is that this shift in strategy will cause a backlash, as the media is reporting these tests to be inaccurate. In addition, some of these tests do have poor quality control, and will fail. There have been folks that have volunteered to validate these tests, but they seem to be working very slowly.

“Science”

I generally avoid medical-related science and technology¹². Nonetheless, I took an interest in determining whether or not these tests were faulty. I acquired SARS-Cov-2 NP antibodies¹³, SARS-Cov-2 Spike S1 antibodies¹⁴, and SARS-Cov-2 Spike (S1 & S2) antibodies¹⁵ with the intent of testing these assays to see if they’d respond to these cloned antibodies. Although they weren’t perfect, as they were rabbit polyclonal antibodies and chimeric monoclonal antibodies, they should do a good enough job at triggering the antigen in the assay.

Unfortunately, the S1 & S2 antibodies were delayed due to customs issues for ~5 days. These antibodies only last for a short period at ambient temperature. When I contacted FedEx and asked them if they could do anything, they said no. In these 5 days, the antibodies were destroyed.

My protocol for testing was to take a sample of the antibody solution, micropipette it using Sciologex micropipettes, and dilute it using phosphate buffered saline at room temperature. Unfortunately, the seller did not know the original amount of antibodies in the solution, so all measurements are done in relative. The antibody solution is stored at room temp. I then took 10-20 μL (based on the kit’s testing protocol) of the solution, and put it into the sample area, and added the test kit’s “buffer” (except in the case of the BRT kit, which was a little more involved). Each test was repeated where the dilution was doubled (1:100 -> 1:200, etc..).

I was unable to get the A, BP, and BRT kits to trigger, even at a dilution of 1:20. I emailed both of them to ask what their kits tested. LBP responded saying that they tested for the spike protein, so that may as well be the S2 protein that I did not have access to. On the other hand, BRT responded saying that they checked for N(P) antibodies, which I should have been able to trigger.

+--------------------+----------+-----------------------+
|    Manufacturer    | Antibody | Max Positive Dilution |
+--------------------+----------+-----------------------+
| Elab               | NP       | 1:1600                |
| LBP                | NP       | 1:100                 |
| Eubo (Orient Gene) | S1       | 1:400                 |
| GH                 | S1       | 1:800                 |
| Xian Henrikang     | S1       | 1:800                 |
| BRT                | N/A      | N/A                   |
| LBP                | N/A      | N/A                   |
| A                  | N/A      | N/A                   | 
+--------------------+----------+-----------------------+

The gradient of results from 1:200 to 1:3200

Conspiracy

Several of these devices looked similar. It turns out that the manufacturers of the GH kit, and the Eubo (Orient Gene) kit both were owned by “Shanghai International Holding Corp. GmbH”. Upon searching for GH in the FDA device listing database¹⁶, it turns out that GH is a US agent for Orient Gene.

I was unable to find BRT and LBP’s registration for a SARS-Cov-2 / COVID-19 rapid test kit with the FDA. I believe that these two companies licensed their product from a different original manufacturer.

With some of the other devices, they were made by the same original manufacturer and they had different thresholds by which they showed a positive result. This variance is worrying, and shows that there is likely poor quality control done on these tests, or that there is some supply chain factor which is degrading these kits.

Worry

My worry is that these kits will prove unreliable. They’ll have a too high FN rate, and worse yet a high FP rate. In turn, the opinion of these lateral flow assays will sour. These could be useful tools in our arsenal, as they’re cheap, and easy to use. If they prove to be unreliable, the public health sector may overcorrect, and these tests may be banned outright.

“The Little Guy”

The other aspect here that’s problematic is that few researchers seem interested in running basic validation of these assays. The FDA has taken a Laissez-faire approach here, which I think is an appropriate approach. Recently, a team of researchers released a pre-print paper showing the wide variability of LFAs¹⁸. Although in their paper they present the results from a small sample size and variety of test manufacturers it is a start, but we still need to know more.

The one manufacturer I tested that tested positive at the highest dilution has had an incredibly difficult time with our bureaucracy. They’ve been able to legally sell in the EU for some time now. They’ve been trying to gain the ability to sell in the US for the better part of 3 weeks now, but they’re not well-equipped to deal with these things. The importer they’ve been dealing with has been trying to lock them into an exclusivity agreement, which would likely impact the availability of their tests.

What’s Next

There’s nothing left for me to do on this avenue. Getting more test kits from different manufacturers is proving to be an expensive, and complicated venture, with a limited idea of how useful it will be. Obtaining more antibodies also seems fraught with peril, as I’m out $350+ dollars due to CBP holding them. There’s no path to obtaining further tests for research purposes. US Customs has decided to deprioritize furthering our capabilities in this area. I donated the remaining test kits to medical personnel. I hope to enable them to give themselves testing capacity so that they are able to make informed decisions around their health.

I hope that researchers takes interest here. I hope that they decide to perform cross-validation with quantitative antibody tests. In addition, I hope that they attempt to reproduce the sensitivity, and specificity that these manufacturers report.

Maybe we can get to a place where we can arm everyone with these tests and make them easily accessible.

I do not have any competing interests.

I would like to thank Nick Lane-Smith, and Talia Trilling for helping with this article.