Why Are Investors Eager to Lose Money on Health Tech?

Photo by Michael Longmire on Unsplash

If you had a million dollars, and someone asked you to invest it in a brand new company that was based on a new scientific development, you might have a lot of questions: Who’s running the company? What are their qualifications and track record? What is the business plan going forward? Who is the target market?

But the most important question of all is: “Does this new scientific development actually work?” After all, if it doesn’t work as purported, none of the rest matters — not the founders’ education or track record, not who else is investing, not the corporate strategy, nothing.

So you’d think that sophisticated investors would put at least some due diligence into finding out whether a new scientific development actually works as promised, in a way that would justify the prices being charged.

Amazingly, that isn’t always the case.

The most recent case is that of uBiome, a high-tech company offering consumers tests to analyze their microbiomes. A featured product: SmartGut, which purports to provide “smart, actionable insights to improve your gut health.” But as reported by Chrissy Farr, the FBI raided uBiome’s offices at the end of April 2019, looking for evidence of as to Biome’s alleged practices of performing too many tests and trying to bill (or double bill) health insurance companies up to $3,000 a test.

This was quite a blow to a company that, in a few short years, had received some $100 million in investments from venture capitalists, including the famed Y Combinator and Andreessen Horowitz.

To be sure, the company until recently claimed to have over 60 top advisers from throughout academia, including Harvard’s George Church. Casual readers of the company’s website might be excused for believing the following claim: “Our scientific advances in artificial intelligence, machine learning, and statistical genetics, as well as the largest database of microbiomes in the world (gathered over five years from hundreds of thousands of people), allow us to create revolutionary clinical products.”

But people who have $100 million to invest are not casual readers of the company’s website. They might have wanted to look into the scientific tests that uBiome was selling, and whether those tests actually provide clinical useful information that improves patients’ lives.

Although I am not a microbiome scientist by any means, I have considerable experience assessing the quality of academic work across many disciplines — including reviewing hundreds of studies that we have funded at Arnold Ventures, and funding some of the most well-known efforts in open science and reproducibility (such as the Reproducibility Project in Cancer Biology). And even with a cursory look at the “Science” section of uBiome’s website, I am unimpressed.

There are a total of nine scholarly articles published by uBiome’s staff, and none of them provide any evidence that uBiome’s tests are worth paying up to $3,000. The list includes:

• An article reviewing the literature on the microbiome. Perhaps unhelpfully for uBiome’s marketing claims, the article concedes that, “Despite a myriad of studies on the composition of the human microbiome of healthy controls and that of diseased patients, and the relative uniform prevalence of the same bacterial phyla among healthy populations, it has been surprisingly hard to define exactly what constitutes a healthy microbiome.” The article ends with the claim that “yet‐to‐be discovered metabolic functions and inter‐individual differences are likely to greatly contribute to the personalized medicine field.” That may be true, but the article doesn’t point to a real-world example (let alone a rigorous randomized trial) where personalized microbiome analysis would improve hard outcomes for patients.
• An article noting that there was one woman who found out about a recurrence of C.diff (a nasty pathogen that causes diarrhea) through a uBiome test. This seems like a nice (if unsurprising) outcome for one person.
• An article reviewing literature from around the world on at-home screening tests for HPV (human papillomavirus). Towards the end, the article has a short section (five paragraphs) on how the vaginal microbiome relates to HPV. Here’s where you might expect uBiome’s authors to bring out all the evidence that their SmartJane test is clinically useful. Instead, all they have are correlational studies showing that women with HPV might have higher levels of this or that microbiotic bacteria. That’s a long way from showing causation (as opposed to correlation), let alone that there’s a way to alter this particular microbiome in a way that provides any clinical benefit. In the end, the most that they are willing to claim is that this is an “an emerging treatment area,” and that “self-sampling for HPV with the addition of associated microorganisms may provide patients and providers with increasingly relevant and actionable clinical information.” It may indeed, but there’s apparently no evidence of it yet.
• Relatedly, an article showing that uBiome’s SmartJane test can accurately detect HPV, bacteria related to sexually transmitted infections (STIs), and several microbiotic species. Detecting HPV and STIs is no doubt a useful thing, but the clinical relevance of the microbiotic species remains to be seen.
• An article showing that a particular type of genetic sequencing “can accurately detect and quantify clinically relevant levels” of 28 types of microbiotic bacteria, including C.diff, salmonella, and bifidobacteria. But this is a long ways from showing that such a test would actually improve the health or lives of real-world patients.
• An article that was originally in the Dutch Journal of Microbiology (the link goes to a translation on uBiome’s site). It’s merely an overview of what the microbiome is, and how it has been correlated with various human diseases. Nothing in the article shows that uBiome’s tests provide a clinical benefit to anyone, and the article concludes with the helpful warning that many companies are selling probiotics “without a scientific basis or quality control.” I’d have to agree that it’s worrisome when companies sell microbiome products without much of a scientific basis for their claims.
• An article that’s basically a short sales pitch for uBiome. This isn’t a scientific paper at all.
• Finally, an article describing bacteria that might help remediate Antarctic soil that is contaminated with diesel fuel, and an article describing a species of bacteria that is related to bioleaching and acid rock drainage. These articles are completely irrelevant to human disease, and the very fact that uBiome listed them comes across as an attempt to inflate the “scientific evidence” in the minds of casual readers.

That’s it. That’s the sum total of the scientific articles that uBiome’s website lists as having been published by their scientists. And six of the nine articles are either irrelevant or are just summaries of the wider literature. Small wonder that scientists have criticized uBiome and other similar companies for oversold claims.

If I had $100 million to invest, I would have wanted to see that uBiome had funded independent investigators to perform multiple randomized trials, and that the independent trials showed that personalized microbiome tests actually helped real-world patients treat or prevent disease. Instead, uBiome has so little published evidence that even a researcher from a completely different field (that is, me) could review and summarize it in couple of hours over a weekend.

After all, without rigorous evidence, health insurance companies might understandably be unwilling to reimburse for such expensive tests. And the lack of such evidence could even be a contributing factor to why uBiome was apparently so eager to do duplicative tests or duplicative billing, just to increase the chance of getting paid at least once.

To say something in defense of uBiome and its investors: at least the tests actually do seem to exist and to accurately diagnose the presence of certain microbiotic strains (albeit perhaps not as well as claimed). So it’s not as bad as Theranos, where investors put in hundreds of millions of dollars without verifying that Theranos’ blood testing equipment worked at all, let alone had any clinical benefit. The better analogy may be to IBM’s Watson, which does exist and which has even had some successes, but as to which a recent article notes that “no study has yet shown that it benefits patients.”

The bigger question is why so many people invest so many millions of dollars in new health-tech products without doing what I would consider basic due diligence, i.e., asking for hard evidence that it actually works for real-world patients. In an area with so many over-hyped claims, it isn’t wise to take shortcuts on scientific due diligence. Hype can carry a health-tech company only so far: if no study shows a benefit to real-world patients, the market will eventually catch on.