What Digital Health Learned From Netflix: How Data Science Is Creating Self-Learning Healthcare
By Eric Williams, Director of Data Science @ Omada Health
In 1991, millions of postmenopausal women were given a very good reason to be in a very good mood. It turns out the same hormone replacement therapies they’d been prescribed to balance their emotions came with an unexpected side-benefit: a much healthier heart.
That year, a meta-analysis in Preventive Medicine breathlessly announced that HRT was responsible for a 50% decline in heart disease. Let that sink in. Fifty percent. The study’s authors became lauded scientists for having effectively uncovered a way to slash the number one cause of death for women in half. Some doctors even began advising female patients to take HRT for healthier hearts alone.
There was only one problem. The study was totally wrong.
It took over a decade to unravel all the flaws of the authors’ meta-analysis. But their mistake was just the beginning. A 2002 randomized control trial proved the opposite of their assertion was true: Estrogen replacement therapy has no effect and, potentially, increases the risk of heart disease.
So what went wrong with the initial meta-analysis?
“Correlation does not equal causation” is probably the most quoted (and neglected) mantra from your Statistics 101 professor or any data scientist in the field. However, the misleading implications from this common stat-trap are particularly dangerous when it comes to health research. In the case of HRT and heart disease, it took over a decade to unravel the fact that affluent women were more likely to get HRT and — here’s the clincher — take care of their heart health. And that was just one of many potential overlapping factors that lead to false conclusions.
For over a century, randomized control trials (RCTs) have been the gold standard scientific methodology for testing not simply correlation, but causation — and rightfully so. But RCTs come with their own sets of challenges and limitations. Clinical trials tend to be slow, labor intensive, and expensive. Even more troubling, results from clinical trials often don’t generalize to broader populations, due to difficulty and biases introduced by patient recruitment.
But today, we are on the precipice of a revolution in healthcare that has the potential to accelerate the discovery of causal links and enable a healthcare company of any size to test connections between courses of treatment and healthcare results for all types of patients.
Call it the “burden of proof” transformation — the increasingly sticky idea that healthcare costs should be paid based on outcomes and not on activity. The rise of outcome-based healthcare reimbursement has aligned the motivations of payers and providers, and has the potential to kickstart the industry toward generating more efficient and effective healthcare solutions. At the same time, the explosion of digital health has given those in healthcare the unique opportunity to leverage data science and capture the full value of the vast amounts of health data, creating self-learning health systems that will lead to more effective healthcare for millions of Americans.
Successful preventative healthcare is dependent on two things: accurately identifying who is at risk and determining how to intervene. The field of data science and its methodologies — namely analytics, machine learning, and experimentation — have the potential to completely change both the identification of those at risk (prediction) and the optimization (personalization) of their care.
Here are three key ways data science is revolutionizing care and providing the potential for precision population health for the first time in human history.
Prediction is the key to any successful preventive strategy — especially in healthcare.
But a predictive model is only as good as the data that underlies it. Google’s Chief Economist Hal Varian is famous for stating: “[Google] doesn’t have better models; it just has more data.”
Until recently, machine learning has mostly benefited the digital marketing space. That’s why your typical online interaction today will be peppered with targeted ads and product recommendations, which maximize the odds that you’ll click-on, subscribe to, or purchase a particular product. Or, google “Target knows you’re pregnant” to see how data analytics and machine learning can beat our brains to even some of the most personal revelations, predicting what products we’ll need even before we realize it ourselves.
Applying predictive modeling to healthcare is revealing itself to be a game-changer. What if, instead of using machine learning to suggest which movies you should watch next in your Netflix queue, it was harnessed to pinpoint those “tipping point” individuals who are most likely to forget to take a medication, miss a critical doctor’s appointment, or fall off the wagon of a diet or new exercise routine?
Clover Health, a technology-enabled Medicare Advantage provider, is making a big bet on predictive analytics impacting preventative care. Starting with roughly 16,000 Medicare Advantage members in six New Jersey counties, Clover’s data scientists use claims and lab data to predict members most at-risk of illness. Once patients are identified, Clover’s nurse practitioners are deployed to their homes as a preventative intervention. Since Medicare Advantage plans receive subsidies from the government to cover both the premium and the claims of their members, effective prevention is fully aligned with Clover’s business incentives. In the first half of 2015, Clover has claimed that these methods have reduced hospital admissions of their members by 50% and hospital re-admissions by 34%.
In concert with predicting who is most at risk, effective prevention needs an effective intervention. The experimental tools of data science, when deployed smartly, can do just that.
The undeniably successful use of marketing optimization and user learning displayed by companies like Google and Netflix have made A/B testing table stakes for most technology companies these days. The process of hypothesis generation, randomization, and evaluation is now common language from developers to CEOs.
Of course, A/B testing is old news, especially in the healthcare field. In fact, it largely originates from healthcare. Some trace the origins of systematic clinical trial design back as far as 1747 when surgeon James Lind tested six different proposed “cures” for scurvy (including, but not limited to: seawater, cider, vinegar, and — thankfully — citrus) on the crew of the HMS Salisbury. Since then, experimental design has solidified the “Randomized Control Trial” (basically, an A/B test) as the widely accepted gold standard for experimental measurement. The goal, no surprise, is to help determine causation between a delivered intervention and a primary outcome.
But as the amount of available data has exploded, so too has the possibility of the “super-charged RCTs” — rapid A/B tests exploring multiple facets of interventions that give unprecedented insight into what works in healthcare. When it comes to preventing chronic disease, super-charged RCTs create new opportunities to understand human behavior, personalize, and optimize interventions to deliver the best health outcomes.
But if a model is only as good as its data, then an A/B test is only as good as its outcome. Which means super-charged RCTs only matter if you can measure actual health outcomes.
In digital health, this is surprisingly rare.
At the time of writing, there are over 165,000 mobile apps available claiming health benefits and very few have any evidence to back up those claims. In fact, the gap between the health claims made by these apps and the troves of data they are collecting has become so enormous that a startup has been created to bridge this gap. Evidation Health, a Silicon Valley company, recognizes this missed opportunity. They aim to align the data collected by digital health interventions with the outcomes captured by health plans. This way, they can validate (or not) the health claims made by those companies and find the most effective interventions for health plans to implement.
Established health companies also need to accept this responsibility. They’ll need to measure their effectiveness against promised outcomes if they want to truly capitalize on their own vast amounts of streaming data and optimize their interventions against these outcomes using iterative, RCT-like methodology. But this responsibility can just as easily be viewed as an opportunity — digital health companies should take advantage of the evolving field of experimental design. We can implement the most flexible, adaptive trial design, and build systems that intrinsically improve with scale. Once more digital health companies embrace this approach, we’ll see massive changes in the ways that technology can influence healthy, sustainable, and scalable behavior change.
The ultimate goal of these efforts is a self-learning healthcare system — one that generates continuous feedback on the most effective approach for populations and individual patients, then incorporates that feedback to create a virtuous cycle of improvement.
The BioMe Biobank Program, lead by The Charles Bronfman Institute for Personalized Medicine in the Icahn School of Medicine at Mount Sinai hospital, is an effort to capitalize on the power of centrally-collected and analyzed data. By pooling genomic, environmental, and lifestyle data from thousands of diverse individuals, small signals — previously undetectable underneath large amounts of noise — can be detected and linked to health outcomes. The goal is precision disease classification and diagnoses, where medication and healthcare are delivered at the level of the individual, customized using each patient’s unique data profile.
At Omada, we share the goal of “precision population health”, and our data science team is focused on personalizing and optimizing our flagship product, Prevent®, using the tools of data science.
Prevent engages participants with an evidence-based curriculum, a supportive social network, the constant guidance of a personal health coach, and digital tracking tools that include a wireless scale and mobile app — all to help reduce a participant’s risk of progressing toward obesity-related chronic disease.
Throughout Prevent, we deploy predictive models to identify those participants who are at risk of gaining weight or dropping out of the program. These models are based on digitally recorded program behavior data. Have you started tracking food less frequently? Are you logging your physical activity at more random times instead of on the schedule you developed with your personal health coach? Has your weight fluctuated recently? What behaviors indicate that a retired, male participant is likely to gain back the six pounds he has already lost, and how can our team intervene at the right moment to make sure it doesn’t happen?
Omada uses a three-step process to maximize the effectiveness of the program :
- Measure The team is equipped with a constant firehose of data, measuring engagement, participant behavior, weight loss, and other key indicators of successful behavioral interventions.
- Optimize Using experimental design, we run continual RCTs and A/B tests, creating a virtuous cycle of program improvement focused on maximizing the health outcomes that matter most for Prevent, including weight loss.
- Personalize Behavioral interventions are most effective when they are tailored to the needs of individual participants. By leveraging the power of big data and clinical rigor, we are focused on maximizing the efficacy of Prevent for every user — delivering the right interventions, at the right times, in the right ways.
Here’s an example of this process in action: Each night our machine learning algorithms are trained on streaming demographic data, as well as longitudinal engagement and weight data. They spend the night assessing and predicting participants at high risk for gaining weight. By morning, these at-risk participants, randomized through our internal clinical trial management system, are surfaced to their health coaches along with specific, personalized intervention suggestions as defined by the predictive model. The effect of the suggested intervention on the participant’s outcomes are captured, studied, and used to iterate on and optimize both the prediction algorithm and suggested interventions — leading to the continual refinement and precision of our ability to keep our participants from falling off track.
As we’ve scaled Prevent over the last 18 months to over 40,000 participants, we’ve amassed a data set containing tens of millions of points on everything from weigh-ins to interactions with health coaches, group members, and curriculum. We’ve assembled one of the largest data sets on behavior change in human history — and can compare all of that data against continuously collected weight-loss results. This combination allows us to determine influences on behavior — and subsequently, influences on clinically meaningful outcomes.
This approach has huge implications for fighting what the Centers for Disease Control and Prevention (CDC) has labeled the public health challenge of our generation: chronic disease.
There’s widespread agreement that the most effective form of tackling chronic diseases is prevention. The biggest remaining challenges are A) how to scale effective interventions to deal with a problem that affects more than one in three American adults and B) how to design interventions for populations or personality types that respond to different incentives.
With the data set we’ve built — which continues to grow every day — we’ve started to discover how small changes to an interface, or tiny shifts in how a health coach interacts with a patient, can have big impacts on essential health outcomes. As our data set grows richer, so will our experiments and results. Our ultimate goal is an adaptive, personalized curriculum that optimizes weight loss and decreases average blood sugar (a1c), while reducing the greatest amount of risk for every participant. Every day, we integrate new elements that drive us towards that goal.
It’s an exciting time for healthcare. And an even more critical moment for the millions of Americans at the tipping point of chronic disease. For many of these people, data science is paving the way to live longer, healthier, more fulfilling lives.
