Validating digital health solutions
How I shifted from the lab to “real world” research at scale
Hi readers. I’m Tomer Gazit, data team lead at Hello Heart. In this post I’ll tell you about our new research which describes the sustained clinical impact that Hello Heart had on users with hypertension. I will describe how this research, which was recently published in JAMA Network Open, came to fruition, what our goals were, the obstacles we faced, major findings and what plans we have for the future.
Introduction
A recent review in The Lancet from the NCD Risk Factor Collaboration estimated that 1.2 billion people worldwide suffer from hypertension¹. This estimate includes nearly half of the US adult population, which contributes to more than half a million deaths each year. Unfortunately, less than a quarter of people with hypertension have adequate control over their condition². Hypertension is often called ‘the silent killer’ because it shows no early symptoms and simultaneously can lead to a host of serious problems including heart attack, heart failure and stroke. Such widespread and long-lasting diseases demand solutions that are both effective and scalable. One of the reasons I joined Hello Heart was to combat this and have the opportunity to make a difference in people’s lives — and do it at scale. Having participated in and managed multiple clinical and research projects in the past, I found that when you want to scale up, you can only go so far with the data you collect in the lab. I believe that large-scale proactive and personalized health maintenance can be achieved by digital health solutions, and to study this, you need to go out into the real world.
Here’s where we come in. Hello Heart is a clinically-based smartphone solution for patients to track, understand, and improve their chronic conditions. Users are able to build healthy tracking habits and improve their health in real-time with an easy-to-use smartphone application. But, does it work? Can the results be sustained over time? Some studies argued that blood pressure (BP) self-monitoring alone is insufficient to lower BP without other co-interventions, such as lifestyle counseling³. In 2017, we published an initial paper describing a population of 5,115 users who enrolled in the Hello Heart app and followed them for 22 weeks. We found a positive correlation between user engagement and blood pressure reduction, and saw a normalization of BP for 69% of users. This initial study was a first step in proving the validity of our mobile solution for hypertension but we’ve gained traction since then, and we decided to take this to the next level. We now have a database of over 150,000 users who used the app for a period of up to five years, including cohorts that utilized our new features such as connecting to their clinic or to Google Fit / Apple Health and more. Our user population has also changed as we moved from B2C to B2B2C and marketed to self-insured employers who offer chronic health management solutions like Hello Heart to their employees. We wanted to reassess our clinical performances on a large and long-lasting cohort, and were interested in learning about other aspects such as the effect of the solution on potential hypertension crisis, and which factors were involved in the observed clinical outcomes.
Cartoon is copyright to Cartoons by Jim and reproduced with permission from HiNZ.
BP change over time — Dealing with random effects
We recently joined forces with Dr. Alexis Beatty, a cardiologist and health systems researcher at The University of California, San Francisco (UCSF) to plan the study. We concentrated on users that (a) registered the app through their employer between January 2015 and July 2020, including 21 companies that participated in the Hello Heart program and (b) started with a systolic BP above what is considered normal (>120mmHg). This resulted in 28,189 users from these 21 companies that met the study criteria of entering at least two BP measurements. Not a bad starting point! We followed them at multiple time points from the first BP reading: first week (week 0),after 2, 4, 6, 12, 26 weeks, and after 1, 2 and 3 years. The 21 companies made up a diverse group, including large Fortune 500 organizations to smaller businesses. Some of the companies were more technologically-advanced while others were from more traditional industries that utilized minimal technologies. We assumed that users from different companies would interact differently with the app so the first challenge was to adopt a methodology that would take this into account. One option was to average within each company and then perform the statistics at a company level, but such analysis would drastically reduce our statistical power. We wanted to take advantage of our large user population while still controlling for the different variables within the company — in other words, a mixed model design. Mixed design models are useful when observations belong to hierarchical subgroups (companies in our case) within a population so that you can model both fixed effect (across entire population) while taking care of random effects, which are group specific (here is a nice conceptual explanation).
With this approach we found a significant reduction in BP across time and this reduction was sustained for the entire period, up to three years. Systolic BP by year 1 was reduced in 53% of users with baseline elevated BP, 69.7% with baseline stage 1 hypertension, and 85.7% with baseline stage 2 hypertension. Participants who continued the program for three years maintained these lower levels, with a mean (SEM) reduction of 7.2, 12.2, and 20.9 mm Hg systolic BP compared with baseline for those starting with elevated, stage 1 and stage 2 hypertension, respectively. In the next figure you can see BP change across time for users who started with stage 1 (systolic >130 mmHg) or stage 2 hypertension (systolic >140 mmHg):
What is engagement?
Of course we wanted to test the effect of user engagement with the Hello Heart app on BP reduction over time, but how do you measure engagement? One disadvantage of doing research on real world data is that you don’t have control over many aspects of the design: how many times and how long the users use the app, what features they will use, etc. It becomes difficult to conceptualize a ubiquitous measure of engagement. For example, is using the app extensively over the first month better or worse than using it occasionally for a longer time period? Clearly, engagement at week 0 comes from a different distribution than engagement at year 1, etc.
While every decision has its pros and cons, we decided to proceed as follows:
(1) Evaluate for each time period how the user is engaged in the app (in terms of number of sessions)
(2) Compare this value to its peers by normalizing each user’s number of sessions by the mean and standard deviation of that specific time period
(3) Average these normalized values across “possible” time points to form an overall user engagement score
(4) Due to the exponential nature of the distribution of the number of sessions, the data was log transformed
This allowed us to take into account the relative engagement of the user at each milestone as well as the duration of use. The reason I use the word “possible” is that not all users had the chance to reach the three year mark. Again, due to the “real life” nature of the study, people registered at different time points; since we stopped the study in July 2020, a user who registered in July 2019 could only reach the one year mark.
To classify users into different engagement levels, we used a simple K-means clustering algorithm. Then, we evaluated different Ks and, in accordance with the “elbow” approach (choosing the point where the curve visibly bends from high slope to low slope), and we found a value of three clusters to be informative (as shown in the following figure). Subsequently, we used this as a categorical variable, but also kept the continuous engagement estimation, both of which produced similar results.
In the mixed design model we saw that greater engagement with the app was associated with lower systolic and diastolic BP. These effects can be biased by confounding factors, so we added any relevant information we could get including age, gender, depression, anxiety, diabetes, high cholesterol, smoking, Area Deprivation Index, and regions in the United States. Even after accounting for external factors, the results still showed a significant effect of engagement on BP reduction.
Engagement and Hypertension crisis — mixed design with categorical variable
A Hypertensive Crisis (HTC) is a severe increase in blood pressure (systolic > 180 mmHg, diastolic >120 mmHg) that can lead to organ damage and/or stroke. In the descriptive analysis we saw there is generally a gradual rise in BP which becomes steep prior to the potential HTC and then a drastic decrease in BP following (see next figure). We wanted to evaluate cases where there was a rise in BP that had the potential to become an HTC, and examine whether being active in the app had a correlation to the number of HTCs in the following period. We evaluated 882 cases where users had a high BP (systolic > 140 mmHg) and were observed to have an increase of more than 10 mmHg in mean weekly systolic BP over 3 weeks (without systolic BP >180 mm Hg). We wanted to evaluate the association of the number of BP measurements entered into the app with the number of potential HTCs in the following month. We still needed a mixed design model to account for the effects of the different companies but our target variable was the number of systolic readings in the following month which were above 180 mm Hg. When you talk about a target distribution which represents the number of events, it smells like Poisson modeling. But it’s not fair to compare a user that usually enters many BP readings to a user that doesn’t, so you also have to normalize by the number of total BP readings. In this case, we used a generalized mixed model with a Poisson distribution with the number of potential HTCs measured as target and log of the number of total measures as an added offset. Interestingly, we found that the more you measure your BP, the lower the probability of potential HTCs.
What mediates the effect of engagement on BP reduction?
We are often asked how our solution works and what changes it ignites that produce the previously described reduction in BP, prevention of potential HTCs and better health in general. This is a difficult question to answer and obviously it is a combination of many actions and lifestyle changes people adopt as part of the solution. This could include any combination of the following: adopting a better diet and losing weight, participating in physical activity, taking their medications more regularly, managing stress better or proactively managing their condition and not waiting for a crisis. Obviously, adding a measurement in the app by itself does not directly reduce BP, but it’s what you do with these measurements and how you manage your health that matters (see this recent report submitted by the Validation Institute describing the reduction in claims cost for employees that chose to use Hello Heart vs. others). In other words, we suspected there are some factors that mediate the effect of engagement on BP reduction. To explore this, we followed users that either entered their weight in the app or connected their app with Google Fit or Apple Health. We found that for each 1000-step increase in daily steps, there was a 0.8 mmHg reduction in systolic BP (P = .03). But increasing the number of steps you walk in a day is one of the factors that mediate the effect of engagement on BP reduction. Meaning that part of the reason why using the app reduces BP is through the effect of walking more. We did not find such an effect for weight loss. So get out there and move, it’s good for your heart health!
Conclusions
After ~15 years of carefully designed lab research, this was my first attempt at doing a “real world” study. As a scientist, you always want to control as many variables as you can, so it’s not an easy transformation to evaluate app usability, but it is definitely rewarding. Once you deal with the obstacles, variability, and unknowns, you get the opportunity to evaluate how people really behave and how an app can improve their health. This study allowed us to describe outcomes from the largest and longest peer-reviewed study on digital therapeutics for management of heart health, showing a sustained and significant reduction in BP for up to three years.
So what’s next? Well, we’re just getting started. This large database allows us to study and conclude numerous insights on user activity and health, which we hope to use for the benefit of our users. Today, we continue to analyze this dataset and leverage the learnings to develop predictive and forecasting models to detect HTCs and other heart health crises ahead of time. These functionalities will enable Hello Heart to notify users and direct them to seek specific medical care in a timely manner. We are also working on personalizing user experience in the app. This means that users will receive tailored messaging and recommendations that will help encourage them to be proactive and to manage their own health. So stay tuned…
Join Us!
If you think this is cool, join us! Visit our careers page to see available related roles.
Cartoon is copyright to Cartoons by Jim and reproduced with permission from HiNZ.
References:
1.NCD Risk Factor Collaboration (NCD-RisC). Worldwide trends in hypertension prevalence and progress in treatment and control from 1990 to 2019: a pooled analysis of 1201 population-representative studies with 104 million participants. Lancet. 2021 Sep 11;398(10304):957–980.
2 https://www.cdc.gov/bloodpressure/facts.html
3 Tucker KL, Sheppard JP, Stevens R, et al. Self-monitoring of blood pressure in hypertension: a systematic review and individual patient data meta-analysis. PLoS Med. 2017;14(9):e1002389. doi:10.1371/journal.pmed.1002389
