Navigating the Post-Model Landscape: A Lesson in Maximizing Impact
As a member of the Patrick J. McGovern Foundation’s (PJMF) Data to Safeguard Human Rights Accelerator cohort, Medtronic LABS developed a machine learning (ML) model capable of identifying the likelihood of patients enrolling in health service programs. However, further analysis demonstrated that re-engaging patients would require a data-informed intervention that was far from one-size-fits-all.
by Kelly Batlle and Kayla Huemer
Building a predictive model is just the beginning of the ever-evolving landscape of machine learning (ML) in healthcare. The real challenge lies in navigating the insights your model provides to implement interventions strategically. Through PJMF’s Data to Safeguard Human Rights Accelerator, Medtronic LABS developed an ML model capable of identifying patients unlikely to enroll in health service programs with 90% accuracy.
As a team, we were excited by the results of the model but quickly realized we were far from finished and the success of the project would be determined by how we could leverage the model to positively impact patients’ lives. In order to generate impact from the model’s intelligence, we needed to figure out how to best integrate this insight into our products to drive change. Through this narrative, we’ll share how we utilized our model’s intelligence to maximize the impact of our interventions and keep patients engaged in our health programs.
Beyond Model Building: A Paradigm Shift
Our goal for this project was to re-engage patients who hadn’t returned to our program after their initial touchpoint with the health system. As the modeling phase of the project wrapped up, we knew we had to design an intervention targeted at improving our platform engagement. Given that we call patients using our internal telecounseling platform, we knew a more targeted approach could significantly boost engagement. Our initial instinct was to intervene by calling patients who were the least likely to enroll. However, our analysis told a different story.
Wanting the data to speak for itself, we took our historical call logs and analyzed which patients had re-engaged with the program after a phone call. We then segmented these results by the patients’ enrollment likelihood scores generated by the model, ranging from 0 to 1. We found phone calls were largely ineffective in re-engaging patients who were less likely to enroll. When a patient had a predicted enrollment score of less than 0.4, only 1–5% of patients enrolled after a phone call. This is compared to the 30% boost to enrollment post-phone call for patients with enrollment likelihood scores between 0.4 and 0.8. With these insights, we designed a rules-based algorithm and started our first month-long experiment re-prioritizing calls to patients with a greater than a 0.4 enrollment likelihood score. This resulted in an almost six-fold increase in effectiveness at re-engaging patients.
While we didn’t want to leave behind the patients who are less likely to enroll, we learned that a phone call is not an effective way to re-engage these patients. Our next experiments will trial other interventions, such as further education at the patient’s first touch point or prioritizing these patients for a follow-up visit by a community health worker.
A Lesson in Responsiveness
The post-model phase demands a level of responsiveness beyond the algorithm itself. A major takeaway from this project has been understanding that interventions are not one-size-fits-all, underscoring the importance of tailoring responses based on the nuanced insights that the model uncovers.
As ML practitioners, the takeaway is clear: building a model is only half the battle. The true impact is accomplished by thoughtfully responding to the insights gleaned from the model. The initial intuition of intervening for the predicted ‘least likely’ may not hold true for every scenario. Embracing this complexity allows organizations to craft interventions that maximize impact and contribute to a more equitable and accessible healthcare landscape. To learn more, read our full insights report.
About Medtronic LABS & SPICE
Medtronic LABS is a health system innovator that designs and scales community-centered, tech-enabled solutions that drive population health outcomes. We partner with health systems to transform primary care by combining cutting-edge digital technology with data-driven care delivery.
Medtronic LABS is the steward of SPICE, an open-source digital platform designed with and for health systems, patients, and communities. SPICE is focused on data-driven, outcomes-focused care at both the community and primary care levels. The platform is certified as a digital public good, and we work with Ministries of Health to plan for long-term country ownership.
