Examining the potential of ART patient risk scoring in managing LTFU

Our team from Palindrome Data and Right to Care during user testing.

The PREDICT Field Study is a project pioneered by a collaboration of Palindrome Data with, Right to Care, Indlela, and HEERO in Mpumalanga, South Africa. The project has leveraged big data and machine learning to produce models that can successfully predict LTFU in 2 out of 3 patients. After obtaining ethical approval, we analyzed over 10 years of de-identified patient data, followed by modelling and the development of easy-to-use clinician job aids to help identify patients who have a higher risk of being LTFU. Segmenting patient risk allows clinicians to triage and tailor interventions based on individual patient attributes and predicted risk. This field study follows the PREDICT group’s initial work on designing a predictive model, field testing some initial scorecards, as well as designing a behavioral nudge for patients (which can be seen at the bottom of the scorecard).

A complete paper-based adherence scorecard.

The user testing forms part of a wider pilot study that will evaluate the long-term impact that the tools have on managing LTFU. The study identified four clinics in Mpumalanga that were allocated different tools.

Study design

The purpose of the user testing was to test the tools and finalize them before the implementation of the study. This was done by establishing how well the tools worked in a clinic and whether they could be integrated into the clinician’s routine workflow. This would allow us to determine their long-term feasibility. Furthermore, it was important to understand the reliability of the risk scores produced by the tools and agreeability among clinicians.

The feedback from the user testing can be presented in four categories:

1. Integration with clinician’s routine workflow: In this category we saw some distinct differences between the paper tools and the digital tools. In the digital sites the nurses felt that the tool was easy to use and going through the scorecard was quick, as a result they concluded that the tool integrates succinctly with their routine workflow. In the paper sites users also considered the tools to be easy to use but there were concerns around the pieces of paper being misplaced and adding additional paperwork to an already paper intensive process.
2. Reliability and agreeability: Across all sites users felt that the scores were reliable and 90% of them agreed with the risk scores produced by the scorecards.

Out of all scorecards, 90% of healthcare workers agree with the risk scores.

1. Facilitating better patient-clinician conversations: Users across sites communicated that the tools (both digital and paper) facilitated better conversations between them and the patient. The nurse going through the scorecards with patients firstly made them feel cared for and prompted them to be more forthcoming with the challenges they are facing which resulted in an easier application of the treatment referral options.
2. Encouraging patient ownership over treatment plan: The scorecards with the treatment referral plan are designed in a way that encourages engagement with the patient to unpack their challenges and select a treatment plan best suited for them. The nurses communicated that this is a process that makes the patients feel like they are partners in their treatment journey, and they have some ownership over it which encourages accountability.