Research methods and Data Science give us a lead in improving mobile services for SA nurses

Authors: Nathan Begbie, Charles Copley, Eli Grant

Photo by Hush Naidoo on Unsplash

Nurses are vital to any national health service, so supporting them is a key component of the health system. One example of this is the NurseConnect programme for South Africa’s National Department of Health (NDOH).

NurseConnect uses WhatsApp to send weekly messages of support and instruction to nurses’ phones. More in-depth information is accessible on a mobisite, with articles and knowledge-testing quizzes. We learned that enrolling nurses onto our platforms and evaluating how to improve the user experience and content is key to a platform’s success.

This blog is about an experiment we conducted to test the effectiveness of SMS advertising to increase nurses’ use of the mobisite, identify bottlenecks and gather user experience recommendations.

In our trial, we implemented a cluster randomised control trial (CRCT) (see https://medium.com/patient-engagement-lab/cluster-randomized-control-trials-in-mhealth-9e6b34a44961) with randomised airtime incentives at the clinic level. The treatment conditions were:

1. No SMS reminder (n = 1511)
2. SMS reminder and no airtime (n = 1758)
3. SMS reminder and R5 airtime (n = 1541)
4. SMS reminder and R10 airtime (n = 4719)

There were a total of 6230 nurses involved in the trial. Of these, 4719 nurses received some form of a request reminder and/or airtime incentive.

We were focused on two experimental outcomes:

1. A user clickthrough from a text message with a web link to the NurseConnect mobisite
2. A user reply to a text message containing a question

An important part of the experiment was that all nurses allocated to the incentivization conditions, received the airtime incentive irrespective of their behaviour. This ends up testing the behavioural response of reciprocity. This occurs when someone is more likely to do something for you because you have done something for them prior to the request.

To deliver on this design, we needed to do a number of things:

1. Send individualised messages to thousands of nurses.
2. Embed a unique clickthrough link to the mobisite in each SMS. A single shared link would not meet our criteria, as each nurse required their own unique link.
3. Have a way to track the user behaviour on the website for each nurse who was part of the study.

To investigate user behaviour, we linked each SMS to subsequent behaviour by that nurse on the mobisite. We considered using Google Analytics to do this, but the non-commercial version does not provide individual-level data. We overcame this challenge by allocating each participant a random UUID (unique user Identifier) to ensure anonymisation of participant information. We then used the URL shortening service Bitly to create a unique shortened link to a website endpoint specific to that user and embedded the short-url in each SMS. We were then able to track the browser behaviour of each individual endpoint using Google Analytics without the commercial version.

We implemented this user tracking system in the general purpose programming language Python, as a part of the Django framework and deployed this to the mobisite. This system allowed us to follow a user throughout their journey, so that we could determine behaviour at an individual level. We could answer our questions about user behaviour. Did the nurse simply click the link and then leave? Did the nurse stay on the website and read articles? Did the nurse leave comments? Did the nurse take a quiz?

After running the survey, we saw clear differences in behaviour between the differently incentivised groups that were consistent with the Theory of Reciprocity. The results are graphically summarised below.

While there was a clear difference in user behaviour across groups, overall response rates were lower than we had hoped.

Below, we visualised the path of users through the site, effectively communicating the extent to which certain design decisions made by the team were affecting how the site was used.

The results are interesting and useful from an operational perspective. It is clear that nurses appreciate financial incentives and demonstrate reciprocal behaviour. Whether this could extend to other forms of non-financial incentivisation would be an interesting experiment to increase uptake in similar services. The participant experience flow above also clearly shows bottlenecks: many nurses did not even reach the mobisite. This may be indicative of slow network connections that lead to timeouts.

As the diagram demonstrates, of those who left the login page, almost half left immediately, and additional attrition occurred while forcing users to sign up. It can also be seen that although some participants clicked the link, they cancelled the request before they were redirected to the site.

An additional insight is that a number of nurses drop off at the login page. When the site was designed, we believed that signing up to the service was necessary to limit users to a nursing population. Now we ask: is a sign-up to the service necessary to view content?

Overall, the experiment demonstrated important technical solutions to the problem of linking participants behaviour from a push message to subsequent behaviour on a mobisite. In this case the message was delivered on SMS — but it could also be WhatsApp or email.

The technical link solution to view participant behaviour could facilitate a number of similar operational improvements in this area. The high mobile phone penetration rate in Low-to-Middle-Income countries suggests this link between a message and subsequent user behaviour could be an essential building block to measure uptake of mobile services in our region.