A Fine Balance: 9 Recommendations for Capturing User Data Through Seamless Digital Experiences
In the work that we do at Praekelt.org, there’s often the risk of a disconnect or delay between the technologies we build and the tools we use to measure their impact. In an effort to address this risk, we practice what we call ‘Seamless MERL’ (Monitoring, Evaluation, Research, and Learning), integrating our products and the evaluation of those products, while creating a smooth, seamless experience for users. There’s no single way to create seamless MERL experiences, but together with Nicki Ashcroft and the Institute for Reproductive Health, Georgetown University, we’ve developed a list of essential considerations which we shared at last year’s MERL conference in Washington DC.
9 Recommendations for Seamless MERL:
- Set a research goal upfront: There are many reasons why you may want to do research or monitoring on your digital product. The first step is refining the questions you want answered and the purpose of your research. Prioritizing questions has helped us be more selective in our data gathering activities and, we hope, resulted in a less intrusive experience for our users. Create a living document that you revisit as the MERL work proceeds: regular reviews and revisions are necessary as we’ve sometimes discovered it’s not always possible to get the data you want. For example we have learned that some of our users in Indonesia are very unlikely to provide an accurate answer to how much money they save and exactly what they spend it on. Stay flexible and allow your questions to change over time in response to what you learn.
- Choose your channel: SMS? Online? In your app? This choice is especially relevant for services and products that engage with end users via multiple channels. Think about the potential burden on the user and the potential richness of the engagement to decide when you use which channel and for what purpose. While we’re able to gather much richer data via mobile web than through USSD on MomConnect in South Africa, users are far more likely to answer questions via USSD (an interactive, text-based technology that does not require an internet connection) because answering is fast, cheap and easy.
- Make it easy for your respondents: Technology is not always intuitive to use. If your survey requires users to navigate through complicated menus or use unfamiliar features, you’re going to be limited to only the most tech-savvy respondents. Accurately assess the capabilities of your respondents through piloting or focus groups and design accordingly, keeping in mind that most people have a low tolerance for technology-induced frustration. In Nigeria, we’ve found that literacy rates have a dramatic impact on the use of text-based technologies. We now rely on Interactive Voice Response (IVR) to collect data from our users there.
- Be smart about interaction frequency: This can be a difficult balance. Too infrequently, and you don’t get good data. Too often, and you risk annoying your users (which can reduce interactions, and reduce the chances of good data). Look at usage patterns and sequencing; most digital products see a steep drop-off of users after the first month, so ask the most important questions at the start. For our youth-focused programs we are careful not to run concurrent surveys. We always ensure that there are periods during which questions and surveys are not being served so that users don’t feel like they’re constantly being mined for data.
- Consider ethics: Seamless MERL doesn’t avoid the normal ethical research questions, and in some ways, it creates more. For example, once a question is sent, you can’t control who has the phone or the user’s privacy settings, which is particularly important for sensitive topics, such as family planning usage or HIV/ AIDS status. Identify the possible risks, mitigate them where you can, and disclose them to the user. Thanks to the research we’ve conducted with Girl Effect we know that most of the adolescent girls we’re working with are borrowers of phones rather than owners and we’re careful not to serve sensitive questions without making sure it’s our user that’s logged in rather than a family member.
- Choose what data you need: Questions that are short, engaging, and easily understood in the local context are the most successful — that’s true with any MERL. However, with digital MERL, expect to ruthlessly decide what data is most important. Our experience is that response rates for online surveys tend to start dropping dramatically after around five questions: those questions really need to count. When we have more questions we try and split up our surveys into several mini-surveys with large-enough gaps in between to avoid irritating users.
- Protect user data: Seamless MERL can create vulnerabilities in data collection and storage. While hacking is fairly rare, the larger the project, the greater the risk. Beyond the risks posed by malicious external parties, it’s important to create internal safeguards too. At Praekelt.org, access to user data is carefully controlled by creating different levels of users for our content management systems and dashboards. Each user has a secure password they use to access the data and interactions with data are also recorded to help identify any potential breaches.
- Know the rules: MERL must follow the appropriate laws and regulations in every country. For our maternal health work in Nigeria, to cite one example, all data collected from users must be stored on servers and computers within the country in order to comply with data sovereignty laws.
- Timing counts: A longer monitoring effort may produce more insights, but engaging users over a period of months or years can be difficult. Strategies like gamification and providing offline user touch-points can increase engagement and response rates. No matter what you do, users will drop off over time, so collect the more important answers first. For our mobile web projects we ask users to register to engage with some of the social features, allowing us to track users over time and gather longitudinal data. More literally, we also take care to send out survey and questions during times we know respondents are more likely to have the time and inclination to answer. In Bangladesh, the girls in our programs are much more likely to interact with us on Friday afternoons
Mobile technologies can overcome many obstacles to learning and data capture. But with each additional point-of-contact and subsequent request for information, we risk losing our users, limiting not just our MERL, but service engagement too. Our ultimate goal should be a seamless experience in which services and research are intertwined. The result, hopefully, are products that collect larger volumes of better quality data than ever before.