Value and validation
How feedback enhances the quality of research outputs
By Tom Wein, Mathilde Schilling, Philippa Hammond, Joel Mumo and Cecile Juma
If there’s one thing participants could change about research, it would be feedback. In a recent qualitative study, Busara’s research participants told us that they would want to learn the results of the studies they take part in. Beyond answering questions, respondents want to learn something new, and as representatives of their communities, share this new knowledge with their friends and family.
Reporting back to respondents is often recommended as a way to conduct more ethical research, because it addresses the power imbalances between researchers and respondents. This is still rarely practiced in quantitative research, likely due to the sheer volume of respondents typically involved. With qualitative research, however, ‘member checking’ has been a standard practice.
Is there any actual impact in giving feedback to research participants? We say yes!
As part of Busara’s work with GSMA, Safaricom and the Kenya Forest Service, we conducted qualitative and quantitative research with members of Kenya’s Community Forest Associations (CFAs). These CFAs help protect the forest, and our research focused on how we could ease the burden of collecting data and reporting their efforts. Our findings were published, but we had not yet presented them to the people who had so generously shared their time and experiences with us. Despite limited time and resources to give detailed feedback to participants from the quantitative part of the study, we did send a response in SMS format to see what difference that would make to their lives. Using SMS, we believe, is the very least researchers can do to provide feedback while keeping costs low. We wanted to make sure that participants in our qualitative sample received proper feedback and got a chance to ask questions and validate the findings. We decided on phone calls for this exercise.
We called the original 21 people who had participated in the qualitative research phase, reaching 19 of them. We talked over the findings with them, and briefly asked which were most important to communicate to the wider group. They told us that they felt the conclusions were true and well founded, and agreed that the feedback was clear but should be in a written format. Busara’s staff were described as open to questions and well-informed. Overall they felt that the experience of participating in research with Busara had been positive, albeit a little time-consuming, and the connectivity issues occasioned by COVID-19 remote research protocols had been troublesome. Ultimately, they felt they had learned new things, and had enjoyed the chance to reflect together. They gave no sense that the research was extractive, uncomfortable or something they reflected negatively upon. The real benefits to the community, participants implied, will result if changes really happen. Nonetheless, the increased knowledge gained during this process should be of value, they said. Participants gave examples of how the research had benefited the community, such as more people planting mangroves, and more people now attending CFA meetings.
Reflecting on these conversations, we came up with a feedback SMS that conveyed the most useful information, and sent it to 200 of the 400 people who had participated in the survey. We then called all 400 participants to ask a few questions, before sending out the text to the remaining 200 people. In total, 338 people agreed to complete the follow-up survey.
The SMS (sent in both English and Kiswahili) read: “In February you participated in our research on CFAs and forests in Kenya. Thank you for participating, your insights were very valuable to us! We have reported the following to Safaricom, KFS and other stakeholders: 1. The digital solution can provide more knowledge on conservation through training and information sharing. 2. More transparent and predictable financial incentives for dedicated CFA members can speed up tree-planting. Safaricom and KFS took this seriously and are considering the best solutions. Please share these findings at your next CFA meeting.”
We looked at the difference in responses to the follow-up survey between those who received SMS feedback in advance, and those who received it after the phone call to measure the impact of sharing research results. We found that:
- Giving participants feedback makes them significantly more likely to say they were treated respectfully (p<0.01), and significantly less likely to say that they find it difficult to speak up in community meetings (p<0.05).
- It has no significant impact on their desire to recommend Busara, their likelihood to change something in their lives as a result of their findings, or their motivation to conserve the forest. (Desire to recommend Busara and the motivation to conserve the forest were already very high).
We take from these findings that giving post-study feedback makes a real and measurable difference to research participants. This is true even if it is only in the form of a brief text message, an extremely low-cost mode that all researchers should be able to do. Soon, supported by Feedback Labs, we’ll be conducting a new study to examine which bits of feedback are most important to people, to help craft the most useful and impactful messages. All this work is being conducted under Busara’s new agenda on research ethics in the Global South, ‘Participant Voice First’.
