Using Rigorous Evidence to Minimize Risk (and Maximize Impact)
By Moitshepi Matsheng & Noam Angrist | Young 1ove
Young 1ove, an NGO based in Botswana, was founded to scale-up evidence-based solutions to pressing local challenges by young people for young people.
Our first program was a 1-hour class revealing the risks of “sugar daddies” — older partners who often give young girls gifts in exchange for sex. There was a dire need: a study found a quarter of young girls were dating partners five or more years older. Older partners pose serious risks: 45% of 40-year-olds have HIV. Despite the risk, 90% of adolescents didn’t know this, classifying older partners as safe and “mature.” While a daunting challenge, there was a promising evidence-based solution. An intervention that revealed that older partners were 5–10x more likely to have HIV reduced teen pregnancy — a proxy for unprotected sex and HIV — by 28% in a randomized trial in Kenya ten years prior. Although the evidence was rigorous and promising, and had been presented widely in academic circles, the intervention had never been scaled.
Motivated to address the spread of HIV and frustrated by the fact that this evidence had not been scaled up, we bridged evidence with action and brought the program back to life. Given our commitment to evidence, before scaling-up, we decided to test the intervention through another randomized trial to ensure the intervention was effective in the Botswana context. We trained peer educators as well as teachers to deliver the program. This was meant to inform both whether the program worked and the most cost-effective delivery model. The trial ran from August, 2014 to December, 2015. We partnered with leading academics at J-PAL and Baylor on the trial, and joined forces with the government of Botswana and Evidence Action to design a scale-up strategy pending results.
Grounding in evidence had many advantages as well as unique challenges. Our evidence-based approach was intended to both deliver impact and to guard against unknown risks. Through this process, we’ve learned a few things about preparing for the unknown, even when armed with rigorous evidence. For all of those NGOs and funders using evidence to make decisions, we’d love to share our experience.
Planning For An Evidence-Based Future
As the trial progressed, we worked with our partners to build a ‘pre-policy plan’ — an analogy to a ‘pre-analysis plan’ used in medical trials and in the social sciences to pre-commit to a set of actions to ensure objectivity. Such a pre-policy plan is highly unusual in the policy sphere and social sector. We found it was critical, helping us to acknowledge and plan for risk and achieve alignment among the partners. Armed with this plan, we were ready to spring into action if results were positive, and had braced for potential alternative outcomes.
Our pre-policy plan outlined three broad categories of outcomes and our response in each:
- Scenario 1: A positive result. In this case, we would scale-up the program in Botswana and beyond as quickly and effectively as possible.
- Scenario 2: A null or negative result. In this case, we would not scale-up the program and pivot in an evidence-based fashion.
- Scenario 3: An ambiguous result. In this case, we would also not scale-up the program and pivot in an evidence-based fashion.
The Results Are In
The results emerged between March-August of 2016. The results were very promising but complex. Slotting them into one of our three scenarios was not straightforward. For example, our main measure of impact, pregnancy (identified by classmates and teachers when a girl was found to be missing from school), decreased significantly, a clear positive result. This measure was the same as the one used in the Kenya trial. However, a novel and alternative measure of pregnancy — visible stomachs — had a more complex interpretation and gave reason for reservation. Did this mean results were positive or ambiguous? The lack of a biological outcome made it hard to reach a definitive conclusion. There was thought-provoking debate but little consensus. In an effort to achieve consensus, we engaged in months of discussions and peer review.
Moreover, a rich array of locally relevant lessons emerged. For example, we found that ‘sugar daddies’ weren’t just in their 40s, they were also in their 20s. These older partners were still 3–5x more likely to have HIV, but required different messaging. In addition, when peer educators delivered the program, the results were strong, but when teachers delivered it, less clear.
Our pre-policy plan proved useful and we had consensus that the results were between positive to ambiguous and that the way forward was an ‘evidence-based pivot’. However, the pre-policy plan had not outlined exactly what an ‘evidence-based pivot’ would entail. We had brainstormed possibilities verbally but not put those in writing ahead of time. Over the course of trial, personnel at funding agencies and partner organizations had changed. This personnel shift and lack of pre-specified and specific planning led to round after round of discussions, new proposals, and meetings in an effort to re-align on the exact path forward. The cost of this realignment ended up amounting to a third of the funding available. As we debated the most evidence-based way forward, our funding was drying up, the government wanted action, and the clock was ticking.
Towards the end of 2016, we decided our evidence-based pivot would be to incorporate lessons learned locally and rigorously to improve the program, such as message risks of older partners of all ages and focusing on youth-led delivery. We would re-test the program measuring impact on HIV/STIs as well as pregnancy and use biological outcomes rather than proxies before a possible scale-up. We recently signed a five-year partnership with UNICEF, a multi-year deal with Grand Challenges Canada, and received support from our Big Bang Philanthropy partners to engage in this exciting R&D process.
A Few Takeaways
Throughout this process, we’ve learned many lessons we continue to carry forward with us. First, the importance of continually generating evidence. Had we moved forward without a trial in Botswana, we would have transplanted ten-year old results from Kenya and lost a rich array of lessons from our local context. Now, our future programs will be informed by this contextually grounded and rigorous evidence and much stronger for it.
Second, scenario planning and aligning in writing at the outset was critical. Our joint pre-policy plan enabled alignment on broad principles among our partners, which ultimately enabled us to move forward together.
Third, we were able to make a decision on our way forward within a few months because we had a coalition that understood evidence and could review the results in real-time. Had we waited for the final academic paper to be released, decision making would have stalled for years. This coalition included multidisciplinary input from specialists in public health, economics, and education; it included academics, donors, policymakers, and implementers. We set up a structure called a Project Review Committee which met quarterly and brought all of our partners together to review the evidence as it emerged. Having the internal capacity and partners to act quickly and in-line with rigorous evidence was paramount.
There are many things we could have done better. Although we had a written pre-policy plan which moved things forward, it proved vague in the face of personnel shifts and complex results. In the future, we plan to be more specific about actionable paths forward and put as much as possible in writing. Moreover, while we made a decision within months, we waited too long for peer-review consensus before realizing it would take years to arrive. Both factors delayed decision-making and alignment.
Throughout this journey we were lucky. We had great partners who provided invaluable support and advice, who pushed us to think critically, act objectively, take risk, who stuck with us in a tough moment, and who provided catalytic support. It paid off: we are now developing a new and improved program informed by rigorous and local randomized trial evidence.
Our challenges and successes are not unique to us. Open Road Alliance’s roadblock analysis found related roadblocks in an analysis of over 102 applications. So, given our empirical roots, we’d like to ensure our experience is ever-more of an outlier and have a few summary call to actions to our friends who use evidence to design for impact and minimize risk:
- Understand how the evidence-base you are drawing on translates to your context and generate new evidence that is relevant.
- Plan for multiple scenarios ahead of time and get specifics in writing. A predefined framework can help leverage evidence to maximum effect.
- Find partners you trust who can interpret evidence for action in a time-bound fashion and be credible spokespeople.
The world is unpredictable: we’re grateful to have been able to keep impact on track.
Moitshepi Matsheng
Moitshepi Matsheng is the Co–Founder and Country Coordinator of Young 1ove. Moitshepi leads stakeholder relations, including the organization’s partnership with government. Moitshepi also leads talent for the organization and strategic initiatives. Moitshepi is a lawyer turned social entrepreneur. She is the first Motswana to receive the Queen’s Young Leaders Award. She was chosen by former President Mogae as a Pioneer of the Champions for an AIDS-Free Generation and was recognized as one of 50 Batswana Change-Makers Under 40 during Botswana’s 50th independence celebration. Moitshepi was a keynote speaker at the UNESCO Youth Forum in 2017.
Noam Angrist
Noam Angrist is the Co-Founder and Executive Director of Young 1ove. Noam is passionate about bridging the gap between rigorous evidence and scaled policy and practice. Noam previously worked and conducted research at the University of Botswana, the Jameel Poverty Action Lab (J-PAL), and the World Bank. Noam has a B.S. in Mathematics & Economics from the Massachusetts Institute of Technology and an MSc. from the University of Oxford. Noam was recognized by Forbes 30Under30, is a Global Shaper with the World Economic Forum, and was a Fulbright and Rhodes scholar.
