Investigating a History of Impact
This post describes a retrospective review into the policy impacts of CEGA-funded research, conducted during Summer 2020. It sheds light on some of the key factors that go into measuring, tracking, and communicating the impacts of our work. The post was written by CEGA Program Manager Sam Fishman, with input from Executive Director Carson Christiano. Special thanks to CEGA Policy Intern Jose Pinilla Bustamante for his help implementing the review.
At CEGA, our mission is to improve the lives of people living in poverty. We do this by collaborating with our network of affiliated researchers to generate evidence that drives better policy.
Which begs the question: what are the impacts of CEGA’s impact evaluation research? It’s time we turned the microscope on ourselves.
CEGA has begun to track — at least in a more systematic way — the extent to which the insights we produce are ultimately taken up by governments, NGOs, and other policy-making institutions. We go beyond a simplistic view of “impact” by asking: Were research findings reviewed in policy meetings, or cited in policy documents? Was the intervention applied, scaled, or adapted in another country or context? Did an activity build the capacity of scholars from low- and middle-income countries to generate evidence? Or the capacity of decision-makers to use evidence? Were impact evaluation or research transparency best practices institutionalized?
Knowing the answers to these questions can help CEGA make better decisions about the kind of research — and research partnerships — we support. We also suspect this information will be useful to our donors as they make consequential investment decisions.
Alas, measuring the broadly-defined “impacts” of CEGA’s work is complex. It’s not every day that a study shows that X leads to Y, which then prompts a policymaker to do exactly X (though this does happen). Often, the pathway from research to impact is circuitous, muddling the precise role of the research in affecting policy change.
For example, a rural electrification study in Western Kenya led by CEGA Faculty Director Ted Miguel and affiliate Catherine Wolfram revealed that the demand for grid connections among rural households was far lower than expected. According to Wolfram, “We showed [Kenya Power] the data, and shortly after, they changed their approach to make connections free. The move seems consistent with our study, but we have no evidence of causality.”
The timing of policy impacts is also important. Effecting change takes time, and impacts often don’t materialize until long after the supporting grant has ended. Recording these deferred impacts requires significant coordination and follow-up, sometimes years after a final report is submitted to the donor.
In an effort to better understand CEGA’s historic policy impact, we set up a retrospective review involving 13 interviews with 9 CEGA-funded research teams — some of whom had completed their studies years ago. This was a largely qualitative exercise involving one-on-one, hour-long conversations with researchers and policymakers. What emerged were more complete stories about the influence of specific projects, as well as salient lessons about how CEGA can better define, track, and drive policy impact.
Here are three key takeaways:
#1: Projects co-created with decision-makers — often after years of relationship-building — yield the most exciting results.
It won’t surprise anyone working in the development research space that patience, long-term commitment, and strong local partners allow researchers to ride the waves of political change and stay on policymakers’ radars long enough to make an impact.
In Mexico, a CEGA-funded research team led by Joyce Sadka (Instituto Tecnológico Autónomo de México) influenced a major labor reform by building connections with bureaucrats and labor courts over the course of many years. This was in part due to Sadka’s decades of experience in Mexico studying labor law and cultivating relationships with local lawyers, judges, and other judicial personnel.
In another interview, we learned that UC Berkeley PhD student Ben Krause spent six years in Haiti prior to conducting a study exploring civic incentives for tax compliance. Despite a volatile political landscape, living in Haiti for an extended period allowed him to forge deep relationships and build trust with decision-makers. As a result, Krause was able to co-create research with committed government partners. By the end of his project, he received numerous calls from local mayors interested in pursuing a similar program.
Multiple researchers reported that making more funding available for scoping meetings with partners would help align projects to real world implementation and policy processes. Additionally, CEGA is investing more heavily in institutional partnerships with low and middle-income country governments and their multilateral counterparts — an effort we hope will benefit our research network when relevant opportunities arise — and prioritizing well-established policy partnership in RFPs.
#2: Don’t underestimate opportunities for policy impact during the course of project implementation.
Publishing evidence can take a long time, and implementers and policymakers often have immediate program decisions to make. Our review illustrated that researchers can generate impact independent of long-awaited study results if they seize on opportunities to inform and assist partners along the way.
One example is a women’s civic training module in India with the NGO Pradan. At first, Pradan was hesitant about implementing a randomized evaluation. CEGA affiliated researcher Soledad Prillaman (Stanford) was willing to listen to the NGO’s needs, and then dive head first into designing the kind of training module that would be most useful for Pradan. In other studies, researchers who were willing to help design and implement the interventions they were studying found similar opportunities for impact by providing technical assistance to their partners.
This insight is one reason CEGA launched our new Targeting Aid Better initiative. The Targeting initiative diversifies CEGA’s approach to generating policy impact by prioritizing direct support to governments (in this case, by helping them dynamically target aid using big data), in addition to collaborating on a learning agenda. In future RFPs, CEGA plans to ask applicants about these direct support opportunities, in addition to the longer-term policy implications of the work.
#3: Donors and partners can be instrumental in amplifying the policy impacts of research.
CEGA Faculty co-Director Ted Miguel’s research on the long-run impacts of deworming, also known as the the Kenya Life Panel Survey (KLPS), demonstrates how partnering with influential donors and partners — including governments, NGOs and multilaterals — can go a long way towards expanding the reach and take-up of evidence. Throughout the KLPS project, major donors like GiveWell have taken interest in the results and helped to emphasize their significance to the broader health economics community. This kind of engagement (and investment) from donors can help research insights disseminate more quickly and influence a wider network of decision-makers and partners in new contexts.
Of course, it can be tricky to track policy impacts even for immensely influential studies, because the results tend to blend in with other, complementary contributions to knowledge and policy. For example, the deworming research led by Miguel’s team would likely not have had the far-reaching health impacts that it has — over 280 million children dewormed in 2019 alone — if not for the tremendous efforts of our partner organization, Evidence Action, to build relationships with policymakers in India, Kenya, Nigeria, Ethiopia, and Pakistan. Rather than attribute specific impacts to one study, we have found that in most cases, it’s important to convey a more holistic picture of how a given policy change came about.
During our interviews, we learned that at the end of a study, many project teams don’t have the funding and incentives to work intensively with policymakers to help them adopt promising research insights.
Throughout our retrospective review, we uncovered countless tidbits of information regarding CEGA’s historic impacts that were missed by traditional, shorter-term M&E processes. While CEGA caught wind of many of these developments through informal channels, we didn’t have a complete picture, or a process for systematically tracking and communicating policy impacts beyond the life of the CEGA grant.
Now, in an effort to better understand the conditions that lead to policy impact, CEGA is formally integrating policy impact tracking into our standard reporting practices. Additionally, we plan to scale up our retrospective review process to a larger sample of CEGA-funded studies in the coming year.
Meanwhile, we seek to constantly integrate lessons learned into CEGA grant-making and dissemination activities. For example, we plan to prioritize research for funding that leverages strong policy relationships, responds to well-defined policy needs and questions, and incorporates activities that meet short-term demand for technical support. Further, we will work with teams to lay out clear plans for evidence synthesis and dissemination, so that opportunities to deliver evidence to decision-makers at the right times are not missed.
Of course, the data we collected through our retrospective review did not yield a definitive formula for how research can drive policy impact. Still, as a result of the “first-mile” and “last-mile” efforts detailed above, we expect the collection of impact stories on our website to grow.
