Stories of Impact: a Research Program’s Policy Influence 5 Years Later
Leah Bridle, Associate Director of Research, and Anya Marchenko, Senior Program Associate, highlight policy impacts from a set of randomized evaluations (RCTs) co-managed by CEGA and Oxford Policy Management, funded via the Economic Institutions and Development (EDI) program (2015–2021)¹, and reflect on why the majority of studies in this portfolio succeeded in informing policy decisions. You can read their full “Impact Report” here.
EDI is a research program designed to test key institutional reforms and impact policy. CEGA helped competitively select studies that responded to the demands of decision-makers while expanding global knowledge. The research agenda was ambitious: to rigorously evaluate how institutions could use information and incentives to more effectively provide public services, collect taxes, mediate legal disputes, and carry out other fundamental state functions.² As the program wraps up we are seeing a significant number of funded studies achieve some kind of policy impact — from using evidence to make elections more democratic to adopting a new software to raise tax revenue nationwide. We believe this level of policy engagement is directly linked to researchers investing significant time in-person with their policy partners, co-creating the intervention with their partners, and in many cases, providing valuable technical assistance beyond data collection for the experiment.³
From the start, CEGA made a concerted effort to select a portfolio of impact evaluations that would be both rigorous and poised to inform policy. Our 2015 and 2016 research competitions were open to any researcher co-designing field experiments with governments or civil society partners in Africa, Asia, and Latin America. This, along with targeted outreach to hundreds of scholars worldwide, yielded a high quantity and quality proposals. We ran a multi-stage blinded review system to carefully distill more than two hundred proposed studies to the set of thirty studies we could fund.
Funded evaluations were technically sound, logistically feasible, promised to contribute to the evidence base, and directly informed ongoing policy debates. In several cases, the experiment was embedded within governmental program implementation at scale. For many evaluations, research designs and trust had incubated over multi-year engagements between researchers and their policy partners. In some cases, the timing of the EDI research competition rounds seeded early-stage partnerships, which were subsequently supported when they grew ready to experiment at scale.
Tracking our impacts
In an effort to better understand the conditions that lead to policy engagement or impact, CEGA formally tracks studies’ contributions in a centralized database, which we populate via standardized, bi-annual research team reporting. Our templates are specifically designed to tease out detail and nuance of researcher-policymaker engagements from various angles, to give us insight (from where we sit in Berkeley) about how funded studies around the world are progressing scientifically, deepening engagements with policymakers, and informing policy decisions. We distill reported information into three categories for basic tracking: i. policy-oriented dissemination (sharing evidence); ii. direct participation in a specific decision-making process (a “direct forum”); and iii. actual policy change (can be programs, budgets, etc.)
To our knowledge, there are few programs that have been tracked in this way in real-time and the impacts reported have been compelling. For example, we estimate that 47% of the thirty EDI-funded studies were closely co-designed with their implementing policy partner, 37% directly participated in a specific policy decision-making process, and 17% have already directly influenced government policy.
Of course not every project was successful; these are field experiments conducted in the real-world that can be undermined by a host of factors (including but not limited to a global pandemic). What’s more, multiple projects are still working with partners to understand and realize policy lessons. And yet, we’ve already seen eighty recorded instances of engagement or impact from our EDI research portfolio. The graph below shows how our database grew, by impact record type, since launching this internal tracking system in 2017:
Stories of Impact
Below we highlight a range of notable impacts from the EDI program, organized by continent. To learn more about each project, the researchers and partners driving them, and their routes to impact, click on the embedded links below or see more detailed discussion in our full Impact Report.
(East and West) Africa
Government planning to scale effective performance pay scheme for teachers in Rwanda
The evaluation showed that newly recruited teachers working under a performance-pay scheme (who were paid based on their attendance, preparation, pedagogy, and student learning gains) attended class more often, had more effective classroom teaching practices, and were not more likely to quit (during the two years of the experiment) than teachers working under fixed-wage contracts. The Ministry of Education has signed a letter of intent to evaluate a potential path to scale for this program and the research team is involved in drafting a national Teacher Recruitment Framework.
Read more about the project here.
Making elections more democratic in Sierra Leone
Researchers worked directly with political parties to vary how much input registered voters had in selecting candidates for the 2018 Parliamentary elections. Results show that learning which candidates voters prefer in primary races increased representation (defined as the candidates selected by the party being those preferred by voters). The nation’s two major political parties are now considering changing their constitutions to make Parliamentary candidate selection more democratic in response to research results.
Read the Impact Story here.
Helping the Senegalese national tax authority modernize property tax collection
Conducting an impact evaluation while introducing a digital tax records and management system is directly helping the Senegalese tax authority increase property tax compliance rates in Dakar. The researchers’ embedded, ongoing collaboration with the Senegalese administration has led to government interest in scaling-up the most effective version of the program per the results of the evaluation.
Read more about the project here.
Improving tax audit selection methods in Senegal
Another randomized evaluation in progress also shows promise to influence Senegalese tax policy. Researchers designed an algorithm that selects firms for tax audits, predicting the likelihood that each firm is evading taxes based on data from corporate income tax, value-added taxes, and other data (rather than relying on human discretion). Their work with the tax authority shows promise to have high-level input into the design of tax auditing policy for all of Senegal, which is crucial to meeting the government’s goal of bringing down the tax evasion rate.
Read more about the project here.
(South) Asia
Gender-based civics training for women scaling across India
Local NGO Pradan worked with a researcher to design and evaluate a new “gender training” to inform women in rural India of their rights and encourage them to participate in local politics. Now, in part based on findings from the pilot and Pradan’s advocacy, the Indian government is testing the feasibility of scaling these trainings across the country.
Read more about the project here and the Impact Story here.
First-ever training of marriage officials to protect women’s rights in Pakistan
A rigorous evaluation of a program designed and run by the Punjab Commission on the Status of Women involved the first-ever registry and training of all marriage registrars in the province of Punjab. The registrar training resulted in a 166% increase in the number of registrars who could correctly identify key rights women have in marriage. The Council on Islamic Ideologies is now reviewing the marriage contract to make it more “woman friendly,” an important step to improve access to legal rights for millions of women.
Read the Impact Story here.
An app effectively improving social protection wage payments extended to Bihar
India’s MGNREGA, one of the largest social protection programs in the world, is chronically delayed in sending wage payments to the fifty million workers who rely on them for subsistence. Researchers evaluated a new app, PayDash, that aims to improve the flow of information and accountability between the individuals responsible for processing payments. Introducing the PayDash app reduced the time required to process wage payments by 24% in high-delay areas. Given these promising impacts and the policy relevance of the issue, the team has already expanded their app to an additional state, Bihar, at the request of senior officials.
Read more about the project here.
Extending a tax compliance tool to two new states in India
Researchers developed an algorithm to more effectively identify “bogus” firms in Delhi that facilitate VAT tax evasion. Specifically, the researchers estimate that if the tax administration had used the algorithm to select which firms to inspect in the past, they could have prevented enough VAT tax evasion to increase revenue by US$15–45 million. Officials from Punjab and another Indian state have approached the research team to adapt this approach.
Read more about the project here.
Latin America
Large labor reform law passed in Mexico
Researchers wrote a part of Mexico’s large labor reform bill that passed in May 2019 using insights from their research. The legislation helps citizens better navigate the Mexico City Labor Court, requiring workers to meet with a conciliator before going to trial, providing workers with more opportunities to settle their cases (lowering the burden of legal fees), and increasing transparency around court proceedings.
Read the Impact Story here.
Stay tuned as additional projects finish their analysis, engage with direct partners on their results, and we post more Impact Stories on this blog and on CEGA’s website.
[1] The Economic Development & Institutions (EDI) program is funded through generous support from the United Kingdom’s Foreign, Commonwealth & Development Office. Oxford Policy Management leads EDI in partnership with Paris School of Economics, University of Namur, Aide a la Decision Economique, and CEGA.
[2] For details, see the research agenda setting “white” paper: At the Intersection: A Review of Institutions in Economic Development
[3] You can read a discussion of these common trends we see in this (and other) programs in the post “Investigating a History of Impact”.
