Ted Miguel on the Merits of Long-Run Impact Evaluation
CEGA Communications Associate Dustin Marshall talks with Faculty Director Ted Miguel about CEGA’s evolution as a research center and its work with non-profit charity evaluator GiveWell.
D: What is CEGA’s role in promoting evidence-based development?
T: Innovation. Researchers at CEGA were some of the first to use randomized controlled trials (RCTs). We‘ve since infused our research with new types of data — from remote sensing data, to cell phone call records, to sensors networks. Our affiliates are pioneering the use of machine learning and other data science tools for development. CEGA’s role has always been to bring new ideas, new methods and new data into the mainstream of the development research community. We see ourselves as an engine for innovation in the movement for evidence-driven development.
D: How does CEGA’s mission align with that of GiveWell?
T: A lot of our early RCTs and other experiments — like deworming for example — didn’t just evaluate impact, but impact per dollar spent. Using our dollars wisely is something that’s baked into CEGA’s DNA. It’s also what Givewell and effective altruists are all about. In order to truly measure impact, and cost effectiveness, GiveWell understands that you need to use the best methods, the best data, and the best tools. Which is exactly what CEGA aims to do.
D: GiveWell recently gave CEGA a grant to study the long-run impacts of deworming. Why is this work so important?
T: It turns out there are very few existing experiments on school-based deworming that are amenable to long-term follow up. Because of this, GiveWell is supporting CEGA to conduct a fourth round of data collection using the Kenya Life Panel Survey (KLPS). This effort began almost 20 years ago, and has since helped shape the design of large-scale deworming programs in Kenya and other countries. Examining the 20-year impacts of the program will help us understand the broader returns to investments in deworming, and children’s health more broadly. In Development Economics and International Development Policy, most of our studies are short-run impact studies or medium-run at best. Understanding long-run impacts could radically change how we measure costs and benefits and ultimately design social programs.
D: GiveWell is supporting a broader effort by CEGA to identify RCTs suitable for long-run follow-up, beginning with two important domains — child health and cash transfers. Why are long-term impacts something we should care about?
T: Timing is a big factor when you consider the trade-offs between following up on old studies vs investing in new research. If you start a new project today and you want to understand impact in 10 years, you have to wait 10 years. Say you identify an RCT, experiment or government program that began 10 years ago and you have the opportunity to follow up with participants now. This approach will help us understand long term impacts in a far more timely and efficient way.
D: How will you go about this and what do you expect to find?
T: We’re putting together a team of faculty, postdocs and others to organize this effort. We’ll figure out what data exist, who has it, and which studies involving cash transfers and/or interventions designed to improve child health outcomes could be useful for understanding long-run impacts. Even if we only find a handful of suitable studies, we’ll be able to generate unique evidence of long-run impacts that could shape our understanding of the costs and benefits associated with major development interventions. There are probably dozens of studies out there that no one has thought to follow up on (or has had the resources to follow up on). We’re determined to do this with GiveWell.
D: What do you see as the long-run potential of this grant?
T: Our hope is that we find at least a few studies amenable to long-run follow-up. In these cases, we will seek funding for in-depth surveys. I’m confident there will be some, although no one knows if it will be 25 percent of studies or 5 percent of studies. Also, as we look at the literatures on cash transfers and child health, we will come up with a process for identifying suitable long-run projects that we can apply to other sectors. Finally, we’ll learn what researchers can do when they set up new project to make sure that long-run data collection is a possibility down the line.
To learn more about GiveWell’s Incubation Grants, and their investments in CEGA, see: https://www.givewell.org/research/incubation-grants.
