“Cash benchmarking”: A Youth Unemployment Program
Preliminary findings from the 18-month follow-up survey in Rwanda and how this methodology may inform how we measure the success of aid programs.
This post features an interview with CEGA affiliate Craig McIntosh (UCSD) and his collaborator Andrew Zeitlin (Georgetown University) on the midline results of their survey comparing the impacts of a youth employment program to unconditional cash transfers in Rwanda, moderated by CEGA Program Manager Sam Fishman.
CEGA affiliate Craig McIntosh (UCSD) and Andrew Zeitlin (Georgetown University) just released the results of an 18-month follow-up survey in Rwanda, comparing the impacts of a USAID youth employment program, Huguka Dukore (HD), to unconditional cash transfers implemented by the NGO GiveDirectly. This new evidence — funded by USAID through UC Berkeley’s Development Impact Lab, and generated in partnership with Innovations for Poverty Action (IPA)— could inform how we measure the success of aid programs at a time when hundreds of new poverty interventions and social protection schemes are being rolled out across the world in response to the Covid-19 crisis.
We caught up with McIntosh and Zeitlin to ask some questions about the results of the study, and what the “cash benchmarking” approach can tell us about the cost-effectiveness of development programs.
Sam Fishman: What does it mean to “benchmark” a program against cash? And why is USAID interested in cash benchmarking?
Craig/Andrew: Cash benchmarking is based on the idea of comparing the effect of a program to what would happen if you simply gave the cost of the program to the same beneficiaries. This makes sense from a welfare perspective because unconditional cash transfers show you what the beneficiaries themselves value. It makes sense from a comparative perspective because cash transfers have emerged as one of the best-documented development interventions and so can serve as a kind of ‘index fund’ for development spending (stock market index funds have no management costs and thus set a baseline performance level which managed funds are pressured to surpass). USAID is interested in the question because it helps them understand how they can allocate scarce resources to deliver benefits in a cost-effective way.
S: In what contexts is cash benchmarking appropriate (or not appropriate)?
C/A: Since cash transfers are typically targeted at households or individuals, they are most relevant to other forms of programming that are micro-targeted and are intended to generate benefits on related outcomes (consumption, asset investment, schooling, child welfare, etc.). For circumstances where markets are lacking (for labor, output, or critical inputs), then simply increasing income may not be effective and interventions centered around infrastructure, training, or market depth may be more relevant.
S: USAID is funding a number of other cash benchmarking studies with GiveDirectly (Malawi, Liberia, DRC), including your earlier study involving a sanitation and nutrition program in Rwanda. What makes this study different?
C/A: This suite of studies will provide vital information about the generalizability of cash transfer impacts across contexts. So together, these studies may help USAID simulate the benefits that would be realized in the Sub-Saharan African context for a cash transfer of any amount. Our two studies in Rwanda uniquely feature a head-to-head comparison of an in-kind intervention and cash in identical samples (the prior study compared a child nutrition program to cash, and this one compares a workforce readiness program to cash). We are also able to examine complementarities because we cross-cut the cash transfer intervention with cash, and we have substantial randomized variation in the amount of cash transferred. This means our study allows us to make an unusually rich set of cost-equivalence and cost-effectiveness comparisons across programs and across scales of investment, while speaking to two fundamental theories about the bottlenecks to productive employment, in particular skills and capital.
S: Based on these (preliminary) 18-month results, it appears cash is driving some positive outcomes for beneficiaries, but not others. What do these results show us about what cash can and cannot do in low-income settings?
C/A: Overall, our results on the impact of cash transfers are consistent with the large pre-existing literature: big improvements in asset values, consumption, savings, and subjective well-being. We find no evidence that cash improves cognitive or non-cognitive skills such as aspirations or business attitudes. Interestingly, the randomization of cash transfer amounts suggests that smaller transfers ($400) provoked larger increases in productive hours and income than larger transfers ($750), indicating that within this range the transfer amount starts to increase recipients’ demand for leisure.
S: How did you integrate intervention costs into the study design?
C/A: Both prior to and subsequent to the study, we costed both interventions in detail (the workforce readiness program Huguka Dukore, implemented by Education Development Center (EDC), and cash transfers provided by GiveDirectly). We built the study to make a comparison between two programs of the identical total cost to USAID, including all operational expenses. We randomized cash transfers in a range around the expected cost of HD, and then use regression adjustment after the fact to compare the HD program to a cash transfer of the same cost.
S: There’s another follow-up planned for 36 months after the intervention. How should we view these 18-month results in light of the pending longer-term follow-up?
C/A: Right now we have a very intriguing picture of a glass half full: the HD program generated really meaningful improvements in outcomes like work hours, productive assets, savings, and subjective well-being, but cost-equivalent cash generated significantly larger benefits across all of these outcomes. These two programs have launched beneficiaries on subtly different work trajectories, with HD moving people into wage employment and cash moving them into entrepreneurship. So we are fascinated to see how this picture evolves over the course of the next year and a half, but it’s still too early to tell.
S: What lessons do you think USAID (and the broader development community) should take away from these early results?
C/A: Taking our two studies together, cash transfers on the order of USD 400 per household will be a hard intervention to beat when one focuses on economic measures of wellbeing over a 12-18 month timeframe. We’ve seen something of a ‘goldilocks’ size: smaller cash transfers don’t change enough to move outcomes, while very large cash transfers hit diminishing returns as they begin to increase the demand for leisure.
S: What do these results tell us about the role cash benchmarking can and should play in evaluating development interventions more broadly?
C/A: We want to advance an agenda of providing the kinds of comparative cost-effectiveness analyses that are most useful to policymakers. One thing that makes our study unique in this regard is that we vary both programming types at cost-equivalent levels, and we vary the scale of cash transfers. This allows us to contrast a cost-equivalent approach to comparing intervention modalities with one based on cost-effectiveness ratios. In fact, in our study, cash transfers appear superior for most outcomes across both approaches, but the most cost-effective cash transfer ($410) is larger than the cost-equivalent amount ($330). One takeaway for policymakers embarking on a cash benchmarking exercise is that they need to think carefully about the kinds of programmatic trade-offs they are willing to make in order to establish the right counterfactual for their program. Are they willing to do more, but for fewer people?
