The Untapped Potential of Impact Evaluations in Humanitarian Contexts
By Ananda Young, HAEC
What is the right modality of program delivery? Does anticipatory action prior to a humanitarian emergency have a positive impact on a household’s ability to cope? Which methods are most cost-effective? Which methods lead to the most positive impact on long-term resilience?
Are you interested in answering questions like these for your program? Then an impact evaluation might be the right choice for you!
Impact evaluations are increasingly becoming a critical research tool to investigate potential policy interventions and programs. Impact evaluations, which causally establish the impact of an intervention on the outcomes of interest, are transforming the development sector.
Over the past two decades, they have been widely applied to improve program effectiveness, test innovative interventions, and guide program replication and adaptations. However, while impact evaluations are an established tool in development, this tool is not yet used regularly in the humanitarian sector.
What is the context?
Around the world, conflict, crisis, and disasters threaten the lives, rights, and security of millions. In 2022, OCHA estimated that 274 million people were in need of humanitarian assistance and protection, an increase of 39 million compared to the previous year. Demand is only expected to rise as climate change exacerbates the frequency and severity of extreme weather events and instances of violent conflict remain high.
By 2030, 46% of the world’s poor are expected to live in fragile or conflict-affected areas. Every year, international donors spend approximately USD 30 billion on humanitarian assistance and emergencies. However, OCHA estimates that it would require $41 billion to reach only 67% of those in need.
Thus, every dollar counts.
However, there is a notable absence of rigorous, high-quality evidence to inform this funding. A study commissioned by USAID in 2012 found that only 3% of the evaluations were impact evaluations.
Why are impact evaluations important?
While most see the need for humanitarian assistance funding as indisputable, significant program design and implementation questions still remain. Impact evaluations present a unique opportunity to increase the quality and amount of data and to stretch every dollar to its maximum potential.
In the past decade, impact evaluations have revealed that the impact of microfinance was being overestimated, that the impact of deworming initiatives has diverse and long-term effects on a child’s life, and that cash transfers are deeply empowering and effective. Just as impact evaluations revolutionized the development sector, they offer similar gains to the humanitarian sector.
The lack of impact evaluations in the humanitarian sector presents a huge missed learning opportunity. The need for such evidence is more critical than ever as demand for humanitarian assistance is set to skyrocket, funding is set to increase, and interventions are set to be scaled.
Are there examples of effective impact evaluations in humanitarian settings?
Suppose you are an implementing partner aiming to deliver emergency cash assistance to those recovering from a recent natural disaster — does providing cash as a single payment or as multiple transfers affect the use of financial resources? Is one more effective in promoting economic recovery? An impact evaluation conducted by Causal Design measuring Mercy Corps’ emergency cash assistance program in the wake of Typhoon Haiyan in 2013 was able to answer exactly that.
A randomized controlled trial compared the distribution of one large single cash transfer to three smaller separate transfers and found that the lump sum transfer resulted in increased ownership of small productive assets for the household. Households within the evaluation that received the single cash transfer invested in more livestock when compared to households that received the same amount of cash in three payments. These findings demonstrated to Mercy Corps that cash assistance as a lump sum payment rather than as multiple smaller payments was more effective in promoting investments in productive assets, and thus in promoting a household’s economic recovery in this context.
So why are there so few impact evaluations in humanitarian settings?
Despite the learning potential, significant skepticism remains around using impact evaluations in humanitarian contexts. The BHA-funded Humanitarian Assistance Evidence Cycle (HAEC) recently conducted a series of consultations with humanitarian stakeholders to find out why.
Through this consultation process, HAEC found that lack of incentives, implementer bandwidth constraints, ethical concerns, short programming timelines, misaligned research partnerships, and lack of funding were some of the core constraints, real or perceived, limiting the use of impact evaluations in this context. While many of these constraints are not unique to humanitarian settings, they are amplified in these contexts.
Who is addressing these challenges?
Recognizing this significant learning opportunity for the humanitarian sector, there is a growing movement to promote the use of impact evaluations as a tool for evaluating humanitarian programming. Some of the trail-blazing actors with innovative initiatives in this space include:
- Humanitarian Assistance Evidence Cycle (HAEC): HAEC is a USAID/BHA-funded activity focused on increasing the use of cost-effective and timely impact evaluations in humanitarian contexts to increase the efficiency and effectiveness of emergency food security activities. HAEC is looking to fund impact evaluations for BHA-funded emergency food security activities. Learn more here.
- 3ie’s Humanitarian Assistance Thematic Window (HATW): HATW aims to produce high-quality evidence to help inform policy and programming in the humanitarian sector. 3ie is committed to generating rigorous evidence focused on developing learnings on what is effective and efficient in delivering programs in fragile and conflict-affected contexts through impact evaluations with a special focus on innovative approaches that are gender-responsive and equity-focused.
- World Food Program’s Office of Evaluation’s (OEV) Impact Evaluation Strategy: The OEV, in partnership with the World Bank’s Development Impact Evaluation (DIME) unit, is supporting interested WFP country offices to conduct impact evaluations for their humanitarian programming. The OEV will support offices to identify opportunities and design and deliver impact evaluations suitable to contexts and planned operations.
- J-PAL’s Humanitarian Initiative: J-PAL, funded by the UK’s Foreign, Commonwealth, and Development Office (FCDO), has committed to generating actionable evidence to improve the effectiveness of humanitarian operations. They have identified priority evaluation questions aligned with the humanitarian sectors’ priorities and plan to foster research and knowledge partnerships, particularly highlighting the applicability of randomized evaluations and building capacities of practitioners.
- IPA’s Peace & Recovery (P&R) Program: P&R supports field experiments and research related to reducing violence and promoting peace; reducing fragility; and preventing, coping with, and recovering from crises. IPA prioritizes studies, including impact evaluations, that develop, illustrate, or test fundamental theories of peace, violence, and recovery.
What can I do?
- Evaluation in Action (EiA) Series: If you are interested in having an ongoing or recently completed impact evaluation conducted in humanitarian contexts featured in HAEC’s EiA series, you can submit information here.
- Apply for funding for an impact evaluation! If you have questions about your humanitarian program that can be answered through an impact evaluation, apply for funding here!
