How do we know if programs have lasting impacts? A groundbreaking study in Jamaica helps make the case for long-run studies
While empirical evidence is increasingly being used to inform important policy decisions, the long-term impacts of most interventions are unknown. Long-run studies that follow up with participants years — even decades — later can shed light on the (cost) effectiveness of all kinds of programs and help officials make grounded policy choices for their constituents. Using a well-known example from Jamaica, CEGA Director of Operations Lauren Russell makes the case for long-run studies.
Early Childhood Development in Jamaica
In the mid 1980s, Jamaica experienced high levels of poverty. One out of every three Jamaicans lived below the poverty line [1], over 25% of the population was unemployed [2], as much as 10.5% of children were malnourished [3], and many families lacked access to quality education, especially early childhood education and development services [4]. Policymakers and researchers alike wanted to understand how to help children from low-resource communities reach their full potential.
The importance of early childhood development (ECD) is well known. Children who don’t receive adequate nutrition, stimulation, and emotional support early in life tend to lag behind more advantaged peers, diminishing their chances for success later in life [5]. Researchers working in Jamaica hypothesized that investments in ECD could foster long-term productivity and wellbeing, and potentially address some of the underlying causes of intergenerational poverty.
And so, the Jamaica Early Childhood Stimulation intervention was launched with the goal of advancing emotional and cognitive development for malnourished children living in poverty, and improving their socio-economic and health outcomes in the short-term and later in life. 129 children aged 9–24 months living in poor neighborhoods of the Jamaican capital, Kingston, were enrolled in a randomized controlled trial (RCT) to test this new approach. Over two years (1987–1989), community health workers made weekly home visits and taught mothers how to play and interact with their children in ways that were known to boost learning, psychosocial development, and self-esteem. For example, mothers were encouraged to talk with their children, to label things and actions, and to play educational games with their children, emphasizing language development and praise.
The Challenge
Long-run studies can be powerful tools. By tracking how an intervention affects people’s lives over many years, long-run studies generate meaningful, policy-relevant evidence on the drivers of economic productivity, living standards, and other measures. They are also necessary to accurately estimate spillover effects — in other words, the total impact of a program, including the impact on those who receive it and those nearby who do not — as well as social and economic returns on programmatic investments. If parallel data collection and tracking protocols across multiple interventions are established, long-run studies may also produce generalizable insights on the lasting impacts of related interventions and strengthen external validity [6].
Importantly, long-run studies can provide a unique environment in which to conduct additional research at a low cost that would otherwise be too expensive. By providing information on a broad set of outcomes for a large sample of respondents over time, long-run panels can create rich opportunities for layered evaluations (e.g. impact evaluations of programs or policies applied randomly to the sample using panel data as the baseline and endline) as well as natural experiments (e.g. assessments of how population-level changes, such as the onset of a pandemic or changes in policy, affect members of various sub-groups).
Unfortunately, it is not uncommon for the short-term impacts of programs that promote welfare for low-income beneficiaries to fade or even disappear over time. By the same token, it can take years for the impacts of programs and policies — especially those geared towards building human capital or supporting cognitive development — to fully materialize. For the Jamaica study described above, some of the key outcomes of interest, such as those related to labor market participation and economic opportunity, were not measurable until participating children reached adulthood, ten to fifteen years after the ECD intervention was delivered.
Despite these realities, most (development) economics studies — including RCTs — don’t measure long-run outcomes. Instead, they measure the effects of programs on a relatively short timeline, with final surveys usually occurring within one to five years. This may be a function of limited funding; the complexities of tracking respondents over time and distance; the high risk of “contamination” of comparison groups; misalignment with policy windows; or some combination of these. The result is that the majority of research projects don’t attempt to rigorously measure the long-term effects of health, education or socio-economic programs, which is critical to understanding which are most cost-effective and generate lasting benefits for people’s lives.
Jamaica, 30 Years Later
To understand the long-term effects of the Jamaica Early Childhood Stimulation intervention, CEGA affiliated professor and UC Berkeley economist, Paul Gertler, along with James Heckman (Nobel laureate, UChicago), Rodrigo Pinto (UCLA), Christel Vermeesch (World Bank), Sally Gratham-Mcgreagor (UCL), Susan Walker (U of West Indies) and Susan Chang (U of West Indies) followed participating children for more than three decades and are releasing powerful results of a 30-year follow-up in the NBER working paper publication series.
Here’s the big finding: participating toddlers — now 31 years old — whose mothers were taught to provide cognitive and psychosocial stimulation, now earn 37% more as adults compared to a control group — even more than at age 22 where researchers observed a 25% increase in earnings.
Essentially, better parenting skills can largely compensate for the early disadvantages associated with poverty.
This is the first long-run, experimental evaluation of an ECD program in a low- and middle-income country (LMIC), and suggests a low-cost way to dramatically reduce inequality later in life. It provides a rare glimpse at the effects of an early childhood intervention over the decades — and gives policymakers and development experts tangible proof of the potential effects of ECD programs in this context and beyond. While long-run studies can be hard to pull off, this example underscores their intellectual and policy value, and suggests that they can be a worthwhile investment.
The Case for Long-Run Studies
There are a handful of other notable examples that showcase the value of long-run studies, especially in the United States. One is Heckman’s evaluation of the iconic Perry Preschool Project — a 1960’s precursor to the Head Start program that targeted disadvantaged African-Americans. The study showed that early childhood education could strengthen families and break cycles of poverty, even 50 years later [7].
While there are fewer examples in LMICs, a suite of consequential long-run studies is emerging. The Kenya Life Panel Survey, which began in the late 1990s, provides critical insights into the effects of deworming on improving school attendance for both treated and surrounding untreated individuals. By following the surveyed group over two decades, the research team has found that treated individuals experienced significant gains in adult living standards and earnings as well as shifts in sectors of employment and residence [8].
Another example is Chris Blattman (UChicago) and coauthors’ evaluation of a Youth Opportunities Program in Uganda, finding that impacts for grant recipients recorded after four years dissipated after nine years [9]. Tania Barham (UColorado) and coauthor’s evaluation of a conditional cash transfer (CCT) program in Nicaragua showed earning, labor market, and family planning gains for men and women after ten years in Nicaragua [10].
These are critical findings with big implications that would not have been identified without the commitment of researchers, donors, and policymakers to long-run follow-ups. It is becoming increasingly clear that very long follow-up periods — sometimes greater than a decade — may be necessary to confidently estimate program impacts on lifetime earnings, health, wellbeing and many other outcomes, and make meaningful contributions to policy.
