Randomised Controlled Trials Won A Nobel Prize — But They’re No Magic Bullet

Experimental approaches to economics led by Abhijit Banerjee, Esther Duflo and Michael Kremer signal a transition away from grandiose, poorly researched foreign aid programmes and instead provide much-needed empirical evidence about anti-poverty interventions.

In the field of medicine, randomized controlled trials (RCTs) are used to assess the efficacy of new medical treatments. In the same way that we wouldn’t use new medicines on the general public without trialling them, Banerjee and Duflo argue we should use RCTs to evaluate the effectiveness of development interventions before implementing them on a larger scale. This involves individuals from a random sample of the population being randomly assigned to different groups: one group receives the ‘treatment’, and the other is a control group which doesn’t receive treatment. Since the groups are randomly selected, they are comparable and differences in their outcomes are caused by the effects of the intervention. By using RCTs across many locations and with different interventions, the range of theories to explain the data can be narrowed down, and economists can implement policies based on verifiable evidence.

Applications of RCTs

Microfinance

Microfinance provides business loans to low-income individuals who would otherwise have no access to financial services, aiming to increase investment and allow recipients to become self-sufficient. To test its effectiveness, a randomised experiment was conducted in Hyderabad, a city in India where there is very low access to microfinance institutions (MFI). 52 areas of the city were randomly selected for opening an MFI branch, and another 52 were part of the comparison group which didn’t access microfinance.

Results found that microcredit incrementally increased small business investment and only increased profits for the most profitable pre-existing businesses. Those who received microfinance were more likely to start a business and purchase large durable goods, such as refrigerators or televisions. However, there was no significant change in income nor household spending (both proxies for material wellbeing) between treatment and comparison groups after 18 months or 36 months. There was also little change in health, education and women’s empowerment outcomes. Therefore, RCTs reveal that microfinance may not be as much of a miracle tool to combat poverty as it has been thought.

Health

RCTs have been instrumental in providing solutions to poverty traps created by poor health. For example, children who are often ill are unable to do well at school, so they may earn low wages as adults and remain trapped in poverty. Despite it being available for free, preventive healthcare against diseases is seriously underinvested by poor families, with 2–3 million people dying from vaccine-preventable diseases every year. In a set of Indian villages, it was found that less than 5% of children were receiving the basic immunization package.

This low vaccine uptake was because staff at health centres were often absent from work, meaning people were deterred from getting jabs. A random experiment was conducted to find whether the use of mobile vaccination clinics, with staff always on site, would make a difference. Indeed, full immunization was 17% in the villages randomly selected to have access to these clinics, compared to 6% in control villages.

It was also found that by offering a free bag of lentils and steel plates after completion of vaccinations, the immunization rate significantly increased to 38%. Paradoxically, offering lentils also reduced the cost per immunization, since the nurses were busier and their efficiency rose as a result. This demonstrates how small, precise policy alterations from RCTs can provide meaningful social benefits.

Criticisms

A serious issue for RCTs is its lack of external validity. Estimates from the effect of a policy from where the RCT is administered cannot always be extrapolated to different places and people, since different populations have unique social, environmental and institutional circumstances. For example, a labour market policy in a high-unemployment country cannot inform or be utilised in a low-unemployment country, or a country with different cultural traditions about income sharing within families. Furthermore, performing experiments in many different places may not resolve this problem since we don’t know whether the places themselves are being randomised. Economists including Angus Deaton have criticised how RCTs are unjustifiably considered as a ‘gold standard’ and placed at the top of the hierarchy of empirical methods in social sciences.

It has also been argued that by focusing on smaller questions, development economics is losing ambition and losing sight of the bigger, more important questions. Excessive focus on randomized experiments may mean that economists miss out on the structural factors which have caused and alleviated poverty, involving politics, markets and institutions. The general decline in global poverty over recent decades was because of reforms in China (resulting in its rapid growth) and changes to global trade, which RCTs did not play any role in and cannot investigate.

In response, the bigger questions are undoubtedly important but, as opposed to smaller questions, there is little meaningful way to study them. The alternative to an RCT in the past was a large uncontrolled experiment where ineffectual development interventions were made without verifying whether they work.

Perhaps the most pressing issue for RCTs is the ethical concerns about treating poor people as mere subjects in experiments. In Kantian deontology, this would be considered as immoral since it uses people as a means to an end, rather than an end in themselves. Or, in Deaton’s words, he worries about the implications of experiments done ‘by better-heeled, better educated and paler people on lower income, less-educated and darker people.’ Therefore, in order to avoid exploitation of those in poverty, it is extremely important that participants of RCTs are told what the study is about, have a voluntary choice to take part and can drop out at any time, as per the standard practice. Researchers also require approval from independent ethics boards, both in their own country and where the study is conducted.

It may also be considered unethical to give out beneficial treatments from RCTs (such as cash transfers) to people on a random basis, when some of them are poorer and in more urgent need of them than others. On the other hand, some experiments could cause direct harm to participants and affect people who never consented to being part of the study. Yet this objection misses out the fact that although intuition may suggest a certain policy is beneficial (or harmful), an RCT could show that the policy has zero overall effect. Also, RCTs give information about the size and cost of the intervention. Without this crucial evidence, a policy may be used which works but it is less cost-effective (and thereby benefits fewer people) than an alternative, which in itself may not be ethical.

Overall, RCTs are useful to gather solid evidence about development policies on a smaller scale, which make a genuine impact when applied more widely. However, we shouldn’t consider them as a revolutionary single idea to solve poverty, nor can we expect them to provide answers to broader questions about the reasons for differences in human development outcomes.