The New Experimental Turn in Economics
RCTs, or Randomised Control Trials have been the rage in Development Economics for over a decade now. Not only spanning several academic papers, it has also entered the public’s imagination through Poor Economics, which won the McKinsey Book of the Year award in 2011. Before that too, it had begun to gain prominence with the setting up of J-PAL (Jameel Poverty Action Lab), the Development Economics research centre at MIT.
An RCT is conceptually very simple — most of us would have heard of it occurring in clinical trials. Essentially, researchers divide subjects into 2 groups, treatment and control, at random. They then administer the drug to the treatment group, and a placebo to the control group. The whole idea is that before administrating the drug, both groups would be equivalent since they were randomly assigned. However, after the intervention, if the treatment group characteristics differ from that of the control group (for example, cancer cells growing at a slower rate), that means the drug has an effect. Using an experimental design like this, medicinal researchers can isolate the effect the drug has, giving them a clear picture of its effectiveness.
So why was this so revolutionary and still so controversial when applied to Economics? A simple question brings this to light — how do we know if foreign aid has actually helped developing countries? This was asked by Esther Duflo, a founding member of J-PAL and co-author of Poor Economics, in a TED Talk. She goes on to argue that the lack of a counterfactual (a country which did not receive foreign aid) means it is impossible to know quantitatively what impact foreign aid has had on the country which received it. What’s worse is that we don’t even know whether aid helped or harmed the country! With their slogan of Translating Research into Action, J-PAL aims to answer fundamental questions in Economics with data from experiments in the field.
However, that is easier said than done. There are several key differences between RCTs in medicine and RCTs in Economics. In clinical trials, RCTs are seen as the gold standard, whereas there is still a lot of debate about the effectiveness of RCTs in Economics.
Do they really work?
This is still an open question, though a mounting body of experiments imply it will continue at least a little longer. If one were to believe Bill Gates, he believes that J-PAL is helping his charity Melinda and Bill Gates foundation exactly identify where the foundation should be spending its money. On the other hand, prominent economists like Angus Deaton are wary of them, believing that development economists of today have overvalued these methods, which have their pitfalls like any other.
What makes them different in Economics?
RCTs in medicine, as mentioned before, are considered the gold standard for establishing casualty. Economics doesn’t have it so easy. Think of this example — imagine you are a Physicist, conducting an experiment on the properties of light in a lab in India. You get certain results. You’d be fairly certain that were you to repeat the same experiment in the US, the results would remain the same. Why would that happen? This is because the physical laws of the world apply uniformly across the planet. What is true in India, is also true in North Korea.
Now think of a simple Economics question — what is the best way to increase immunization of children in India? The founders of J-PAL conducted an RCT in Udaipur to find a solution. They came to the conclusion that providing women with 1 kg of lentils (daal) was enough incentive for them to bring their children to immunization camps. But do we know for sure this will work in another country, like say Uruguay? Was the experiment, and all the expenditure worth it, if the results can only be applied in a local context and has no external validity? We’ll revisit this in a bit.
Dealing with Human Subjects
This is perhaps the biggest drawback of RCTs, but one that simply cannot be resolved. In all physical sciences, the subjects they are dealing with do not know they are being observed (although suddenly, the Heisenberg Uncertainty Principle makes me wonder if this is really true).
Even in clinical trials, where human beings voluntarily sign up and know they are being tested upon, they cannot control their internal systems to react in a particular way. But in most randomised controlled evaluations (basically, RCTs in Economics), human subjects know they are being tested upon. This can lead to the Hawthorne Effect, in which subjects being observed alter their behaviour during the experiment, thereby compromising its internal validity.
This need not be deliberate — it is simply a natural reaction to being observed, like how a student feels if a teacher is watching while they write their exam. There is simply no way around this because there are strict rules concerning using humans as subjects for an experiment — you must explicitly take their consent. Hence, all RCTs suffer from the same drawback of aware subjects — one does not know if they are exhibiting their natural behaviour, which can render entire experiments meaningless.
Working with Human Beings
Almost as critical as the above point is that RCTs usually have a local partner organization, typically an NGO, and local field staff to run the experiment. A typical RCT has several layers of people — the Principal Investigator, the Research Manager, the Research Associate, Field Manager, Monitor and finally surveyors. While the PI designs the experiment, it is typically the responsibility of the field manager and below to handle work on the ground.
There is a strong case for the message to get diluted along the way, to the extent that the field staff could compromise the validity of the experiment. There is an important line to draw here — J-PAL, IPA, ePoD and the sort are not NGOs. They are research organizations, with professors and other people trying to experimentally answer questions of economic importance. However, they almost always have to partner with local NGOs, who can help them run the experiment on the ground by telling them which villages to target and getting the requisite permissions. But these are NGOs and are involved in trying to maximise social output, unlike J-PAL, which aims to randomise.
Even if an RCT involved significant benefit to the treatment group, the treatment group should be randomly selected, even if that means that needing people miss out on it. Local NGOs (and field staff, who are usually hired as J-PAL contract employees) may not feel the same way, especially if the need and importance of randomisation is not made explicitly clear. In a way, the result of the experiment depends on how well the field staff are trained to do and say the right things. However, papers on RCTs rarely go into much implementation detail of these experiments, meaning results simply cannot be trusted, however revolutionary they may be.
In the Search for a Universal Answer
If you were asked a question — how do we end terrorism in the world, do you think there is a simple, elegant solution which we’ve missed all along? Is there a magic pill to end all of the world’s poverty? How you answer these questions probably answer how you think about RCTs. If you believe that there does exist such an answer, you’d see RCTs as the ideal way of studying development economics, since it is currently the strongest way of assessing causality, despite all its pitfalls. If you believe that no such answer exists, RCTs probably seem like an expensive waste of time, with questionable results.
I personally believe that the answer lies in between. While RCTs cannot find a universal answer, it is simply because such a universal answer does not exist. Human beings in India behave differently than their counterparts in Africa, who behave differently from those in Iceland. They respond to different incentives, have differing notions of education and relaxation.
We have to recognise that Economics is more of a social science than a hard science, meaning it is closer to sociology, psychology and anthropology than it is to physics, chemistry and biology. As is the case with social sciences, there is no unique answer, or fundamental equation to problems. While my Physicist friend disagrees, saying we have just not found it yet, I do not think we ever will.
But while RCTs may not have absolute universality in terms of results, they do have important lessons to teach. It is reasonable to assume that something that works in India is likely to work in places with similar social conditions, like Bangladesh and other South Asian countries, where cultures and people are similar. This is not to say that all work in development economics should turn into RCTs — certain problems are best solved by time series analysis and case studies, and not by expensive RCTs. But in the absence of a universal answer, the best we can do is embrace the new experimental turn, since it is helping us answer the bigger questions, little by little.
References
- The Power and Pitfalls of Experiments in Development Economics: Some Non-random Reflections, Christopher B. Barrett and Michael R. Carter
Cover Image Source: Pixabay.com
Originally published at www.freelunch.co.in.
