Confronting Economic Data: “Natural” Randomized Controlled Trials
In an ideal world we could observe a set of agents with a policy intervention and then observe the same agents during the same time period without the intervention. We could then directly measure the effect of the policy intervention. Randomized controlled trials are an approximation of this gold standard which I have already discussed. The second alternative is an approach that economists have developed which makes use of “natural” experiments. These are situations where treatment occurs randomly.
The classic example of this approach is the paper by Angrist on the effects of the Vietnam war on labour market outcomes of draftees. The idea is simple enough: young men were determined to be eligible to go to Vietnam or not by a random lottery. The Random Sequence Number generated a random element to whether you got drafted or not.
Why is this useful in this setting? Imagine that being sent to Vietnam was not determined by lottery but rather by choice. Certain people would decide to enter the military, we would expect to observe people with bad outside options to enter the military. A good outside option would be to go to college and get a high paying job. We would expect those people to skip the military. As a result, if we looked at wages of those who went to Vietnam and those that did not, we would see a negative correlation. This would be driven by the fact that many non-Vietnam veterans would have gone to college and not due to their lack of military service. In this setting we have no hope of figuring out what the Vietnam war did to peoples’ earnings.
So, in reality we have randomized assignment of treatment and non-treatment much like in a Randomized Controlled Trial. Angrist then suggests using a Wald estimator, a fancy name for simply taking the average earnings between the draftees and non-draftees and then dividing that by the difference in proportions of each group that actually entered the military. The last step (dividing by proportions) corrects for the fact that some people who were eligible did not go to Vietnam and some that were not volunteered to go.
It’s a neat approach to identifying the effect of the Vietnam war on earnings. Unsurprisingly, Vietnam veterans have lower earnings than non-veterans. However, we have a problem now, which also affects RCTs: there is no clear way to interpret our findings. What is the mechanism that generates these lower earnings? Without understanding the underlying mech- anism it is not clear what the policy response should be? Do we pay soldiers more? Do we pay them for education? How do these findings apply to our world today?There are too many possible explanations for this discrepancy in wages. However, what we have learned is that there is a systematic difference in earnings between these two groups and this finding is robust.
So, where do we go now? I have basically highlighted the major problems with RCTs and “Natural Experiments”. The first is that, unlike in clinical trials, RCTs cannot create convincing counterfactual worlds. “Natural Experiments”, such as the Vietnam lottery, allow us to use randomization that exists in the world, to identify patterns in the data, but again it is not clear how to interpret these results (this criticism applies to RCTs as well). The final step is to convince you that it is possible to use data to estimate policy invariant parameters of economic models and provide deeper insight into real-world phenomena, without costly experimentation or the search for “natural randomization”.
