An econometrician’s reply to “How do economists figure out how the world really works?”

Usually not with linear regression

Mark Thoma has a new post/column on econometrics, “How do economists figure out how the world really works.” Usually this would be great, but he kind of overstates how awesome we are and I want to clear a few things up. All of the quotations below are from Thoma’s column.

Experimental vs. observational studies

The essential difference between statistics and econometrics is the inability of economists to perform laboratory experiments where the effect of one variable on another can be examined while holding “all else equal…”

Thus, while data from laboratory experiments is usually confined to two variables and the experiments can be performed repeatedly to ensure a single observation is not a statistical fluke, economists must use historical data — a single observation — where all else is definitely not equal.

This is an oversimplification. We don’t have very much experimental data in economics; that part is true. But lots of statisticians work primarily with nonexperimental data. Surveys and polls are not experiments. Most public health studies are not experiments and epidemiology is nonexperimental. Any time-series data is usually not coming from an experiment. I could go on and on (but won’t).

There is an important distinction that I think Thoma’s trying to make here, though. Econometricians usually worry much more about how their subjects make decisions than statisticians do, and how those decisions will affect the outcomes of interest. Here’s an example of what I mean: suppose we want to know how getting a college degree affects labor market outcomes — to make it even more specific, suppose we want to know whether the starting salary of college graduates is higher on average than their salary would be if they had started to look for a job immediately after high-school. This is an example of the sort of thing labor economists research all the time.

Now, this would be easy to answer if we could do an experiment: match up pairs of comparable college applicants, admit one of them to college, and prevent the other from going to college. But that experiment is unethical and would never happen, so a naive alternative approach is to track high-school students who go to college and students who don’t go to college, and compare their salaries in four years.

Of course, if we do this, we might find that the students who attend college have on average better SAT scores, better grades… leading to the next point.

Linear regression/multiple regression analysis

For this reason, economists use a technique called multiple regression analysis. Essentially, what this means is that the effect of the treatment on the outcome is examined by including a (sometimes large) set of controls to account for all of the variables that cannot be held constant.

“Multiple regression analysis” is another way of saying “linear regression” so I’ll use that term. And every empirical researcher in every discipline learns linear regression. It’s usually the first method taught! So I don’t know why Thoma’s trying to claim that it’s a special approach used by economists alone.

Here’s how it might work in the college example. Instead of comparing all of the students who went to college against all of the students who didn’t, we might match them up based on SAT scores, GPAs, demographic information, and anything else we thought was relevant. So we look at all of the students who have a GPA on graduation between 3.8 and 4.0, an SAT score between 2200 and 2400, are white females who live in rural areas, etc., and compare the salaries in 4 years of the students in that category who went to college to those who did not. Then do the same thing for every other way of splitting up these variables. What I’ve described is a version of linear regression. If you have strong evidence of exactly how a variable like GPA affects salaries in 4 years, you can do a more sophisticated version of this approach, and that’s probably closer to what Thoma has in mind.

Again, this is a statistical approach that was invented by statisticians and is used by researchers in every empirical discipline for observational data and for experimental data alike. It is in no way unique to economics.

It also doesn’t work very well in economics, and ironically this is where econometrics starts to diverge from more conventional statistics. And the reason that it doesn’t work very well, of course, is that rural white females with high GPAs and high SAT scores who choose to go to college tend to be different from those who choose to immediately look for a job. And we know they’re different, first and foremost, because they choose to go to college.

All else equal, if someone expects to benefit from college, he she is more likely to attend. And if someone expects to benefit less from college, he or she is less likely to attend. Unless these students are completely unable to predict whether they’ll benefit from college — and “completely unable” isn’t just rhetoric here, “mostly unable” isn’t enough — the high-school graduates attending college are more likely to benefit than the high-school graduates who aren’t attending.

This biases any estimates of the effects of college attendance. Instead of getting the pure effect, we’d be estimating a mixture of the effect of attending college and the effects of all of the unobservable factors that make someone benefit from college. Even if college itself would offer no benefit to the high-school graduates who have chosen not to go, we would still (incorrectly!) measure a positive benefit.

So much for linear regression.

Now, econometricians aren’t the only people concerned about these selection issues. Paul Rosenbaum, just as one example, has written a lot about these issues, and he’s a statistician. But these selection issues are front and center in econometrics: they are the biggest and most important concern. So we don’t use linear regression very often (or, at least, we don’t trust linear regression very much. It still gets used.), and a large part of econometrics has been devoted to finding alternatives.

Empirical research in Macroeconomics

But there is one important caveat, something that is particularly problematic for testing macroeconomic theories. Most macroeconomists know the data on GDP, employment prices, interest rates, productivity and so on fairly well. So it is not very useful to build a theoretical model to explain these data, and then test to see how well it fits. Of course the model would fit. After all, why build a model that is inconsistent with the data you already know about?

And that is the key — to use data researchers did not know about when the model was built. Testing models against data that is revealed only after the model is built is the best way to do this….

Most macroeconomic models do not fit well by any measure. The point that I suspect Thoma’s trying to make is that models look unrealistically good when you evaluate them with the same data you used to build the model. When you evaluate them with new data, they start to look much worse. And this is completely true. The go-to reference is Meese and Rogoff (1983), a study that looked at whether the exchange rate models of the day could out-forecast a simple benchmark model: the number zero. (tl;dr: the models couldn’t, “no change” won. This was very surprising.)

Last thoughts

I’ll wrap up by summarizing my own thoughts. Econometrics has been very influential in macroeconomics, but not because of linear regression or out-of-sample evaluation, and it’s benefited enormously from the contributions of dyed-in-the-wool statisticians. And econometrics has been really useful in learning how to estimate and build macroeconomic models.

But we really learn a lot about the economy when the existing models fail: the Great Depression and recovery taught economists a lot about the business cycle; the sudden increase in inflation in the late 70s and the subsequent recession and disinflation in the early 80s taught us a lot about inflation; and the current financial crisis is teaching us a lot about the importance of the financial sector in the economy; etc. These are extremely expensive experiments to run and we badly need to figure out how to avoid them.