A few days ago, Paul Krugman made reference in one of his columns to some data compiled by the US Energy Information Administration on trends in energy use over the past few decades. The data touch on the question of how much energy different nations use to generate $1 of GDP. Are we getting more or less efficient in our use of energy? The numbers, as Krugman argued, show we’re generally getting more efficient. Below I’ve listed the numbers for US energy usage from the year’s 2001 through 2011, in sequential order, from left to right, separated by commas, the units being BTUs per dollar of GDP:
8,482.307, 8,459.179, 8,274.763, 8,178.463, 7,944.349, 7,688.294, 7,671.837, 7,543.901, 7,414.716, 7,503.361, 7,328.424
So you see, the amount of energy used to generate each bit of GDP is going down. Same is generally true for other nations. Fair enough. I’m not going to question that.
But isn’t there something fishy about these numbers? The energy units are BTUs, and the final entry says we used precisely 7,328.424 BTUs per dollar of GDP in 2011. There are 7 specific digits reported in this number, implying that we know our energy/GDP figure to an accuracy of 1 part in 10 million. It’s incredibly impressive. Think about that “.424" at the end. It’s not “.425" or “.423" but exactly “.424".
Is this at all meaningful? Of course not. It’s ridiculous. Unfortunately, this kind of illusory accuracy infects economics and finance quite widely. It may not be the most important issue in the world — even writing about it makes me feel like a grumpy old man — but we’d all think more clearly if we paid more attention to the numbers. So, what’s wrong here?
I’m not sure how accurately we know GDP, but it’s not one part in 10 million. By Googling I found this study from the early 1990s looking at the typical revisions that get made to GDP figures as people use incoming data to correct initial estimates; they get revisions on the order of about 1–2%. That seems quite plausible. I’m sure there are other significant sources of error in calculating GDP, not least of which is that some economic activity goes unreported, whether to dodge taxes or for other reasons. If we know it to one part in 100, that’s pretty good.
So, let’s be overly generous and suppose that we know our energy use EXACTLY, no error at all. We divide that number by our GDP to get energy per unit GDP. The result can’t be known any more accurately than we know our GDP, so there should again be at least 1% certainty in the result. This leads to the conclusion that in the figure 7,328.424 BTUs per dollar of GDP for 2011, not only is the “.424" completely meaningless but so is everything after the “7" and “3". We use about 7,300 BUTs per dollar of GDP, plus or minus about 100. Take uncertainty in our knowledge of energy use into account and things will get even worse.
So, in the data to which Krugman referred, we have to conclude that almost all of the numbers that appear there are meaningless! The only meaningful digits are the first two in each figure. Just to be clear, Krugman didn’t make this table and I’m sure he recognizes this. I’m not criticizing him.
I know this is a little boring, but the same issue turns up frequently in other areas of economics — if you spend more time looking than you probably should. Earlier this year, after Thomas Piketty published his now famous book, I had the unfortunate experience of actually trying to read some economics papers that some people claimed (in trying to push back against Piketty) had “already explained” the level of wealth inequality in the US. The most painful for me was this monstrosity, published in 2011 in the very prestigious economics journal Econometrica. The paper proposed and analyzed a model by which wealth might grow in a population following a fairly simple random process.
It took me a long time — a week at least—to fight through the jungle of algebra and theorems. And the payoff was definitely not worth it. Wade into the jungle, and you eventually discover that the model just assumes what it wants to show. It assumes that a few people in the population, about 1%, increase their wealth over one generation (taken as 45 years) by a huge amount — about 100 times more (and possibly a lot more) than almost everyone else in the population. This gets passed on to their offspring. In essence, they show that in a model in which the wealth of 1% grows very much faster than the bulk of the population — and where this imbalance persists over generations — you end up with lots of inequality. Hardly surprising (which I think explains all the theorems and dense mathematics).
But that’s an aside — back to the sloppy use of numbers. The authors understandably had to choose some plausible values for various parameters to enter into their model, defined to either two or even just one significant figure (T=45 years, the duration of a generation, and ρ = 0.04, the rate of discounting future income). But this didn’t stop them from going on to report values characterizing the shape of the wealth distribution to four significant figures — it’s 1.796 or 1.256. What certainty — and on an issue of such importance!! It’s the same thing as in the table from the US Energy Information Administration.
I’m not sure it’s connected, but Michael Lewis writing for Bloomberg recently offered his views on some of the “occupational hazards” of working on Wall St. One of the things he bemoaned was general unwillingness to admit uncertainty, and a strong desire to be certain and knowledgeable even in cases when it is impossible:
Anyone who works in finance will sense, at least at first, the pressure to pretend to know more than he does. … It’s not just that people who pick stocks, or predict the future price of oil and gold, or select targets for corporate acquisitions, or persuade happy, well-run private companies to go public don’t know what they are talking about: what they pretend to know is unknowable. Much of what Wall Street sells is less like engineering than like a forecasting service for a coin-flipping contest — except that no one mistakes a coin-flipping contest for a game of skill. To succeed in this environment you must believe, or at least pretend to believe, that you are an expert in matters where no expertise is possible. I’m not sure it’s any easier to be a total fraud on Wall Street than in any other occupation, but on Wall Street you will be paid a lot more to forget your uneasy feelings.
To be honest, I don’t think this observation is connected to what goes in with numbers in economics. I think it’s just that people don’t get the training they need early on to use numbers accurately, and especially to protect themselves from projecting an unwarranted certainty. Felix Salmon touched on this issue in financial journalism last year.
And, to be fair, many economists do work hard to make the uncertainty in their predictions and assessments clear. For example, the economists behind this recent report making projections for US economic growth and inflation deserve credit. I’m no fan of the DSGE models of macroeconomics these economists use to make their projections, but at least they’re up front with what they claim to know. Their model predicts a 70% likelihood that GDP growth will be somewhere between 4.2% and -1.3% in 2015. That’s next year. There’s a 30% chance it will be outside that range. Now that’s a prediction I can believe.