The tyranny of national averages
Marshall Burke and Apoorva Lal, AtlasAI
Signed four years ago, the Sustainable Development Goals represent an ambitious effort to achieve improvements in human livelihoods around the world. The SDGs set explicit targets to be reached by 2030, and designate a large set of indicators that are to be tracked to understand whether appropriate progress is being made.
These indicators — when they are able to be tracked — are tracked at the national level (e.g. see this very nice SDG tracker). There are at least two reasons for this country-level measurement. First, the national level is the level at which data are historically available. Countries periodically conduct nationally-representative household surveys, or utilize systems of national accounts, that are designed to generate reliable country-level statistics. However, most do not collect (or do not publicly release) representative data below the country level.
A second argument for country-level measurement is a large academic literature that suggests that country-level institutions, such as respect for the rule of law, protection of private property, and inclusive decision-making, are paramount in explaining why some parts of the world are much wealthier than others. Under this argument, it makes sense to measure things at the country level, because between-country differences are the main source of overall differences in prosperity.
But is national-level tracking what we should be doing? What if there were other credible ways to track indicators below the country level? And what if the variation in outcomes of interest was actually larger within countries than between them? And without reliable subnational data, what do you do if you want to target a program to poor people, or figure out whether a particular anti-poverty program is actually working?
Unfortunately, even national-level tracking of key indicators such as poverty rates is difficult in many parts of the world. In recent decades, published estimates indicate that over 50 countries in the world have either zero or one poverty estimate available, which is below the necessary two observations that would allow even a basic understanding of how poverty is changing. And, again, these are national-level estimates that reveal nothing about how livelihoods are changing within countries. Given this data landscape, saying that we are going to achieve the Sustainable Development Goals is a little like signing up for a weight loss program but not having a scale on which to weigh yourself.
Other credible options for measuring SDG-relevant outcomes are now emerging, however, and these could change the game in terms of how we understand and spur progress towards achievement of the SDGs. In particular, there have been a host of efforts in recent years to use non-traditional data sources to study patterns of economic development below the county level. Data from mobile phones, social media activity, street-level photos, and satellite imagery have all shown promise in accurately measuring local-level economic activity. These efforts suggest that the first reason for country-level tracking — that the country level is the only level at which relevant data are available — is beginning to make less sense.
Analysis of the granular economic estimates produced from these new data sources also suggests the second reason for measuring things at the country level — that most of the variation is between countries rather than within them — is also not very compelling.
Here’s some evidence that our team at AtlasAI has generated on this point. At Atlas, we’re combining satellite imagery with deep learning models to generate accurate, local-level estimates of a range of outcomes relevant to the SDGs. We’ve trained a deep learning model to predict village-level asset wealth across Africa from satellite imagery. Here we are measuring asset wealth as an index, where higher values correspond to households owning more assets. In our training data, households in the top quintile of this index almost all have electricity and own a radio, tv, fridge, and phone. In the bottom quintile, almost none have electricity nor own any of these items.
In held-out villages — i.e. villages in countries that the model was not trained on — the model is able to explain an impressive three-quarters of the variation in asset wealth measured in household surveys, using only satellite images as inputs. This means that a model can be trained in one country where data are available (e.g. Uganda) and accurately applied to a nearby country were imagery is available but household survey data aren’t (e.g. South Sudan).
For the five most populous African countries, the figure below shows how the distribution of asset wealth changes as we move from country level averages, to state level averages, down to district level averages. It’s immediately clear that country-level averages hide enormous sub-national differences in asset wealth. For instance, Ethiopia is the poorest country on average among these five countries, but we estimate that some of the richest districts in Ethiopia are wealthier than the average in Egypt, the richest country in the sample. Conversely, some of the poorest Egyptian districts are poorer than the Ethiopian country average.
Using these data across all African countries, we calculate that 43% of the overall variation in asset wealth is within countries, and nearly a quarter is within states within countries. This number is even higher if we focus on just sub-Saharan Africa — 65%. Similar numbers have been found for other outcomes, including child health. These within-country differences are starkly apparent in the corresponding map, which shows how wealth estimates change when country-level averages are disaggregated down to the state and then district level.
These data illustrate the growing ability — and clear importance — of measuring outcomes below the national level. We believe that data such as these can have a large range of applications, from better tracking progress towards the SDGs, to better targeting of anti-poverty programs, to better evaluation of whether these programs are working. Look for a public data release on our website in the coming months!
