What to Know Before Citing Gilens and Page (2014), “Testing Theories of American Politics”

Take care when referencing the “oligarchy” study.

Because this highly influential study about representation and income is still regularly cited in reporting and commentary, I’ve summarized below what I found when I tried to replicate it. My results have been published in a peer-reviewed journal and then examined by Dylan Matthews at Vox. I have not seen convincing responses to these points; I apologize if I have missed them.

• Gilens and Page (2014) claim to reveal through statistics which groups truly have “independent impact” on policy outcomes in the United States. The paper concludes that America is merely a “democracy by coincidence,” since “ordinary citizens get what they want from government only when they happen to agree with elites or interest groups that are really calling the shots.”
• However, the authors do not report that, in their own dataset, median-income Americans received their desired outcome 47% of the time that the policy process had to pick a winner between ordinary citizens and wealthy elites during the time period examined (1980–2002). This can be confirmed simply by counting cases.
• In other words, one who believes that economic elites have a “very large, positive, independent impact” on policy while average citizens have “near zero” impact must also believe that elites chose not to use their supposedly dominant influence roughly half of the time they were opposed by median-income Americans. (Note also that in most cases, elites preferred the same outcomes as ordinary citizens. The correlation between high-income and median-income preferences was 0.91.)
• The results do not change if interest groups are incorporated as follows. Median-income Americans’ favored outcome occurred 30% of the time that they faced combined opposition from interest groups and the wealthy. In comparison, the rich won their favored outcome at a slightly higher rate of 32% when opposed by the other two groups.
• The “democracy by coincidence” claim is thus clearly false. Furthermore, the authors’ statistical model of American politics, which includes wealthy Americans’ and interest groups’ preferences, can only explain about 7% of the total variation in policy outcomes: knowing what elites and interest groups wanted didn’t tell us very much about which policies were enacted.
• A status-quo bias in American policymaking emphasized by the authors affected both income groups, and median-income citizens preferred status-quo outcomes more often than elites did in the data.
• A secondary finding in the original study is that business interest groups have more influence than “mass-based” interest groups. The authors do not make clear that supposedly weak “mass-based” interest groups, by their definition, include the National Rifle Association, AIPAC, Christian Coalition, and National Right to Life Committee — not just labor groups as widely interpreted.
• The authors do not report that when median-income and rich citizens disagreed on campaign finance, it was the median-income citizens who opposed progressive reform — successfully. See also another critique by Branham, Soroka, and Wlezien, who find that “even when the rich do win, resulting policies do not lean systematically in a conservative direction.”
• It is virtually impossible to replicate the authors’ statistical approach. The authors did not post their replication code publicly even though a footnote in their published paper suggests that they had. (At last check, it remains the case that only pre-processed replication data, not code, is available in the repository at Perspectives on Politics.) They did not write down the statistical model that they used, nor did they explain how the study’s figures were generated.
• Replication is especially important where methods are new or improvised, and they certainly were improvised in this case: the approach consists of a modified linear regression of a dichotomous outcome variable on highly correlated independent variables representing median-income and high-income preferences, made possible only by a forced reduction in that correlation.

If you would still like to cite this research favorably — as of January 2017, many continue to do so — the propaganda outlet Russia Today has helpfully reproduced several excerpts from the original study for your convenience.

Social science does not progress without scholars putting forth new and potentially provocative results that are then scrutinized. But journalists cannot feasibly follow post-publication debates as closely as academics can, and it shows: press coverage of this misleading study continues to sap faith in liberal democratic institutions at the worst possible time. When they are interviewed, most recently in the Washington Post on December 15, the authors of the original study have a responsibility to inform journalists that fundamental weaknesses have been identified in their viral research.

My work was not funded by any individual or organization outside of Princeton’s Department of Politics.