On the importance of footnotes
Keep an eye on the fine print and glean worthy insights into social science methodology.
In undergraduate studies, one of the most commonly overlooked aspects of social science research is the role footnotes play in a careful reading of source material. If a student is lucky, she will take a class where the professor mentions the importance of paying close attention to the footnotes. However, for many other students, footnotes are simply an afterthought: text at the bottom of the page that simply cites sources or explains unimportant background information.
Although often considered tedious, I’d venture footnotes are important to consult when the main text just doesn’t add up. They can inform research in meaningful directions.
Take for example literature on the effects of Proposition 47, a criminal justice reform bill passed in 2014 in California which changed many low-level, nonviolent, felony offenses to misdemeanors, both retroactively and for future cases. Falling within a larger context of criminal justice reform, thousands of prisoners were released from jail due to the law. Three years later in 2017, the law’s effects on crime and recidivism rates have been extensively studied.
Mark Males contributed to this research in his report, Is Proposition 47 to Blame for California’s 2015 Increase in Urban Crime? The study looked in depth at crime rates before and after the passage of the Proposition, using crime reports from the Federal Bureau of Investigation (FBI) for 66 of California’s cities of 100,000 people and additionally Fresno and Oakland (so 68 total cities falling into 22 counties) and prison discharge statistics from the California Department of Corrections and Rehabilitation (CDCR).
Males based his study on the theory that if the Proposition caused increased rates of crime, one would observe the highest crime rate increases after the proposition’s passage in the cities and counties where the most people were discharged from jail. According to Males, the results of his study were as follows:
The cities in 11 counties with the largest decreases in both total jail populations and felony jail populations showed equivalent changes in violent crime, and smaller increases in property and total crime, than the cities in 10 counties with the smallest decreases in jail populations.
In other words, the data suggested that larger decreases in jail populations were associated with no changes in violent crime and smaller increases in property and total crime. However, Los Angeles was depicted separately in the report’s main table which compared changes in jail populations to changes in crime rates, as one can observe below.
Why was Los Angeles depicted under the average and not as part of the average data results? The answer to this questions exists in the sources footnotes under the table, which hints that Males treated Los Angeles County uniquely in this study and did not include it in his final comparisons between those counties most and least affected by the Proposition. To blunt this criticism, Males does speak to Los Angeles specifically in the report and explains:
Los Angeles County (shown separately due to the unreliability of its 2014 crime statistics) had a lesser decrease in total jail ADP and an average decrease in felony jail ADP, while the city of Los Angeles saw more unfavorable crime trends than the state as a whole.
In other words, Los Angeles County, despite its average decreases in ADP, saw larger than average spikes in crimes after the passage of the Proposition, “more unfavorable” than the state as a whole. However, these statistics were not included in Males’ comparisons because of Los Angeles’ purportedly unreliable crime statistics.
But why were these crime statistics deemed unreliable? This question is left unanswered by Males. Without a clear outline of methodology, we’re left to speculate why Males left the data out of his analysis.
Is it that had Los Angeles had been included, the data would have suggested another conclusion? As one can see below from a Stanford Law School’s Proposition 47 Progress Report: Year One Implementation, Los Angeles County was, by far and away, the County that saw the most inmates released after passage of the Proposition. Disregarding this large percentage of the overall population undoubtedly biased Males’ data.
It’s plausible that had Los Angeles been included, the correlation would have pointed in the opposite direction. This apparent bias undermines Males’ conclusion and suggests his findings may not be as strong as they seem.
We’re in asterisk territory now with this study — it’s simply unreliable due to an unexplained, significant methodology gap described in a table footnote.
Faulty social science to be sure. So, keep an eye on the fine print, folks.
