On the importance of comparing apple to apples
In the social sciences, it’s important we acknowledge the biases that occur when one compares apples to oranges
Last week, I looked at how footnotes in a study informed my research into Proposition 47 and its effects on crime after its passage. Combing through the vast amounts of Proposition 47 academic research also led me to another observation — statistics sure can be misleading. Before taking a statistic’s meaning solely on its face, it’s important we delve into how that statistic is calculated. This week, let’s turn to the same Stanford Law School study I mentioned last week as a demonstration of why understanding statistical contexts, and making appropriate apples to apples comparisons, is so important.
The Stanford Law School research studied Proposition 47’s effects, notably impacts on recidivism rates, and came to the following conclusions:
According to the Department of Corrections, to date 159 of 4,454 state prisoners awarded reduced sentences under Proposition 47 and released early have been returned to state prison for new crimes. A prison return rate below 5 percent indicates that any increase in crime over the past year should not be attributed to inmates freed from prison under Proposition 47. According to the latest recidivism data released by the Department of Corrections, 42 percent of all inmates released from state prison were convicted of a new crime and returned to prison within one year. However this data reflects recidivism prior to Proposition 47, which reduced several common felonies to misdemeanors and has resulted fewer new prison sentences.
So, in regards to recidivism rates, Proposition 47 was a wild success, right?
Not so fast. Let’s unpack that last sentence. The 5% value “reflects recidivism prior to Proposition 47, which reduced several common felonies to misdemeanors and has resulted fewer new prison sentences.”
So, let’s imagine a methamphetamine possession offender who gets out of state prison because Proposition 47 reclassifies his felony offense to a misdemeanor. After he gets out, he again is convicted for possession of methamphetamine in an increment worth less than $950, but he’s no longer sent to state prison because this offense was reclassified under Proposition 47 — so he’s not included as a recidivist offender in the 5% rate calculated by Stanford.
The 42% recidivism rate, calculated before the passage of Proposition 47, would have included our offender as a recidivist.
So, to be clear, this 5% rate does not factor in offenders who commit crimes, once released, that were reclassified under Proposition 47. The 42% rate does factor in these offenders, as it was calculated before the passage of Proposition 47.
In other words, this study did not compare apples to apples — apples to oranges, in fact.
It may be that Proposition 47 offenders do return to state prisons at lower rates, but the statistics in this study are calculated under two completely different sets of legal contexts. That makes their comparison unreliable.
Further, as the study explains itself, “recidivism may be measured in many different ways (e.g. new arrests or new probation violations). The recidivism data provided here — new convictions that result in returns to state prison — is the only statewide recidivism measure available in California.”
If we wanted to study other recidivism data to control for Proposition 47 effects, it just doesn’t exist. Perfect.
Ultimately, we’ve got non-similar contexts within which we’re trying to make a statistical comparison and a lack of reliable data to make that comparison. Stanford Law School tries its best, but social science realities make for messy, uncertain conclusions.
So, while it’s important we understand Proposition 47’s effectiveness, it’s also important we acknowledge the inherent biases that occur when one compares apples to oranges. From designing the research, to lending a critical eye towards a review of the research, gleaning worthy insights from social science is hard work to be sure.
