Debunking the Scanlan Doctrine — Part 1

“A successful pseudoscience is a great intellectual achievement. Its study is as instructive and worth undertaking as that of a genuine one.”
— Frank Cioffi, Freud and the Question of Pseudoscience (1998)

Dear reader, you’ll do well to bring a healthy sense of humor to this post. I write this piece in the spirit of inquiry attested in my epigram, which I’ve lifted from a paper by Boudry & Braeckman. In Part 2 of this post, I revisit this paper because it offers valuable perspective on the disorienting phenomenon I attempt to unravel for you here.

Back in October, I enjoyed an exchange on a discussion forum of the American Statistical Association, regarding a doctrine promulgated by attorney James P. Scanlan for some 30 years now. Thanks to the contributions of several thoughtful participants in that exchange, a healthy critical discourse arose that I believe visits upon this doctine a well-deserved debunking. Because this ‘Scanlan Doctrine’ (as I’ll call it) implicates ongoing threats to our democracy, in matters including voting rights and racial disparities in policing, I think it worthwhile to undertake the effort to share the fruits of this discourse with a broader audience of engaged citizens who are not statisticians.

Let us begin with one of the less ‘tortured’ sentences from Mr. Scanlan’s prodigious output:

“… I explained the statistical pattern, inherent in other than highly irregular risk distributions, whereby the rarer an outcome, the greater tends to be the relative (percentage) difference between the rates at which advantaged and disadvantaged groups experience the outcome and the smaller tends to be the relative difference between rates at which such groups avoid the outcome.” [emphasis mine]
— Scanlan JP, “Race and Mortality Revisited” Society (2014) 51:328–346

I take this statement to embody the actual content of the Scanlan Doctrine, and parse it as follows:

1. The phrase “inherent in other than highly irregular risk distributions” unmistakably asserts a claim of genericity. Thus, Scanlan presents us here with a claim having something like the character of a theorem from high school geometry, “applicable to all triangles” on the understanding that highly irregular cases (such as completely flat ‘triangles’ formed from 3 collinear points) don’t count.
2. The clause in bold type, which I term the limit clause of the Scanlan Doctrine, embodies the nontrivial substance of this doctrine. I will show below that this clause is equivalent to a simple geometrical intuition, and also that this intuition does not hold generically as claimed.
3. The clause in italics, which I term the arithmetical clause of the Scanlan Doctrine, embodies by contrast a trivial statement. Although true, this statement has no substantive implications.

Before we proceed to an objective exposition of the Scanlan Doctrine, it may prove helpful to explore its emotional content. I think you’ll better appreciate what substantive content the doctrine has, once you can hear the ideological ‘dog whistle’ that non-statisticians may have trouble picking up.

The limit clause, as it turns out, embodies not only all of the nontrivial substantive content of the Scanlan Doctrine, but also its essential emotional content. Not to put too fine a point on this, the limit clause raises the following spectre:

Even as our society bends over backwards to accommodate the agenda of the social-justice warriors, their demands will only become more and more shrill as our accommodations must necessarily exacerbate the unfair and tendentious measures of injustice that they wield against our liberties and freedoms!
—The ‘dog whistle’ of the Scanlan Doctrine, at an audible frequency

Having thus heard the limit clause’s primal, ultrasonic scream, you may now appreciate the wail of desperation that runs through Mr. Scanlan’s decades of copious output on this topic. The dread and frantic anxiety that seize you when you read Mr. Scanlan’s writings may even come into focus, as manifestations of the psychoanalytic concept of countertransference.

But enough with casting gentle jibes in Mr. Scanlan’s direction! An objective treatment of his Doctrine will serve us far better.

Developing this treatment properly will require cumulative distributions. For whatever reason, cumulative distributions just don’t get the love they deserve. Distributions of the density type — the famous ‘bell curve’ being the canonical example — hog the limelight. Don’t get me wrong; densities are fine, as far as they go. The following density plot for heights of girls and boys from England’s NCMP shows how nicely and intuitively a density displays the central tendency of a random variable.

Figure 1: Heights of boys and girls aged 130 months, from England’s National Child Measurement Programme (NCMP)

But serious statistical work that involves comparing distributions inevitably requires cumulative distributions. (The well-known Kolmogorov-Smirnov test, for example, compares two distributions by checking the biggest gap between them cumulatively.) In the density above, certainly you can see that the girls are shifted to the right of the boys — corresponding to the well-known fact that at this age (6th graders), girls average taller than boys. But consider how much clearer and more quantitative the picture becomes in a cumulative format:

Figure 2: Percentiles of height for boys and girls aged 130 months. Source: English NCMP

Straight off a plot like this, you can read the fact that girls at the top quartile (75th percentile) are about 1 cm taller than top-quartile boys. You can also see that the shortest 15% of boys and girls have almost identical height distributions.

One further technique that will prove essential to my treatment of the Scanlan Doctrine is standardization. Consider that, instead of expressing childrens’ heights in absolute units of cm, we could just as validly express them in the relative terms of percentiles of boys’ height. Rescaling the height axis of Figure 2 according to this principle, we obtain Figure 3:

Figure 3: Distribution of heights of 130-month-old girls relative to boys of the same age.

To help you get your bearings in the somewhat abstract space of Figure 3, I’ve added dotted lines showing how you can read the following facts off it:

• The median (i.e., 50th-percentile) height for boys is just about 45th percentile for girls. Thus, 100% – 45%=55% of girls were taller than the median boy.
• The top-quartile height (75th percentile) for girls hits the top quintile (80th percentile) for boys. Thus, among children aged 130 mos, a girl who is taller than 3 out of 4 girls is also taller than 4 out of 5 boys.

A connection with the Scanlan Doctrine begins to emerge as we consider the solid gray policy lines drawn at 12th and 95th centiles of boys’ height in Figure 3. Consider 2 stories:

• Suppose these kids go on a school trip to an amusement park. One of the rides has a minimum height limit of 4½ feet (137 cm). This corresponds to the 12th centile for boys, which is drawn as a policy line toward the left side of Figure 3. The diagram makes clear that boys and girls are not disparately affected by this policy.
• Imagine a school’s highly competitive coed basketball team sets a strict lower height limit of 156 cm. This corresponds to the 95th centile of boys’ height, drawn as the policy line on the right side of Figure 3. Clearly, girls enjoy an advantage in the application of this policy: whereas only 5% of boys meet this criterion, nearly 8% of girls do. Note (for later reference) that the ratio 8/5 is also the slope of the girls’ curve at the top right of the plot.

With this ‘machinery’ in place, we may now proceed to give a graphical account of the Scanlan Doctrine. Appropriate to the dynamic narrative of the limit clause (which, remember, embodies both the nontrivial positive part and the essential emotive content of the Doctrine), I have built a Shiny app that lets you slide the policy line left or right. If you’re a hands-on type, by all means please try the app now (it will open in a separate browser window). If you’re the impatient type, you may prefer to scroll right down to Figure 6 below, where you’ll find an animation that demonstrates how and why the limit clause fails.

Figure 4: Graphical Exposition of the Scanlan Doctrine.

The essential setup is this:

• The Scanlan Doctrine typically concerns itself with some kind of bad outcome, such as being arrested or disenfranchised or suspended from school. The percentiles in Figure 4 simply represent group percentages experiencing this outcome.
• Group experiences of the policy-driven outcome are considered in comparison to those of a ‘reference group’, labeled “Ref” in Figure 4. The outcomes curve of a group that is strictly disadvantaged relative to the reference group will lie strictly above the Ref line; the opposite will hold for a group that is strictly advantaged relative to Ref.
• I attach special terms ‘Scanlan limit’ and ‘elite ratio’ to the slopes of outcomes curves at their left and right ends, as indicated by dotted red and green tangent lines in Figure 4. The term ‘elite ratio’ makes sense in light of the basketball team story: girls in Figure 3 have an ‘elite ratio’ of about 8/5, so they enjoy an 8:5 advantage in getting selected for the team. The ‘Scanlan limit’ — and its relationship to the ‘Scanlan ratio’ — receive an extended discussion below.
• The power of this formulation arises chiefly from the simple guidance it provides on how to draw a valid outcomes curve. Valid outcomes curves (1) run from the bottom left corner of the plot to the top right, and (2) slope upward the whole way. Any curve you draw meeting conditions (1) and (2) is legit. (The app doesn’t let you draw ‘freehand’, but it does allow you produce a whole family of curves by varying the slopes at the left and right endpoints, from 0–90°.) Thus, our framework readily admits the possibility of a group with mixed advantage. And indeed, if you look very closely at the bottom left-hand corner of Figure 3, you can see the girls’ curve does just barely cross the boys’. Thus, the girls technically have a mixed (rather than strict) advantage over boys with respect to height.

Completing our debunking of the Scanlan Doctrine requires now revisiting its limit clause. Recall that this states the following:

“… the rarer an outcome, the greater tends to be the relative (percentage) difference between the rates at which advantaged and disadvantaged groups experience the outcome …”
— Scanlan JP, “Race and Mortality Revisited” Society (2014)

Figure 4 helps us to appreciate this as a basically geometrical intuition. Where the policy line intersects a group’s outcomes curve determines the prevalence of the outcome for that group. Of course, you can read absolute prevalences directly off the vertical axis of Figure 4. But the figure also makes relative prevalence directly accessible through a simple geometrical construction. Consider the solid red chord in Figure 4, labeled “Scanlan ratio=2.98”. Because the slope of the Ref group has been standardized to 1, the slope of the red chord (≈3) equals the relative prevalence of the outcome in the red group. (This follows directly from the definition of slope as ‘rise over run’.) Thus, in Figure 4 the Ref group experiences the outcome at a rate of 5% while the disadvantaged group experiences it at 3 times that rate: 15%. (Meanwhile, the green advantaged group experiences the outcome at a rate of just 0.61×5%≈3%.)

Scanlan’s limit clause addresses what happens ‘in the limit’ as the policy line shifts leftward toward zero. If Scanlan had stated simply that the Scanlan ratio approaches the Scanlan limit, he would stand on solid ground. But instead, he advances a claim about the direction from which this limit is approached. In particular, he claims falsely that the Scanlan ratio for a disadvantaged group always rises to meet the Scanlan limit. To see where Mr. Scanlan’s imagination has failed him, consider Figure 5:

Figure 5: A strictly disadvantaged group with concave curvature in the rare-outcome regime.

Clearly, the red group in Figure 5 meets our definition of ‘strict disadvantage’: sitting entirely above the reference line, it always experiences the bad outcome more often — no matter where we set the policy line. And indeed, the geometry shows as before that the Scanlan ratio (which stands at 1.18 in the Figure) does approach the Scanlan limit of 1 as the policy line slides left. But because the red outcomes curve in Figure 5 has a curvature that is concave upwards (to the left of 10%, anyway), this limit will be approached from above. That is, the Scanlan ratio will actually decrease as the policy line slides leftward in Figure 5. The Scanlan Doctrine’s limit clause therefore hinges vitally on a presumed convexity of the kind displayed by the disadvantaged group of Figure 4. The animation in Figure 6 below summarizes our case up to this point.

Figure 6: Animation showing that the Scanlan Doctrine’s limit clause does not hold generically. The green and red groups are both strictly disadvantaged relative to the Reference group, but their Scanlan curves have different curvatures — at least to the left of 20th percentile of the Reference group. In this region, the green group is ‘convex upward’, while the red group is ‘concave upward’. Consequently, as the as the policy line shifts leftward, the ‘Scanlan chords’ of the two groups tilt in opposite directions as their slopes (the ‘Scanlan ratios’) converge to their respective ‘Scanlan limits’.

One of the participants in last October’s discourse drew our attention to a highly technical but basically equivalent treatment of the Scanlan Rule given by Lambert & Subramanian. (Free version of paper here.) The L&S treatment differs from mine mainly in flipping the figure around the 45° line, which has the salutary effect of putting outcomes curves where they seem to belong intuitively: advantaged groups above the reference, disadvantaged groups below. Unfortunately, this also has the effect of converting the Scanlan limit and Scanlan ratio into the reciprocals of slopes in the L&S figure, severely compromising the accessibility of my geometrical objectification of the limit clause.

What makes this objectification is so critical is that Mr. Scanlan’s interlocutors may now put to him questions such as:

• What evidence have you put before the court, to demonstrate that the disadvantage in this case is of the convex type to which your Scanlan Rule is applicable?
• Briefly, please, how is this convexity material to the matter at hand?
• What is your estimate of the Scanlan limit in this case? How have you arrived at that estimate? How should the magnitude of the Scanlan limit you quote be taken to weigh upon the court’s judgment, against the considerations introduced by [opposing party]?

One bit of tidying-up remains: to complete Part 1 of this post, we must dispose of the arithmetical clause. Fortunately, the triviality of this task matches the triviality of the clause itself. Let’s return to it by way of an example.

“… and the smaller tends to be the relative difference between rates at which such groups avoid the outcome.”
— Scanlan JP, “Race and Mortality Revisited” Society (2014)

In Veasey v. Abbott, 2% of Anglo, 5.9% of Hispanic and 8.1% of African-American registered voters lacked IDs required by a Texas Voter ID law under review. In this context, the arithmetical clause of the Scanlan Doctrine would seem to direct our attention to “the rates at which such groups avoid the outcome” of disenfranchisment. Indeed, the “relative difference” between these rates is to be examined. The ratio 91.9% : 98% = 0.938, it seems, warrants equal consideration alongside 8.1% : 2% = 4. Thus, alongside a statement like African-American voters are 4 times as likely as Anglos to be disenfranchised by SB 14, Scanlan seems to instruct us to note: but African-Americans achieve nearly 94% of the enfranchisement of Anglos under this bill.

I can do no better than to leave this absurdity dangling mid-air. (As I’ve said, the Scanlan Doctrine’s arithmetical clause has no substantive implications.)

Through our efforts here — I say “our” because I know this wasn’t easy for you, either! — the Scanlan Doctrine has morphed from a mystifying screed into an objectively realized and criticizable set of propositions. We have revealed a geometrical intuition lying at the Doctrine’s core, and demonstrated that the claim of genericity ascribed to this intuition does not hold water. We have also disposed of a purely arithmetical sham component of the doctrine.

In Part 2 of this post, I re-examine the Scanlan Doctrine specifically as a pseudoscience.

My readers’ civil comments are of course most welcome.