Understanding Reactivity: Law Schools as a Case Study
The paper Rankings and Reactivity: How Public Measures Recreate Social Worlds by Wendy Nelson Espeland and Michael Sauder is a canonical text on reactivity that demonstrates the phenomenon of reactivity as evidenced by the relationship between US News World Report Law School Rankings and the schools that they rank. This blog will focus on their work and conclude by touching on the work’s implications for the Global Indices Network (GIN) Project’s research on the power and pathologies of indices and other metrics that rate and rank entities in the world. Information about reactivity and law school admissions are references to their work. Direct citations will only be used for direct quotes or when another piece of work is referenced.
When the University of Texas at Austin’s pre-law fraternity hosted a question-and-answer event with a visiting law school dean, questions mainly focused on the application.
“How much do extracurriculars matter?”
“How heavily weighted is the personal statement?”
“I heard that GPA and LSAT are the only things that matter. Is that true?”
The dean’s answers varied in wording, but essentially communicated a single sentiment. In more or less words, “GPA and LSAT matter most.”
When the listening students applied to UT as prospective undergraduates, they were assured that their applications would be evaluated holistically, not just numerically. Words were not so kindly minced in law school admissions. However, the drastic difference in approach results from something more complex than higher standards for older, more experienced applicants. The emphasis on LSAT and GPA over all other application components is the result of a complex phenomenon known as reactivity. In this case, the convergence towards the near dominance of GPA and LSAT scores in law school admissions decisions reflects a change in law school behavior shaped by the impact that average GPA and LSAT scores have on the national and international rankings of law schools.
Reactivity describes a relationship between a measure and the thing that is being measured. More specifically, a reactive measure is one that “modifies the phenomenon under study, which changes the very thing that one is trying to measure.”[1]One example would be when an individual taking part in a research study knows that they are being observed and behaves differently than they would in the same situation if they were not being observed. Reactivity is most often regarded in the social sciences as a methodological problem that shapes the behavior of actors targeted by rankings and ratings. Sensitive to status and other reputation concerns, entities such as law schools will change their behavior — in this case, admissions standards and norms — in order to maximize their ratings on some comparative metric. In simplest terms, it means the metric (relative rankings) comes to dictate the choice of measures. In order to explore reactivity in greater detail, we will look to Espeland and Saunders’ account of the effects of reactivity on law schools ranked by The US News & World Report Law School Rankings.
Reactive responses fall into two general categories: self-fulfilling prophecies and commensuration. The term “self-fulfilling prophecy” refers to “a false definition of the situation evoking a new behavior which makes the originally false definition of the situation come true.”[2]This was the situation that the visiting dean described to UT’s pre-law students when explaining why LSAT scores and GPA were so important.
“Idon’t care about US News Rankings,” the dean said with some consternation. “Law schools don’t really care, they just think that students care. But the students say that they don’t really care, they just think that employers care. And employers say that they don’t care, they just think that students care. For the life of me, I can’t find anyone who will admit to caring about them.”
But as long as everybody thinks that everybody else cares, the US News Rankings will continue to have significant impact on student, employer, and university administrative behaviors. What’s more, the effects of self-fulfilling prophecy can also be self-reinforcing. “Reputation” accounts for 40% of schools’ US News Ranking scores and is determined using responses to two surveys, one sent to practitioners and one to academics. However, even experts in the field cannot intimately know the details of all 190 schools being ranked. The result is that knowledge of past rankings heavily influences survey responses, which influence future rankings. Past rankings have thus been shown to be the strongest predictor of current reputation score. Self-fulfilling prophecy explains how ranking systems take on significance merely by seeming important and thus inciting reactive tendencies.
The other category of reactive responses is commensuration. Self-fulfilling prophecy classifies behaviors, but commensuration describes cognition. Commensuration transforms qualities of a thing into measurable quantities, in turn shaping “what we pay attention to, which things are connected to other things, and how we express sameness and difference.” Examples of commensuration include “prices, cost-benefit ratios, survey responses, and standardized tests.”[3]All information under the mechanism of commensuration must either be disregarded, and thus rendered irrelevant, or converted into a measurable metric. The US News & World Report Rankings obscure many relevant characteristics about schools, like whether schools are public or private, what kind of work students do after graduation, and schools’ missions, demographics, and financial aid quality. Instead, the index favors student body test scores and the aforementioned reputation survey responses. The result is a ranking system that disregards many differences between law schools, wrestling a few qualities into numerical measurements, and producing a single number meant to indicate some kind of objective standard of overall quality. This is the process of commensuration.
Together, the imperative to act on an illusion of importance and an aggressively quantified ranking system produce a range of reactive behaviors in law school administrations, students, and even employers. However, there remain two sub-categories of behavior that are of special significance to the GIN Project’s research: maximization and manipulation.
In the context of law schools, maximization includes acts designed to maximize indicators, and thus maximize scores, potentially at the expense of the school’s mission or other objectives. For example, career counselors are pressured to push students into the first job offer the student can secure in order to guarantee the best job-placement percentages, rather than helping the student explore alternative options that could help them better achieve their career goals. While making budgets, law schools are inclined to prioritize initiatives that will improve rank and reject those that will not despite initiatives’ potential to improve student welfare. Virtually all schools have carved out large portions of their budgets to create merit scholarships to entice students with high test scores. All of these actions, despite often being antithetical to both the schools’ and the students’ interests, are done in the name of maximizing US News Rankings scores.
Manipulation is a form of maximization, but the term refers specifically to acts of gamesmanship. Manipulating or gaming rankings involves efforts to maximize numbers that “are unconnected to, or even undermine, the motivation behind them;” these actions aim toward “managing appearances… without improving the characteristics the factors are designed to measure.”[4]For example, law schools count all graduates with any job, legal or nonlegal, toward their post-graduate employment numbers. This adds another layer to the aforementioned pressure on career services departments to push students into work; career counselors are often encouraged to push students into any kind of work that they can find post-graduation, even if it is a low-skilled position (such as a barista) with no connection to law or a student’s long-term plans. Other examples include funneling students with low LSAT scores, but otherwise strong applications, into temporary “probationary” or night programs in order to improve student-to-faculty ratio without counting them toward the school’s median LSAT number. Manipulation is an insidious effect of reactivity because it pushes subjects of scoring to play to the numbers rather than improving the quality of what the numbers seek to measure. This undermines the integrity of a rankings system that is already highly alienated from its own mission because of commensuration.
Self-fulfilling prophecies, commensuration, maximization, and manipulation all affect the actions of law schools. More significantly, when taking a view of reactivity as not only a research problem to be solved but a tool that can be harnessed, it becomes clear that the influence of reactive forces grants US News & World Report Law School Rankings an inordinate amount of power over law schools. This power dynamic between the index producer and its subject of scoring can be seen unfolding on much larger playing fields, which is the primary concern of the GIN Project. Our team is concerned with this same vein of reactive behaviors, but analyzes them in governments.
State reactive responses can be similar to those of law schools, as when state leaders use rankings as focal points for policy priorities and new initiatives. However, state responses seem to indicate that rankings on the international level can be a two-way source of power, depending upon the relationship between a given state and an index producer. This is evidenced in the way that some states will tout their high or improving scores as a sort of publicity stunt in speeches and international publications. Doing so is often an attempt to improve credibility with other nations or their own citizens. Meanwhile, some state actors will attack indices when displeased with their scores (or the fallout from their scores). This the case when European Union officials threatened to ban credit rating agencies from issuing downgrades during the Euro-crisis because they were deemed “counterproductive.”[5]These examples only begin to scratch the surface of the expanse of reactive state behaviors to indices. However, they make clear that reactivity is not a passive problem, but a pervasive power dynamic that must be taken seriously.
About the Author
Abby is a sophomore at UT Austin pursuing a degree in Politics, Philosophy, and Economics and a certificate in Human Rights and Social Justice. She is interested in international development, refugee issues, and foreign and economic policy.
Works Cited
Cooley, Alexander. “The Emerging Politics of International Rankings and Ratings; A Framework for Analysis.” Ranking the World. Edited by Alexander Cooley and Jack Snyder. Cambridge University Press: 1–38.
Espeland, Wendy Nelson, and Michael Sauder. “Rankings and Reactivity: How Public Measures Recreate Social Worlds.” American Journal of Sociology113, no. 1 (2007): 1–40. doi:10.1086/517897.
[1]Saunder & Espeland. Page 3.
[2]Saunder & Espeland. Page 11.
[3]Saunder & Espeland. Page 16.
[4]Saunder & Espeland. Page 29.
[5]Alexander Cooley. Ranking the World. Page 4.
