Trust & Numbers: Part 2
This is the second of a three-part series on Trust, Power, Objectivity, and Truth in evaluation and learning for the social sector, based on a talk I delivered at Evaluation 2018. Read Part 1 here or skip to Part 3.
Trust
To explore trust and credibility in evaluation, it helps to understand how the history of the scientific method has closely paralleled the history of demographic statistics, or the way we measure and describe groups of people.
Demographic statistics, the numerical description of population groups, rose to prominence first in France, when rulers began to control such large swaths of land that they could no longer personally account for all the people they ruled over. Thus, demography emerged alongside the rise of European nation states.
Historian of science Theodore Porter wrote in Trust in Numbers — a book whose title I’ve adapted for these posts — about the historical association between objectivity and quantification. Porter observed that “quantification is a technology of distance,” writing,
Since the rules for collecting and manipulating numbers are widely shared, they can easily be transported across oceans and continents and used to coordinate activities or settle disputes. Perhaps most crucially, reliance on numbers and quantitative manipulation minimizes the need for intimate knowledge and personal trust. Quantification is well suited for communication that goes beyond the boundaries of locality and community.
Put another way, numbers were far less likely to generate controversy (or resistance to the decisions of rulers) than words. This is because the mathematical calculations that produce numbers so often appear to be independent of human judgment. This apparently disinterested quality of numbers grants them power, even today.
As Porter noted, the language of quantification proliferates in conditions of mistrust.
Power
What makes the transportability, comparability, diversity-flattening, fact-ness of numbers so powerful? Is disinterestedness always a more valuable platform for truth? Why do numbers have this power?
It’s now time to distinguish between numbers — mere counting — and statistics, which involve more or less complex mathematical calculations and the theoretical construct of probability. If we want to know why numbers are powerful, there’s no one better to ask than political theorist Hannah Arendt. She wrote in 1969 — at the height of political unrest in the US — that “the power of the government depends on numbers.”
She contrasts this with tyranny, “the most violent and least powerful form of government. Indeed,” she says, “one of the most obvious distinctions between power and violence is that power always stands in need of numbers, whereas violence, up to a point, can manage without them because it relies on implements [often weapons].”
Arendt was arguing against what she saw at the time as “a consensus among political theorists from Left to Right…that violence is nothing more than the most flagrant manifestation of power. ‘All politics is a struggle for power; the ultimate kind of power is violence,” said C. Wright Mills.” The phrase “speak truth to power,” in some ways echoes this notion. But if we listen to Arendt, we need to unpack the association between violence and power. She says,
“Power corresponds to the human ability not just to act, but to act in concert. Power is never the property of an individual; it belongs to a group and remains in existence only so long as the group keeps together.” It relies on “Authority… [which] can be vested in persons … or it can be vested in offices” like the presidency, or the senate. “To remain in authority requires respect for the person or the office. The greatest enemy of authority, therefore, is contempt, and the surest way to undermine it is laughter.”
Objectivity
Statistics are powerful in part because of their historical association with objectivity. While demographic statistics facilitated the rise of European nation states, the Enlightenment idealized objectivity as the ultimate standard of scientific validity.
The importance of objectivity was championed by Robert Boyle, known as the “father of chemistry,” and even more importantly, as the humble patriarch of the experimental method.
Historians of science tell us that Boyle achieved the feat of objectivity through two innovations. First, he established conventions for the scientific method, such as mandating the elimination of all variables but the ones being tested, and requiring replication of results through repeated experiments. Second, he championed a way of validating experimental truth: through a witness. Witnesses to truth could only be European men of means, whose wealth enabled them to resist bribery. These men were also considered objective witnesses because they were unencumbered by all things female: bodies, emotions, and ornamental flourishes.
This painting, known by many as “The Experiment” depicts Robert Boyle demonstrating the creation of a vacuum using his version of the scientific method. He has isolated his experimental subject, a dove, in glass, to protect it from interfering variables. He then uses his famous air pump to extract the air from the glass until the dove suffocates, proving the existence of the vacuum. Note how the woman on the right cannot clearly witness the experiment, as distress has caused her to cover her eyes and turn away, signaling her unreliability as an objective witness.
In his scientific method, Boyle represented the figure of the “Modest Witness.” Modest Witnesses spoke and wrote without flourishes; their descriptions of experiments communicated authoritative, impersonal, disembodied truths. In this painting, note the faces of the men.
A heated argument with my best friend Sam gave me a more personal experience of these concepts. We went to graduate school together and briefly taught at the same college. He was teaching a course on masculinity that I had designed for a joint sociology and anthropology department. Not all of the syllabus was familiar to him, and some of it he found downright annoying. He called me one morning while he was preparing to teach a book called Stone Butch Blues by Leslie Feinberg.
The book describes how Feinberg suffers violence at the hands of people who mistook her for male before realizing she was in fact female. My friend Sam — an economically privileged white guy — disputed the truth of the narrator’s account. He said he had lived in New York City during that time and he knew there wasn’t this kind of rampant gender-based violence going on. I reminded him that I too had lived in New York City then and this kind of violence was plentiful enough for me to write my bachelor’s thesis on hate crime. I asked him, “What makes you think your memories of the city are the only ones that matter? Why are you so confident that your perceptions are just authoritatively True?”
My friend Sam was like The Modest Witness — unconscious of the particularity of his own experience. But this is all old-hat, now, right? We’ve had social movements that fundamentally challenge the idea that objectivity is a view from nowhere. We’ve had cultural relativism, feminism, and on and on. But sometimes those critiques end up simply flipping the script: instead of power and privilege speaking truth, we get into oppression derbies where the more oppressed or violated a person is, the more accurate their perspectives are — by definition. Even when the definition of power is flipped, we still end up reinforcing the idea that truth is singular.
The commonsense version of objectivity that we’ve inherited from the Enlightenment refers to a “view from nowhere,” “untainted by perspective,” “disinterested,” “irrefutable,” and “identical regardless of changes in viewpoint.” If a view from nowhere is pure, like clear glass, then views from somewhere, from a particular person or type of person, are correspondingly tainted and delegitimized.
In resisting a view-from-nowhere objectivity, we sometimes err instead on the side of a view that comes from a correct somewhere, whether that somewhere is female, or disabled, of color, or working class. It is certainly the case that systemic experiences of discrimination often widens one’s perspective. You see from your own eyes, while also being unable to escape knowledge of how the world looks to those with more power than you. Conversely, when one occupies a position of privilege, one can sail through life unencumbered by knowledge of what it’s like to see the world from a less privileged place. Still, it’s dangerous to rely on any one perspective.
I can now reflect on the ways that disembodied numbers fundamentally transformed how I viewed my brother. Years after he died, I took a job working with young children on the autism spectrum. This job required me to learn how to complete these rubrics called “individualized education programs,” or IEPs, used to plan educational goals for children with cognitive and other disabilities. At the time, IEPs were (primarily) constructed of a series of quantifiable assessments: indicators and benchmarks. Ironically, these individualized educational plans only allowed for a fairly narrow range of variation among children. IEPs enabled us junior teachers to monitor and evaluate students’ progress toward intended outcomes — often signifying distance from a neurotypical norm. We needed to help Benjamin practice holding eye contact. More seconds of eye contact? Benjamin’s improving! By redirecting Lissy when she was obsessively twiddling shoelaces, we were decreasing the amount of time she spent “perseverating.” With Jonathan, it was the number of “meltdowns” he had per day that positioned him on a bell curve of distress tolerance.
By design, our understanding of individual children’s growth morphed according to these indicators. What counted was what we were counting. In this context, I learned the special education definition of “profoundly retarded” — a phrase that had defined my brother’s educational needs for as long as he lived. He too had an IEP. Retrospectively comparing him to the children with autism with whom I worked, made him appear to be nearly inanimate, closer to a vegetable than a normal, healthy child — still less a child with secret super powers.
As someone who has personally grappled with the challenge of deriving the truth from multiple conflicting perspectives, I have thought deeply about what is at stake in our attempts to find and proclaim the truth, especially those truths that intimately affect other people’s lives.
If we want our research to be useful, we must provide evidence that is credible to audiences that matter. The figure of the Modest Witness might be instructive to us as evaluators trying speak truth to power. But this history raises some really tough questions the field needs to grapple with — particularly around the acronym of the day: DEI (or diversity, equity and inclusion). Who gets to be read as “modest?”
Whose witnessing is credible to whom? What kinds of evidence persuade? We are navigating these questions in a moment of epistemological crisis over what counts as — and sometimes the very existence of — the truth. So given this incredibly fraught history, and this complex moment, where does that leave us as evaluators, who want to be as accurate as possible, who want to speak clearly and concisely, but also to capture an elusive sense of truth, one that reckons with power and multiple perspectives?
Answers to these questions in Part 3.
