Hollaback and Why Everyone Needs Better Research Methods
And Why All Data Needs Theory
A few days ago, a video produced by a third party “viral video creative agency” for the anti-harrasment NGO, “Hollaback,” went viral. The two-minute clip showed a conventionally attractive, white actress walking the streets of New York for “10 hours”, and being repeatedly catcalled. But you couldn't help noticing that all men catcalling to the white actress were people of color — actually almost all black men. At the end of the video a text said that “100+” instances of “verbal street harassment” had occurred during these 10 hours, from “people of all backgrounds”
So, what gives?
In effect, this was a research project, and it had an implicit research question (“Do conventionally attractive white women get verbally harassed in New York?”) and produced an answer: the video. However, in doing so without any reflection on its own method, it amply demonstrates the crucial substantive and political importance of research methods.
I’ve taught “introduction to research methods” to undergraduate students for many years, and they would sometimes ask me why they should care about all this “method stuff”, besides having a required class for a sociology major out of the way. I would always tell them, without understanding research methods, you cannot understand how to judge what you see.
The Hollaback video shows us exactly why.
The Hollaback video also shows why “data” without theory can be so misleading—and how the same data can fit multiple theories. Since all data collection involves some form of data selection (even the biggest dataset has selection going into what gets included, from what source), and since data selection is always a research method, there is always a need for understanding methods.
First, let’s list all the hypotheses compatible with the “data”, this video:
Hypothesis 1- Men of color are disproportionately more likely to catcall, especially to a white, conventionally attractive female.
Hypothesis 2- All men are equally likely to catcall but the makers of the video were biased, consciously or unconsciously, against black men (and edited out men of other races on purpose.)
2.a Consciously: they are racists and are playing to the “white women endangered by black men” trope — which has a long and ugly history, hence the concern raised by many over the past week.
2.b Unconsciously: There is a methodological twist to the research which creates this outcome.
2.c. Both 2.a. and 2.b are true.
Hypothesis 3- It’s a spurious correlation: there is some other reason that caused these two events to go together
The important methodological point is that the video, without further reflection, can support all three wildly incompatible propositions. In other words, if you just look at the video, you can believe any three, and you will likely choose whichever fits your existing conclusions and prejudices..
Let’s start with 3, the easiest to dismiss.
A spurious correlation occurs when a third, unrelated variable, causes a change in other variables, which then seem like they are causally connected even though they are not. And since the human brain is a narrative writing machine, seeing A and B together makes people write stories that tie A and B. A silly, but correct, example is the correlation between ice cream and murder: during months when more people eat ice cream, there are more murders. This is not because popsicles are good murder devices. This spurious correlation caused by a confounding variable: the season. There is more crime in summer, and there is more ice cream in summer.
While it’s so obvious in summer and murder example, many major errors have been made historically because researchers missed confounding variables. A famous one is the Hormone Replacement Therapy which researchers thought decreased cancer risk when in fact, it increased it. The confounding variable was that healthier women were more likely to get the therapy in the first place. This mistake has certainly caused many deaths.
The most obvious confounding variable here would have been if Hollaback had their actress walk the streets of Harlem, or other Black neighborhoods, and skipped over white ones. But they claim they walked all of New York, and did not just pick minority neighborhoods.
Thankfully, someone did the leg work, and mapped out where the shots were—something any researcher producing a two minute video from 10 hours should have done, and a great example of why “anecdotes” rather than systematic research can be so misleading.
Take a look at their pie-chart:
As the authors correctly state, here’s the question this distribution raises:
The filmmakers claim to have shot this video while walking the streets of Manhattan for 10 hours, but over half of the shots in the video are actually taken from just one street, namely 125th St. in Harlem. It makes one wonder whether the filmmakers intentionally chose to concentrate their filming on a couple of neighborhoods, or if, out of many locations, these are the only places where harassment occurred.
Let’s now take hypothesis 1. This will obviously appeal to many people, as it plays to long-rooted prejudices, and people may well believe that this is their experience. Most people, especially white people, are more likely to make note of a person’s race if they are not white. In other words, a man who is a black man is more likely to be described as a “black man” than a white man is likely to be described as a “white man” instead of … just a man. This differential salience of information is why research studies try to carefully systemize their observations rather than rely on people’s impressions which bias towards whatever is personally salient—whatever you care about, you will notice more of.
It’s clear many viewers jumped to the “hypothesis 1” conclusion: that the video was a representative sample of catcalling. For example, prominent author Joyce Carol Oates seems to believe this. She’s certainly not the only one.
This is a great example to why personal casual observation and anecdotes, which is so important to our sense of identity, cannot and should not be trusted blindly. Systematic study aimed at removing our biases is crucial to understanding the world.
Removing the means of implicit biases can be eye-opening. For example, after decades of lack of women in major orchestras, some big symphonies started doing their interviews blind — musicians played their instruments behind a curtain. Lo and behold, women, previously greatly lagging in professional employment in symphonies, started being evaluated as performing much better.
Hence, given great methodological distortion in selecting examples, the video will remain compatible with some people’s pre-existing idea that catcalls are racially disproportional, but will not actually provide a methodologically sound way of evaluating it.
To further evaluate hypothesis 2, let’s turn to what the producer of the video said. According to a Slate article, he explained what happened like this:
“We got a fair amount of white guys, but for whatever reason, a lot of what they said was in passing, or off camera,” or was ruined by a siren or other noise. The final product, he writes, “is not a perfect representation of everything that happened. “
And that right there, is indefensible, methodologically or substantively. The only neutral explanation is that there is a lot of construction, ambulances and sirens going on in more white parts of New York, and somehow they just cannot catch a catcalling white guy. That sounds implausible to me, but if—a big if—that were the case, an ethical researcher would redo their study because the data is no longer valid because of the confounding variable—noise at non-minority neighborhoods. At a minimum, some effort would need to make sure the presentation of the findings were not tainted by this methodological obstacle.
There is also a whole discussion about the impact of the choice of a conventionally attractive white, young, actress on the findings (even if they were presented perfectly which they are not) and other choices made. In fact, every element of research design requires hard thinking—and that’s what good social science does, and what bad anecdotal, jumbled, biased viewpoints miss.
Over at Twitter, David Chen makes another important point: This video was designed to “go viral” and a video that started with a few white guys doing the catcalling might not have.
That is a great point, and a great discussion about the context of “going viral”; however, the point here is that this is methodologically unsound regardless of intent or conscious (or unconscious bias).
Research methods, a topic that is seemingly so dry, are the heart and soul of knowledge. Most data supports more than one theory. This does NOT mean all data supports all theories: rather, multiple explanations can fit one set of findings. Choosing the right underlying theory, an iterative process that always builds upon itself, requires thinking hard on how data selection impacts findings, and how presentation of findings lends itself to multiple theories, and how theories fit with existing worldviews, and how better research design can help us distinguish between competing explanation.
A good research project consciously grapples with these.
A bad one? It flunks, as would this one.
