It’s Not What It Looks Like

Or, How The Representativeness Heuristic Is Ruining Every Fucking Thing

Possible prereq: Buckets and Bayes

“Evaporative cooling” is a metaphor describing one way that groups can become more extreme over time. Basically what happens is that zealots or militants act in ways that the most moderate people find distasteful, those moderate people leave, and the average level of zealotry or militancy goes up. That alienates the remaining moderates, who also leave, setting up a self-reinforcing spiral that drives away more and more people until all that’s left is an echo chamber full of die-hards.

This essay isn’t exactly about evaporative cooling, but it’s about another slippery slope that feels pretty similar to me. It’s about representativeness, and how (as the title foreshadowed) I think it’s ruining every fucking thing.

The representativeness heuristic is when, as a shortcut, you decide that something is what it looks like. It’s using prototypes and archetypes to simplify the world around you, divvying things up into predefined categories that you already know how to deal with:

• Does Linda have brightly dyed hair? Did she attend college on the West Coast? Is she single, outspoken, and intelligent? Then she’s probably a feminist. After all, she looks a lot like the archetype of a feminist.
• Does Richard wear football jerseys? Does he drive a truck and listen to country music? Is he a tradesman in his thirties with three kids? Then he’s probably a Republican. After all, he looks a lot like the archetype of a Republican.
• Is Jamal over six feet tall? Is he joking and laughing loudly on the street corner with a group of his friends? Do any of them have tattoos or sagging pants or rap music playing on their phones? Then you should probably clutch your bag or briefcase tighter as you walk past them. After all, they look a lot like the archetype of street hoodlums.

Whoopsie, right? That last example is clearly Not Okay—it’s at least implicit bias and probably outright racism. Situations like that one are places where we’re not allowed to use representativeness, even if (say) we’re cops in an economically blighted city and those details are reasonably strong predictors of danger.

That’s because, as a society, we’re slowly learning that it’s inaccurate and unfair to judge individuals according to the properties of the groups they happen to resemble. Even if it were true that 70% of the people matching Jamal’s description present a clear and present threat (and it’s absolutely not true), it would still be unacceptable to treat the innocent 30% like criminals. That’s how you end up with systemic oppression and police shootings and people dropping out of school because why bother, there’s no escaping the stigma anyway, my dad’s a Harvard PhD and they still pull him over because of his skin color.

So we give ourselves sensitivity training, and paper over our knee-jerk judgments with slightly embarrassed second thoughts, and add caveats and qualifiers to the things that we say—not just about race, but about gender and religion and sexuality. What would’ve been “that’s so gay” in high school in the 90’s becomes “I dunno, it seems kind of effeminate to me,” if we say anything at all.

And this is progress. It’s a band-aid, since our brains remain jump-to-conclusions machines no matter what we do, but at least we’re trying.

The problem is, it’s a blacklist, not a whitelist. As a society, we’re carving out exceptions to a general standard of representativeness—choosing specific places where we remind each other that it’s not allowed. The vast majority of the time, representativeness is not only rampant, it goes completely unnoticed. People don’t even notice that they’re doing it when you point directly at it. We’re immersed in it, like fish in water.

And it’s drowning us.

A few days back, I was scrolling through Facebook and I came across some photos posted by a friend of mine, of people drinking and dancing at a party. The photos were accompanied by a caption that went something like “Taking a risk, here, since I’m between jobs and I’ve heard potential employers look down on this sort of thing. But come on—we’re all adults.”

In the replies, a couple of people questioned why there’s such a strong taboo against posting party pics. I mean, really—do employers think that people don’t drink on the weekends? Do they think that people who don’t know how to unwind and relax are somehow going to be better workers?

No, answered another mutual friend. It’s more complicated than that:

I think it’s more that employers can often see things in party pictures which they have good reason to disapprove of — for instance, if there’s pictures of you doing illegal/stupid/dangerous things whilst drunk, breaking drug laws, or going out when you’re supposedly on sick leave. Because many people can’t tell where to draw the line (dancing pictures are fine, rave pictures with visible mydriasis are maybe not so fine, nonconsensual creep shots of drunk people getting naked are very bad to post, shots of you doing property damage while wasted are definitely a bad idea) it’s become very common to loudly advocate a “no party pictures” heuristic. Then [if you do it anyway] other people get meta-judgmental that you’re not following the heuristic they think is sensible.

Picture Jordan, about to post some photos of a wedding afterparty to Facebook while their roommate Kelley looks on.

Kelley: “I wouldn’t.”
Jordan: “What? Why? There’s absolutely nothing wrong with this picture.”
Kelley: “Yeah, but it just looks kind of bad, you know? Like, they’re looking at your résumé, and then they look you up online, and they see this, and the other applicant has, like, just pictures of their cat, or whatever.”
Jordan: “There’s nothing wrong with this picture.”
Kelley: “I know, but it kind of makes you look like you party, you know? Like, if you’re posting this, what other pictures are you not posting.”
Jordan: “That’s so fucking stupid.”
Kelley: *shrug*

The problem is, when Kelley is offering feedback on the photo, there are a bunch of different levels all getting smushed together. There’s what Kelley thinks of the photo (probably fine), what Kelley thinks employers will think of the photo (probably not fine), and what Kelley thinks Jordan thinks that employers will think of the photo (that it’ll be fine, except that Jordan’s naive to think that), etc.

And the further Kelley goes imagining what people imagining other people might imagine, the dumber and less accurate the predictions get, until finally the only things that work are simple, black-or-white rules. “Just don’t do that [because everything else is too subtle or complicated for me to be confident about it].”

Just don’t do that, because then you’re going to look like the idiot partiers that your future boss doesn’t want to hire. You’ll be painted with the same brush.

It’s not hard to see how this happened. Start with a few high-profile cases of stupid people posting pictures of themselves doing stupid stuff and then getting fired (or not-hired). That draws attention to the idea that employers care about online pics, and are looking out for them, which leads to the creation of a heuristic: don’t post pictures of yourself drinking alcohol.

The heuristic is overly restrictive; it’s erring on the side of safety (just as our knee-jerk racist errs on the side of safety by clutching their bag). That’s true of most heuristics, since humans are risk-averse and loss-averse.

In response, the most cautious people—those who are the most anxious, and were the most hesitant to post a party pic in the first place—just stop outright. Not worth the risk, they tell themselves, even though the actual risk from their brunch group selfie is essentially zero.

Meanwhile, employers who’ve caught the trend start paying more attention, and catching more stuff on the other end—things that might’ve slipped by before, but which are now defensibly objectionable.

These two dynamics put pressure on the system from both ends. Moderate, well-behaved people are posting fewer party pics, and more party pics are being punished. The overall population of party pics shrinks, and it shrinks such that it becomes more extreme. Fewer of the non-objectionable pics are being posted, and so the overall impression “party pics are generally bad” gets stronger, and “correctly” so.

Lather, rinse, repeat. The stigma is stronger, so more people who are sensitive to the stigma drop off, leaving behind only those who are oblivious or rebellious. The oblivious and the rebellious tend to post worse pictures anyway, so the stigma gets strengthened and seems to be more justified, causing more people to stop, until eventually there are approximately zero party pics that aren’t objectionable, and then everyone just goes “see?”

(Except for a handful of my Facebook friends, who wonder despairingly how on Earth this happened.)

So, you can’t post party pictures on Facebook, even though doing so would be essentially harmless the vast majority of the time. No big deal, right?

If you have the time, read Motherhood in the Age of Fear, an opinion piece recently featured on nytimes.com. If you don’t have the time, here’s the story behind the story:

• A few neglectful, bad parents do a few horrible things, like leaving their children in the car with the windows rolled up for hours in the middle of a hot summer day. Someone gets hurt, or even dies. The police get involved. The story is on the news.
• A few bad things happen to other children, as is basically inevitable in a country with thirty million children. Someone drowns in a lake, or is kidnapped from a local park. These stories rise to the top, thanks to perverse incentives.
• Over time, all of the stories smush together in the viral narrative, cohering into a stereotype of a neglectful parent—lazy, selfish, uncaring, irresponsible. That image has a few key triggers attached to it, like kids in the car, kids coming home alone, kids wandering through the neighborhood unsupervised.
• Nobody pays attention to base rates (the fact that for every one disaster, there might be a thousand or ten thousand or a hundred thousand kids who come through a similar-looking experience just fine). The vigilant and the self-righteous begin to police their neighbors according to the representativeness heuristic—if it looks like neglectful parenting, then it is neglectful parenting.
• The real police get called (or social services, or the neighborhood watch). People whose job it is to take these things seriously—people who will be held accountable if they don’t take things seriously, and then something bad happens. So naturally, they do take things seriously, and charges are filed, or people are put on probation, or reputations get ruined.
• The more wary parents stop taking the risk. They cave to the pressure and start hovering over their kids all the time—not because they want to, but because if they don’t, they’ll be punished for it. They need to not only be attentive, caring parents, but also to look like attentive and caring parents, all the time, because it just takes one moment of looking like the negative stereotype for someone else to call you out.
• The problem gets worse as more and more conforming, law-abiding, I-don’t-want-any-trouble parents drop out. Of the remaining parents, a higher percentage are actually neglectful (still not a high percentage in absolute terms, but higher than before). Another couple of kids get kidnapped or hospitalized, and the jump-to-conclusioners point to this as evidence that they were right all along. Parents who originally thought “that doesn’t seem right” now switch sides, and the cycle accelerates.
• Mothers who leave their kids in the car in plain sight, in nice neighborhoods, on cloudy days, with the windows cracked, for five minutes while they get a cup of coffee, are being screamed at and taken to court and often charged or forced to do community service even when they’ve broken no law, because the other people watching them have a picture of what a bad mother looks like, and their little picture-matching algorithm went “ding!”

It all starts when we, as a society, let go of the distinction between what it looks like and what it is—when we do the mental equivalent of clutching our bag just because the person in front of us is tall, young, and black. In particular, it starts when we let others get away with forgetting that distinction—most of us aren’t going around calling Child Protective Services on parents whose kids are obviously doing just fine in the parking lot, but we’re also not proactively stepping up to defend those parents from attack.

(It’s like how the world would look if we all stopped shouting down people who are openly racist or sexist. The number of actual racists and sexists would remain a tiny minority, but the prevalence of racist and sexist behavior would skyrocket. Those things are shrinking in today’s society, not because the bigots are going away, but because the rest of us are abiding by a contract where we don’t let them get away with it.)

I did a quick scan for other examples of this dynamic. They weren’t hard to find:

• My colleague Valentine likes to use shorthand from mythic and mystic traditions as he teaches rationality. These terms are evocative and memorable, and they often point at very real phenomena—a lot of the wisdom of the ancients is actual wisdom, for all that we’ve overturned the more farfetched bits. Yet because it sounds like stuff that only crazy mystics say, he often has to spend half of his time and effort just proving that he’s not crazy. The alternative is to spend even longer spelling it all out step by step, only to have someone say “isn’t this just chakras?” at the end of a forty-five minute lecture and ruin everything anyway.
• I once reposted a Facebook comment that I found deeply insightful, only to receive criticism because the original comment was referencing an article from The National Review, and my cousin couldn’t believe I was agreeing with something printed in that garbage fire.
• Single fathers in the United States have a hell of a time avoiding accusations of pedophilia when they take their kids to the park or the playground. Doesn’t matter that for every child molester there are literally hundreds or thousands of good dads, or that the vast majority of child abuse doesn’t involve random kidnap. If you’re a lone adult male at a playground, you look like people’s mental image of a child molester, and many people will treat you as such. (Indeed, this same downward spiral plus a similar spiral around touch can be blamed for the almost complete disappearance of all healthy, non-familial adult-child relationships in middle class America over the past few decades. Many elementary school teachers are now explicitly forbidden to give a comforting hug to their crying first-grade students after a skinned knee!)
• For some reason, we have a narrative that it’s okay to dismiss or criticize people who are being defensive?? Which, in my theory, would be explained by some small fraction of defensive people going over-the-top, seeing threats where there weren’t any or resorting to poor reasoning or emotional blackmail in response to an attack. We-as-a-culture get an image of an unjustifiably defensive person, we remember what that person looks like in stereotype, and then anyone who’s even a partial match for that picture gets treated as if they’re an instance of the bottom quartile.
• I’ve been wanting to host an event at my house called “naked colloquium,” in which we stand up and give informal, ten-minute talks to one another on intellectual topics, followed by questions and discussion. I’ve wanted us to do this naked in part because society’s views on clothing and nudity are bass-ackwards to begin with, and it’s low-hanging fruit (haha) for expanding people’s comfort zones and practicing public speaking under stress and building up confidence that doesn’t depend on expensive suits and makeup. But mostly, I’ve wanted to do it naked specifically because I know that, whenever people hear the idea, they’ll immediately think it must be a weird sex thing, even though no.

It’s worth pausing for a moment to point out that representativeness is not, in itself, a bad thing. Indeed, the fact that our brains do this is a huge part of how we manage to successfully navigate a horrendously complex world.

The key is, representativeness should inform your prior belief. It’s like betting “red” if I’m pulling marbles out of a bag that’s 70% red and 30% green. Once you have more information, you need to update to a more accurate posterior belief, just as you wouldn’t plug your ears and deny it if the marble was actually green.

(I mean, most of the people who talk about chakras really are kind of nuts. It’s not a bad bet to make. The problem is when people are so sure of their bet that they miss the opportunity to notice that Val’s smart and sane and trustworthy and trying to teach them something they’ll be better off knowing.)

Unfortunately, a lot of people really don’t want to be wrong. They don’t want to back the losing horse, and they don’t want to have defended someone who turns out, in fact, to have actually been bad. In situations like the ones described above, the most attractive option is to go with the flow, and judge what everyone else is judging. The second most attractive option is to say some middling, non-committal thing that is inoffensive no matter what the truth turns out to be.

The thing that approximately nobody wants to do is go out on a limb to actively defend a guy who looks creepy at the playground, but probably isn’t. There’s non-negligible risk in that, and very little reward.

So what do we do instead?

My attempt at finding a solution: split and commit.

“Split and commit” is a phrase I want people to throw around at each other. It’s a request I want all good people to be able to make of one another, just as we currently ask one another not to act on any unjustified knee-jerk bias that might pop up when we’re dealing with someone of another race or gender or culture.

It’s a simple maneuver: you split your impression into what-it-is-if-it’s-what-it-looks-like, and what-it-is-if-it’s-not-what-it-looks-like. For instance, if you catch your significant other in what seems to be a rom-com-style compromising position with another person, you put forth whatever mental energy is necessary to form two impressions of the situation, one compromising and the other charitable.

Then, you commit to a course of action in each scenario. Maybe you do this publicly, if (for instance) one of your colleagues is being accused of improper conduct, or maybe you just do it quietly, in your own head. But you think to yourself, on purpose, okay, if this is what it seems to be, my response should probably be something like X. But if this is just a misunderstanding, if it just looks bad but it isn’t actually bad, then a correct and appropriate response would look more like Y.

“I hear that Blake has accused Cameron of sexual misconduct, and I absolutely agree that if Blake’s accusation is true and accurate, this is serious. In that case, I’d be comfortable taking Cameron’s name off the door, sending out an email to all of our clients, even testifying if necessary. But look, right now we don’t have any firsthand knowledge, and no one’s done an investigation, so I think we should also keep in mind what we need to do if this accusation turns out to be false, or even just inaccurate or incomplete. In that case, Cameron’s going to be out in the cold, and will need defending. We’ll need to figure out some kind of disciplinary response for Blake. In fact, now that I think of it, even putting Cameron on leave today feels a bit premature—that was my first instinct, because it kind of looks bad and I want onlookers to know we’re taking things seriously, but since we don’t actually know we should think for a minute and see if there’s a way that we can protect Blake without sending any premature negative signals about Cameron…”

What this leads to (given sufficient buy-in) is a culture that’s got just a bit more critical thinking at the critical moment. It’s a culture where people don’t fall in love with a single course of action. It’s a culture where people start thinking, in advance, about what bits of evidence distinguish World A from World Not-A. It’s a culture where we remind each other that the vast majority of hoof-sounds belong to horses, not zebras, and that even in areas where zebras are in the majority, you’re still allowed to be a horse, and claims of horsehood should not be sneered at by default.

Furthermore, it’s a culture where people can’t do the waffling, noncommittal, “well, we don’t really know so let’s not form an opinion yet” move that serves only to protect one’s own butt from attack while leaving involved parties out in the cold.

(Or worse: the “we don’t really know yet, but we want to err on the side of safety and prudence” reaction, which is really just “we’re going to conform to whatever we think the loudest possible complainers/accusers would demand that we do, so they don’t come after us for being negligent” and which greatly accelerates the downward spiral by reinforcing the implicit cultural narrative that you should respond to what it looks like even if you don’t know what it is.)

People often want to hedge their bets and take stances that feel like they’ll be okay in both possible worlds—they like being able to strategically switch between “that looks bad” and “well, looks can be deceiving,” and it’s hard to do that if you’ve nailed yourself down to one position. But if you practice nailing yourself down to two or more positions, you can not only dodge that risk yourself, but also help reduce the social availability of motte-and-bailey attacks.

So the next time someone says “I don’t know, Steve, that sounds like bullshit,” or “Listen, Sarah, you seem pretty defensive,” don’t just sit there and let it happen. Stand up, hit pause, and make them take ten damn seconds to check their assumptions, and form a clear mental model of the other possible explanation. That we do this already with race and gender and sexuality and religion is a heck of a good start, but there are a bunch of other good things in our society that are circling the drain and at risk of disappearing forever.

Further reading: Common Knowledge and Miasma