What’s In a Face

An essay about the return of physiognomy and why we should welcome it.

The story begins with a weird piece of science.

R. Thora Bjornsdottir, Nicholas O. Rule. The Visibility of Social Class From Facial Cues

In a new twist on first impressions, the study found people can reliably tell if someone is richer or poorer than average just by looking at a “neutral” face, without any expression.

Writes the University of Toronto. And the researchers not only present the experimental results — they also have a theory at hand for how we do the trick:

This is due to visibility of the positions of muscles that become etched in the face over time as a result of repeated life experiences.

Positions of muscles? Life experiences? Sounds a bit vague, honestly. But they do see the social implications of this: “People talk about the cycle of poverty, and this is potentially one contributor to that,” says Rule. Fair enough — the findings are certainly noteworthy, for reasons we will see later. Depending on the accuracy of the guessing game, of course. “People can reliably tell it someone is richer or poorer than average” — well, say: how reliably exactly?

They were able to determine which student belonged to the rich or poor group with about 53 per cent accuracy, a level that exceeds random chance.

Say that again? If I flip a coin a 100 times and it shows heads 53 times, do I suppose what just happened exceeds random chance? Probably not. If I do it a 1000 times and I get 530 heads? Still nothing crazy. Maybe if I do it a million times and still get 53 percent heads, I start suspecting there is something fishy about my coin. Did Bjornsdottir and Rule run their test a million times? I have not checked in the data section of the original paper, but I guess not.

So can we please just dump that hilarious example of sloppy science and forget about the whole face reading magic and all the unpleasant implications? Not so fast. Even though they probably got it all wrong, Bjornsdottir and Rule’s work is full of the uncanny stuff the nightmares of sociologists and responsible AI researchers are made of. Are we really judging people from their physical appearance? We do, of course we do. To what extent do we do it? What is in a face exactly and how does it relate to the way we categorize people? And, next level uncanniness: If we do it, can machines do it too?

Let us talk about this nightmare a bit before we come to the man/machine part of the story. The concept is called Lookism, but it is completely missing in the discussion of the results. But the authors in fact do make reference to it — implicitly — in the abstract of the paper:

Further investigation showed that perceivers categorize social class using minimal facial cues and employ a variety of stereotype-related impressions to make their judgments. Of these, attractiveness accurately cued higher social class in self-selected dating profile photos.

So what they have found (let’s be nice and suppose there actually was a signal and the 53 percent were not just random noise) is that participants seemed to be quite good at recognizing attractiveness and that they could use this to rate the photos. The arithmetics of this is quite simple: more attractive people = richer people. And this correlation is studied quite well indeed.

So here is for the positive side of lookism. If you happen to be tall and attractive, a very favourable bias will lift you up the ladder to better jobs and higher salaries. People will like you, people will trust you, they will think you are capable of amazing things.

Surprisingly (or not so surprisingly, maybe), the negative side of lookism is something of a scientific black hole. And that is where AI comes into the story.

About a year ago, two chinese researchers published the result of an AI experiment in which they taught a computer to distinguish between criminals and innocent people, with a series of photos from Chinese records — the criminals’ portraits were taken in prison.

We study, for the first time, automated inference on criminality based solely on still face images, which is free of any biases of subjective judgments of human observers.

The method used is nothing of a secret, in fact it was pretty simple and straightforward supervised machine learning: From a training set of pictures, the computer learns to extract qualities that describe the two different classes (criminals/non criminals) in the best possible way. The result was so frightening that there was hardly any serious discussion but rather a damnation of the research idea as such: Given a new set of photos, the AI could distinguish between criminals and non criminals with an accuracy of about 90 percent.

Sure, it is actually quite easy to identify serious methodological flaws in Wu and Zhang’s work. Blaise Agüera y Arcas did it in detail and with much expertise here, I will leave it up to the interested reader to dig into that. Wu and Zhang do have quite a naive understanding of their methods, that’s for sure. A sort of new positivism actually very widespread in the big data and machine learning community: machines can see things better and clearer than humans.

we adopt the approach of data-driven machine learning to fully automate the assessment process, and purposefully take any subtle human factors out of the assessment process. Unlike a human examiner/judge, a computer vision algorithm or classifier has absolutely no subjective baggages, having no emotions, no biases whatsoever due to past experience, race, religion, political doctrine, gender, age, etc., no mental fatigue, no preconditioning of a bad sleep or meal.

But maybe that is not the point. Let us for moment assume that, given enough good quality data, the machine can actually do the trick — if not now then in a very near future. So let us assume it can find distinctive signs that tend to be present in criminals but not in non criminals (or the other way round) —which would of course imply that these distinctions really exist. And they might indeed: A simple example could be scars — but that is maybe not relevant, as it does not consist of an inheritable trait. Wu and Zhang actually tried to account for these kind of physical deviations. So how about facial symmetry? Maybe a wry face actually is more common in criminals than in people with successful careers?

Success: Now we have a point. It’s actually these over achievers that open up the discussion and lead us to the more interesting and more unsettling implications of Wu and Zhang’s research. Maybe what they have found is a striking case of negative lookism. Lookism is a societal bias like sexism or racism, only that it sorts people not for gender or skin colour, but for beauty, or to be more precise: attractiveness. The effect has been well studied in a positive way: It is well known that taller people earn more, and researchers have estimated it at around 5000$ a year per 10 centimeters body height. The same effect can be found with faces. Symmetry seems to be of main importance there. If we acknowledge this kind of positive lookism as an observable fact, shaping our society in a way that gives some of us an unfair advantage, why not extrapolate the same effect to the other side of the spectrum, to the dark side of things — and assume that some of us are born with a probably even more unfair disadvantage. In fact, if lookism is strong enough, we should not be surprised to find a societal bias, correlating unattractiveness with likeliness to commit crimes. That is not to say that there is a causal connection: less attractive people are not inherently prone for crime, but it might as well be that society pushes them over to that side.

And we do know the logic: Blacks are more likely to end up in prison than white people? It’s a fact but nobody would dare to say it’s in their genes — well, nobody reasonable at least. If we have second thoughts about some people being more prone for trouble, it’s a self fulfilling prophecy of the very unfair kind. So what Wu and Zhang’s AI might have found is not a confirmation of the old ideas of Lombroso and sorts, but a very contemporary and very worrying effect nobody seems to be willing to talk about.

So if a machine can be taught to learn our notion of beauty/attractiveness, it can probably get a sense of the biases linked to these concepts as well. We should embrace these results and try to understand the implications before we condemn the whole thing right away because it brings up dark memories of the past. Maybe Lavater and Lombroso were not so wrong in the end. They just did not connect the dots in the right way. If lookism is real and if it has a powerful negative lever as well, we better include it into the Isms we are trying to get a hold of — and get rid of — these days. What if there were some sinister blind spots hidden behind sexism and racism?