Do Facial Expressions Accurately Represent Feelings?
Facial recognition technology is about to get an upgrade that could change the way we measure the impact of advertising. And of course it’s coming from Facebook.
In May, the US patent office approved two new Facebook patents for technologies that analyze the facial expressions of it’s users so that it can have a better understanding of their emotional response to content. Facebook will then use that data to serve up more relevant content.
News coverage of the patents focused on the changes that could come to the user experience on Facebook, like the the ability to like and dislike content with an expression, but there’s another application of this technology that could have significant implications on the advertising industry: the use of facial expressions to measure the creative impact of advertising.
Hypothetically, Facebook could use the facial data it’s already collecting to tell it’s advertisers how people are reacting to ads. Facebook hasn’t claimed any plans to use the technology this way but it would make sense if they did, given their growing investment in measurement and transparency.
And there’s precedent for the use of facial expressions in ad testing; Affectiva and Imotions are two relatively new companies that record and analyze facial expressions as people watch ads to measure emotional response, and consequently an ad’s creative impact. Ali Goldsmith wrote a great piece on this technology last June.
Why are research companies so hot on facial recognition right now?
The first reason that facial recognition is on the rise is that advertisers are starting to realize that emotion is a better predictor of advertising’s business impact than classic metrics like ad recall and awareness. The 2013 IPA paper “The Long and Short of It” famously showed that emotional campaigns perform better than rational campaigns. This is a massive subject that we’ve covered before, and it’s worth diving into the topic if it’s new to you.
The second reason is that advertisers are starting to doubt the accuracy of self-reported metrics, like the ones you find on most market research surveys, and they’re looking for more passive ways to measure consumer response to advertising. The concern with self-reported surveys is two fold:
- People aren’t always consciously aware of advertising’s impact and therefore have a hard time recording it in a survey. Research has shown that people tend to remember rational messages more than emotional messages because emotional messages are processed at an almost subconscious level. Robert Heath’s book Seducing the Subconscious, provides a great explanation of the theory behind this behavior.
- People tend to give answers that researchers want to hear even if they’re not true. In a 2014 study of church attendance, the Public Religion Research Institute found that people who were surveyed in-person tended to exaggerate how often they go to church more than people who were surveyed online because people are generally more compelled to give socially-acceptable answers, even false ones, when talking to real people.
Can the expression on your face actually predict the success of an ad?
The evidence is far from definitive but a new book by Lisa Feldman Barrett, an author and psychology professor at Northeastern University, is casting doubt on the reliability of facial expression data. “How Emotions Are Made” puts forward a controversial new theory that emotions aren’t basic universal things we’re born with, rather they are mental constructs made from a more basic set of sensations (arousal, calmness, unpleasantness and pleasantness) and our cultural understanding of the world around us.
This theory, called the “Theory of Constructed Emotion,” contradicts the classic model of emotion that was outlined by Psychologist Paul Ekman forty years ago. His theory claims that all of us are born with six basic emotions that are hard wired into our brains (anger, disgust, fear, happiness, sadness, and surprise).
To come to this conclusion, Ekman conducted a study where he showed pictures of different facial expressions to cultures around the world to find out if they were universal signifiers of basic emotions. He found that they were, and built something called the Facial Action Coding System to categorize human expressions according to emotion. He’s been seen as a leader in the science of emotion ever since, and was even used as a consultant on the Disney movie about emotions, “Inside Out.”
But Barrett’s book highlights a methodological flaw in Ekman’s study: he wasn’t just showing pictures of faces, he was describing western emotions to participants prior to showing the facial stimulus, and then providing context around what happened to cause the facial expression they were seeing.
Barrett reproduced the study, without the flaw, with a remote tribe in Namibia, Africa, called the Himba, who have had almost no exposure to western culture. She found that participants described behaviors when they were shown the facial stimulus, not emotions. Even when they were asked to categorize photos by emotion.
“Smiling faces were not ‘happy’ (ohange) but ‘laughing’ (ondjora). Wide eyed faces were not ‘fearful’ (okutira) but ‘looking’ (tarera).”
Barrett goes on to explain that the brain uses more than just a facial expression to understand how others are feeling. We use context — what happened before and what’s going on around the face — and past experiences to create a mental categorization of the emotions people are feeling. Without context and a cultural knowledge of what that context could lead to, we can’t accurately tell what emotions people are experiencing by the expression on their face.
“In every waking moment, your brain uses past experience, organized as concepts, to guide your actions and give your sensations meaning. When the concepts involved are emotion concepts, your brain constructs instances of emotion.”
The two pictures below illustrate this dynamic well. If you only look at the picture on the left, you might think that the woman it shows is angry or terrified. But when you look at the same face with more context, you realize it’s an elated Serena Williams, who just beat her sister in the 2008 US Open. And you use your cultural knowledge of sports to understand this, even if you don’t realize it.
Barrett’s research is still very fresh, so companies like Affectiva aren’t going to go out of business tomorrow. But her argument is compelling. And it could create a problem for Facebook if they look to commercialize the technology as a stand-alone solution.
Facial scanning technology is usually used in tandem with other measurement tools like self-reported surveys, because it’s still new and unproven; market research giant Millward Brown has a partnership with Affectiva where they bundle facial scanning technology with an array of standard surveys. I expect Barrett’s book to reinforce that doubt and help keep facial scanning in it’s place: as a supplement to more established methodologies.
References:
Barrett, L.F. (2017) “How Emotions Are Made: The Secret Life of the Brain,” Houghton Mifflin Harcourt, Boston and New York.
Binet & Field (2013) The Long and Short of It, Balancing Short and Long-Term Marketing Strategies, IPA.
CBI Insights (2017) “Facebook’s Emotion Tech: Patents Show New Ways For Detecting And Responding To Users’ Feelings,” retrieved August 11, 2017 from https://www.cbinsights.com/research/facebook-emotion-patents-analysis/.
Heath, R. (2012) “Seducing the Subconscious: The Psychology of Emotional Influence in Advertising,” Wiley-Blackwell.
Paulson, M. (2017) “Americans Claim to Attend Church Much More Than They Do,” New York Times. Retrieved August 10, 2017 from https://www.nytimes.com/2014/05/18/upshot/americans-claim-to-attend-church-much-more-than-they-do.html.
