Facial Recognition and Mood Detection in Media: How the trend is emerging in advertising, marketing, music, and more
--
In our current situation, everything we could possibly need or want is essentially converging onto one entity: The Internet. Specifically, people have access to music they like, connections to friends and family, access to TV shows, movies, news and videos all across social media. Most of these platforms allow you to sign up for free, and monetize themselves in order to make money, which essentially means ads are paying for a myriad of these services that people love and can use. In a way, these services take something you like (a video on YouTube, a song on Spotify or Pandora) and show you an ad while you’re engaging in something enjoyable. Additionally, services like Facebook are taking your data and showing you ads based on what it thinks you’d be interested in. This is what we like about the current situation, as it is a step forward in personalizing ads toward consumers. However, what we don’t like is that there is not much direct engagement between the ads and the consumers. Sure, these platforms can show us the ads we MAY want to see, but they don’t know EXACTLY what we want to see and when. Also, ads are very expensive to mass market, so people usually see a lot of the same ads if they are using that platform often. In turn, most people find ads annoying and tedious. For years, the method of surveys and focus groups has been used to try and increase consumer engagement by finding out how those participants may feel about those ads or find out what they want. Also, if they do participate in focus groups or surveys, they expect some form of compensation and as an effort to keep customers from getting bored or frustrated, the surveys or focus groups have to be short and sweet, so not much data can be collected. According to Terry Lawlor in his article on MarketingResearch.org about the future of market research, “Marketing researchers are under increasing pressure to deliver real business value to their customers. Adding to that pressure, however, are the ongoing decline of survey response rates and challenges with collecting data from specific demographic groups. The challenge to find ways to complement panels, focus groups and surveys is on…” The true responses that most advertisers try and look for are emotional and personal connections with the target audience and how to further engage them with what’s being marketed.
A lot of the problems advertisers face is that there is no personal, real-time connection to consumers. Essentially, there is no way to show/give the consumer what they want, when they want it. “With a traditional view-then-report approach, some fleeting emotions may not even be recognized by respondents who are more likely to remember how they felt at more memorable points in the advertisement, especially at the end. If the emotions of the respondent are being observed, the danger is that different observers may interpret emotions differently.” (Lawlor, marketingresearch.org) We can apply this concept of real-time engagement with a variety of different media industries, for example, in the digital music service space, platforms like Spotify and Soundcloud try to curate playlists or a selection of music based on past listening history or what you’re in the mood for. However, the user has to go out and find the music themselves, instead of the service being able to know exactly what the user wants in real-time.
Real-time, consumer engagement with ads and other services can be seen as an issue for as long as businesses have been servicing consumers in our world’s history. However, the need for this engagement can be related to the fact that the current ways to show ads are simply not working anymore. If we look at the K-wave principle, which is the cycle of economic developments that either experience periods of great technological and economic growth or periods of decline each lasting around 40–60 years, we can see why there is need for change. Professor W. Thompson of Indiana University wrote an influential essay about how K-Waves have influenced technological development since as far back as the 900s. He claims we are heading into a sort of depression until around 2020 (zerohedge.com). This would mean we are starting to experience a K-wave “Winter” (a period of economic decline). Additionally, economic cycle theorist Harry Dent wrote a book called Demographic Cliff: How to Survive and Prosper During the Great Deflation of 2014–2019, which mentions how the boomers in the US are getting older as the “everyday consumer never came out of the last recession,” how young people “cost everything and produce nothing,” and how only the rich ones are spending money (Dent, Business Insider). In looking at these theories, we can determine that ads are not really working and the youth population that businesses need to target now, are not spending enough money. In addition to the K-wave theory actually taking action, we can attribute why it is happening to the fact that what marketers consider to be high value engagement, is not always thought of the same way by consumers.
In a recent report from Forbes Insights and Turn called “The New Rules of Engagement: Measuring the Power of Social Currency,” 250 marketing executives and over 2,000 consumers were surveyed. The report showed the disconnect between what marketers think is engaging and what consumers think. For example, the surveys showed that marketers think that gaining followers or experiencing activity on social media by consumers is a big way of engagement. However, when consumers were asked when they feel engaged with or invested in a brand, only 15% said it’s when they share an ad, while 49% of marketers said sharing an ad strongly influences their engagement measurement.
As a result of these factors, we can see why in the current situation, consumers are not as engaged as they should be. Other factors that are keeping this situation in place are legal obligations, such as privacy. It’s hard to engage someone without breaching their privacy and getting their consent first, so we cannot exactly see real-time consumer reactions in every moment of their daily lives. It has to be on a request basis, so making focus groups and sending out surveys have been the norm. We have also not really had the technology to evaluate a person’s reaction or feelings about something on a moment-by-moment basis. We have solely a view and response system, so other than reaching into their sub-conscious and seeing what emotions need to be triggered, we can only see how someone feels at the end of seeing/doing something. The growing media problem, need and opportunity for more consumer engagement and insight is here. If we look at how the changing consumer in the C-scape applies to media companies, we see how consumers are in control now more than ever, and a divide between marketers and the changing consumer does not help these media companies at all. Also, we need to utilize consumers who are creating user-generate-content as a way to get the word out and promote your business or whatever it may be. We see another aspect of the C-scape, convergence, as a big change in this way since all forms of media are converging onto one platform. We can look at any form of media being part of the storytelling process and there is now an opportunity to have a random Facebook or Twitter user be a major source of marketing for a business. It seems as though the advertising/marketing industry is moving in the direction of getting through to one of those users in the form of engagement and empathizing with that user. Essentially, you need to catch their eye and give them control over what to do with an ad or form of promotion. In addition to this, we can see a need for consumers to see exactly what they want in real-time, as marketers need to know about consumers on the spot just by looking at them instead of waiting for responses from the declining practices of surveys and focus groups. This need will rely on Content and curation, the third and fourth big business changes in the C-scape, and if the company can engage with a consumer more, the consumer will want to follow the company further.
Prediction of the new trend that MAY be coming to Media.
When I talk about these needs for marketing and advertising, I am referring to how businesses need to find new and better ways to promote themselves and give consumers a better experience. So, this can apply to various businesses across almost all media industries. We can apply these needs to when consumers are watching music videos, gaming, DJing at a party or even just walking around an area trying to find out more information about someone or something. There are not necessarily current issues with all of these actions, but there is definitely room for innovation. The new trend that may be coming to media is facial recognition technology used as a way to increase consumer engagement with any machine it’s interacting with. This engagement, could be based off instant and real-time mood detection in the consumer. For example, facial recognition technology seems to be moving toward acting as a replacement to surveys, focus groups and social media feedback (likes, comments, sharing, viewership, etc.). The technology can do this in combining with other technologies such as mood detection and augmented reality. Continuing what Terry Lawlor said earlier about the decline of focus groups and surveys in his article Facial Recognition in Market Research: The Next Big Thing, he also mentioned that emotion detection and technology that monitors facial expressions will provide some more opportunities by “capturing data as the respondent views a video…using technology throughout the viewing stage [enables] advertisers to understand how the tiniest elements of their video may impact audience response.” (Lawlor, marketingresearch.org) So, although marketers have been using focus group, survey tactics and online review methods for years, there are many things they still don’t know about individual consumers. Referencing one of Edward Cornish’s Supertrends from his book Futuring: The Exploration of the Future, technological progress can explain why companies will improve their information gathering techniques, as “technological progress…enables humans to achieve their purposes more effectively.” (Cornish, 23) For example, how a user may feel at a particular moment in experiencing a product, message, or advertisement or even when their attention is lost or gained, can help companies achieve more of a sense of engagement with that individual consumer. Also, a lot of people experience things differently, so if there was a technology that could pick up on an individual’s emotions and know their exact responses in real-time, it would make companies able to target the individual consumer a lot more efficiently. Looking at Cornish’s DEGEST trends, we can reference Technology in relation to the Supertrend of technological progress, as it is the big way for a business to stay current and relevant. With a technological innovation like facial recognition and mood detection, there is an opportunity for the business to do so. Also part the DEGEST acronym of business trends is Demography, as the world population is growing, and while we have groups of people to target, “each person is an individual with a unique set of abilities, needs and aspirations.” (Cornish, 83) Therefore, it is becoming very important for businesses to know specific ways in which people are changing.
Facial recognition is emanating primarily in the advertising industry, as companies are taking steps toward using it to increase consumer engagement when looking at or watching ads. Big media/technology companies like Microsoft and Amazon have created their own image recognition AI’s that use algorithms to give an almost instant judgment of what is seen in the picture. The Microsoft AI can detect the age and gender of the person in the picture. Amazon’s image recognition machine learning AI is called Rekognition, which has been trained on thousands of objects and scenes, and can comprehend objects, faces and scenes all in one image. For example, it can identify that an image is of a dog down to what its breed is.
It can also identify faces to the point of recognizing specific facial features of happiness or sadness. Amazon is saying future implications of what it’s working toward in relation to media are what they call “Smart” marketing billboards, that collect demographic data about viewers, their reactions and what they think (Jeff Barr, Amazon Web Services Blog).
A lot of businesses have started to use the technology as good brand recognition and as a good digital marketing tool. For example, Usher released a music video in 2015 for his track “Chains,” which was an exclusive of the digital music service Tidal and turned on the user’s webcam while they were watching the video. If turned their head away, the message “Don’t Look Away” popped up, along with the pausing of the music and video. This could be further brought into the music industry, by giving people incentive to watch a full music video or listen to a full song. In addition to moves toward being good promotion for a business, an Atlanta based ad agency created the Facedeals project. The project is still in beta testing, but essentially the users sign up online, giving the company access to their Facebook so it can learn the user’s facial features and then synch with a facial recognition camera placed outside any business. That business would then be able to send the user a text when he/she walks in by sending a text to their phone and notifying them of any deals/coupon offers. Also, MasterCard announced “Selfie Payments,” where you can take a selfie to pay.
Also, the first steps toward giving consumers what they want when viewing an ad through engagement have arrived! For example, Virgin Mobile launched their Blinkwashing campaign in 2013, which was a choose-your-own-adventure video, where the clip changes the scene based on when the user blinks, altering the storyline. Another example is Douwe Egberts, a South African Coffee company, which placed an ad at an airport that used a facial recognition camera to detect if the person looking at the ad yawned or not. If it detected a yawn, it would dispense a free coffee. Lastly, Nike used facial recognition technology for an online ad campaign to promote its new Free-run shoe. The consumer could make different facial expressions and based off that, the shoe would mimic the face as a way to highlight how flexible the material was. In these ad campaigns, the consumer was given control over what it wanted from the ad.
We’re also seeing companies utilizing facial recognition technology already to evolve from the focus group/survey method. Companies can already pretty much test consumer reactions to advertisements in real-time with new platforms such as Affectiva, which is a start-up from MIT that created a database of over a billion facial expressions. It uses this to train algorithms to recognize typical human emotions with 90% accuracy. It is already starting to be used by Fortune 500 clients like PepsiCo, as they have been able to start bypassing the issues of high cost. “The cost is very low as the service is over the web, and it can be turned around almost immediately after the data is collected.” (Nick Langeveld, CEO of Affectiva) It’s competitor, Emotient, is starting to use facial recognition in the retail sector as a way to monitor emotional reactions of staff and customers, as well as demographics such as age and gender. It is on trial in various stores, with the objective of creating conclusions of “customer satisfaction and employee morale.” (Kieron Monks, CNN) In addition to this, there are companies like Dumbstruck who can record people’s reactions to digital content using the technology. Essentially, it enables companies to send out a custom link to pretty much anyone with a webcam and web browser on their phone or computer. From there, it can analyze moment-by-moment when users enjoy a video, when they hate it, lose interest, etc. It can also see the user’s race, gender and age. Companies like IHeartMedia and NBC are already using the technology to measure audience engagement and reaction to promotional content. This could develop into a way for advertising companies and brands to get consumer reactions to videos, commercials or even movie trailers before spending millions to mass market the material.
Lastly, the Mind Lab at Syracuse is developing an alternative technology for mood detection and inferring mental states. The Lab seems to have combined technology like fMRI brain imaging and EEG into a headset that can detect blood pressure, hormonal changes and brain activity. It is being used to have a better idea of what’s going on inside your body when experiencing things like 3D virtual reality and gaming. It’s a way to get consumer reactions in real-time, other than using facial recognition.
Moving past consumer interaction with advertisements and collecting real-time reaction data, the technology is being explored in machines people use in their daily routine. AutoMotive, a research group at MIT is exploring humans’ emotional connection with machines and developing a car that can detect the driver’s mood and emotions. The group is developing sensors on the door handles and steering wheel that can pick up electrical signals from the skin such as stress or exhaustion. So, the car can use this emotional data to pick-up warning signs. The group is also developing a facial recognition camera that can be mounted on the windshield to analyze facial expressions. Examples of what the car can potentially do with these reactions are if the driver is stressed, the sensors can soften the light and music. Also, it could potentially change the color of the vehicle’s outer paint as a warning sign to other drivers that the person operating the vehicle is in a distressed state.
Facial recognition is also delving into the social media space. As Facebook is already showing interest in collecting data other than likes or dislikes from posts in adding the “reactions” option users can choose from. Also, Sammi Krug, the product manager at Facebook, said that “Overtime we have to learn how the different reactions should be weighted differently by News Feed to do a better job of showing everyone the stories they most want to see.” (Krug, Screenwerks.tv) Technology like facial recognition could very well be used for this to measure real-time reactions. Also, think of anyone that has a profile on social media whether it be Facebook, Twitter, Tinder that has information about them. A new app called Blippar is using facial recognition technology to allow your phone camera to scan for faces of people who are signed up for the app, which then allows users to create profiles for someone that appears. You can add anything from favorite music to mood and beyond. It is combining the technology with AR, as you can see the person and their information around them from wherever they are sitting.
Prescription/Scenario of the new trend. What the new world situation COULD be.
We can now see that facial recognition and mood detection technology can be used to detect people’s emotions and reactions in real-time, allow them to engage with advertisements, places of business and even each other. However, what if facial recognition was able to allow a person or machine using the technology to cater to someone’s needs, wants, desires and mood right there on the spot? This is what current media companies could be steering toward in the future. We see current media companies already using the technology to detect moods and reactions in people, so it wouldn’t be too big of a step for the rest of them to adapt and adopt this trend. It seems as the technology was originally used in the security industry, with building security and police stations alike. It was also used, for example, at a music festival in the UK called Download last year (2015) to find the faces of past organized criminals in the area. However, I don’t think the new players in the media space would come from that side of the industry. Although, the security field will probably continue to develop the ability to utilize the technology for security reasons, there are more opportunities available and already being developed/looked at by companies in the media space. Most likely, the major players to get on this trend would be the big computing/tech companies like Google, Amazon and Microsoft who have already expressed interest in facial recognition technology with AI’s like Google Home and Amazon Echo, which are able to talk to you and give you essentially what you want already by connecting to the Internet. However, combining those with facial recognition and mood detection technology can make for some major innovations and these companies have the money to do it. Looking at the current innovations arising from the trend, we can use various scenario methods to talk about other potential future realities.
If the AutoMotive group’s research and technology at MIT can be fully developed at MIT and go commercial, car dealerships can start to use the technology as a way of marketing themselves and creating brand recognition by saying their cars can be your best friend. It could choose your music for you based on your mood, adjust your lights based on vison sense, alert other drivers if you are tired/in distress to keep you and others on the road safe. There’s already semi-autonomous cars being developed and sold by Tesla, Mercedes and BMW. So, whose to say you can’t hop in your car when your feeling sad and have it automatically know to take you to some of your favorite or “happy” places without even lifting a finger or saying a word?
Also, many ad agencies are already using technology like Dumbstruck that can detect real-time reactions of people when watching an ad. If ad agencies partner with a major satellite provider like Direct TV or Verizon Fios, all these TVs could have facial recognition technology to report this data back to the providers instead of having to find people to participate. Aside from ad agencies retaining information from television users, what if Samsung were to develop TVs that had facial recognition cameras installed and can detect your mood when watching? It too, just like your car, can be your best friend! It could change the channel if you’re uncomfortable watching something, change it to something you haven’t seen before and if it detects you are falling asleep, it’ll slowly lower the volume until you fall asleep, causing it to shut off, instead of you having to set the sleep mode to 30 minutes as a guess to how long it’ll take you to fall asleep. Even major Airlines could adopt this same technology to install on every screen in the aircraft.
What if the technology they are using in the Syracuse Mind-Lab could be developed into something the consumer takes home and uses while they’re playing a game? Xbox could pair the headset they use at Mind-Lab with their games or even develop a wireless scanner that can configure your brain activity and bodily functions (hormonal activity, blood pressure) with your mood and combine that with a facial recognition camera in the game system. This could create endless possibilities for games that can change the adventure based on your mood in real-time, so not everyone who buys the game would play the same storyline!
In thinking about the implications in the music industry, if facial recognition can detect your age, gender, mood, potentially your brain activity, then when you go onto your phone or computer to listen to music, it could do the work for you. Spotify could most likely get into this first, as they appear to be the most technologically advanced digital music service at the moment, able to automatically generate playlists and music based on your listening history. You could get onto your computer and with a quick scan of your face, the service will know whether you’re about to need music for studying or to play music at a pregame. It could have a playlist ready for you just by detecting activity on your face and even in your brain, keeping track of your listening history so you’re not hearing the same music as well as giving you access to the millions of songs out there. Additionally, if Blippar AR tech starts to allow users to put information about themselves through the app that other users can see, this could be a great factor for any business that is trying to create a positive experience for their customers. The Blippar app is supposed to show general information people can put about themselves like favorite song/music, hobbies, mood, etc. So, say a DJ at a club was wearing augmented reality goggles like Oculus Rift technology or like the RideOn ski goggles (that can show you where to go through the goggles, find friends, etc.), and let’s say the goggles are configured with facial recognition and mood detection technology…that DJ can look around at everyone’s faces and see their favorite type of music, artists, current mood, etc. They can make a set-list based off that information or even have a service like Spotify make it for them. This could apply to not just a venue with a DJ, but any business that plays music. Another example could be if you walk into a hotel elevator, which has a camera that can can detect your mood or know what music you like, it’ll play elevator music based off that.
We can even take this technology into the retail sector, as Emotient is already trialing facial recognition cameras to measure emotional reactions and demographics of customers and employees in stores. From there, we can see the future could be just as Philip K. Dick predicted in his 1956 sci-fi novella Minority Report, as we see a young Tom Cruise, in the 2002 film rendition of the novel, walking into a store that greets you using your name and has any data of previous purchases or typical habits to guide you through the store.
Finally, if we combine facial recognition technology and technology that measures biological features of the human body (such as X-rays or MRI’s), we get a simple camera that could essentially do a quick scan and automatically know biological facts about the person. For example, what if a restaurant could tell how hungry you were right as you walked in the door, what your mood was, allergy information and diet information? It could pick your meal for you right on the spot. In turn, knowing what to make, how many courses to make, whether you need a beer, etc. This could be a huge digital marketing and brand recognition tool for whichever food chain/restaurant is able pick it up.
Although all of these innovations with the facial recognition and mood detection trend may seem nothing but positive for society, there are many problems that can arise. The primary issue would be privacy violations all across the board. Giving technology this much access to a person on a daily basis is a lot more personal than the world has ever gotten in the media space. A lot of people may not want to feel they are constantly being watched 24/7. Also, if we are getting everything we want and more, it kind of takes away any element of surprise that media has brought us in the past. For example, someone walking into a random bar and loving the DJ without expecting to or seeing a commercial you didn’t expect to see and conducting an “impulse-buy” for the product or service. Also, if you were able to see personal information about someone anywhere you go (Blippar AR technology), it kind of ruins the point of “meeting someone new on the spot.” While there is social media to tell us information about someone ahead of time or after the fact, there is no spontaneity in seeing someone attractive at a bar, seeing an icon for country music floating above their head and immediately being turned off. If this trend starts to catch on the way it could, everything in the media space could become too predictable, thus making life boring. Lastly, if the facial recognition or mood detection technology doesn’t give you what you want, yet is using data based on your race and gender, all sorts of racial and discrimination issues could arise.
What would make this trend not occur is mainly government law. There are so many issues with privacy and freedom of speech with these innovations where they could be stopped with a simple Bill set into place or certain regulations set into effect by the FCC. If government were to stop these innovations from happening, the alternatives would simply be bringing us back to square one in terms of using surveys and focus groups by having people take part in this technology voluntarily. However, we would still be more technologically advanced and see elements in consumers we hadn’t yet seen before.
Going back to the 4 C’s in the C-scape, if we consider all four, we can see the need and opportunity for facial recognition and mood detection technology to play a big role in the media world. Allowing the consumers to have an emotional connection with machines and other technology would be making everything more personalized toward the individual. This is what companies looking into facial recognition and mood detection technology are aiming at, as this is what media is lacking and what businesses need to adapt to as consumers take more control. Also, all of these facial, mood, brain and biological detection technologies could potentially be converged onto one piece of technology that can automatically curate material like music or entertainment to your liking. It could curate content for you without having to tell you where to look for it. Technological innovations in media are making the future look brighter than ever, and if you look into trends like facial recognition and mood detection close enough, you can see an unimaginable world that is not so far away.
