How Facial Recognition Works And How it Can Harm The Future If Misused
There’s a massive bait-and-switch at the centre of facial recognition technology.
If you upload a photo into Google’s reverse image search, it’ll find websites where that picture has appeared, or provide “visually similar images” that have the same colouring and composition. The leading search engine in Russia, called Yandex, has reverse image search too but it doesn’t work the same way. It’s not looking for visually similar images. It’s looking for similar faces, the same face.
The difference between these search engines is that Google hasn’t switched on facial recognition and Yandex has. On Google, you can enter a name and look for a face. But on Yandex, you can enter a face and look for a name. And that distinction represents a potentially enormous shift in our offline lives, where we usually decide who we introduce ourselves to.
Now that computer scientists have created tools that can turn faces into nametags, it’s worth reflecting on how we got here and what we stand to lose.
A computer’s facial recognition system has broadly the same components as your own facial recognition system. You see someone with your eyes, your mind processes the features of their face, and recalls their identity from your memory.
Now imagine if you could have eyes in lots of places and could download and store memories from other people, then you have something more like the automated version of facial recognition, which has only come together in the past 5 years or so. Its eyes are digital cameras, revolutionary machines that turn light into data. “It’s a state of the art digital model, which records images on memory chips instead of photographic film.”
Digital imagery arose in the early 2000s, which coincided with the arrival of
the social internet. So right when we were able to take an unlimited number of pictures, Facebook, Flickr, Youtube, and other sites told us our images had a home online. “100 million photos are being tagged every day on Facebook.”
Professional photography also went up on websites, news articles, and photo libraries, and Google’s web crawlers gathered them into Image Search. And then the computer vision researchers went to work.
The millions of digital photos posted to the internet, like the Facebook pictures where we tagged our friends or Google image results of celebrities — they were used to build the “mind” of facial recognition systems. That mind is made up of a series of algorithms. They locate faces in an image, map facial
features to correct for head rotation, and then take over 100 measurements that define that individual face. Those measurements are usually described as
the distance between the eyes, the length of the nose, the width of the mouth.
But the truth is, nobody knows exactly what’s being measured. That’s determined by a deep learning algorithm looking for correlations in raw pixel data. To train that algorithm, engineers give it sets of triplets: an anchor photo, another photo of the same person, and a photo of a different person. The algorithm is tasked with deciding what to measure so that the statistical difference between the two matching photos is as small as possible while the distance between the non-matching photos is as large as possible.
These algorithms are refined through millions of examples, but they still don’t perform equally well on all types of people or all types of photos. That hasn’t stopped them from being packaged and distributed as ready-to-use software. But whoever uses that software won’t be able to identify you until you’re in their database of known faces. That’s the “memory” of the system — and it’s separate from the training images.
In the case of the iPhone's faceID, it’s a database of one — you volunteer to store your face on your device in exchange for easily unlocking your phone.
Companies like Facebook and Google also keep databases of their users. But it’s governments that typically have access to the largest databases of names and faces, so facial recognition significantly expands the power of the state. They collected these images for other reasons and now they’re repurposing them for facial recognition without telling us or obtaining our consent, which is why several US cities have banned government use of facial recognition.
Retail stores, banks, and stadiums can create or buy watchlists of known shoplifters, valued customers, or other persons of interest, so they’re notified if one of those people shows up. And then there’s another source of labelled
photos. Those are the ones we’ve been labelling ourselves by setting up profiles on social media networks. It’s typically against the terms of use to programming bots that can download faces and names from Linkedin, Twitter, of Facebook, but it’s doable. And what’s at stake is something that most of us take for granted: our ability to move through public spaces anonymously.
“So we typically think of public and private as being opposites. But is there such a thing as having privacy when we’re in public?” “I would like to think so.”
Evan Selinger is a professor of philosophy who argues that facial recognition is a threat to “obscurity,” Which is the idea that personal information is safer when it is hard to obtain or understand. So he says that “ We have natural sort of limitations in what we can perceive and what we can hear. Even the human mind has a sort of basic limits in how much information it can store.”
So one of the things that technologies do is they reduce the transaction costs of being able to find information, being able to share information, and being able to correctly interpret information. And so facial recognition is probably the most obscurity-eviscerating technology ever invented.” We don’t have to imagine how this could play out. It’s already happening with photos from the Russian social media network VK.
A journalist who covers Europe for Bellingcat showed how it works with a random video of Russian soccer fans picking fights in Poland. “There are about 10 or so of these soccer hooligans in this image and for every single one of them you can find their profiles on VK.” So if you click the photo here it will take you directly to the photo’s link. So this is probably like his buddy who uploaded a photo.
So it does feel weird when you do this and you have access to way more information than you should, is what it feels like. But also publish what we’re like one thousand percent sure of and if possible we may be done include the names of the people. How you feel about this technology probably depends on how much you sympathize with the person being identified.
Bellingcat has used these tools to find to identify people linked to the attack on flight MH17 in Eastern Ukraine. It’s also been used to doxx police officers accused of brutality, anti-corruption activists protesting against Vladimir Putin, random strangers as part of an art project, and sex workers, porn performers, and others who have posted anonymous photos online.
“The way that we share our images and our names on social media, LinkedIn Twitter, Instagram, it seems to suggest that we don’t want to be obscure or we’re not really looking to be anonymous. Are we allowed to want to share and connect with other people online and still be able to expect not to be recognized when we’re offline in our regular lives?”
I would say absolutely. In fact, I would go further and say if we ever create a society where that’s not a reasonable expectation, a lot of the things that are fundamental to being a human being are really going to be compromised. Having any individuality requires experimenting in life and experimenting requires the protections of some obscurity. But also intimacy requires obscurity. Right.
If you want to be able to share different parts of your life with different people, and I think most of us do right. We don’t want to come to work and behave the same way we do with our friends. We don’t want to treat our partners in the same way we do acquaintances. And the concern, when
you lose too much obscurity, is that these domains bleed into one another and create what’s called context collapse.
And it doesn’t mean that one is more real or one is more authentic. Leading a rich life requires us to be able to express ourselves in these diverse ways.” The photos we took to share with friends, or document history, or simply get a government ID have been used to build and operate a technology that strips away the protections that obscurity has always provided us. It’s nothing less
than a massive bait-and-switch. One that could change the meaning of the human face forever.
