Is this Tom Hanks or Bill Murray? Let’s try facial recognition.
This 2013 photograph has resurfaced recently, with people wondering whether the man in orange is Tom Hanks or Bill Murray. Before I reveal his identity, let’s consider another way to find the answer: facial recognition technology.
Facial recognition is the way that Facebook is able to suggest who is in the photos you’ve uploaded. Most cameras these days can detect faces, but telling different people apart is a more complex problem. Happily, it’s also a well studied one, and there are plenty of open source approaches we can follow.
For this example I’m relying entirely on Adam Geitgey’s code from ‘Modern Face Recognition with Deep Learning’. Read that link for an in-depth description of what’s happening here. But basically: I’ll train a model to ‘understand’ what Tom Hanks and Bill Murray look like, by measuring their facial geometry through dozens of examples. That’s simple enough: just look up each actor on Google Image Search, and save the page. My criteria: one face per image, good lighting where possible, and no obscuring features like glasses:
Now that we have a training set of images for each actor, we run some code that isolates the face in each image, and standardises the pose. This means warping the face so the eyes and mouth are in consistent locations, and gives slightly unsettling results:
Then we generate an embedding for each image, which is a set of 128 facial measurements and looks something like this (plus 123 more columns):
Now, using the magic of Support Vector Machines, we can ask the computer to generate 128 measurements for any other photo, and check whether those measurements are closer to Tom’s or Bill’s. Let’s test it with a photo of Bill Murray that’s not in our training set:
Sweet. A perfect match would be 1.0, so that’s pretty good. The computer recognises Bill Murray because, well, it looks like other photos of him. Now let’s classify the unknown face.
So there you have it, the unknown person is Tom Hanks! Only, hang on — it’s actually Bill Murray:
“We are finding this very funny. I personally don’t see the resemblance to Tom Hanks but we’ve noticed that people have been mentioning it for years,” [DiMichele-Ross] told the Guardian.
How did we get it wrong? And not just wrong, but so wrong the algorithm was 95% convinced of its answer?
It’s hard to know, and that’s an issue that goes to the heart of machine learning: it’s possible to get an incorrect answer with very high confidence, depending on what’s in your dataset and how you train your model. That’s trivial when you’re trying to recognise celebrities, but facial recognition mismatches have the potential to put innocent people at the scene of a crime. And as I’ve just shown, it’s really easy to apply an off-the-shelf classifier without understanding the model’s deeper workings. Everyone will be doing this in five years — police, retail, even smart doorbells that recognise who is knocking.
In other circumstances, machine learning errors could determine whether you get a loan approved, the length of your prison sentence, or whether an autonomous weapon thinks you’re a threat.
In this case, there are a few clues in the training set. My hunch is that a particular Hanksian form of open-mouth smile was judged as quite similar to Bill Murray’s mock crying:
The training set contains many more photos of Hanks with his mouth open than Murray, who apparently prefers more of a closed-lip smile. If we trained with more or different images, we might have gotten a different result.
Side note: now that we’ve got a trained model, let’s throw some other faces into it and see who they look like!
Well, at least that last one makes sense.
Thanks to Adam Geitgey for sharing his code and making this post possible. Contact me on Twitter: @flashman
