I Made a Thing!
Facial Recognition Edition
What’s the Big Idea? In verifying someone’s identity you can look to three broad categories. Something you know, like a password. Something you have, like a keycard. Or, something you are, like your facial features.
This project concentrates on that last one. I wanted to explore the idea of using facial recognition to authenticate users. I built a basic Rails app that allows users to sign in using either a password and email, or using their email and a webcam captured image.
I encourage you to play with it here. (It’s currently on Heroku’s free tier and there may be a slight delay loading the landing page.)
Using the app. When a user first registers their account is created in much the same workflow as just about every site on the internet, name, email, password and confirmation. Account created:

The next part is where it gets fun. After account creation, the user is prompted to enroll a reference image that will be checked against future sign in attempts:

At this point we have a user, and we have a reference image that can be checked against. So when a user wants to log in they can use either something they know (password), or who they are (facial features). Like this:

From there, you can use it for the same purposes as any web app that has a user model.
Technical talk. Github repo here. This is built on top of a pretty bog standard user authentication model using the Bcrypt gem. This is an excellent write up of that process so I won’t explain it here.
Starting from there I wanted to add in the ability to use facial features as an authentication factor. I originally started playing with the excellent OpenCV library, which would allow me complete control, and do the processing locally. I quickly realized that face recognition is an exceptionally complex task, and for any degree of reliability, I would probably want to reach for an external service to handle facial recognition.
After playing with some of the features at Amazon’s Rekognition, and Microsoft’s Face API, I decided on Kairos. They offer an easy to use API that comes with a great free tier for playing around on personal projects.
I then wrote a service class for my Rails app. This does two things (okay three, but the third one is an easter egg for other developers, look at the linked file if you’re curious). 1. It has an enrollment method that takes that signup image, and has the Kairos API create a facial profile for that user. 2. It has a visual auth method that takes a login image and has Kairos reference it against that particular user’s enrolled image. Kairos then tells us what the likelihood is that they are the same person. You can set your own rejection threshold with the ENV[‘confidence’] variable. I found that 0.7 worked well, although for higher security, bumping it up to .85 or .9 seemed to yield good results without too many false rejections.
The results of calling the Kairos service class can then be used in the session controller to create a logged in session, or reject the attempt (worth noting is that the create_webcam method is called through AJAX, which is why we are returning a js redirect):

Challenges: I’m not a JS wizard. I explored using the getUserMedia functions to access the webcam, but ended up reaching for WebcamJS as it very neatly wraps up getUserMedia into some very nice pre-written functions. It also provides an easy fallback to a flash alternative for browsers that don’t support getUserMedia.
Testing was also a challenge. For obvious reasons, it is really hard to automate testing something that requires a real human face. I simply was unable to find a way to mock webcam images for an integration test. That’s not to say that it isn’t possible, just that it isn’t easy.
Handling photos with no recognizable faces, or too many faces was also an issue. Luckily Kairos provides enough information in the returned information to set up some error handling. Right now, it isn’t handled particularly well to the view, only telling the user that something didn’t work, but not giving context.
Some closing thoughts. While I haven’t yet been able to spoof my way into an account, I suspect that it would not be hard. It rejected my driver license photo, although that may be a reflection of the Colorado DMV’s photographic ability. More interestingly, it rejected a photo of me displayed on a phone and held up to the camera. I suspect, but haven’t tested, that a color print would fool the system.
With that in mind, you want to use this system for purposes where certainty is not required, or use it as part of a two factor system.
Certainty not required… what could you possibly use this for, Jack?
Scenario: I used to run a youth experiential education program. We would take HS and college aged kids out on trips from 3 weeks to 90 days. One of the best ways to keep parents informed was with a blog of photos and daily activities. However, parents are largely interested in seeing just the photos that have their kids in them. At our peak we could have as many as 300 different students on 4 or 5 continents pumping out hundreds or thousands of photos per day in a dozen or more blog posts. This system could be built out to check all blog photos in a day, and find all of the registered users in them, and send out an email summary of photos of the student.
Scenario 2: Use it to create an Oauth like system for IOT devices, access control and access logging. More on this in a week or so.
