Onfido Hack Day — Orwell
On November 9th, Onfido held a 24-hour long hackathon. Engineers, designers and product managers formed teams and competed for four awards: most useful hack, most viable, most ambitious and most innovative/creative. We knew we were all in for some tough competition, but it sure was fun!
This is the first in a series of blog posts, which will describe some of the projects that came out of the hackathon — the tech used, programming language, framework or library.
To kick things off, we’ll start with a project which aimed at easily identifying Onfido employees. We could leverage this work to unlock our facilities, similar to what we achieve with a fob nowadays.
Project Orwell
For face recognition, we decided to go with an out-of-the-box solution. We went with Amazon Rekognition — we had wanted to try it since it was first released, and the hack day seemed like the perfect excuse. Team Orwell had one front-end (Gautier), one mobile (Sérgio) and two back end developers (André and Daniel). The aim was to have an Android mobile app identifying Onfido personnel in real-time with face-tracking technology and a web app that served as an identifying photo booth. Both of them would be communicating with the face recognition back end we were building.
For the web app, we used React. In order to gain time (this was a hackathon after all) we used create-react-app, an npm library allowing us to create the project with different tools already pre-configured (webpack, babel, jest…).
The beauty of React is the intrinsic independence of each react component. We used the react-webcam component to capture the user’s selfie, something we hadn’t tried before. It was super easy to use and in less than 5 minutes we had the front-end app uploading base64-encoded images. Since React is not really a framework, it doesn’t have everything baked in. For that reason, client-server communication was dealt with using Axios, a “promise-based HTTP client for the browser and node.js”.
Another use case for our application was populating the database with a new individual given a photo, first name, last name, date of birth, etc. For that, we leveraged the simple FormData interface.
On the mobile side, we found in Google’s Play Services Vision all we needed to put in place a real-time face-tracking system. It offers callbacks for when a new face is detected on the camera preview, and it even supports multiple faces simultaneously.
We had some challenges from the get-go: the back end was only ready to handle a single face match per selfie, while the Android device detected multiple faces. Our heuristic was to use the largest face on the screen for the identification attempt. This was possible because our face-detection mechanism returned the position and size of the detected faces on the mobile screen.
So at the time a new face was detected (let’s call it T), we should send the next frame we received after T + delay, in which delay = 750 ms. This is due to the fact that the face detection mechanism would detect faces too early, while they were not yet properly centered on the camera or front-facing it, resulting in poor comparisons when sent to Amazon Rekognition. Also, whenever a new face was detected, and while we were waiting for the back end response of whether it was recognized or not, we would present a question mark inside a colored circle to the user, indicating the recognition was in process.
Finally, if the face was recognized, the question mark would turn into a check sign and the circle would become clickable to show the identified user’s details.
The back end was a simple Sinatra app which used the aws-sdk-rekognition gem to communicate with Amazon Rekognition. We had an endpoint to upload a new individual to our system and another endpoint for subsequent identification within our search space.
Top problems were agreeing on a format that served both the React app and the Android app (base64-encoded image with some specificities for each, no biggie), implementing a custom endpoint to our internal image retrieval service (Imago) so we could get employees’ selfies via its API, and the non-obvious need to delete photos after identification had been carried out. In hindsight, we think this could well be an option in the Amazon API (i.e., if we want to keep the photo we’re searching for a match against in our Rekognition collection or not).
Unfortunately, we didn’t get to the part of actually integrating our face recognition system with a piece of hardware capable of unlocking the front-door of our offices, but we’re confident with a little bit more work that would be achievable. The hackathon was a good way to experiment with some tech we’ve been meaning to trial. We’re already planning for the next one!
In part 2 of this series, we will take a look at Kubernetes Unicorns. Stay tuned!
Part 2 is now available here.
