Face Recognition Authentication using Flutter and Tensorflow Lite

The growth of processing power in devices and Machine learning allows us to create new solutions that a few years ago couldn’t have been achieved. In this case, I want to show an interesting way to perform authentication using Flutter and Tensorflow Lite with face recognition.

Source: BBC post

Introduction

I will explain step by step, how is the process of building a fairly simple facial recognition app that contains 3 functionalities: Sign Up, Sign In and Clear DB.

Process summary

Sign up

1. The user takes a photo.
2. The ML models process it and create an output (array of numbers) to be stored in a database.
3. A name and a password are requested (the name is not actually necessary, it’s requested just to show a greeting in your profile).

Sign in

1. The user takes a photo.
2. The ML models process it and creates an output.
3. The output will be compared against the outputs already stored in the database (it compares by proximity the closest one it finds). As condition, the proximity has to be under the threshold (minimum distance), if overcomes it, it will process it as a non-existent user.
4. If the user exists (face already processed) it requests the password for that user, validates and authenticates it.

Clear DB

This functionality is just for debugging, It deletes all the data saved in memory.

Note: The purpose of this application is simply to show the main functionality (which is facial recognition). Tha’s why the records are not stored in a database on a server, but saved in a json file in the device memory.

How it works

It works with two computer vision models working together, the Firebase ML vision model to perform the face detection and preprocessing in the image, and the MobileFaceNet model to process, classify and transform into a data structure ‘savable’ by a database (an array of numbers).

Tensorflow Lite:

To integrate the MobileFaceNet it’s necessary to transform the tensorflow model (.pb extension) into a file with .tflite extension. It’s a painful process explained in this series: part 1, part 2. part 3. But if you want to skip it, you can directly download the .tflite file from my repo, link.

More info

Note: In this post I explain what is and how works Tensorflow, Tensorflow Lite and more.

About MobileFaceNet models:

MobileFaceNets are a set of CNN models, which use less than 1 million parameters and are specifically tailored for high-accuracy real-time face verification on mobile and embedded devices.

Under the same experimental conditions, MobileFaceNets achieve significantly superior accuracy as well as more than 2 times actual speedup over MobileNetV2 (which is explained in this post).

For more info check out this paper.

It receives a matrix of 112x112 inputs and returns as output a matrix of 7x7 with values adjusted according to importance, as you can see, the corner unit should be of less importance than the center unit, because of the noise in the edges.

Firebase ML vision:

With ML Kit’s Face Detection API, you can detect faces in an image, identify key facial features, and obtain the contours of detected faces. It works very well preprocessing the image to detect the zone to be cropped and then processed by the MobileFaceNet model.

Installation

In the official package page is very well explained how to install it step by step.

Spoiler: You’ll need to create a Firebase project 😒.

more info

Flutter implementation

Note: I’ll skip a lot of code, because if I explain step by step the full code, this post will be too long, anyway I will explain the fragments that I consider most important.

If you want to see the full code check out the repo.

Description

As soon as you enter the sign up or sign in option, the ML vision model detects existing human faces in the frames that the camera preview, the class Face contains the coordinates of the points that make up the frame around the face.

If the sign in or sign up button is pressed (as the case may be). The detected face of the last frame is captured, cropped and then pre-processed to be processed by the MobileFaceNet model.

The MobileFaceNet model returns an output (array of numbers).

• If it’s the Sign Up process, the application requests a name and a password, and later saves the three data (name, password and ML output).
• If it’s the Sign In process, the application performs a search in the database comparing the Euclidean distance between the ML output of the image and the ML outputs stored for each user, the one that matches or is close enough (accuaricy above the threshold) the application brings the data and requests a password.

Euclidean distance formula.

Programmer-friendly definition:

The user enters the password, the application validates if the password corresponds to the password of the detected user and then… voilà! you’re in.

The diagram represents the described process. Source image: Unsplash

Example

Process ordered from left to right

1. I try to log in without registering
2. Registration process.
3. Successful login after registration

Conclusion

In this article, I didnt make any reference to the security of the authentication mechanism presented, since the idea of asking for a password is precisely to avoid attacks such as showing a photo of other person to the camera. But there are other interesting ways to achieve secure authentication without the need of a password, as you can see in the following examples:

• Asking the user to perform a pose and validate it using the PoseNet model.
• Voice command associated with the user through the Speech Recognition model.
• Associate a unique object that the user uses as a ‘totem’ such as a dice or a spinning top, and then validate it using the MobileNet model… (it doesn’t seems secure at all, but sounds cool).

If you’re going to perform inception, you need imagination…

I encourage you to try to perform a secure userless & passwordless authentication based on these ideas!

Repo: https://github.com/MCarlomagno/FaceRecognitionAuth