Facial Recognition with Android (2/4)
In this series of articles I’m going to cover how you can add facial recognition (not just facial detection) to your Android app.
In the previous article I covered 2 ways to perform facial detection on Android, Google Vision and Firebase ML Kit. Lets recap the 4 steps we’ll be covering
- Facial Detection on Android
- Facial Recognition using Google AutoML (off-device)
- Facial Recognition using TensorFlow (off-device)
- Facial Recognition using TensorFlow Lite (on-device)
So now we’ll look at Cloud AutoML and how to classify images and then make calls to AutoML from our Android app.
Setting Up Accounts
The first thing you’ll need to do when using Cloud AutoML is to setup a Service Account for your project and download the account key json file.
This guide is the best resource for the commands you’ll need to execute using the gcloud tool.
As part of this setup you’ll also need to enable Billing on your project. This is because you only get 1hr of free training time for your model, if you upload a large dataset then you’ll need to pay for the extra training time.
Once you have your account setup, you can download the service account key json from Firebase (you’ll need this to generate an access token for the REST service later, or to use the SDK directly).
Go to Project Settings > Service Accounts and click “Generate new private key”.
Training Our Model
Now that we have all the accounts and permissions setup, we can go to the AutoML website to upload our images and train our model.
https://cloud.google.com/automl/ui/vision
We start by creating a dataset, give it a meaningful name. You can choose to upload your images here from your computer, or you can specify a Cloud Storage bucket with images and a CSV that maps labels to images.
In this sample we’ll just upload images from the computer.
Importing may take a while depending on the number of images and your network speed, sit back and relax, you’ll receive an email once the upload has completed, so there is no need to watch the screen like a hawk.
Once the images are uploaded you can define the labels that you want to use to classify the images. In this example I’ve uploaded some photos of friends, so the labels will be their names.
Next step is to go through and apply each label to the correct photos. This can be a long process if you’re doing it by hand, if you know you have a large dataset then maybe the Cloud Storage bucket option earlier is better for you. I don’t have many images or labels in this example, so it only takes a few minutes to apply.
Now we want to train our new model on the uploaded images and labels. We see a warning here that I should have at least 100 images per label, so the results aren’t going to be stellar, but for the purposes of the demo it should be good enough.
Getting hold of the large quantities of data needed to train an accurate model can be one of the harder parts of machine learning.
We get 1 compute hour for free, which should be enough for a small sample size like this, if you have larger datasets then you’ll need to pay for more computing time. This is why when you setup your accounts earlier you needed to have the Blaze payment model selected.
Time to grab a cup of tea! Once again, you don’t need to sit and watch this screen, you’ll be notified by email when training is complete.
When the training is complete you are presented with an evaluation of how the model performed.
The Confusion matrix here shows that it correctly classified all images marked as Andy as Andy. If there had been some percentage of images labeled as Andy classified as Cameron that might be an indication to upload more images of both these people to help train the model further.
Using a new photo that wasn’t part of the initial dataset, we can now use the web GUI to test our model. If we upload a new image using the Predict tab, we can see if the model correctly classifies this new image….and in this example, I get a confidence score of 81% that this is a photo of Andy.
Sweet! So we now have a Cloud AutoML model trained on photos of some friends. There is a Cloud AutoML service which we can now call in 2 different ways to actually use this model in our Android app from part 1 of this series.
Let’s look at the Android app and the 2 ways we can call the Cloud AutoML service.
- REST endpoint
- Cloud AutoML SDK.
We’ll take a look at both and add code for these to our GitHub project.
REST Endpoint
The Predict tab of the AutoML Website gives us the following definition of the REST service to invoke our trained model
curl -X POST -H “Content-Type: application/json” \
-H “Authorization: Bearer <access_token>” \
https://automl.googleapis.com/v1beta1/projects/devnibbles/locations/us-central1/models/ICN3704829353327390855:predict -d @request.json
For the REST Endpoint we’re going to be using the popular Retrofit library, using the info above we’ll create the following Retrofit Service interface.
Based on the sample json we expect back, we also define the CloudResponse model here too.
The json and model we need to POST to this service looks like this.
Using the Android Architecture Components, we’ll create a ViewModel that can be reused by both our Cloud Vision and Firebase ML-Kit detection code. Here is the part that creates the REST service, we include a logger to help diagnose any problems in our demo code.
To classify an image we call it using the following code inside a Coroutine.
You can see the complete code for the CloudAutoMLViewModel here
Cloud ML SDK
In order to use the Cloud AutoML SDK, we need to add the following to our build.gradle file.
// Cloud AutoML SDK
implementation 'com.google.cloud:google-cloud-automl:0.55.1-beta'
implementation 'io.grpc:grpc-okhttp:1.13.1'We can then add another method to our CloudAutoMLViewModel
The service account json is downloaded from your Firebase project, it defines the private key which is needed to access the CloudML SDK, just add the whole json contents of the file to this variable.
Normally you wouldn’t put this into your app, you’d create a REST service of your own, and call that REST service from your app. Your own REST service would then forward the request to Google, thereby hiding your private key on your server.
DO NOT SHIP YOUR PRODUCTION APP WITH PRIVATE KEYS INSIDE!!!
Looking at the code we can see its very similar to the REST implementation earlier, we generate a payload with our image bytes and we then call the predict() method passing the same config as earlier for the project, location and model.
Once our prediction is done, we parse the results as before, passing similar values back to our LiveData results.
Once again, you can view the complete CloudAutoMLViewModel on github.
Now that we have our ViewModel that can call out to either the REST or SDK version of the AutoML service. We can finally take a look at how the Android app has been modified to use this ViewModel to display the name of the person on-screen.
In both of the classify methods above, you’ll notice that we pass in a faceId parameter. This parameter is the unique id of a face in a frame and is generated by the Detector.
We use this faceId to link the image being passed to the ViewModel to the classification results. When a FaceGraphic is created this id is now associated with it. we can then perform a lookup later to assign the name.
Take a look at this snippet from the GoogleVisionActivity
Our ViewModel is created in the onCreate() method and looks like this.
As you can see, we observe the results of any classifications that may have been performed, and when a new result comes through we find the associated face and set the name.
Conclusion
In this article we covered how to setup Google Cloud AutoML, how to train a model based on some photos of friends and then how to use that model from within our Android app.
As you can see the setup of Google Cloud AutoML is very simple once you have your dataset ready to go.
Calling the service from your app is very easy with tools like Retrofit.
In part 3 of this series I hope to migrate from Google Cloud AutoML to using TensorFlow directly. This is all in the leadup to our final part of the series to using TensorFlow-lite on the device itself.
Stay tuned, and please do let me know what you thought of this article.
