Building CelebritySleuth — A Serverless Framework Application

CelebritySleuth

A celebrity face recognition service I built with Serverless Framework using Twilio, Amazon Rekognition and the IMDbPy API.

This project provides an overview of Serverless computing and drills the concept down by helping build a compelling example of Shazam for Celebrities, dubbed CelebritySleuth using the Serverless Framework.

For more information on Serverless architecture and Serverless Framework check out this whitepaper.

Introduction

We are at a turning point in the age of computing where machine learning is being commoditized, allowing average devs to wield the power of artificial intelligence to process natural language, recognize faces, and convert text to speech without having to dabble with arcane neural networks or other deep learning-based algorithms. Services such as Amazon’s Rekognition, Polly and Lex are cases in point, allowing individual devs to design applications which until recently were the domain of research labs and large technology corporations.(To top it off, these are 100% serverless. You only pay for the sliver of time you use them and it is incredibly scaleable.)

In an effort to understand the capabilities of these services, while having fun building applications using a serverless model, I was searching for an interesting problem to solve. The first problem I tackled was counting the value of coins placed on a table. The solution was easy to build using Python and various Python imaging libraries. Unfortunately, when it came to cramming the code into AWS Lambda, I ran into a hard-stop. AWS Lambda had an upper limit on the size of code that can be deployed and the imaging libraries I employed were huge. This quickly turned out to be a no-go.

Around this time, my little one was faced by a vexing problem of her own. While watching a show on TV, she couldn’t place the actresses face with her name. She knew that she had seen the actress before in another Disney series, but who was she? While discussing this over the dinner table, we quickly decided that I shohuld build something to solve her predicament. And CelebritySleuth was born.

Audience

Developers or Solutions Architect who want to:

1. Learn more about the Serverless Framework and how to use it for developing auto-scaling, pay-per-execution, event-driven applications.

2. Build applications that matter to business objectives instead of spending time configuring, deploying, and maintaining infrastructure.

3. Explore the efficiency boost that serverless architecture provides.

The CelebritySleuth Application

The CelebritySleuth application is an event-driven application taking advantage of:

1. The user’s mobile SMS/MMS for the presentation tier,

2. Twilio in the middle-tier to bridge the SMS world and AWS Gateway

3. A set of AWS Lambda functions written in Python making use of AWS Rekogniton for image processing and IMDB for gathering information on the celebrities.

In a typical usage, the user snaps a picture of an celebrity (on TV?!) using his/her phone camera and sends the image along with the instructions to a phone number hosted on Twilio. The system validates the instructions and responds immediately if there is an error in the instruction set or with a message to await the actual response if the validation passes. Within moments, the user receives a response with the name and biography of the celebrity.

To start, the application has to be trained to recognize the faces by building a collection of celebrities. You do this by sending a random sample of celebrity pictures (image URLs) and their corresponding names to the application. The more pictures of an individual celebrity sent, the more accurate the recognition will be.

The CelebritySleuth application consists of two services:

- Twilio Communication Service

- Face Recognition Service

The services are decoupled to allow for using different presentation tiers in future.

  • The Architecture

The CelebritySleuth application uses Lambda functions for compute needs. As a result, the application components are provisioned just before usage and brought down after use resulting in a low-cost, highly-scalable application.

The above picture illustrates the high-level architecture of the application. Details are as follows:

1. User sends a picture and commands to add/match face to a collection. Alternatively, the user can create a collection, in which case a picture is not required. The SMS/MMS is sent to a telephone number hosted by Twilio.

2. Twilio intercepts the message and forwards it to an API Gateway based on the user’s Twilio configuration.

3. API Gateway translates TwiML to JSON and calls the Request Processor lambda function.

4. The Request Processor lambda validates the commands and put a message to the appropriate topic on SNS. If the validation fails, it returns the error message to the user via Twilio.

5. When a message arrives in the Create Collection topic, a lambda is triggered which adds the named collection in AWS Rekognition via Boto libraries. A success/error message is put in the Response Processor topic.

6. When a message arrives in Add Face topic, a lambda is triggered which identifies the most prominent face in the image and adds the metadata for the face to the given collection. If there is no faces identified, it creates an error message and sends the response to the Response Processor topic.

7. When a message arrives in Match Face topic, a lambda is triggered which identifies the most prominent face in the image and matches the metadata for that face with known faces in the collection. If a match is found, the corresponding person’s name is returned. The Lambda then uses IMDB to lookup the biography of the person.

8. The various lambda-based processors drops the response message on the Response Processor topic.

9. The Response Processor picks up the response and constructs a SMS message and calls Twilio’s SMS service.

10. Twilio validates the From number and sends the message to the corresponding To number.

  • The Components

The application consists of the following components:

1. Python — Python is a programming language that lets you work quicklyand integrate systems more effectively. We will use Python 2.7 for building the CelebritySleuth Application.

2. Twilio — Twilio Messaging Service for having the user communicate with the application through SMS.

3. AWS Lambda — AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume — there is no charge when your code is not running.

4. AWS Rekognition — Amazon Rekognition is a service that makes it easy to add image analysis to your applications. With Rekognition, you can detect objects, scenes, and faces in images. You can also search and compare faces.

5. IMDb — IMDbPY is a Python package useful to retrieve and manage the data of the IMDb movie database about movies, people, characters and companies.

Setup Instructions

1. Installing Python

If you are on a Mac or Linux machine, you probably already have Python installed. On Windows you will have to install Python.

Regardless of your operating system, you should use a virtual environment for running Python. Anaconda, or its terse version Miniconda, are Python virtual environments that allow you to manage various versions and environments of Python. The installers come with Python and the package manager conda with it.

Follow the instructions here to install Miniconda. Note: For this project we will use Python 2.7.

2. Creating a Python Virtual Environment

After installing Python 2.7, create an virtual environment as follows. Note: I am calling my virtual environment faces:

$ conda create -n faces python=2

3. Initialize Your Python Virtual Environment

To start working in your new Python virtual environment:

$ source activate faces

If you are working in Windows, use:

$ activate faces

4. Install Git

Git is a popular code revision control system. To install Git for your respective operating system follow the instructions here.

5. Install CelebritySleuth

To install CelebritySleuth from Git, follow the instructions below:

$ mkdir DevFestDC
$ cd DevFestDC
$ git clone — recursive https://github.com/skarlekar/faces.git

6. Twilio Setup

If you do not already have a Twilio number, signup for Twilio and get a trial phone number with MMS capability. You can use this number for 30-days during the trial period.

· Setup Twilio Environment Variables

Update the setTwilio.sh in the repository with your credentials from Twilio and setup your environment. Instructions on using vi are here.

$ cd faces
$ vi setTwilio.sh
$ source ./setTwilio.sh

Windows users, use:

$ cd faces
$ edit setTwilio.cmd
$ setTwilio.cmd

· Test Twilio Setup

To test your Twilio setup, run the Python program sendmessage.py under twilioTester. This program simply sends a message to your mobile number using your Twilio credentials. Note: Make sure you are running this in your Python 2.7 environment.

$ python twilioTester/sendmessage.py

If you receive a message with an image on your mobile, your Twilio is setup is working.

7. Install Node.js and Serverless Framework

Serverless Framework is a Node.js application. To use Serverless Framework and run the CelebritySleuth application you will need to install Node.js. Follow the instructions from Serverless Framework website to install both Node.js and the Serverless Framework.

Ensure your Serverless framework is operational using the following:

$ serverless –version

8. Amazon AWS Setup

1. Sign into your AWS account or sign-up for one.

2. Setup your AWS credentials by following the instructions from here.

9. Testing your Serverless Setup

Now that you have setup AWS, it is time to test your Serverless setup by creating a mock function using the Serverless Framework.

Create a test directory. In the test directory, create a Lambda function from the default template as follows:

$ mkdir sls-tester
$ cd sls-tester
$ sls create — template aws-python — name sls-test

This should create two files in the current directory:

> serverless.yml
>
> handler.py

The serverless.yml declares a sample service and a function. The handler.py returns a message stating that your function executed successfully.

To deploy the function, simply type:

$ sls deploy — verbose

This should deploy the function. The verbose option provides extra information.

To test your function, type:

$ sls invoke — function hello

If you get the following message, your Serverless setup is working.

WARNING: You are running v1.9.0. v1.10.0 will include the following breaking changes:

- Some lifecycle events for the deploy plugin will move to a new package plugin. More info -> https://git.io/vy1zC

You can opt-out from these warnings by setting the “SLS_IGNORE_WARNING=*” environment variable.

{
“body”: “{\”input\”: {}, \”message\”: \”Go Serverless v1.0! Your function executed successfully!\”}”,
“statusCode”: 200
}

10. Twilio Communication Service

The Twilio Communication Service bridges Twilio’s SMS messaging service with the Face Recognition Service. When the user sends a message to his/her Twilio number, the message is intercepted by Twilio’s Messaging service. The Twilio Messaging service will forward the SMS message contents to AWS API Gateway URL. The AWS API Gateway in turn will invoke the Request Processor (process_request) Lambda function in the twilioCommunicationService.

The TwilioCommunicationService supports two functions:

1. The processRequest function validates incoming requests and sends a response synchronously if the format of the message is correct, or an error message giving the correct usage if it’s not — see the Usage section for the details on how to use this service. It then invokes the faceRecognitionService asynchronously through SNS to process the command.

2. The sendResponse function composes a response from the faceRecognitionService and sends the response back to the number from where the request originated.

· Deploy Twilio Communication Service

Assuming your local Serverless setup is complete, and the test above passes, follow the instructions below to deploy the twilioCommunicationService using the Serverless Framework:

Set your Twilio credentials by running the shell script you updated earlier.

$ source ./setTwilio.sh
Windows users, use:
$ setTwilio.cmd

Change directory to the twilioCommunicationService directory and deploy the service by running sls deploy as shown below:

$ cd twilioCommunicationService
$ sls deploy — verbose

Ensure there are no errors in the deployment process. You can also head on to your AWS Console and verify that the API Gateway has been created. You should see an API Gateway called dev-twilioCommunication.

Also, ensure the Lambda functions are created by verifying that the twilioCommunication-dev-processRequest and twilioCommunication-dev-sendResponse lambda functions is available in the AWS Lambda console.

Ensure your Twilio credentials are setup as environment variables by clicking on each of the Lambda functions and verifying that the TWILIO_AUTH_TOKEN and TWILIO_ACCOUNT_SID have been created with the correct values in the Environment Variables section in the Code tab.

· Setup Twilio Messaging Service

Follow the instructions below to setup the Messaging Service in Twilio and associate your Twilio number to the Messaging Service:

1. In the Twilio website, login to your account and head to the Messaging Services.

2. Click on the + under Messaging Services with Copilot to add a new Messaging service.

3. Give a name to your service and click on Create. Let us say, CelebritySleuth.

4. Under Properties in the USE CASE drop-down, select Chat Bot/Interactive 2-way.

5. In the Inbound Settings section, ensure PROCESS INBOUND MESSAGES is checked.

6. Copy and paste the AWS API Gateway URL from your AWS console into the REQUEST URL field and add /process_request to the end of the URL. Your URL should resemble this: https://netxnasdfda.execute-api.us-east-1.amazonaws.com/dev/process_request

7. Select HTTP GET in the drop-down next to the field.

8. Leave rest of the fields to its default value and click SAVE.

9. Head to the Numbers section in the Twilio console.

10. Click on the number assigned to you. This will take you to the section where you can configure what should happen when an SMS message is sent to your Phone Number.

11. Under the Messaging section, select Messaging Service under the CONFIGURE WITH drop-down.

12. In the MESSAGING SERVICE, select the Messaging service that created in steps 2–8 above and click SAVE.

11. Face Recognition Service

The Face Recognition Service currently supports three functions. They are:

· createCollection: To add a collection.

· addFace: To add a face to an existing collection.

· matchFace: To match a face in a collection and look up the biography of the matched person’s name in IMDb.

A collection is a logical group of face indexes (face vectors) and their corresponding names. When you create a collection using the createCollection Lambda function, it is empty. On the back-end, createCollection creates a collection in AWS Rekognition. You can then add new faces to the collection and associate it with a given name. The collection thus created can then be used for searching images for known faces with high degree of confidence. Physically, the face indexes are stored in a database on a server managed by AWS Rekognition, you do not have direct access to this database.

Once you have created a collection you can add faces to the collection using the addFace Lambda function. To add a face to a collection, you have to provide an image, a collection name, and name you want to associate with the face. If there are no faces in the given image, or if the collection does not exist, an error message is returned. The addFace function uses AWS Rekognition to detect faces in the given image, extract features from the face, and persist information about facial features detected in the image to AWS Rekognition. The facial features are stored as searchable image vectors.

Once you have some faces indexed using the addFace function, you can then provide images of the person indexed using the matchFace function.

12. Deploy Face Recognition Service

Change directory to the faceRecognitionService directory and deploy the service by running sls deploy as shown below:

$ cd faceRecognitionService
$ sls deploy — verbose

Ensure there are no errors in the deployment process. You can also head on to your AWS Lambda Console and verify that the Lambda functions faceRecognition-dev-addFace, faceRecognition-dev-matchFace and faceRecogniton-dev-createCollection have been created.

13. Testing the Application

To test the application, create a collection by send the following SMS message to your Twilio number:

face addcol celebs

Does it work? Are you getting a message back from yonder? If not why? Look at the CloudWatch logs for your processRequest Lambda function. What is the issue?

If you look carefully, you will find that the Lambda functions do not have enough permissions to operate the SNS topic or AWS Rekognition resources.

14. Fixing twilioCommunicationService

Open the serverless.yml file in the twilioCommunicationService and uncomment the section iamRoleStatements, save and deploy the service again.

$ cd twilioCommunicationService
$ vi serverless.yml
$ sls deploy — verbose

Repeat adding a collection again.

face addcol celebs

This time, you should get a message saying that you will get a response back momentarily.

All good. You should receive your response momentarily!”

You can wait till the cows come home, but you are not getting a response are you? This is because, you have to provide more permissions to the Lambda functions in the faceRecognitionService.

15. Fixing faceRecognitionService

Open the serverless.yml file in the faceRecognitionService and uncomment the section iamRoleStatements, save and deploy the service again.

$ cd twilioCommunicationService
$ vi serverless.yml
$ sls deploy — verbose

That should have fixed it. Now if you add a collection:

face addcol celebs

You should not only get the following response:

“All good. You should receive your response momentarily!

You should also get the following message:

“Collection celebs was created successfully”

16. Adding a Face to the Collection

To add a face to the collection, along with an image of the person, send the following as part of the same SMS message:

(image)
face addimg celebs firstName_lastName

You should receive the following response:

“Picture of firstName_lastName added to collection celebs”

17. Matching a Face in the Collection

Now, find another image of the person and test the face matching power of AWS Rekognition by sending an image and the following commands through SMS to your Twilio number:

(image)
face match celebs

You should not only get the following message:

“All good. You should receive your response momentarily!”

You should also receive the following message, along with a biography of the person.

“Face matched firstName lastName with xx.xx similarity and yy.yy confidence.

….

….”

Usage

Following are the commands currently supported by CelebritySleuth.

· Add Collection

To add a new collection, type the following in the SMS body. You don’t have to add any image with this command. Any image added will be ignored.

face addcol (collection-name)

where collection-name is the name of the collection you want to create.

Example:

face addcol celebs

· Add Image

To add an image to a given collection and associate the face in the image to a name, type the following in the body of the SMS. You should include an image in the same SMS message.

face addimg (collection-name) (first-name_last-name)

where collection-name is the name of an existing collection and first-name_last-name is the full name of the person. Note: the first name and last name should be separated by an underscore.

Example:

face addimg celebs Albert_Einstein

· Match Face

To match a face in a given collection:

face match (collection-name)

where collection-name is the name of an existing collection. This command must be accompanied with an image in the same SMS message.

Example:

face match celebs

Application in Action:

Following is a sample animation of the application in action:

https://gist.github.com/skarlekar/c27ae9fef725683c0f969e2b02649454/raw/6749a5103b006e0b313f4dd510e9bd9b664c33b3/mel-fuzzed-less.gif

Conclusion

Artificial Intelligence Services offered by providers such as Amazon, Google, and Microsoft are cornerstones for building intelligent applications with a short runway. It provides developers with an ability to add intelligence to their applications using pre-trained services rather than build their algorithms and models from scratch.

On the other hand, Serverless Framework is an accelerator for adopting serverless architecture. It promises significantly less DevOps, lower cost, high scalability, and multiple deployment options across a variety of providers.

In my testing, I could not deploy the CelebritySleuth application on Microsoft Azure because the Serverless Framework does not currently support deploying functions written in Python. Aside from this, using Serverless Framework I was able to build the CelebritySleuth application from start to finish in a couple of hours, compared to half-a-day for setting up the components manually through AWS console.

Disclaimer: Capital One is not affiliated with the companies mentioned in this blog post. All trademarks used are the property of their respective owners.

One clap, two clap, three clap, forty?

By clapping more or less, you can signal to us which stories really stand out.