week 6&7 | GSoC’19 | CCExtractor Development

Image Source: https://ccextractor.org

For the past two weeks I was involved in two major tasks I worked upon

• Training facial expression deep learning model
• Implementing TensorFlow Serving

For facial expression recognition, I used fer2013 dataset and Google colab to train the model. You may find the Colab Link here. I had initially planned to integrate this model into the existing Django app, But deploying models this way was a very naive approach and inefficient way to deploy deep learning models. The second choice that I had was TensorFlow Serving. TFS(TensorFlow Serving) is a flexible, high-performance serving system for machine learning models, designed for production environments. TFS allows us to seamlessly serve our machine learning models. It also provides the option to serve multiple version of the same deep learning model.

A small overview of TFS , Manager is responsible for loading, unloading, lookup and lifetime management and handle the full lifecycle of Servables, Servables are the underlying objects that clients use to perform computation (for example, a lookup or inference). Manager may wait to unload until a newer version finishes loading, based on a policy to guarantee that at least one version is loaded at all times. Below is the workflow TFS.

source: https://www.tensorflow.org/tfx/serving/architecture

For more information, How TFS works, You may visit the link provided in the Resources down below.

The first step is to make Docker(recommended) ready environment for the deployment of the model. Here is the link that you will find it helpful in making your Docker environment ready. Now ,it’s time to make the trained model ready for TFS below provided gist will help you to make your model ready for Tensorflow Serving.

x and y are input-output tensor nodes respectively

Once it’s successfully run, you may check the signature definitions of the model by running the below command in terminal:

saved_model_cli show — dir {export_path} — all 

{export_path: is the base directory which should be same as it was defined while saving the model}.

It will show output like below, Here you will find information about the servable.

MetaGraphDef with tag-set: ‘serve’ contains the following SignatureDefs:
signature_def[‘serving_default’]:
 The given SavedModel SignatureDef contains the following input(s):
 inputs[‘input_image’] tensor_info:
 dtype: DT_FLOAT
 shape: (1, 48, 48, 1)
 name: prefix/image:0
 The given SavedModel SignatureDef contains the following output(s):
 outputs[‘final_output’] tensor_info:
 dtype: DT_FLOAT
 shape: (1, 6)
 name: prefix/softmax:0
 Method name is: tensorflow/serving/predict

Now it’s time to check How our deployed model is working. Now let’s make our TFS ready for work

docker run -it -p 8500:8500 -p 8501:8501 -v /Users/amit/Desktop/test/exportpath/:/home/ tensorflow/serving:nightly-devel

• -p 8501:8501 : Publishing the container’s port 8501 (where TF Serving responds to REST API requests) to the host’s port 8501
• -p 8500:8500 : Publishing the container’s port 8500 (where TF Serving responds to gRPC requests) to the host’s port 8500
• -v /Users/amit/Desktop/test/exportpath/ : where the models are saved
• /home/ : This directory is mounted under which the model directory will be present
• tensorflow/serving:nightly-devel : Running a Docker container based on the serving image “tensorflow/serving:nightly-devel”

go to root dir (using cd ..)

tensorflow_model_server — rest_api_port=8501 — model_name=fer2013 — model_base_path=”/home/fer2013/models/”

Now TFS is ready to listen a client request.Below is the python client to call a REST API requests .

import json
data = json.dumps({“signature_name”: “serving_default”, “instances”: img.tolist()}) #img is of shape (1,48,48,1)
import requests
headers = {“content-type”: “application/json”}
json_response = requests.post(‘http://localhost:8501/v1/models/fer2013:predict', data=data, headers=headers)
predictions = json.loads(json_response.text)[“predictions”]

Hola! You will get the result.

Future work may include the integration of kubernetes cluster to scale as the load starts increasing on the app.

I did face some challenges while implementing the TensorFlow Serving but I hope going through official documentation and doing one or two online tutorial examples you may find it easy to work with TFS. I hope you will find this article helpful.

Thanks for reading :)

LinkedIn GitHub Twitter

References:

https://www.tensorflow.org/tfx/serving/

https://medium.com/tensorflow/serving-ml-quickly-with-tensorflow-serving-and-docker-7df7094aa008