Operationalizing TensorFlow Object Detection on Azure — Part 3: Serving using TensorFlow Serving

Photo by NeONBRAND

In this series of blog posts, we are going to be learning about operationalizing TensorFlow Object Detection API on Microsoft Azure.

Part 1 covered TensorFlow Object Detection API and how to setup our training and evaluation workflow using Docker containers and virtual machines.

Part 2 covered how to train and scale using Kubernetes and distributed TensorFlow.

This part, Part 3, will cover how we can serve our trained model using TensorFlow Serving as a web service, and we will be deploying a simple client to get results from our service.

You can find the project repository at https://github.com/sozercan/tensorflow-object-detection/

Serving using TensorFlow Serving

Now that we have trained our model, we will start making inferences with it. This part of the series will talk about how to accomplish this with TensorFlow Serving.

Let’s get our Docker container ready. You can find the below Dockerfile in the project repo.

For convenience, base image is available in my Docker Hub as sozercan/tensorflow-serving-devel. Pre-built image is also available at sozercan/tensorflow-serving-model-server . It includes both the TensorFlow Serving server and a sample client in Python.

Let’s deploy this to our Kubernetes cluster using Helm:

STORAGE_ACCOUNT_NAME=[your storage account name]
STORAGE_ACCOUNT_KEY=[your storage account key, see part 2 for details]
# from tensorflow-object-detection
helm install tensorflow-serving/server --name tensorflow-serving-model-server --set azure_storage_account_name=$STORAGE_ACCOUNT_NAME,azure_storage_account_key=$STORAGE_ACCOUNT_KEY

This will deploy and start our Tensorflow Serving container that is serving the model in our file share at /data/export/saved_model by default.

This will also create an external IP address so we can reach our service. Let’s grab our external IP:

kubectl get services

Let’s test this with a simple client app written in Python and send this an image for object detection:

helm install tensorflow-serving/client --set server_url=[external IP from above],image_url=[JPEG image of your choice]

Other parameters and defaults you can set are:

model_name: model name of the Tensorflow Serving model server (serving by default)

label_map_path: path for the label map(/data/pascal_label_map.pbtxt by default)

save_path: path for the output of processed image (/data by default)

server_port: port for the Tensorflow Serving model server (9000 by default)

num_classes : number of classes you are detecting (1 by default)

Since this is a job, our client app will run and then will quit when finished processing our image. You can see finished jobs with kubectl get pods --show-all

Here is the code for our client app:

After this, our client app will send image to your TensorFlow Serving model server, it will visualize the bounding boxes and labels using our trained model and output the image to the Azure file share. You can find the output image with the output- prefix.

sample output image (input from here)

Conclusion

In this part, we learned how to setup a TensorFlow Serving model server to serve our trained model, and set up a client app to process, and output images with bounding boxes and labels.

Through this guide, we set up an end-to-end workflow of gathering and exporting our dataset, training and evaluating using virtual machines and Kubernetes cluster using distributed TensorFlow and finally, serving our trained model using TensorFlow Serving in our Kubernetes cluster.

If you have any questions or comments, please leave a comment below or reach out to me on Twitter @sozercan