Building Robust Production-Ready Deep Learning Vision Models in Minutes

A comprehensive hands-on guide to TensorFlow Serving

Introduction

What is Serving?

Source

TensorFlow Serving Overview

TensorFlow Serving Architecture

Life of a Servable (Source)

Model Serving Methodology

Main Objective — Building an Apparel Classifier

Fashion-MNIST samples (by Zalando, MIT License).

Loading Dependencies and Data

1.14.0
Train_images.shape: (60000, 28, 28), of uint8
Test_images.shape: (10000, 28, 28), of uint8

Training a simple CNN Model

Train_images.shape: (60000, 28, 28, 1), of uint8
Test_images.shape: (10000, 28, 28, 1), of uint8

Fine-tuning a pre-trained ResNet-50 CNN Model

Source
Train_images.shape: (60000, 32, 32, 3), of float32
Test_images.shape: (10000, 32, 32, 3), of float32

Saving models for TensorFlow Serving

Serving models with CPU Inference

docker pull tensorflow/serving

Install TensorFlow Serving

!echo "deb [arch=amd64] http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" \
| sudo tee /etc/apt/sources.list.d/tensorflow-serving.list && \
curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg \
| sudo apt-key add -
!sudo apt-get -y remove tensorflow-model-serverReading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
tensorflow-model-server
0 upgraded, 0 newly installed, 1 to remove and 39 not upgraded.
After this operation, 0 B of additional disk space will be used.
(Reading database ... 124893 files and directories currently installed.)
Removing tensorflow-model-server (1.13.0) ...
!sudo apt-get update && sudo apt-get install tensorflow-model-serverHit:1 http://packages.cloud.google.com/apt cloud-sdk-stretch InRelease
Get:2 http://security.debian.org stretch/updates InRelease
...
...
Preparing to unpack .../tensorflow-model-server_1.14.0_all.deb
Unpacking tensorflow-model-server (1.14.0) ...
Setting up tensorflow-model-server (1.14.0) ...

Start Serving Models with TensorFlow Serving

model_config_list: {
config: {
name: "fashion_model_serving",
base_path: "/home/jupyter/tensorflow_serving/tf_saved_models",
model_platform: "tensorflow",
model_version_policy {
specific {
versions: 1
versions: 2
}
}
}
}

Serving Model Inference Requests

Model Warmups

Model 1 warmup complete
Model 2 warmup complete

Benchmarking Model Serving Requests

CPU times: user 7.64 s, sys: 612 ms, total: 8.26 s
Wall time: 9.76 s
Out[116]: 10000
CPU times: user 21.6 s, sys: 1.38 s, total: 23 s
Wall time: 36.4 s
Out[122]: 10000

Serving models with GPU Inference

Pulling TF Serving GPU image

!docker pull tensorflow/serving:latest-gpulatest-gpu: Pulling from tensorflow/serving
Digest: sha256:fd54edb56a7b
Status: Image is up to date for tensorflow/serving:latest-gpu
docker.io/tensorflow/serving:latest-gpu
!docker images

Start Serving Models with Docker TensorFlow Serving GPU

!docker ps -all
!docker logs 7d4b091ccefa | tail -n 15

Model Warmup

Model 2 warmup complete

Benchmarking Model Serving Requests

CPU times: user 23.5 s, sys: 1.87 s, total: 25.3 s
Wall time: 31.3 s
Out[155]: 10000
100%|██████████| 10000/10000 [01:55<00:00, 86.56it/s]CPU times: user 39.1 s, sys: 2.2 s, total: 41.3 s
Wall time: 1min 55s
Out[157]: 10000
100%|██████████| 10000/10000 [03:04<00:00, 54.12it/s]CPU times: user 3min 8s, sys: 17.2 s, total: 3min 25s
Wall time: 3min 4s
Out[159]: 10000

Bonus: Building an Apparel Classifier API with Flask & TensorFlow Serving

Create API with Flask

Start Docker Container for TF Serving

!docker start 7d4b091ccefa
!docker ps -all

Start our Apparel Classifier Web Service

(200, 'API Live!')

Serve Sample Apparel Classification with Web Service

plt.imshow(cv2.cvtColor(cv2.imread('sneaker.jpg'),    
cv2.COLOR_BGR2RGB))
{'apparel_type': 'Sneaker'}

Benchmark our Web Service

100%|██████████| 10000/10000 [05:26<00:00, 30.66it/s]CPU times: user 1min, sys: 3.17 s, total: 1min 4s
Wall time: 5min 26s
Out [58]: 10000
Inference time per image: 32.599999999999994 ms

Conclusion

Google Developers Experts

Experts on various Google products talking tech.

Dipanjan (DJ) Sarkar

Written by

Data Science Lead @Applied4Tech, @Google Developer Expert — Machine Learning, Author, Consultant, AI Advisor @Springboard, Connect: http://bit.ly/djs_linkedin

Google Developers Experts

Experts on various Google products talking tech.

Dipanjan (DJ) Sarkar

Written by

Data Science Lead @Applied4Tech, @Google Developer Expert — Machine Learning, Author, Consultant, AI Advisor @Springboard, Connect: http://bit.ly/djs_linkedin

Google Developers Experts

Experts on various Google products talking tech.

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store