TensorFlow Serving using Docker

Ashish Jain
Aug 31, 2018 · 3 min read

Docker is an open source software platform to create, deploy and manage virtualised application containers on a common operating system (OS), with an ecosystem of allied tools. It unlocks the potential of your organisation by giving developers and IT the freedom to build, manage and secure business-critical applications without the fear of technology or infrastructure lock-in. With Docker, you have the solution that helps you manage the diverse applications, clouds and infrastructure you have today while providing your business a path forward to future applications.

With accelerated advancements in AI more and more companies are emphasising on products, technologies and platforms , but Google with its advanced technology stack has put a step ahead by announcing Tensorflow Serving, which enables data scientists, researchers and developers to launch a pre-built super fast RESTFUL API to serve their models in a production ready environment.

Features of Tensorflow Serving which distinguishes it from any other serving platform are :

  • Multiple models, or indeed multiple versions of the same model, can be served simultaneously.
  • Architecture is highly modular, parts of its architecture can be used individually
  • Lifecycle management and version control of models is possible
  • Provides out-of-the-box integration with TensorFlow models, but can be extended to serve other types of models and data
  • Provides high performance API with low latency and maximum throughput
  • Model updation can be done without service downtime

How to configure Tensorflow Serving ??

Step 1 : Download and install Docker

Docker is an open source software platform to create, deploy and manage virtualised application containers on a common operating system (OS), with an ecosystem of allied tools. With Docker, you have the solution that helps you manage the diverse applications, clouds and infrastructure you have today while providing your business a path forward to future applications.To install docker on your system click here .

Step 2 : Clone TensorFlow Serving repository

For original Tensorflow Serving Repo click here .

The documentation and procedure to successfully build and run Tensorflow Serving image is complex using the repo mentioned on Tensorflow website.

There are lots of dependencies issue and errors which makes it difficult to setup. Hence i’ve used this repo as the dependencies are pre-compiled here, and it offers some pre-trained model to test our architecture.

https://github.com/ashish4669/tensorflow_serving.git

Step 3 : Build TensorFlow Serving Docker image

Move to tensorflow_serving repo and do a docker build for the image. This will use the Dockerfile present in the repo to build image and manage dependencies.

docker build — pull -t tensorflow-serving .

Step 4 : Run TensorFlow Serving Docker image

Once the image is built, we'll run it. Use the absolute path of the repo and replace it with my path /Users/Desktop/tensorflow_serving/model_volume/ in the command below.

docker run -it -p 9000:9000 -p 9001:9001 -v /Users/Desktop/tensorflow_serving/model_volume/:/home/ --name tensorflow-serving tensorflow-serving

Step 5 : Run TensorFlow Serving API

This step will start the server and serves the rest api on port 9001. It uses the configuration from this directory on the virtual machine /home/configs/models.conf

tensorflow_model_server — port=9000 — rest_api_port=9001 — model_config_file=/home/configs/models.conf

Step 6 : Test the above architecture

Once the architecture is ready and server is up and running, we can test it using POST request.

curl -X POST \
http://localhost:9001/v1/models/iris:classify \
-H 'cache-control: no-cache' \
-H 'postman-token: f7fb6e3f-26ba-a742-4ab3-03c953cefaf5' \
-d '{
"examples":[
{"x": [5.1, 3.5, 1.4, 0.2]}
]
}'

You’ll get the following response

{
"results": [
[
[
"Iris-setosa",
0.872397
],
[
"Iris-versicolor",
0.108623
],
[
"Iris-virginica",
0.0189799
]
]
]
}

Once this architecture is up and ready, you can experiment with your own customised models and update model.conf file to test them.

With great powers come great responsibility !!!

Tensorflow Serving is a powerful, high performance platform for deploying models in production. BUT !! There’s lack of documentation available on how to configure it on system and use it. This blog is an effort to provide basic understanding of this serving platform, and steps to configure and use it.

Leave a comment if you have questions or suggestions. Looking forward to see you in next blog “How to update models in TensorFlow Serving” !!

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade