How to Setup Tensorflow Serving For Production

Google is once again leading the space and defining how we build and develop technology. The rush to be the leader in AI is no joke, and they are not waiting for anybody. They recently announced Tensorflow Serving, which is an API that easily enables Data Scientists to launch a pre-built super fast RESTFUL API to serve their models in a production ready environment. It includes the ability to serve different models, AND the ability to serve different versions of the same model at the same time. Too good to be true? Its Google, so no.

Getting Started

Google has made it super easy to get started by providing us a docker file.

For those not familiar with docker before this, like I was, let me tell you, it simplifies your life tremendously especially for things like this. So if you do not have Docker installed, click here to install it.

Start by cloning the Tensorflow Serving Repo

git clone https://github.com/tensorflow/serving
cd serving/tensorflow_serving/tools/docker

In that docker directory you will see 3 files.

  • Dockerfile, which is a minimal VM with TensorFlow Serving Pre-Installed and Compiled.
  • Dockerfile.devel, which is a minimal VM with all of the dependencies needed to build TensorFlow Serving.
  • Dockerfile.devel-gpu, which is a minimal VM with all of the dependencies needed to build TensorFlow Serving with GPU support.

I wanted you to clone this repo to get familiar with the code and examples. However, I have made my own Dockerfile which is better suited for learning and testing quickly. Click Here for the repo link. That repo also has pre-built models in it, that we will later build in this series.

git clone https://github.com/brianalois/tensorflow_serving_tutorial.git

Do a docker build using the Dockerfile in my repo. This will take about 3–5 minutes.

cd tensorflow_serving_tutorialdocker build --pull -t test-tensorflow-serving .

Run that docker newly built docker image, and enter into the virtual env.

Or (if you want to compile it yourself)

I suggest most people just build from my docker image and skip this part.

If you want to compile the tf serving from source you can. In the original tf serving repo where they have the docker images run this command(it will take 30 min — 2 hours, depending on your computer):

cd serving/tensorflow_serving/tools/dockerdocker build --pull -t test-tensorflow-serving -f Dockerfile.devel .

However, based on your computer that command might fail. It did for me and I have a Top of the line Mac currently, so you will probably need to run this command, to compile it yourself.

#however based on the ram of your computer you may have to run this
docker build --pull --build-arg TF_SERVING_BUILD_OPTIONS="--copt=-mavx \
--cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0 --local_resources 2048,.5,1.0" -t \
test-tensorflow-serving -f Dockerfile.devel .

Run the Docker Image

Now after it is built regardless of the way you did it, we can begin to run it.This next command is not a direct copy and paste. You must use absolute paths here to the volume you want to use for your docker machine. We will be using the model_volume directory in the repo just cloned. My path is “/Users/Brian/Desktop/model_volume/” Replace to what ever your path is.

Tip you can run this command in your directory to find the path

pwd

Then run this command with your absolute path:

docker run -it -p 8500:8500 -p 8501:8501 -v /Users/Brian/Desktop/tensorflow_serving_tutorial/model_volume/:/home/ test-tensorflow-serving

Now if everything worked you should be in your docker machine, and your shell should look something.

root@15954c4d0666:/#

Now we can get to the fun part and run the tensorflow serving api:

tensorflow_model_server --port=8500 --rest_api_port=8501 --model_config_file=/home/configs/models.conf

This basically starts the server and serves the rest api on port 8501. It uses the configuration from this directory on the virtual machine /home/configs/models.conf, but that is actually referring to the file in the repo in this directory

cat model_volume/configs/models.config

That file contains config information for the server to be able to serve the models in the repo in the right way. Lets look at that real quick:

model_config_list: {
config: {
name: "xor",
base_path: "/home/models/xor",
model_platform: "tensorflow",
model_version_policy: {
all: {}
}
},
config: {
name: "iris",
base_path: "/home/models/iris",
model_platform: "tensorflow",
model_version_policy: {
all: {}
}
}
}

This is pretty self explanatory, the part that is important is if you want to serve all versions of your model this part is necessary, if you do not have it than it will only serve the latest model:

model_version_policy: {
all: {}
}

The model versions are determined by name of the nested directories, in that specific model directory.

Now Lets Test It!!

Open up another terminal, to make some curl requests. You could/should use Postman if you prefer.

Iris Classify

curl -X POST \
http://localhost:8501/v1/models/iris:classify \
-H 'cache-control: no-cache' \
-H 'postman-token: f7fb6e3f-26ba-a742-4ab3-03c953cefaf5' \
-d '{
"examples":[
{"x": [5.1, 3.5, 1.4, 0.2]}
]
}'

Response:

{
"results": [
[
[
"Iris-setosa",
0.872397
],
[
"Iris-versicolor",
0.108623
],
[
"Iris-virginica",
0.0189799
]
]
]
}

Iris Predict

curl -X POST \
http://localhost:8501/v1/models/iris/versions/2:predict \
-d '{
"signature_name": "predict-iris",
"instances":[
[5.1, 3.5, 1.4, 0.2]
]
}'

response:

{
"predictions": [
[
0.872397,
0.108623,
0.0189799
]
]
}

There you go! We’re all set. Now you must be wondering well how to a build, train and save my models in a format that this api will recognize. Click here for part 2 of this series that answers those questions.

Best of Luck

— — Brian Alois Schardt

Associate Director of Engineering Warner Music Group

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store