Serving TensorFlow Model - Hello World

You have trained your TensorFlow model. What is next? Serving a TensorFlow model for the first time often requires some curdles to jump over. It is always nice to start with a “Hello World!” before facing Thanos. At least for me, the hello word used on the official website: Serving MNIST was not felt like a true hello world. After some search, I found TensorFlow Serving 101 pt. 1 which uses a simple model to add a number with 10 and to return the result. Even though that is a kind of hello world I was looking for, the article hides a lot of actual practice from readers. Instead, you will deploy the model using a Docker image and access it using a third-party client library. I do not say it is wrong but I decided to serve the same model in native Google’s way.

Step 1: Install TensorFlow Model Server using apt install I prefer installing this way to avoid all the hassles of compiling from source. In Step 3, if you get /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.22' not found error in Ubuntu 16.04, follow this StackOverflow answer to fix it.

Step 2: Create a new Python script tf_server_hello_world.py with the following code:

Above code create a model to receive two integers: x and y and generates and output ans which is the addition of x and y. Once you run this script using python3, you will get a new directory models/hello_world/1 . For more details, please check the parent article.

Step 3: Open a terminal in models directory and start tensorflow_model_server.

tensorflow_model_server --port=9000 --model_name=hello_world --model_base_path=$(pwd)/hello_world

Step 4: Create a new tf_client_hello_world.py with the following code. Note that TensorFlow serving API is released for Python 2. Therefore we are using Python 2 here.

Step 5: Running tf_client_hello_world.py should give an output similar to this:

$python tf_client_hello_world.py
outputs {
  key: "ans"
  value {
    dtype: DT_INT32
    tensor_shape {
    }
    int_val: 30
  }
}

As you can see, we get 30 which is the addition of our inputs 10 and 20.

Hope you find this Hello World simple enough to start your adventure.