Exporting deep learning models from keras to tensorflow-serving

Combining Keras and Tensorflow-Serving
  • Keras 1.2.2
  • Tensorflow 1.3
  • Tensorflow-serving-api 1.0.0
  • Python 2.7 (Tensorflow-serving only supports 2.7 currently)
Python source for exporting keras on a tensorflow backend to tensorflow-serving.
  • Format of the signatures — Documentation on the tensorflow side wasn’t clear and no examples were provided. Confusing things further several tutorial suggested just passing the tensors. For Keras running on a tensorflow backend the signature inputs for prediction look like: prediction_signature = tf.saved_model.signature_def_utils.predict_signature_def({“image”: Resnet50model.input}, {“prediction”:Resnet50model.output})
  • Variable initialization — init_ops sets up the model on the tensorflow_serving side, and both global and model parameters need to be initialized here or tf-serving will complain about uninitialized variables at runtime.
  • Keras has a variable called learning phase that also needs to be separately initialized. To further confuse matters the initialization has to happen before the model is instantiated. (K._LEARNING_PHASE = tf.constant(0)
    K.set_learning_phase(0)
    )
  • sess.run(init_op) can be called only once. I was setting up the model in a jupyter notebook and on at least one occasion called this twice, which caused errors in tensorflow-serving.
  • Keras associates itself with the tensorflow session that it’s running on implicitly and most operations are transferred. For some reason in the above code (possibly due to the use of pop?) the final dense layer added to the model, the one that does the transfer learning, was being randomly initialized and wasn’t being loaded when all the other weights were loaded. This lead to an incredibly challenging situation where the model would predict correctly in python but would be incorrect and would usually consistently predict one of the classes once the model was loaded into tensorflow serving. In order to fix this I had to explicitly associate the Keras backend with the active session using the command: K.set_session(sess)

--

--

I’m a research scientist working at NVidia on deep learning for tabular data.

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Even Oldridge

Even Oldridge

I’m a research scientist working at NVidia on deep learning for tabular data.