Automated Model Deployment with BentoML and Kubeflow

How to automate model deployment and even model retraining in your own setup.

Benjamin Tan Wei Hao
DKatalis
Published in
9 min readSep 8, 2022

--

One of the things that I’ve been dissatisfied with so far with our current workflow is that model deployment is not as automated as it could be. This is partly because in model deployment there are several things to consider:

  1. How the model was built, which affects how the model would be served
  2. How the served model would be consumed
  3. How to scale workloads
  4. How to monitor the service and implement logging
  5. How the model can be retrained and automatically deployed if it performs better

I’ve been experimenting with BentoML and Yatai for a few weeks now, intending to come up with a proof-of-concept that would be able to address the above points. I’ve already covered my initial impressions of BentoML/Yatai in a previous post.

While the proof-of-concept that I will be presenting is by no means complete, I think it proves that BentoML is indeed a very compelling solution to address most of the pain points when it comes to model deployment.

Note that your tech stack most likely would look different, it doesn’t matter because you should be able to find analogue to what you’re already using, except of course for BentoML/Yatai!

Brief Outline

Here’s a breakdown of how I approached solving the problem:

  1. Model training with Kubeflow and triggering CI
  2. Building Bentos and Deploying to Yatai via CI
  3. Serving, Monitoring, and Logging with Yatai

Step 1: Model Training with Kubeflow and Triggering CI

This is what the pipeline looks like. Yes, your pipeline would most definitely be more sophisticated but this is purely for pedagogical purposes. Just imagine that all the data preparation, transformation, model training, evaluation, and other steps are all wrapped up in the Short train model

In the following sections, I’d describe the pertinent parts of each component.

--

--

Benjamin Tan Wei Hao
DKatalis

Author of The Little Elixir & OTP Guidebook, Mastering Ruby Closures, Building an ML Pipeline in Kubeflow. | Currently: Product Owner at @dkatalis.