Can you deploy an ML Model within an hour ? Well, I just did.

Developing a machine learning model is fun as it includes a lot of research and experimentation around it. But keeping track of those experiments is also important. This is where mlflow helps us. After all the experiments, model building, model selection, we get the final model in hand. Now we need to deploy it on the cluster, which is a slightly complex step. At this point, we have our hero — Seldon, to rescue us.

In this article, we will review both mlflow and Seldon. Both the tools are open-source and hence anyone can use them.

Introduction to MLflow

MLflow is an open-source platform for managing end-to-end machine learning lifecycle. It tackles four primary functions:

• Tracking experiments to record and compare parameters and results (MLflow Tracking).
• Packaging ML code in a reusable, reproducible form in order to share with other data scientists or transfer to production (MLflow Projects).
• Managing and deploying models from a variety of ML libraries to a variety of model serving and inference platforms (MLflow Models).
• Providing a central model store to collaboratively manage the full lifecycle of an MLflow Model, including model versioning, stage transitions, and annotations (MLflow Model Registry).

MLflow is library-agnostic. You can use it with any machine learning library, and any programming language, since all functions are accessible through a REST API and CLI. For convenience, the project also includes a Python API, R API, and Java API.

MLflow components

MLflow provides four main components:

• MLflow Tracking: Record and query experiments — code, data, config, and results
• MLflow Projects: Package data science code in a format to reproduce runs on any platform
• MLflow Models: Deploy machine learning models in diverse serving environments
• MLflow Registry: Store, annotate, discover, and manage models in a central repository

Tracking

This API helps in managing all the experiments a data scientist performs. Usually, these experiments are messy. The data is hard to track and it’s difficult to understand what parameters are used in the experiment. By using this API data scientists can log all the information and compare all the experiments performed. The UI too is easy to understand.

This API allows data scientists to log hyperparameters, metrics, models, and other related artifacts. All this information is stored on the disk under mlruns folder as the default option. However, we can use other databases or remote-tracking servers. To log runs remotely, set the MLFLOW_TRACKING_URI environment variable to a tracking server’s URI or call mlflow.set_tracking_uri(). Accessing the UI is just one command away (like mlflow UI).

Projects

Even after tracking the experiments and all the related data and files, when these projects are shared among data scientists, it is hard to reproduce the same results. MLflow Projects have a standard format for packaging reusable data science code. These projects constitute a simple directory with the source code and an ML project file that specifies the dependencies and how to run the code. Projects can contain a conda.yaml file for specifying a Python Conda environment.

Below is a simple example of MLproject file

name: My Project 
conda_env: conda.yaml 
# Can have a docker_env instead of a conda_env, e.g. 
# docker_env: 
# image: mlflow-docker-example 
entry_points: 
     main: 
         parameters: 
             data_file: path 
             alpha: {type: float, default: 0.5} 
             l1_ratio: {type: float, default: 0.1} 
         command: "python train.py {alpha} {l1_ratio}"

Reproducing the code is then one command away with `mlflow run .`

Models

MLflow Models offer a convention for packaging machine learning models in multiple flavors, with a variety of tools to help you deploy them. Each Model is saved as a directory containing arbitrary files and a descriptor file that lists several “flavors” the model can be used in.

The directory of the saved model under artifacts looks like the following:

model 
    |--MLmodel 
    |--conda.yaml 
    |--model.pkl

Further, using this model is very easy and all the information about the model parameters and dependencies are readily available. MLflow also provides tools to deploy many common model types to diverse platforms, for example: to a Docker-based REST server, to cloud platforms such as Azure ML and AWS SageMaker, and as a user-defined function in Apache Spark for batch and streaming inference.

Registry

The MLflow Model Registry component is a centralized model store, set of APIs, and UI, to collaboratively manage the full lifecycle of an MLflow Model. We can register any of the logged models. A registered model has a unique name, versions, associated transitional stages, model lineage, and other metadata.

Users can easily register the model using UI and find the model under the registered model. However, we need a database-backed backend store to store the model.

Documentation of MLflow

Mlflow documentation is quite rich with enough details about the API with good examples. Users can easily start using the MLflow with the example provided in the documentation.

Summary

Mlflow provides components for tracking experiments, make reproducible projects, model registry, etc. However, it lacks the components to manage and monitor deployments of the models in production. Users still need to depend on other tools for productionizing the model.

Introduction to Seldon

There is one more component named Seldon which is different from mlflow. Mlflow works around the whole ML lifecycle, while Seldon is more focused on MLOps to package, deploy, monitor, and manage the production ML models.

Seldon is an open-source platform to deploy machine learning models on Kubernetes at a massive scale. It converts the ML models into production REST/GRPC microservices.

Seldon handles scaling to thousands of production machine learning models and provides advanced machine learning capabilities out of the box including Advanced Metrics, Request Logging, Explainers, Outlier Detectors, A/B Tests, Canaries, and more.

Experience of using Seldon

Using Seldon is not very hard. The process of deploying the model on the cluster is divided into the following steps.

• Wrap model using language wrapper
• Containerize the model
• Define inference graph
• Deploy model on kubernetes cluster

Although setting up Seldon for Kubernetes requires some knowledge of Kubernetes, we can use the already provided jupyter notebook to easily set-up Seldon on Kubernetes.

Language wrapper

When we have a custom use-case where pre-packaged inference servers cannot cover, we can leverage language wrappers to containerize our machine learning model and logic. Seldon core provides the following language wrappers

• Python [Graduated]
• Java [Incubating]
• R [Alpha]
• NodeJS [Alpha]
• Go [Alpha]

Currently, for the python language wrapper, there is enough documentation to define inference class and how to implement other logic. Installing Seldon-core is also easy for the python wrapper.

After wrapping our model we need to create a docker image. We have two options here using Dockerfile and s2i. s2i is a source-to-image app. There are a bunch of other files we need before containerizing the model, such as inference class(wrapped model file), dependencies file, and environment file.

Inference graph

Seldon Core extends Kubernetes with its custom resource SeldonDeployment where you can define your runtime inference graph made up of models and other components that Seldon will manage.

When it comes to the documentation for defining the inference graph, we do not have many details provided. The user needs to jump between the Kubernetes PodTemplateSpec and source code to properly define the inference graph.

Deploy model on kubernetes

For Kubernetes, we first need to set up a cluster of our choice with one of the supported Ingress (Istio or Ambassador).

Once we have created our inference graph, we can deploy it to Kubernetes with kubectl. For example, if the deployment is packaged in my_ml_deployment.yaml. we can deploy the model using this single command.

kubectl apply -f my_ml_deployment.yaml

Summary

Seldon-core is not a stand-alone tool as it has some pre-requisites. The user still requires some knowledge of Kubernetes as it can help the user in debugging the failures of deployment.

Conclusion

MLflow is easy to install and use for most users. It provides full control over the whole ML life cycle. It provides API for multiple languages, but users still need to depend on other tools for managing and monitoring the deployments.

On the other hand, Seldon-core is more focused on model deployment and monitoring. It requires other tools to be installed to use it properly. Users can version the deployment through an inference graph, but unlike mlflow there is no way users can keep track of the experiments. So it is not as easy to install and use as MLflow, but it still solves some complexity at the deployment level.

When it comes to documentation mlflow has better documentation as compared to Seldon.

P.S. We at Censius are also looking for engineering and growth wizards who are enthusiastic about modern technology operations, and willing to join a team that fosters creativity, generates stellar solutions and delivers engineering excellence. The opportunity to make an impact and contribute to a complex application suite of products, is ticking. Apply with us https://www.censius.ai/careers