Photo by chuttersnap on Unsplash

Building Kubeflow Hybrid Cloud Pipelines

Kubeflow Pipelines is an open-source project to simplify operationalizing Machine Learning workflows. This blog post series will look at an Industrial Image Classification use-case and we’ll use Transfer Learning to train a MobileNet model. We’ll go through the steps needed to operationalize the training, evaluation and serving the model across multiple clouds.

For some background, this blog post covers Kubeflow Pipelines and components: https://towardsdatascience.com/kubeflow-components-and-pipelines-33a1aa3cc338

For the first part of the blog, the focus is on building an deploying a simple pipeline with two steps. Each deployed in different clusters.

Key takeaways for the blog:

• Build and deploy a Kubeflow Pipeline that spans across two clouds.

Pre-requisites:

• A Kubernetes cluster on-premise with Kubeflow (Source cluster).
• A Kubernetes cluster in the cloud with Kubeflow (Target cluster).

Refer to the Kubeflow docs for instructions to install Kubeflow.

Overview of Tasks:

• Create Kubernetes resources for the pipeline.
• Create a secret for accessing the cloud cluster.
• Define the basic two step pipeline.
• Execute the pipeline from a notebook.

Task 1: Create Kubernetes resources for the pipeline

Note: All these have to be run on the on-premise / source cluster.

Create a NFS Storage Class that can provide ReadWriteMany volumes. Deploy the helm chart below depending on the name of the cluster’s default storage class:

• For clusters with the default Storage Class named standard, run the first set of instructions.
• For clusters with the default Storage Class named default, run the second set of instructions.

Helm chart to create a NFS Storage Class based on existing Storage Class.

Create a Persistent Volume Claim called nfs with the Storage Class nfs

RWX Persistent Volume Claim for the pipeline.

Task 2: Create a secret for accessing the cloud cluster

For Google Kubernetes Engine, create a service-account key file corresponding to the project and a service account name under which the Kubernetes cluster was created.

This way cloud access to the cluster can be added or removed at any point. Also, this has an added advantage of being able to check-in the pipeline code into VCS without worrying about committing the kubeconfig along with it. Even when the pipeline is containerized, the kubeconfig files would not be checked in inside them.

Task 3: Define the basic two-step pipeline

Task 4: Execute the Pipeline from a notebook

There are two ways to execute a pipeline:

1. By compiling the pipeline definition and uploading to the KF Pipelines dashboard.
2. Through a notebook with a KF Pipelines client.

Note: This notebook has to be run on a notebook server provisioned from the Kubeflow Jupyter Web App.

Upload this notebook to a Jupyter instance on a Kubeflow cluster and run through the cells to run the Pipeline.

This example can be extended to training a model on-premise and deploying the model to the cloud.

In principle, it follows the same workflow but adds a few more steps to the pipeline.

If you’re interested in solving these kinds of problems, please consider contributing to the Kubeflow project!