Using Google Cloud Build for Kubeflow Pipelines CI/CD

Or how we used kubectl port-forward within Google Cloud Build to integrate with private k8s services

Here at Imagr we are experimenting with Kubeflow Pipelines for some of our machine learning workflows.

In order to automate Kubeflow pipeline deployment we’ve setup a simple Cloud Build job that compiles the pipeline and uploads the new version.

The following snippet starts a docker container that runs thekubectl port-forward command

Invokes kubectl port-forward within a docker container exposing it as a service in the cloudbuild network

The 2 most important things to notice is:

1. --network cloudbuild
2. --address 0.0.0.0

In the next step you should be able to access the endpoints in the exposed service. In the example below I’m using curl to list the existing Kubeflow pipelines.

Invoke the Kubeflow Pipeline API on port 8888

Of course this is not very useful, so here is another example that uploads a new pipeline version:

Uploads a new pipeline version

The same strategy could easily be adapted to communicate with other internal kubernetes / kubeflow services.