Tilt Blog
Published in

Tilt Blog

A tarot deck, suitable for choosing Kubernetes topologies in CI. Via Wikipedia.

Three Ways to Run Kubernetes on CI and Which One is Right for You!

Strategy #1: Local Cluster, Remote Registry

  1. I created a dedicated gcr.io bucket for us to store images, and a GCP service account with permission to write to it.
  2. I added the GCP service account credentials as a secret in our CI build.
  3. I forked kubeadm-dind-cluster, a set of Bash scripts to set up Kubernetes with Docker-in-Docker techniques.
  • When drive-by contributors sent us PRs, the integration tests failed. They didn’t have access to to write to the gcr.io bucket. This made me so sad. Contributors felt unwelcome. I never figured out a way to make this secure.
  • We didn’t reset the gcr.io bucket between test runs. So it was hard to guarantee that images weren’t leaking between tests. For example, if image pushing failed, we wanted to be sure we weren’t picking up a cached image from a previous test.

Strategy #2: Local Registry On a VM

  1. Creates a VM.
  2. Installs all our dependencies, including Docker.
  3. Creates a kind cluster with a local registry, using their script.

Strategy #3: Local Registry On Remote Docker

  1. Creates a container with our code.
  2. Sets up a remote Docker environment outside the container. (This avoids the pitfalls of running Docker inside Docker.)
  3. Starts a kind cluster with a local registry inside the remote Docker environment.
  4. Uses socat networking jujitsu to expose the remote registry and Kubernetes cluster inside the local container.

Putting it Together




Kubernetes for Prod, Tilt for Dev https://tilt.dev/

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Nick Santos

Software Engineer. Trying new things @tilt_dev. Formerly @Medium, @Google. Yay Brooklyn.