Local development with Kubernetes

Originally published at https://mtpereira.com/local-development-k8s.html

Lately I’ve been working with Kubernetes in order to provide a self-service deployment platform. This ranges from setting up a production-ready cluster, to figuring out how to fit it in a development workflow.

Although it’s getting easier, cheaper and faster to use cloud-provisioned environments for development, it can be a productivity-killer to have the cognitive overhead of dealing with remote hosts and what-not for running your code.

Also, when developing Docker containers, we often do not wish to publish them, which can be a pain if you don’t have a private registry already setup. The following is the setup that I’ve found to deal with these issues, using minikube running on macOS.

Setup minikube

We’ll be using xhyve to run minikube. This alternative is lighter and smaller than the default VirtualBox VM and also has the advantage of using Plan9 filesystem over Virtio for mounting your local filesystem on the VM. So far, this setup has been much more reliable for me when it comes to dealing with ownership and permissions of shared files than using vboxfs.

Install all the required packages to run minikube on a xhyve VM:

# Check https://github.com/kubernetes/minikube/blob/master/DRIVERS.md#xhyve-driver
brew update && brew install xhyve docker-machine-driver-xhyve kubernetes-cli minikube
sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve

To start a cluster on minikube using xhyve, run the following:

minikube start --vm-driver xhyve --insecure-registry localhost:5000

This will create a VM and configure a single-node Kubernetes cluster inside it. We’ll see in a bit why we need the --insecure-registry flag.

I’d recommend to add this command to your shell’s aliases so that you do not forget to start your cluster with these options:

alias minikube-start='minikube start --vm-driver xhyve --insecure-registry localhost:5000'

Using minikube’s Docker daemon from our localhost

Since we’re already running a Docker daemon inside minikube’s VM, we should take advantage of this instead of relying on another VM. This really saves up some resources on your host and keeps things simple.

In order to do that, you just need to run the following and you’re ready to go:

eval $(minikube docker-env)

Running a local private Docker registry

Kubernetes provides an add-on for running a private Docker registry inside your cluster, which is documented here. However, there are a couple of caveats that you need to consider when running it inside minikube.

Currently, there’s limited support for adding add-ons automatically to the Kubernetes cluster running on minikube. As such, our registry manifest will be a little different from the official one:

apiVersion: v1
kind: ReplicationController
metadata:
name: kube-registry-v0
namespace: kube-system
labels:
k8s-app: kube-registry
version: v0
spec:
replicas: 1
selector:
k8s-app: kube-registry
version: v0
template:
metadata:
labels:
k8s-app: kube-registry
version: v0
spec:
containers:
- name: registry
image: registry:2
resources:
limits:
cpu: 100m
memory: 100Mi
env:
- name: REGISTRY_HTTP_ADDR
value: :5000
- name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
value: /var/lib/registry
volumeMounts:
- name: image-store
mountPath: /var/lib/registry
ports:
- containerPort: 5000
name: registry
protocol: TCP
volumes:
- name: image-store
emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
name: kube-registry
namespace: kube-system
labels:
k8s-app: kube-registry
kubernetes.io/name: "KubeRegistry"
spec:
selector:
k8s-app: kube-registry
ports:
- name: registry
port: 5000
protocol: TCP
---
apiVersion: v1
kind: Pod
metadata:
name: kube-registry-proxy
namespace: kube-system
spec:
containers:
- name: kube-registry-proxy
image: gcr.io/google_containers/kube-registry-proxy:0.3
resources:
limits:
cpu: 100m
memory: 50Mi
env:
- name: REGISTRY_HOST
value: kube-registry.kube-system.svc.cluster.local
- name: REGISTRY_PORT
value: "5000"
- name: FORWARD_PORT
value: "5000"
ports:
- name: registry
containerPort: 5000
hostPort: 5000

I’ve also decided to keep this local registry ephemeral, which is okay for me at this point, given that I’m dealing with really small images. This can be easily changed to use a persistent volume as described on the registry add-on documentation.

The --insecure-registry localhost:5000 that we’ve used on the minikube start command is relevant here, because it allows for the Docker daemon running on the minikube node to connect to an insecure Docker registry, without using SSL. This simplifies the setup process without any real disadvantage, since we’ll be using this same Docker daemon to build and push our images, so nothing will be sent over external networks.

Execute this command to deploy the registry on your cluster:

kubectl apply -f local-registry.yml

TL;DR

# Setup minikube
brew update && brew install xhyve docker-machine-driver-xhyve kubernetes-cli minikube
sudo chown root:wheel $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
sudo chmod u+s $(brew --prefix)/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve
# Setup minikube
minikube start --vm-driver xhyve --insecure-registry localhost:5000
# Using minikube's Docker daemon from our localhost
eval $(minikube docker-env)
# Running a local private Docker registry
kubectl apply -f local-registry.yml

Conclusion

At this point, you’ll now be able to build and push containers to a local private registry, as well as deploy them to a local Kubernetes cluster. I believe this a great setup to develop applications which will run on a similar environment in production, simplifying the development flow and setup and, at the same time, reducing the possibility of “works-on-my-machine” issues.

If you’d like to discuss this post, please drop me a line using any of my contacts.

Resources

  1. “Configuring the Ultimate Development Environment for Kubernetes”
  2. “Private Docker Registry in Kubernetes”