Deploying Docker Registry on Kubernetes
In a previous post we set up a Bare Metal Kubernetes cluster, and we followed it up with another post showing how to deploy a Redis cluster on it. Now we will deploy a Docker Registry on our our cluster.
Why deploy a Docker Registry on a cluster? Sometimes I like to run a cluster with custom images that only run on that particular cluster. In other cases we may want to air-gap the cluster and thus have a docker registry available where each cluster node can get the images instead.
Thankfully, running a Docker Registry on Kubernetes is not too difficult to pull off. To keep this guide simple and focused, we will deploy a Docker Registry with a self-signed certificate. After doing this we can always add user authentication and/or LetsEncrypt certificates later.
Create Key and Certificate
As described in the Docker guide, create a new self signed certificate as follows:
$ mkdir -p certs
$ openssl req \
-newkey rsa:4096 -nodes -sha256 -keyout certs/registry.key \
-addext "subjectAltName = IP:10.211.55.250" \
-x509 -days 3650 -out certs/registry.crt
The important bit here is that we set the
subjectAltName to an IP number our MetalLB load balancer will assign. In our cluster we will assign
10.211.55.250 to the Docker Registry service (see later on).
Store Key and Certificate in Cluster
The nice thing of Kubernetes is that it allows you to store secrets and make them accessible to all nodes safely. Let’s store the created key and certificate on our cluster now:
kubectl create secret tls registry-cert \
The above command will store
registry.key in a TLS secret called
registry-cert in the namespace
test. We will be deploying our Docker Registry in this namespace.
Create Persistent Volume for Docker Registry
It is a good idea to use a persistent volume for the docker images stored in our registry. This way if the Pod in which our registry runs gets rescheduled to another node, we will not lose all the images we stored in it.
Create a file called
registry-pvc.yaml with the following:
The above will create a persistent volume claim on Longhorn for 50G in the
test namespace. Create it now:
$ kubectl create -f registry-pvc.yaml
Deploy Docker Registry on Cluster
Create a deployment descriptor called
registry-deployment.yaml with the following contents:
- name: registry
- containerPort: 5000
- name: REGISTRY_HTTP_TLS_CERTIFICATE
- name: REGISTRY_HTTP_TLS_KEY
- name: registry-certs
- name: registry-data
- name: registry-certs
- name: registry-data
Here we configure one pod to create a container from
registry:2 image using the TLS key and certificate we created earlier. We tell the container that it can find those under the
/certs directory. How do they get there? For this we use volume mounts. We define a read-only
volumeMount with the name
registry-certs to be mounted under
/certs in the container. This
registry-certs volumeMount references the secret volume
registry-cert in the
We also mount the persistent volume claim
registry-data-pvc under directory
/var/lib/registry in the container.
Go ahead and deploy the registry on our cluster as follows:
$ kubectl create -f registry-deployment.yaml
You should now be able to see the registry pod running on the cluster in the namespace
test. You can check this as follows:
$ kubectl -n test get all
Create Registry Service
Now that our registry pod is up and running, we need to define a service for it so that our pods can access it. Remember the IP number we used when generating the key and certificate? We will use it now. Create a service descriptor called
registry-service.yaml with the following content:
- name: registry-tcp
This will instruct the load balancer to expose port
5000 on IP
10.211.55.250. Go ahead and create the service:
$ kubectl create -f registry-service.yaml
Great! now you should be able to access the registry. Let’s test it out:
$ curl --cacert certs/registry.crt \
Load Certificate in ca-certificates
Since we are dealing with a self signed certificate, we need to instruct the host operating system to trust this certificate. Normally you would do that with a Certificate Authority (CA) certificate, but since this is self signed, the certificate is both the certificate for the registry service, as well as its own CA. So let’s load the
registry.cert as a new CA on all the cluster nodes. We need to do this because the k3s service on all the cluster nodes will need to be able to pull from this repository, and it must trust this certificate. We can easily do this using Ansible:
Note that we are using a
hosts file here that we defined in our previous blog post Bare Metal Kubernetes.
With the certificate now loaded on each cluster node, we will need to restart the k3s services so that it picks up the change. First restart the worker nodes as follows:
ansible -i hosts workers -b -K -m shell \
-a "systemctl restart k3s-agent"
Next restart the control (a.k.a master) nodes:
ansible -i hosts control -b -K -m shell \
-a "systemctl restart k3s"
Test it Out!
With the workers and controllers restarted, let’s test out our new registry! Create a new image tagged with our new registry as follows:
$ docker pull alpine
$ docker tag alpine 10.211.55.250:5000/alpine:latest
$ docker push 10.211.55.250:5000/alpine:latest
If all went well the retagged alpine image should now be in our local docker registry on the cluster. Let’s now try to deploy a pod with this new image. Create a file called
alpine-test-pod.yaml with the following content:
- name: alpine
command: ["sleep", "60s"]
$ kubectl create -f alpine-test-pod.yaml
Check if the deployment went well:
$ kubectl -n test describe pod alpine-test
If all went well the pod should have been created and started up successfully! Now you can store images ‘locally’ on your cluster!