Your very own private Docker registry for Kubernetes cluster on Azure (ACR)

One of the important parts that was not covered in previous blog post about Kubernetes cluster on Azure is how privately deploy your Docker payload to the cluster. Well the answer is simple. You need Your Own Private Docker Registry™ somewhere near of your cluster.

There are few options that you can go with:

  1. You can use private repositories on hub.docker.com, this is probably most easy way. You register, you are doing `docker login` to your new registry and then push.
  2. You also can instantiate docker/registry image inside your cluster. I do not know if this is good idea or not.
  3. Kubernetes has registry addin that you might want to try. Again I never used it, so no comments here.
  4. And finally, the option that I would like to show in this article is just to use Azure Container Registry (ACR).

Docker Private Registry using ACR

Azure provides Container Registry as a Service called Azure Container Registry (surprise!). And apparently if you more or less comfortable with azure cli or portal, it is really super simple to use.

Using new Azure CLI, issue something like this (I assume you already did `az login`):

az acr create -n myveryownregistry \
-g my-very-own-k8s-cluster \
-l westus # "West Europe" is not yet supported

In my case, I have had cluster in “West Europe”, but for the time being ACR is not supported here, so it is required to specify location where ACR is available — “West US”.

After command is done, you will see a little bit of meta information and some scary red text.

ACR registry created

I decided to be brutal and ignore suggestions. So I just enabled myslef as admin user to have access to it:

az acr update -n myveryownregistry --admin-enabled true

Hurray! We have our shinny new registry! And it already very-very own! Next step is actually use it for good.


First use of your private registry

First thing you need to do is actually let docker to know what registry you use. In most cases command line will look something like this:

docker login yoursuperregistry.somewhere.com -u username -p password

But we can do better.

New Azure CLI provides really awesome `--query` parameter. In simple words, any structured output of the tool could be transformed with JMESPath expressions. Now lets put this to work for us.

Azure CLI provides us with `az acr credential show` that will return your generated registry credentials in structured way:

{                                                 
"password": "=oo1+PRLZDxwHyJLFzm8R7GTkB2fE/34",
"username": "myveryownregistry"
}

Using credentials,` — query` and ` — output tsv` parameters, we can construct `docker login` command automatically and then pipe to any command interpreter(powershell — iex, bash, sh, cmd -c):

az acr credential show -n myveryownregistry \
--query "join(' ', ['docker login myveryownregistry-on.azurecr.io', '-u', username, '-p', password])" \
--output tsv \
| sh

I have gist with this stuff for you. After this command executed, your local docker will point to your registry, you are logged in and ready to push your valuable stuff!

Docker logged into your new private registry!

Let’s push something to our private registry!

docker pull hello-world
docker tag hello-world:latest myveryownregistry-on.azurecr.io/hello-world:latest
# Push!
docker push myveryownregistry-on.azurecr.io/hello-world:latest

Well! Hello-world is already in our registry. But we need something useful there. I will use my previous example — simple application with two microservices packed as Docker containers and third Docker container with top level router to pass traffic to microservices. Simple static router based on nginx. The only thing it is doing, is configuring alpine nginx Docker container with two upstreams and routes traffic to them doing some rewriting. If you would like to try it, just clone git repository, cd to working directory and run command bellow (alternatively you can find Makefile with all this stuff already baked):

# Build Docker image from router folder
docker build --rm=false -t chaliy/play-azure-kubernetes-router:latest ./router
# Retag image to your repo
docker tag chaliy/play-azure-kubernetes-router:latest $myveryownregistry-on.azurecr.io/chaliy/play-azure-kubernetes-router:latest
# Push image to the registry
docker push myveryownregistry-on.azurecr.io/chaliy/play-azure-kubernetes-router:latest

Okay. If all this succeeded we have our router in place and ready to deploy it to the cluster for real use.


Use private registry from your Kubernetes cluster

To ensure that Kubernetes can access to the private registry, we should pass Docker credentials securely to the nodes. Kubernetes has special mechanism to pass sensitive data to the nodes — called Secrets and even more special mechanism for storing Docker credentials. In essence you need to use `kubectl create secret docker-registry` command. Of course we will use better way :).

az acr credential show -n myveryownregistry \
--query "join(' ', ['kubectl create secret docker-registry registrykey --docker-server myveryownregistry-on.azurecr.io', '--docker-username', username, '--docker-password', password, '--docker-email example@example.com'])" \
--output tsv\
| sh

The only thing to note, is that Docker email will be or already obsolete and I honestly I do not know where it used right now. So in my scripts I just hardcode it. Tadam!

Docker credentials secret created

Next step is actually define application that will use our private image. Lets create router.yml. It defines Kubernetes Service to be a proxy behind load balanced application instances and Deployment that actually specify image we created earlier.

apiVersion: v1
kind: Service
metadata:
name: router
spec:
ports:
- port: 80
selector:
app: router
type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: router
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: router
spec:
containers:
- image: myveryownregistry-on.azurecr.io/chaliy/play-azure-kubernetes-router:latest
name: play-azure-kubernetes-router
ports:
- containerPort: 80
imagePullSecrets:
- name: registrykey

Important thing to note here is imagePullSecrets, it actually define key of Secrete with Docker credentials. So when engine will try to pull image it will use right credentials of your private registry.

Last step is actually create and run demo application. Use kubectl create command.

kubectl create -f ./ner-uk.yml,./nlp-uk.yml,./router.yml

Technically that is it. Now you can expose router outside of the cluster and test if it runs successfully.

kubectl get services/router
curl http://EXTERNAL-IP/ner/mitie/uk
Check if service is actually working

Source code of example could be found on Github — https://github.com/chaliy/play-azure-kubernetes, be careful there are small changes compared to the article. If you have any questions or suggestions, please welcome!

Conclusion

I hope this gave you good source to start using ACR for Kubernetes. Most important notes are:

  1. Use ACR when you need private registry in Azure. It is quite simple;
  2. Use ` — query` of Azure CLI for automation. It is really powerful. Do not forget ` — output tsv` otherwise output will not be consumable to next command in pipeline;
  3. Use Kubernetes Secrets for Docker credentials.

The End. See you next time!