Reducing build time on OpenShift using Kaniko

Not only do you need a working CI/deployment pipeline to have a quick and smooth way to get your code into production, you also need to make sure building images does not slow you down.

OpenShift comes with support for building images using OpenShift Builds, which might work well for you. But I have experienced slow builds (both start-up time and actual build time), unpredictable behaviour and it requires you to install MiniShift if you want build your image locally.

So what do you do? You look at options such as Buildah, orca or Bazel, but they all have drawbacks, like running with root, not being further developed or require complex configuration. Note that Buildah will be supported in OpenShift in the future — whenever that is.

But for now, having something that would make building container images easy and fast would be great! Hint: Try Kaniko.

Even if Kaniko is not optimal in your case, it may be worth trying out, as getting started with it is pretty straight forward and does not require a lot of time. Plus, the documentation is good and the usage is widespread on Kubernetes. However, OpenShift usage is not very widespead yet, although it’s very similar to running it on Kubernetes.

Getting your hands dirty

First, you need to start a build Pod either from Jenkins or somewhere else.

The Pod definition should look something like this:

apiVersion: v1
kind: Pod
metadata:
  generateName: kaniko-
  namespace: foobar
spec:
  containers:
    - args:
        - '--dockerfile=/opt/example/Dockerfile'
        - '--context=dir://opt/example'
        - '--destination=registry.example.org/foobar/example'
        - '--skip-tls-verify'
      image: 'gcr.io/kaniko-project/executor:latest'
      name: kaniko
      volumeMounts:
        - mountPath: /opt/example
          name: example-app-volume
        - mountPath: /kaniko/.docker
          name: docker-config-volume
        - mountPath: /root/.docker
          name: docker-config-volume
  restartPolicy: Never
  securityContext:
    runAsUser: 0
  serviceAccount: example
  volumes:
    - name: example-app-volume
      persistentVolumeClaim:
        claimName: example-app
    - name: docker-config-volume
      persistentVolumeClaim:
        claimName: docker-config

First you have to specify the dockerfile, build context (e.g. the directory you want to build from) and the destination to push the image. In this example the first two are located using a PersistentVolumeClaim.

The base Dockerfile should contain as few layers as possible because the layers will be extracted in turn, and more layers means longer build time!

FROM mhart/alpine-node
COPY hello-example.js hello-example.js
CMD ["node", "hello-example.js"]

Another PersistentVolumeClaim is used to hold the Docker config, here called docker-config. <auth> should be replaced with a base64 encoded string of username:password, or if the registry is an OpenShift registry, a base64 encoded service account token with access to the specified registry project.

{
  "auths": {
    "registry.example.org": {
      "auth": "<auth>"
    }
  }
}

What is important to note, is that you have to specify a securityContext with the value runAsUser: 0. Also make sure you have a ServiceAccount with SCC (Security Context Constraint) anyuid.

$ oc adm policy add-user-to-scc anyuid <service-account>

Build and publish your image

Now that you set everything up, you are ready to test it out. Start the Kaniko pod as follows.

oc create -f kaniko-pod.yaml

Check the output

INFO[0000] Downloading base image mhart/alpine-node
INFO[0004] Executing 0 build triggers                    
INFO[0004] Extracting layer 0                         
INFO[0015] Extracting layer 1                            
INFO[0015] Extracting layer 2                            
INFO[0017] Extracting layer 3                                                      
INFO[0087] Taking snapshot of full filesystem...  
INFO[0098] COPY hello-example.js .
INFO[0098] Taking snapshot of files...                  
2018/10/20 19:40:46 existing blob: sha256:77187f02cbeff1e8de9cdd8e850f300e7267ddf19991dcbc588a498c14df3ff0
2018/10/20 19:40:46 existing blob: sha256:8ace823f4875e44b3379a651fb8663b963287e09cda6127ea78d3e64433a0b3b
2018/10/20 19:40:46 existing blob: sha256:0eeb656bc1e64b6c3ba63f2fa9450feaef3c60159d48eb2171ad1f25f5e655d9
2018/10/20 19:40:56 registry.example.org/foobar/example:latest: digest: sha256:9d96fe3c08c3707b0ed52d31f54a9f917d2320751bc803a62e506e9b544e9a7b size: 1575

If successful, the pod should be marked as completed ✓.

Congratulations! Your image containing your app should be published to the registry, hopefully much faster than using OpenShift Builds.