Dockerized GitLab CI: Build Docker Images using Google Kaniko

Source: https://unsplash.com/photos/a-forklift-is-moving-a-large-container-behind-a-fence-uXc4PvF53fE

In our last 2 posts, we used Docker socket binding from host docker engine and Docker-in-Docker service to build docker images inside a GitLab pipeline.

Former way is giving access to host docker engine through its docker.sock file while later is using docker in privileged mode.

Job executor using /var/run/docker.sock from host docker engine

Job executor using Docker-in-Dockerservice

Both ways of getting access to docker engine has some known issues as highlighted by GitLab docs here and here.

You can notice that in both ways of executing jobs, GitLab runner create child containers either directly on the host docker engine or as child containers inside a service container (dind).

Kaniko comes into rescue here by building docker images inside a GitLab pipeline without getting access to host Docker either by docker.sock or in privileged mode.

What is Kaniko

kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster.

kaniko solves two problems with using the Docker-in-Docker build method:

Docker-in-Docker requires privileged mode to function, which is a significant security concern.
Docker-in-Docker generally incurs a performance penalty and can be quite slow.

To use kaniko with GitLab, a runner with one of the following executors is required:

• Kubernetes
• Docker
• Docker Machine

Using Kaniko in GitLab Pipeline Jobs

Since we have setup our GitLab Server and Runner using docker compose, make sure your docker-compose.yml looks like this.

version: '3.8'
services:

  gitlab-server:
    image: 'gitlab/gitlab-ce:latest'
    container_name: gitlab-server
    environment:
      GITLAB_ROOT_EMAIL: "admin@buildwithlal.com"
      GITLAB_ROOT_PASSWORD: "Abcd@0123456789"
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://localhost:8000'
        nginx['listen_port'] = 8000
    ports:
      - '8000:8000'
    volumes:
      - ./gitlab/config:/etc/gitlab
      - ./gitlab/data:/var/opt/gitlab

  gitlab-runner:
    image: gitlab/gitlab-runner:alpine
    container_name: gitlab-runner
    network_mode: 'host'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Then run

docker compose up

Once your GitLab and Runner is ready (healthy) after you confirm it using this command

docker ps

Go to your GitLab Dashboard here at http://localhost:8000

Register New Runner For Kaniko

Create a new Repository

Go to Repository → Settings → CI/CD → Runners

Click on New runner button and fill out the runner details.

Click on Create runner button and it will redirect you to this page

Login to your GitLab Runner container using this docker compose command

docker compose exec -it gitlab-runner /bin/bash

And paste the gitlab-runner register …command from the above Step 1

gitlab-runner register \
              --url http://localhost:8000  \
              --token glrt-Y9BCAyZZyFhrrkezJbC8 \
              --executor docker \
              --docker-image "gcr.io/kaniko-project/executor:v1.23.2-debug" \
              --docker-network-mode "host"

Make sure to add --docker-network-mode “host” so job executor container can access the repository in GitLab server using http://localhost:8000

Since Kaniko requires the GitLab runner executor to be one of these:

• Kubernetes
• Docker
• Docker Machine

We are passing --executor as docker and setting the default docker image to the Kaniko docker image as well

gcr.io/kaniko-project/executor:v1.23.2-debug

Once this is processed. Go to Repository → Settings → CI/CD → Runners

and you should have a new runner registered for Kaniko

Now we are good to start adding project pipelines.

Add Pipeline to .gitlab-ci.yml

Before adding our pipeline, lets add a quick Dockerfile to our repository by adding this snippet so we can test building from Dockerfile using Kaniko.

Dockerfile in repository root directory

FROM python:3.10-alpine

RUN python --version

Then Go to Repository → Build → Pipeline Editor → Configure Pipeline

and add the following snippet to .gitlab-ci.yml file

build with kaniko:
  stage: build
  tags:
    - kaniko
  image:
    name: gcr.io/kaniko-project/executor:v1.23.2-debug
    entrypoint: [""]
  script:

    # conventional project with Dockerfile inside root dir
    - /kaniko/executor --no-push

--no-push means that Kaniko will only build the docker image from the Dockerfile and will not push to any container registry which is the default behavior using Kaniko.

Once you commit the above changes, your pipeline should run immediately and it should look like this after successful execution.

docker.sock vs Docker-in-Docker vs Kaniko

While the pipeline is running, you can check on the host docker engine that there is only an executor container running. But any container created by the job itself should not be created on the host docker engine which is what we are trying to avoid using Kaniko.

Using docker socket binding, we have seen that all of the containers were created on host docker engine including executor container and job’s created containers.

Using Docker-in-Docker, we have seen that executor container and service (dind) container were created on host docker engine but any child container created by a job was a child container of the service (container) which required privileged mode

More on Kaniko CLI

If you have your Dockerfile in some nested directory that the project root directory, you can refer to the Dockerfile like thi

# Dockerfile is in some other directory than root dir
- /kaniko/executor --no-push --dockerfile "${CI_PROJECT_DIR}/src/Dockerfile"

Or if you Dockerfile is named something else than conventional name i.e Dockerfile

# Dockerfile is in some other directory than root dir
- /kaniko/executor --no-push --dockerfile "${CI_PROJECT_DIR}/mydockerfile"

If you want to build your docker image by passing a different context than the root directory

- /kaniko/executor
      --context "${CI_PROJECT_DIR}/src"
      --dockerfile "${CI_PROJECT_DIR}/src/Dockerfile"

If you want to push your docker image to some container registry

- /kaniko/executor
      --context "${CI_PROJECT_DIR}/src"
      --dockerfile "${CI_PROJECT_DIR}/src/Dockerfile"
      --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"

Slow uploads when pushing large images

When you push large images with kaniko, you might experience long delays.

This is typically a result of a performance issue with Kaniko and HTTP/2. The current workaround is to use HTTP/1.1 when pushing with kaniko.

To use HTTP/1.1, set the GODEBUG environment variable to "http2client=0" in your gitlab-ci.yml file

variables:
    GODEBUG: "http2client=0"

More details can be found on Kaniko here

Now since we can build docker images in our GitLab CI, we need some registry to store our docker images so we can deploy it to some target servers during the deployment stage.

Setup GitLab Container Registry — GitLab Managed

Source Code

gitlab-in-docker/08. build-docker-images-using-kaniko at master · BuildWithLal/gitlab-in-docker