Using buildah in GitLab CI

Building images in CI is pretty common task and it always brings questions about technologies and workflow. I’d like to show you how we solved these questions by using buildah together with GitLab CI.

Let’s first take a look at our CI infrastucture:

• We are using Gitlab to manage source code and Gitlab CI to run tests, linters and build images.
• Private Gitlab runner with Kubernetes executor is executing our CI. It means that every job/build is running in pod on top of managed Kubernetes cluster.
• Every successful pipeline produces container image with current version or branch build and pushes container into the registry.

There are different ways to build container image, and probably the most common is creating Dockerfile and running docker build -t myimage . However, this command isn’t supposed to be used alone, since it requires docker daemon and it can get more complicated when this build is already running in pod/container as we can’t run another docker daemon.

There is pretty ugly and very offten used solution called dind (Docker-in-Docker). It mounts docker socket into the containter so you are effectively allowing your in-container docker command to manage docker daemon running in the host. You don’t want to do this because:

• All builds on the host are mixed together by sharing single docker daemon
• Build command isn’t respecting any limitations for build pod because build itself isn’t running in build container but on docker daemon outside

Another funny problem with image building is incompatibility of Dockerfile commands in different docker version. You can’t use multi-stage builds with docker version <17.05 and our managed cluster is running 1.13.1. Using similar docker version in host as well in container is required for build to work properly.

There is nice project called buildah which is providing powerful and easy way to build container images, even when you are running in container.

Buildah image

We used to be usingdocker:latest for dind but this tool requires docker daemon on the host and mounted socket.

First step necessary for switching builds to buildah is building buildah buider image. You can use buildah to build this image but using Dockerfile may be easier for the first steps. This image should contain buildah for building images and podman which is used for container registry login and running containers. Following Dockerfile will give you buildah image ready to create container or you can use my tomkukral/buildah.

FROM fedora
RUN dnf install -y \
  buildah \
  podman \
  make \
  && \
 dnf clean all

Kubernetes Gitlab CI runner

Gitlab runner is using native Kubernetes features for starting builds, i.e. build is running as Kubernetes pod and we need to make sure executor is configured properly. Example of config.toml file can be found below. You can see there are two volume mounts:

• /var/run/docker.sock for docker builds because we haven’t switched everything yet
• /var/lib/containers for buildah. Containers (layers) are stored in this directory and buildah requires this to be a regular filesystem. More information can be found in issue #158.

[[runners]]
      name = "Kubernetes Runner"
      url = "https://gitlab.com/ci"
      token = "{{ .gitlab_runner_token }}"
      executor = "kubernetes"
      privileged = true
      [runners.kubernetes]
        namespace = "gitlab"
        
        [[runners.kubernetes.volumes.host_path]]
          name = "docker"
          mount_path = "/var/run/docker.sock"
          read_only = false
        [[runners.kubernetes.volumes.host_path]]
          name = "buildah"
          mount_path = "/var/lib/containers/"
          read_only = false

You may be asking why is mounting directory better than mounting socket? It makes a huge difference because you you can mount any host directory into buildah container and provide project separation. Another big difference is that /var/lib/container is just a storage and no management socket is there.

That’s a pity that build container must be still running in privileged mode but there is ongoing effort to switch to unprivileged.

Building the image

Now we have builder image and Gitlab runner ready to build using buildah and next step is to setup CI pipeline. These steps are pretty straightforward because only few commands must be changed:

• docker build to buildah bud
• docker login to podman login
• docker push to buildah push

Don’t forget to add docker:// to push target if you are pushing to docker container registry because buildah can push to other transports as well.

variables:
  # REGISTRY_USERNAME - secret variable 
  # REGISTRY_PASSWORD - secret variable
  # REGISTRY_SERVER - secret variable
  IMAGE_NAME: ${REGISTRY_SERVER}/proj/${CI_PROJECT_NAME}
container:
  stage: publish
  image: tomkukral/buildah
  dependencies:
    - build
  before_script:
    - podman version
    - buildah version
    - podman login --username "${REGISTRY_USERNAME}" --password "${REGISTRY_PASSWORD}" "${REGISTRY_SERVER}"
  script:
    - buildah bud -t ${IMAGE_NAME}:${CI_COMMIT_SHA} .
    - buildah push ${IMAGE_NAME}:${CI_COMMIT_SHA} docker://${IMAGE_NAME}:${CI_COMMIT_SHA}
  after_script:
    - podman logout "${REGISTRY_SERVER}"

Now let’s try to run pipeline and check your new image.

Resume

Now you are able to build container images using Gitlab CI with Kubernetes executor and it’s up to you to choose right container build tool. There are another interesting features in podman/builah, e.g. image signing. I can write post about it … just let me know if you are interested.

Happy building!