Dockerized GitLab CI: Docker Executor as a Docker in Docker (docker:dind)

Source: https://unsplash.com/photos/a-large-ship-in-the-water-2JNNpq4nGls

Continuing the series of Dockerized GitLab, in this post i’ll show you how to register Docker executor as a GitLab runner with your GitLab server for building, testing and deploying your dockerized projects.

But instead of mounting and using docker socket var/run/docker.sock from our host docker engine, we’ll use Docker in Docker aka dind for building docker containers inside our project pipelines.

In our last post, we registered docker executor as a GitLab runner using docker socket /var/run/docker.sock from our host machine’s docker engine and we have seen that all of the containers including job executor container and containers created inside pipeline’s jobs are actually created on our host docker engine. This way of managing docker executor has some potential issues which we discussed as well.

In this post, we are going to register docker executor inside our GitLab runner service using another way called Docker in Docker aka dind for building, testing and deploying our dockerized projects.

Docker in Docker

Docker in Docker simply means that you are running a docker engine (Daemon) inside a docker container instead of running docker engine on the host machine. Docker in Docker could be useful if you want to manage containers in its own dedicated docker engine, isolated from from the host docker engine and where giving access to the host docker engine could have potential issues.

Update Docker Compose File

Before we start registering a docker-in-docker executor, we need some minor updates to our docker-compose.yml

version: '3.8'
services:

  gitlab-server:
    image: 'gitlab/gitlab-ce:latest'
    container_name: gitlab-server
    environment:
      GITLAB_ROOT_EMAIL: "admin@buildwithlal.com"
      GITLAB_ROOT_PASSWORD: "Abcd@0123456789"
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://localhost:8000'
        nginx['listen_port'] = 8000
    ports:
      - '8000:8000'
    volumes:
      - ./gitlab/config:/etc/gitlab
      - ./gitlab/data:/var/opt/gitlab

    # new changes for adding gitlab server to custom network
    networks:
      - gitlab-in-docker

  gitlab-runner:
    image: gitlab/gitlab-runner:alpine
    container_name: gitlab-runner
    network_mode: 'host'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

# new changes for adding custom docker network
networks:
  gitlab-in-docker:
    name: gitlab-in-docker
    driver: bridge

The update we have made here is that we have put GitLab server container on a custom docker network called gitlab-in-docker with driver bridge. The reason we are doing this is, that docker executor with docker in docker doesn’t work with docker host network as docker in docker is using container links and host network cannot be used with container links.

at least this was what happening to me when trying network host with dind service

So to clone our project repository inside our pipeline job, job’s container will access our GitLab server using its container name i.e http://gitlab-server:8000 . GitLab server cannot be accessed by using http://localhost:8000 from within the job’s container because we are not using the host network.

For this reason, we need to put both GitLab server container and Job executor container on a custom network called gitlab-in-docker so they can access each other by its container names instead of localhost.

Runner needs access to Server container when registering a runner executor. It will get access by localhost as Server is accessible by http://localhost:8000 and Runner container is on the host network so it has access to localhost.

Job needs access to Server when cloning the repository. It will access it by using Server container name gitlab-server as both job container and Server container are on the same network i.e gitlab-in-docker network.

Getting bored from neworking? lets jump into the runner registeration part

Register Docker in Docker Executor

Go to Repository → Settings → CI/CD → Runners

Click on New project runner button

Add Tags so a job could be picked by the runner using its matching executor by tags. Give some description and click on Create runner button. It will redirect you to the below page where you can see the Runner registeration command in Step 1

Copy runner registeration command from Step 1 and login to your GitLab runner container using below docker compose command.

docker compose exec -it gitlab-runner /bin/bash

Once logged in to the Runner container, paste the above command from Step 1 inside Runner container and add these additional arguments.

Make sure to update your --token value from your Step 1 in the below register command

gitlab-runner register  --url http://localhost:8000 \
                        --token glrt-qL_FTjkAGqy7SHcYYStx \
                        --executor docker \
                        --name "Docker in Docker Runner" \
                        --docker-image "docker:27.2.0" \
                        --docker-privileged \
                        --docker-volumes "/certs/client" \
                        --docker-network-mode "gitlab-in-docker" \
                        --clone-url "http://gitlab-server:8000"

We are setting executor to docker the same way we did in our previous post. Then giving it a meaningful name and setting the default docker image. We are using docker image docker:27.2.0 here so we could have docker CLI available for running docker commands inside our pipeline job.

To start the build and service (docker in docker) containers, it uses the privileged mode. If you want to use Docker-in-Docker, you must always use privileged = true in your Docker containers.

We are also mounting /certs/client for the service and job containers, which is needed for the Docker client to use the SSL certificates for connecting in TLS mode.

And lastly, we are setting the network to gitlab-in-docker which is our custom network and clone URL to http://gitlab-server:8000 which we discussed earlier.

Once your command is ready, hit Enter and keep hitting Enter while asking for the inputs unless you want to update any value while prompting.

Once this command runs successfully, you should have a success message on your runner page.

Click on the View runners button and you should see all of your runners including your newly registered docker in docker runner.

Now we have our job executor ready. Lets update our .gitlab-ci.yml by going to Repository → Build → Pipeline Editor → Configure Pipeline

And then paste the following snippet.

build with docker in docker:
  stage: build
  image: docker:27.2.0
  services:
    - docker:27.2.0-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"

  tags:
    - docker-in-docker
  script:
    - docker ps
    - docker run -d --rm --name nested-container1-in-pipelinejob alpine sleep 20
    - docker run -d --rm --name nested-container2-in-pipelinejob alpine sleep 20
    - docker ps
    - sleep 20

services: docker:27.2.0-dind is actually the Docker in Docker provided by Docker officialy. When we define a service for a job, the job will first wait for that service to come up and once that service is ready then the actual job container will start and will process the job’s script part.

variables will be passed to the job container as env variables for accessing the docker engine from within the job container when running docker CLI commands. DOCKER_TLS_CERTDIR will set the SSL certificates directory while DOCKER_HOST will set the default connectivity to the docker engine via SSL port i.e 2376.

If you want to connect without TLS, set the variables as below.

variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""

Once the docker in docker container/service is ready, the executor will start the job container by pulling the docker:27.2.0 image and then run the script commands.

Meanwhile, you can check on your host docker engine that you have 2 new containers. One for the service (dind) and one for the job execution.

Once the job reaches the script part and start running the docker containers by running these 2 docker commands

- docker run -d --rm --name nested-container1-in-pipelinejob alpine sleep 20
- docker run -d --rm --name nested-container2-in-pipelinejob alpine sleep 20

This will run 2 new containers inside docker in docker service. You can confirm these 2 new containers by login into the service container

docker exec -it 0e7a68f4dfbf sh

0e7a68f4dfbf is the docker in docker container ID

And then run this command to get all containers running inside dind service.

docker --tlscacert=/certs/client/ca.pem \
       --tlscert=/certs/client/cert.pem \
       --tlskey=/certs/client/key.pem \
       --tlsverify=1 \
       ps

Which should look like this.

So here you can see that we have separated our parent containers i.e gitlab runner, job executor and its child containers from each other by separating from the host docker engine.

Known issues with Docker in Docker

Docker-in-Docker is the recommended configuration, but there are some known issues highlighted by GitLab documentation here.

• The docker-compose command: This command is not available in this configuration by default.
• Cache: Each job runs in a new environment. Because every build gets its own instance of the Docker engine through its services, concurrent jobs do not cause conflicts. However, jobs can be slower because there’s no caching of layers.

Alternatives to Docker Executor

To build Docker images without enabling privileged mode on the runner or mounting the host docker socket, GitLab has these 2 alternative which you can explore here.

• Kaniko
• Buildah

If you want to avoid touching host docker engine completely but still want to build docker images in a CI job, Kaniko can help you which i have explained in detail with a demo here in this post.

Dockerized GitLab CI: Build Docker Images using Google Kaniko

Source Code

gitlab-in-docker/05. gitlab-runner-with-docker-executor-dind at master ·…