Jenkins Docker Plugin Blues … And a Solution

Intro

Jenkins is one of the best tools for Continuous Integration (CI) due to it’s maturity, huge number of plugins, support for distributed builds, etc.

One of the key features of Jenkins is its support for Docker containers (via the Docker plugin), as it makes it easy to create CI pipelines that rely on specific tools without having to install the tools in the host itself.

In fact Jenkins itself is distributed as a Docker container, which is very convenient because you avoid having to install Jenkins and its dependencies directly on the host machine.

Turns out however that there are several problems that arise when running Jenkins in a container and using the Jenkins Docker plugin in your pipelines.

This article describes these problems and shows how they can be addressed in a simple way by using a Nestybox system container.

If you are planning on using Jenkins with Docker for your CI, this article could save you hours of debugging and help you setup things in a few minutes.

Problems with Jenkins inside a Container

When installing Jenkins on a host, the easiest way to do it is to run a Docker container that contains the Jenkins server in it. This saves you the trouble of having to install Jenkins and all it’s dependencies on the host machine directly.

Jenkins provides a Docker image called jenkins/jenkins that has the Jenkins server in it.

However, if your Jenkins pipeline uses Docker as an agent (which is common and convenient), the jenkins/jenkins image won't work.

For example, if your pipeline has something like this:

pipeline {
    agent { docker { image 'golang' } }

    stages {
        stage('build') {
            steps {
                sh 'go build main.go'
            }
        }
    }
}

then Jenkins will request Docker to run the golang image. But since Docker is not present in the jenkins/jenkins container, the pipeline will fail.

To solve this, you must first create a custom Docker image that has Jenkins and the Docker CLI in it. Here is sample Dockerfile for such an image:

FROM jenkins/jenkins

# Docker install
USER root
RUN apt-get update && apt-get install -y \
       apt-transport-https \
       ca-certificates \
       curl \
       gnupg2 \
       software-properties-common
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
RUN apt-key fingerprint 0EBFCD88
RUN add-apt-repository \
       "deb [arch=amd64] https://download.docker.com/linux/debian \
       $(lsb_release -cs) \
       stable"
RUN apt-get update && apt-get install -y docker-ce-cli

USER jenkins

You can build it with:

$ docker build -t jenkins-docker .

Now, when running this custom jenkins-docker image you must now mount the host machine’s Docker socket into the Jenkins master container.

That’s because this image only has Jenkins and the Docker CLI in it, but not the Docker daemon. Thus, you must connect the Docker CLI in the Jenkins container to the Docker daemon on the host by mounting the daemon’s socket into the container:

$ docker run --rm -d -v /var/run/docker.sock:/var/run/docker.sock -P jenkins-docker

But even this is not sufficient. Turns out that the jenkins/jenkins image (which the Dockerfile above uses as its base image) defaults to a non-root user inside the container. This user will not have permissions to access the Docker daemon's socket mounted into the container. Thus, when you run your pipeline with Jenkins, you'll get this error:

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock ...

The solution is to run the image above with the following command:

$ docker run --rm -d --group-add $(stat -c '%g' /var/run/docker.sock) -v /var/run/docker.sock:/var/run/docker.sock -P jenkins-docker

With this command, the Jenkins master will run inside the container and will be able to connect to the Docker daemon on the host to create the agent containers (such as the golang container for the pipeline example above).

The following figure shows the configuration.

While this configuration works, notice that all containers spawned by Jenkins are created at the host level. This is fine in many cases, but causes headaches when your pipeline has Docker commands in it as described in the next section.

Problems with Jenkins Docker Agents

Sometimes it’s useful for a pipeline stage to invoke Docker commands. For example, the pipeline definition below configures a Docker agent with the docker image and runs a docker build step in it.

pipeline {
    agent { docker { image 'docker' } }

    stages {
        stage('build') {
            steps {
                sh 'docker build -t my-container .'
            }
        }
    }
}

Ideally this should run without problems. But unfortunately that’s not the case. Running this pipeline results in a “permission denied” error reported by the Jenkins docker agent:

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock ...

The problem here is that when Jenkins uses Docker to create the agent container, it mounts the host’s Docker socket on the agent container such that the agent can issue Docker commands (such as the docker build command in the pipeline shown above).

The following figure shows the setup:

Mounting the host’s Docker socket into the agent is fine, except that Jenkins also sets up the default user within the agent container to be a non-root user. This non-root user does not have permission to access the mounted Docker socket.

As a result, when the pipeline step with docker build runs within the agent container, we get the permission error shown above.

To solve this, you are now forced to create (yet another) custom image, this time for the Docker agent, in which the non-root user is added to the Docker socket’s group.

How painful … all you wanted to do was run a simple Jenkins pipeline with Docker!

Yet More Problems

Even if you create a custom image for the Docker agent to get around the problem described in the prior section, you may still face more problems.

For example, in the following pipeline configuration, assume image my-docker-agent is a custom Docker agent image that solves the problem in the prior section.

The pipeline has a single step that requires running a Docker container named my-container within the Docker agent container.

pipeline {
    agent { docker { image 'my-docker-agent' } }

    stages {
        stage('build') {
            steps {
                sh 'docker run --name my-container my-container'
            }
        }
    }
}

This looks perfectly fine, but the pipeline step docker run --name my-container my-container will fail if the host already has a running container named my-container.

This failure occurs because the Docker daemon in the host is the sole entity creating containers. The Jenkins agent container is simply connected to the Docker daemon on the host via the host’s Docker socket. Thus my-container is created by the Docker daemon on the host, and that creation will only succeed if there is no other container with the same name.

It’s easy to solve the naming problem (i.e., don’t assign names the containers), but the problem is not specific to container names only. It occurs when naming any other Docker resources too, such as networks or volumes.

And related problems occur if your pipeline steps need to run Docker containers that use volume mounts or port mappings, because those will need to be with respect to the host, even though the entity launching the container is the Jenkins instance running within the jenkins master container.

All of this adds to the pain points previously discussed.

Recap

Let’s quickly recap the problems that we’ve faced up to now in our quest to simply run Jenkins in a container and use the Jenkins Docker plugin:

1) We had to create a custom Docker image for the Jenkins master. The official jenkins/jenkins image did not suffice.

2) We had to run that Jenkins master container with a volume mount of the host’s Docker daemon, and with the --group-add flag that sets correct permission for the volume mount. This essentially makes the jenkins master container root in your host.

3) If we want to run Docker commands in our Jenkins pipeline, we had to create (yet another) custom Docker image, this time for the Jenkins agent. Image docker did not suffice.

4) In the pipeline steps, Docker build or run commands are issued by the Jenkins instance inside the jenkins master container, are carried out by the jenkins agent container, but result in containers created at the host level (by the host’s Docker daemon). This causes all sorts of context related issues that are *hard* to debug.

These problems will likely take you several hours (or days) to figure out, as they took me while researching Jenkins interactions with Docker, taking precious time away from your real objective of running your Jenkins pipeline while leveraging the power of Docker containers.

Simple solution with Docker + Nestybox

Nestybox has developed a container runtime (runc) that enables Docker to deploy containers that run system-level software in them (such as systemd and Docker), securely, without resorting to privileged containers.

These containers are called “system containers”.

It’s possible to completely avoid problems (2), (3), and (4) described in the prior section by running Jenkins and Docker inside a system container.

Problem (1) is inevitable because we need a container image that has Jenkins and Docker within it whenever we want to run Jenkins in a container and configure it with its Docker plugin.

The solution consists of using a system container image that acts as a “Jenkins sandbox” and has the following software in it:

• Jenkins master
• Docker (not just the CLI, the Docker daemon too!)
• Supervisord (the system container’s process manager)

We have such an image in the Nestybox Docker Hub, together with it’s Dockerfile. It’s called nestybox/jenkins-syscont and it's here.

The Dockerfile and supervisord configuration for that image can be found in the Nestybox GitHub site here.

Here is how things would work.

First, you need to install Nestybox’s “Sysbox” container runtime on your host (it’s free and installs in seconds as shown in the Nestybox website).

Then, you launch the system container with this simple Docker command:

$ docker run --rm -d --runtime=sysbox-runc -P nestybox/jenkins-syscont

This command tells Docker to use the Sysbox container runtime to launch the system container.

When the system container starts, supervisord will start Jenkins and the Docker daemon inside of the system container. You can think of the system container as a container-based virtual host: in many ways similar to a VM but faster, easier to use, and much more efficient.

Notice how the simpler command avoids problem (2) above (i.e., no volume mount or permission to the host’s Docker socket are required). In fact the Docker daemon inside the system container is completely isolated from the Docker daemon on the host.

Notice also that even though the system container runs the Docker daemon within it, it’s not an unsecure privileged container. That’s a key feature of Nestybox system containers. It protects your host.

The following figure shows the setup.

Once the system container is running, simply access Jenkins via its web based UI and configure it to run your pipeline. You can obtain access the Jenkins UI by directing your browser to your host’s IP and port associated with the system container.

For example, on my host, Jenkins is listening on port 32789 as shown below:

$ docker port zealous_taussig
8080/tcp -> 0.0.0.0:32789
50000/tcp -> 0.0.0.0:32788

Thus, if I direct my browser to that port, I’ll see Jenkins there:

I can then access Jenkins by pulling the credentials from the system container:

$ docker exec -it zealous_taussig cat /var/jenkins_home/secrets/initialAdminPassword
10762f55c17d42bfb7a1e98fb2ee278c

You can then configure and run your pipeline as usual. For example, the pipeline configuration for the golang image we showed earlier (repeated below for convenience) works perfectly fine.

pipeline {
    agent { docker { image 'golang' } }

    stages {
        stage('build') {
            steps {
                sh 'go build main.go'
            }
        }
    }
}

In this case the docker container with the “golang” image would be created by the Docker daemon running inside the system container.

Now, what if your pipeline steps need to run Docker commands (e.g., to build or run a container from within your pipeline)? That’s easy now: simply specify the pipeline’s agent configuration as follows:

pipeline {
    agent {
        docker {
            image 'docker'
            args '-v /var/run/docker.sock:/var/run/docker.sock'
        }
    }

    stages {
        sh 'docker run hello-world'
    }
}

As shown, the Docker agent uses the docker image (the official Docker-Hub image containing the Docker CLI in it). No custom image needed anymore. Problem (3) above is gone.

Also, the Docker agent’s docker image runs with a volume mount of the Docker socket ( args '-v /var/run/docker.sock:/var/run/docker.sock'). This is required since the Docker agent's docker image only has the Docker CLI in it, not the daemon.

But here is the key: that’s a volume mount to the Docker daemon running inside the system container, not the Docker daemon on the host. And the Docker daemon inside the system container runs at the same level as Jenkins inside the system container. This solves all the context related issues in problem (4) above.

Why does this work?

Fundamentally, this solution works because the system container image is acting as a virtual host (or sandbox environment) inside of which Jenkins can create containers by leveraging a dedicated Docker daemon, in complete isolation from the underlying host.

The permission issues are gone, because all work inside the system container occurs under user “root” (which is mapped to a non-root user on the host by the Nestybox system container in order to enhance security).

The context issues are gone, because within the system container, Jenkins and Docker and running at the same level.

Since the pipeline is running inside the system container, you can be sure that you’ll never collide with Docker resources on the host itself, which means you can use the host for other tasks too, without ever worrying about colliding with Jenkins and it’s container operations.

You can even run Docker Compose inside your pipeline steps, by simply installing the Docker Compose binaries inside the nestybox/jenkins-syscont image and then calling docker-compose from within the pipeline steps (I’ll post a blog on this soon).

With this approach, you can launch Jenkins, configure it via it’s web UI and have your first pipeline running in minutes.

You can get access to the Nestybox container runtime at the Nestybox website. It’s free. Give it a shot, it will likely save you lot’s of time when dealing with Jenkins and Docker.

Hope this helps!

Some useful links …

• Nestybox website
• Nestybox blog site (more real-world examples)
• Sysbox Quick Start Guide (installation + examples)

Originally published at https://blog.nestybox.com on September 29, 2019.