Open in app

Sign in

Write

Sign in

Thoughts About Docker Security

Chris White
15 min readJul 31, 2018

--

Salton Sea Sunken Dock Beyond Sand Dunes — Chase Lindberg (https://flic.kr/p/9PaNAb)

Table of Contents

Requirements and Other Notes

  • Most docker commands were run using Docker version 18.06.0-ce, build 0ffa825
  • While some software can be supported on Windows/OSX/BSDs most of the recommendations assume a Linux host/access to that host
  • For the easiest tooling it’s recommended that your docker host be a popular Linux distribution (Ubuntu is probably least effort)
  • If nothing else docker containers themselves may be useful for running the tooling
  • Most GitHub links to direct files are with commit hash should the repository evolve
  • Most solutions should have an opensource version available, though I do state commercial options here and there
  • Requests for support on any tooling in the comments will most likely not get a response, so please bring your issues up with the upstream in question

Introduction

Docker security provides an interesting look into securing systems that can diverge from traditional datacenter controls. Ars Technica reported on a case of a malicious image which essentially ran a bitcoin miner. There’s a lot of finger pointing that can be done in this case, and while there are some efforts Docker can make towards better security, it’s still only a part of security as a whole. First you really need to understand security boundaries, a process that often analyzes the shared responsibility model.

What Is The Shared Responsibility Model?

Shared responsibility is, as the name suggests, responsibility shared between parties. If you’ve worked with cloud computing in any capacity it’s a very core concept to ensuring overall security and compliance. AWS, for example, lays out their shared responsibility model on their compliance site. They have responsibility in keeping the datacenters in availability zones secure. On the other hand, they are not responsible for ensuring that your IAM policies are suitable for your business and compliance needs. This is because such needs are very business centric, and it would be infeasible for AWS to set guidelines for every single compliance case.

Docker shares a similar shared responsibility model. They provide Docker Store and Docker Hub as methods to register and download docker images. It’s important to note that the Docker Store will be more vetted out than the Docker Hub. In terms of shared responsibility, Docker maintains the infrastructure for Docker Store and Docker Hub. They also have some degree of responsibility for many of the official images. Your containers being properly fire-walled, however, would not be something Docker would assume responsibility for. So what can be done for securing Docker from our side of the shared responsibility model?

Usage Of Official Images

Official images are vetted through a process described on the official Docker website. Notable features include:

  • Signed images
  • Best practices Dockerfile
  • GitHub repository as source of truth

In cases where you need customization, you may want to consider utilizing official images as your base and building off of that using a custom Dockerfile.

Pull Signed Images

Docker has a “Trusted Content” feature which essentially lets you download signed images only. The official images will all have a signed image that you can download to ensure integrity. For example, you can download the official erlang signed image as follows:

$ DOCKER_CONTENT_TRUST="1" docker pull erlang
Using default tag: latest
Pull (1 of 1): erlang:latest@sha256:e58ee9f4580e7a4e613e4b6d9c4f3f16bd32b23b0abd053adca88458a40c5063
sha256:e58ee9f4580e7a4e613e4b6d9c4f3f16bd32b23b0abd053adca88458a40c5063: Pulling from library/erlang
55cbf04beb70: Pull complete
1607093a898c: Pull complete
9a8ea045c926: Pull complete
d4eee24d4dac: Pull complete
b59856e9f0ab: Pull complete
0b3a71b3b7c5: Pull complete
acc07f483b22: Pull complete
ef7231cbed65: Pull complete
Digest: sha256:e58ee9f4580e7a4e613e4b6d9c4f3f16bd32b23b0abd053adca88458a40c5063
Status: Downloaded newer image for erlang@sha256:e58ee9f4580e7a4e613e4b6d9c4f3f16bd32b23b0abd053adca88458a40c5063
Tagging erlang@sha256:e58ee9f4580e7a4e613e4b6d9c4f3f16bd32b23b0abd053adca88458a40c5063 as erlang:latest

Note that this is not bulletproof security as you can setup a private trusted content repository outside of official Docker sites. It is, however, a good way to add another layer of security for downloading official images. I also recommend checking the Docker Labs repository for a pretty decent overview of trusted content.

Docker Trusted Registry

If you’re in an enterprise environment in an on-prem/cloud/hybrid type setup, Docker Trusted Registry can help with the business side of things (I tried to write this in a way that didn’t sound like a blatant ad but here we are). This is beneficial if you want to use Docker for in house IP solution. It also handles trusted content signing, security scanning, and can even integrate with a company LDAP/AD setup (guess I’ll start freelancing marketing copy). That said, you’re really going to want to utilize this if you’re a business entity and is probably a bit much for average user/open source projects.

Dockerd With User Namespaces

This utilizes Linux namespace capabilities with regards to user IDs. Normally a container will run in a very privileged environment. This could potentially have impact on the host system depending on how the container is configured. To avoid this attack surface, you can utilize user namespaces. This will map container bound users to unprivileged users on the host. Given the way users are mapped you can avoid potential attack vectors involving volumes bound to top level host directories (/ for example). The official Docker site has information on how to setup user namespaces with dockerd.

Hyper-V Isolation

For containers running on a Windows host you can use Hyper-V isolation for your containers. This will essentially isolate the container from the host system reducing the attack vector. It primarily does this by wrapping it in a virtual machine. Running a container on a Windows host with Hyper-V isolation can be achieved through a simple command line parameter:

docker run -it --isolation=hyperv microsoft/nanoserver cmd

For more information you can consult the Microsoft documentation on container isolation.

Dockerfile Verification

A Dockerfile designates how a docker image is built. It’s essentially the blueprints to the entire process. Vital information from Dockerfiles include:

  • What, if any, image was used as the base of the Dockerfile (alpine, ubuntu, etc.)
  • How the image is built, in particular any OS packages that might have been installed
  • Any commands and/or entrypoints used when running the container (this could be a startup script or an init program)

You can see the Alpine Linux Dockerfile for Erlang as an example of one of the official Docker images. In particular it shows that files downloaded are part of a sha256sum verification to reduce attack surface. Some items of concern you want to keep in mind when validating Dockerfiles:

  • Prefer images where the Dockerfile used for building is in a source code repository for change validation
  • Ensure that the base image from used for the Dockerfile is a trusted image (or really, that it’s linked back to an official image)
  • Be suspicious of something trying to bind ports
  • Scrutinize OS package repositories from non-official sources
  • Validate scripts that pulled down from remote sources, especially if there’s no checksum involved

If you’re uncertain for any reason, there’s always the option of utilizing an official image and rolling your own Dockerfile. In the event you’re not sure if an image was actually built by a Dockerfile, there’s always the option of pulling it down and running docker build against it locally. You can also validate some of the commands run to build a fetched image using docker history:

$ docker history --no-trunc -H --format="{{.CreatedAt}} | {{.CreatedBy}}" erlang
2018-07-17T07:55:02-05:00 | /bin/sh -c set -xe  && REBAR3_DOWNLOAD_URL="https://github.com/erlang/rebar3/archive/${REBAR3_VERSION}.tar.gz"  && REBAR3_DOWNLOAD_SHA256="40b3c85440f3235c7b149578d0211bdf57d1c66390f888bb771704f8abc71033"  && mkdir -p /usr/src/rebar3-src  && curl -fSL -o rebar3-src.tar.gz "$REBAR3_DOWNLOAD_URL"  && echo "$REBAR3_DOWNLOAD_SHA256 rebar3-src.tar.gz" | sha256sum -c -  && tar -xzf rebar3-src.tar.gz -C /usr/src/rebar3-src --strip-components=1  && rm rebar3-src.tar.gz  && cd /usr/src/rebar3-src  && HOME=$PWD ./bootstrap  && install -v ./rebar3 /usr/local/bin/  && rm -rf /usr/src/rebar3-src
2018-07-17T07:54:31-05:00 | /bin/sh -c #(nop)  ENV REBAR3_VERSION=3.6.1
2018-07-17T07:54:31-05:00 | /bin/sh -c set -xe  && REBAR_DOWNLOAD_URL="https://github.com/rebar/rebar/archive/${REBAR_VERSION}.tar.gz"  && REBAR_DOWNLOAD_SHA256="577246bafa2eb2b2c3f1d0c157408650446884555bf87901508ce71d5cc0bd07"  && mkdir -p /usr/src/rebar-src  && curl -fSL -o rebar-src.tar.gz "$REBAR_DOWNLOAD_URL"  && echo "$REBAR_DOWNLOAD_SHA256 rebar-src.tar.gz" | sha256sum -c -  && tar -xzf rebar-src.tar.gz -C /usr/src/rebar-src --strip-components=1  && rm rebar-src.tar.gz  && cd /usr/src/rebar-src  && ./bootstrap  && install -v ./rebar /usr/local/bin/  && rm -rf /usr/src/rebar-src
2018-07-17T07:54:24-05:00 | /bin/sh -c #(nop)  ENV REBAR_VERSION=2.6.4
2018-07-17T07:54:24-05:00 | /bin/sh -c #(nop)  CMD ["erl"]
2018-07-17T07:54:23-05:00 | /bin/sh -c set -xe  && OTP_DOWNLOAD_URL="https://github.com/erlang/otp/archive/OTP-${OTP_VERSION}.tar.gz"  && OTP_DOWNLOAD_SHA256="81ed829f829d53ce7dd7e3808eb3162ef672d52bd3ebc1ad1b6c6dafc06cc324"  && runtimeDeps='libodbc1    libsctp1    libwxgtk3.0'  && buildDeps='unixodbc-dev    libsctp-dev    libwxgtk3.0-dev'  && apt-get update  && apt-get install -y --no-install-recommends $runtimeDeps  && apt-get install -y --no-install-recommends $buildDeps  && curl -fSL -o otp-src.tar.gz "$OTP_DOWNLOAD_URL"  && echo "$OTP_DOWNLOAD_SHA256  otp-src.tar.gz" | sha256sum -c -  && export ERL_TOP="/usr/src/otp_src_${OTP_VERSION%%@*}"  && mkdir -vp $ERL_TOP  && tar -xzf otp-src.tar.gz -C $ERL_TOP --strip-components=1  && rm otp-src.tar.gz  && ( cd $ERL_TOP    && ./otp_build autoconf    && gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)"    && ./configure --build="$gnuArch"    && make -j$(nproc)    && make install )  && find /usr/local -name examples | xargs rm -rf  && apt-get purge -y --auto-remove $buildDeps  && rm -rf $ERL_TOP /var/lib/apt/lists/*
2018-07-17T07:39:39-05:00 | /bin/sh -c #(nop)  ENV OTP_VERSION=21.0.3
2018-07-16T22:16:18-05:00 | /bin/sh -c set -ex;  apt-get update;  apt-get install -y --no-install-recommends   autoconf   automake   bzip2   dpkg-dev   file   g++   gcc   imagemagick   libbz2-dev   libc6-dev   libcurl4-openssl-dev   libdb-dev   libevent-dev   libffi-dev   libgdbm-dev   libgeoip-dev   libglib2.0-dev   libjpeg-dev   libkrb5-dev   liblzma-dev   libmagickcore-dev   libmagickwand-dev   libncurses5-dev   libncursesw5-dev   libpng-dev   libpq-dev   libreadline-dev   libsqlite3-dev   libssl-dev   libtool   libwebp-dev   libxml2-dev   libxslt-dev   libyaml-dev   make   patch   xz-utils   zlib1g-dev     $(    if apt-cache show 'default-libmysqlclient-dev' 2>/dev/null | grep -q '^Version:'; then     echo 'default-libmysqlclient-dev';    else     echo 'libmysqlclient-dev';    fi   )  ;  rm -rf /var/lib/apt/lists/*
2018-07-16T22:13:48-05:00 | /bin/sh -c apt-get update && apt-get install -y --no-install-recommends   bzr   git   mercurial   openssh-client   subversion     procps  && rm -rf /var/lib/apt/lists/*
2018-07-16T22:13:02-05:00 | /bin/sh -c set -ex;  if ! command -v gpg > /dev/null; then   apt-get update;   apt-get install -y --no-install-recommends    gnupg    dirmngr   ;   rm -rf /var/lib/apt/lists/*;  fi
2018-07-16T22:12:45-05:00 | /bin/sh -c apt-get update && apt-get install -y --no-install-recommends   ca-certificates   curl   netbase   wget  && rm -rf /var/lib/apt/lists/*
2018-07-16T19:27:24-05:00 | /bin/sh -c #(nop)  CMD ["bash"]
2018-07-16T19:27:24-05:00 | /bin/sh -c #(nop) ADD file:370028dca6e8ca9ed228549d52231cf8139515cc3a14c00aaed75a60b679775f in /

Basic Virus Scan

A virus scan can be nice for identifying basic viruses and malware. ClamAV will be used for this. While it does detect some Linux variants it really shines for Windows based container guests. From a security point of view however, such scans are best run when the system is not live to avoid malicious pre-existing malicious software from interfering. Essentially we need a snapshot of the container’s filesystem and run a scan against it. Using docker save you could get a snapshot of the image. Unfortunately, it uses docker’s incremental storage called “layers” as a result. This means you have to use various JSON metadata files to figure out the exact order to apply layers in.

Now there is a docker export which will provide a way to export a container’s filesystem. This command requires a container, not an image, to utilize so it requires a bit of preparation. docker create thankfully, can create a container without running it to avoid the possibility of malicious software/services interfering with the scan process. The initial setup and scan is fairly simply when all the pieces are in place:

$ docker create --name erlang_scan erlang
6c97429524ebcf9de4f40758c595db1b85345ca84e1228cfdd7f337a9141cd05
$ docker export --output="live_system.tar" erlang_scan
$ clamscan live_system.tar
live_system.tar: OK----------- SCAN SUMMARY -----------
Known viruses: 6590083
Engine version: 0.100.1
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 0.00 MB
Data read: 1029.54 MB (ratio 0.00:1)
Time: 9.586 sec (0 m 9 s)
$ docker rm erlang_scan
erlang_scan

The first step is creating a named container based off of an image. Names here are used because it involves less typing and is a lot easier to work with than a container ID string. Next docker export pushes out the live filesystem to a tar archive. Finally clamscan checks the archive for any malicious files. Note that your clamscan (or clamd) needs to support archive scanning for this to work properly. When the whole process is done we remove the created container (this is assuming the container was created solely for the process of scanning the filesystem).

Low Level Running Container Analysis

This is where a lot of the burden of securing will be pulled from the docker client and moved over to external tooling. While Dockerfile analysis and official image use is a good barrier, there’s still some work you can do to reduce the attack surface. Your container at a low level will utilize various system calls to the kernel in order to do what it needs to. Network calls will generally be utilized as well to pull down packages or contact APIs.

A tool called sysdig can be used to track such calls and traffic (this does assume a *NIX type system that sysdig supports). It has the functionality of many low level tools you’d expect on a traditional Linux system while supporting underlying container functionality such as namespaces. There’s a decent overview on sysdig with containers you can utilize to get an idea of what you can monitor. This could potentially be used in a CI/CD type environment with a workflow somewhat like:

  1. Run a container
  2. Execute sysdig against the container
  3. Let the container run for enough time to get all required services running
  4. Stop the sysdig processing, then the container immediately afterwards
  5. Manually analyze the sysdig output to see various system and network calls, as well as what processes were run
  6. Setup automation in your CI/CD which compares expected container interactions in step 5 with the result of step 1–5 in a completely clean run
  7. Fail or flag CI/CD if the expected system calls/network traffic does not match your expectations

This is a great way to help protect against malicious code that is difficult to detect through traditional static code analysis.

Seccomp Profiles and Capabilities

Analysis of low level calls can help you in another method of securing containers: seccomp profiles and capabilities. seccomp is a way for limiting system calls from your container. This is utilized in docker through JSON seccomp profiles. There is also a default seccomp profile that docker uses to cut out various system calls. It’s important to note that this method of locking down a docker container can generally be more effort than needed for standard container usage. If you’re doing strict compliance such as PCI on the other hand, this can be a nice attack surface reduction for systems dealing with cardholder data.

Kernel capabilities are ways to implement much of the system control handling on a higher level. Essentially you drop all capabilities, then slowly whitelist them. A list of all capabilities are generally tied to your kernel, but the man page will show general ones. Docker labs has a nice overview on using capabilities with docker. You should consider this option if you want to restrict highly risky admin level calls on a high level.

CVE Scanning

This is a fairly interesting prospect when it comes to container security. Many distros are checking security issues and fixing them in package updates. This means adding a cron entry that simply updates repository listings and installs updated packages may be sufficient. Many package managers also have a security fixes only option to reduce the chance that an upgrade breaks your underlying custom applications.

For a more robust system consider using a CI/CD for your docker images. When your containers are taking on serious production tasks you can utilize Blue/Green deployment. This system allows you to use DNS switchovers after confirming that a certain deployment works as intended. Doing so allows for a safety fallback in case OS package upgrades cause your application to fail, versus waiting until everything is deployed to production where customers notice the breakage.

That said, your distro version has an effect on getting security updates. Many OS vendors have an end of life policy for distro versions where they no longer obtain security updates. Utilizing a CVE scan can help track down the security state of your container in terms of listed CVEs. CoreOS has made a service daemon called Clair to help with this effort. Really it’s an HTTP(S) API server that scans images from supported distros. You’ll need a proper client to actually do the image scanning, which can be found on the integrations page. Here’s an example of checking the erlang image:

$ CLAIR_ADDR=localhost CLAIR_OUTPUT=High CLAIR_THRESHOLD=10 klar erlang
clair timeout 1m0s
docker timeout: 1m0s
no whitelist file
Analysing 8 layers
Failed to analyze using API v1: push image https://registry-1.docker.io/v2/library/erlang:latest to Clair failed: can't even read an error message: invalid character 'N' looking for beginning of valueGot results from Clair API v3
Found 504 vulnerabilities
Unknown: 40
Negligible: 187
Low: 149
Medium: 107
High: 21CVE-2018-2562: [High]
Found in: mariadb-10.1 [10.1.26-0+deb9u1]
Fixed By:
Vulnerability in the MySQL Server component of Oracle MySQL (subcomponent: Server : Partition). Supported versions that are affected are 5.5.58 and prior, 5.6.38 and prior and 5.7.19 and prior. Easily exploitable vulnerability allows low privileged attacker with network access via multiple protocols to compromise MySQL Server. Successful attacks of this vulnerability can result in unauthorized ability to cause a hang or frequently repeatable crash (complete DOS) of MySQL Server as well as unauthorized update, insert or delete access to some of MySQL Server accessible data. CVSS 3.0 Base Score 7.1 (Integrity and Availability impacts). CVSS Vector: (CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:H).
https://security-tracker.debian.org/tracker/CVE-2018-2562
-----------------------------------------
CVE-2018-2612: [High]
Found in: mariadb-10.1 [10.1.26-0+deb9u1]
Fixed By:
Vulnerability in the MySQL Server component of Oracle MySQL (subcomponent: InnoDB). Supported versions that are affected are 5.6.38 and prior and 5.7.20 and prior. Easily exploitable vulnerability allows high privileged attacker with network access via multiple protocols to compromise MySQL Server. Successful attacks of this vulnerability can result in unauthorized creation, deletion or modification access to critical data or all MySQL Server accessible data and unauthorized ability to cause a hang or frequently repeatable crash (complete DOS) of MySQL Server. CVSS 3.0 Base Score 6.5 (Integrity and Availability impacts). CVSS Vector: (CVSS:3.0/AV:N/AC:L/PR:H/UI:N/S:U/C:N/I:H/A:H).
https://security-tracker.debian.org/tracker/CVE-2018-2612
-----------------------------------------
CVE-2016-2779: [High]
Found in: util-linux [2.29.2-1+deb9u1]
Fixed By:
runuser in util-linux allows local users to escape to the parent session via a crafted TIOCSTI ioctl call, which pushes characters to the terminal's input buffer.
https://security-tracker.debian.org/tracker/CVE-2016-2779.
<snip>

As seen this image contains quite a number of CVEs. It might be a good idea to consider another distro as base and building an erlang image from that. Another option is to consider using a slim distro such as Alpine Linux to reduce overall attack vector through a considerable reduction in the number of installed packages. Note that Docker Trusted Registry has a vulnerability scanning system as well if commercial options are considered viable.

CIS Benchmark

Even when your OS packages are up to date there still may be security issues with how you configure your system as a whole. For example, the SSH server could be configured with poor cipher settings or allow password based access. For the docker daemon itself there is a benchmark you can scan against by using the docker-bench-security tool. Note this does not scan the actual OS of your containers. To do this on a container level there’s a few options:

  1. Use a commercial tool that handles docker containers
  2. Use OpenSCAP with docker
  3. Search for Ansible Playbooks/Chef/Puppet pre-made solutions for your distro and CIS benchmark in question (chance of being outdated)
  4. Download the CIS benchmarks and perform the tasks manually

There are also a number of OpenSCAP profiles which may not be CIS but give you a reasonable compliance based setup. If you’re willing to pay for it, CIS-CAT Pro will allow you to download OpenSCAP compatible files to make the whole process easier.

Language Specific Package Scanning

While this is a bit out of the container realm, it’s important to remember that your dependencies for applications may be outside of what can be checked by the OS. For example, you might use npm or pip to install various language specific packages. If you’re using GitHub consider setting up dependency scanning in your repository. You can also utilize package specific solutions such as safety for pip packages, or if you’re really willing to throw the money around for it Synk can provide a commercial solution for a number of language specific packages.

Static Code Analysis

This is completely out of bounds for container security, but just like package dependencies the container bound application code you write can be vulnerable as well. Static code analysis essentially looks at the code and attempts to locate poor coding practices that could lead to a security vulnerability. If you’re running a PHP app for example, there are a number of tools available to check code. Given how language centric this is you’ll have to do searching on your own to find out the best solution.

Conclusion

That wraps up our look at some core points you can utilize to help secure your docker containers. Do note that there are other ways you can secure containers, even more that aren’t listed here. This includes items such as firewalling, IDS, monitoring, and all other safeguards at different levels. Going to conferences and continually researching the InfoSec world can provide great benefit to your overall security knowledge as well. I’ll leave you here with a list of some references to look over.

References

Docker
Security
Cve
Containers

--

--

Chris White

Written by Chris White

149 Followers

Working for a major AWS MSP. Bringing you writings on tech meets business and interesting code techniques.

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams