Container Security and Performance

I had the privilege to attend OSCON 2016 in Austin, Texas, the biggest open source conference of the year. The conference was a 4 day event jam packed with in-depth tutorial sessions which were around 3-hours in length and shorter talks that were around 50 minutes in length. There was an obvious theme of open source surrounding the conference, but rivaling that theme was containers.
Containers seem to be the buzziest-buzzword of them all in recent years mostly attributed to Docker. The idea and practice of containers have been around for a long time with tools like FreeBSD Jails and Linux LXC. Docker’s fame has largely spawned from its user experience and integrated ecosystem. Docker isn’t just a containerization technology, it deals with the major issues that hinder operating containers in production such as data persistence with volumes, easy networking, configuration management with dockerfiles, and a registry of community created containers.
Even with the extremely quick adoption rate of container technologies, some people are still concerned about how new it is. I often hear both extremes of the argument: “Docker solves all of my security and performance problems”, or “Docker isn’t really secure at all and has a huge performance penalty”. As with any technology, new or old, neither of those statements are true, in general. Docker does NOT solve your security problems and Docker CAN be performant.
Now I’ll explain what I learned from OSCON about this argument on Docker security and performance so that you can have a more intelligent discussion when it comes to operating containers.
First, we’re going to take a deeper look at network performance within a container. Docker has three main networking options out-of-the-box: Bridge (default), Host, and None. The Docker Bridge network creates a bridge interface on the host (docker0) and allows containers to communicate via IP addresses within the container in its own IP space. The Host network mode in Docker allows the container to share the Host’s network stack. This means that they share all interfaces and IP addresses associated. Last, the None network mode within Docker simply creates a new network stack and does almost no configuration on it. The None mode is good for custom networking, but not much of anything else. I’ll also mention that the three modes I mentioned are docker created networks, but users can also create networks and network plugins that are custom to their application and environment.
I used the iperf tool to measure the bandwidth of container-to-container communication using different docker networking modes on Docker for Mac. First, the Docker Bridge showed the lowest bandwidth at 4.13 Gbits/sec. The test I ran involved one docker container serving the iperf server and another connecting to the iperf server with the iperf client. Note that publishing ports to the host machine and running iperf locally results in about the same bandwidth reading. Second, Docker Host networking saw the highest bandwidth of 61.5 Gbits/sec which matches the bandwidth of running iperf server and client on the same host. Next, using the Docker — link option to link two containers (instead of publishing ports or using the host’s stack) saw a bandwidth reading of 43.8 Gbits/sec. Last, running the iperf server in a container using — net=host and publishing ports using the -p option and running the client locally saw a bandwidth of 17.8 Gbits/sec.
The results of the network bandwidth tests I ran will hopefully help you to tune your containers operating in production environments. A simple reconfiguration of the network can immensely increase your network performance especially when communicating with containers on the same host. Keep in mind that there are security implications with sharing the Host adapter when using the “Host” networking mode. For example, privileged containers are able to send shutdown commands to the host…
Most skepticism of Docker involves security. The short truth is that the same security implications apply inside the container that apply when running on a host directly. Follow normal application security, OS hardening, and networking security guidelines. A container can provide enhanced security if configured properly AND when the host is also configured properly.
First, Containers by definition share the host’s kernel in contrast with VMs where there is a hypervisor layer. Therefore, the kernel must be carefully patched and protected against vulnerabilities. Further worry with the kernel is resource utilization. If a container is compromised which is running on the same host, it could launch a denial-of-service attack and take down the whole host.
Second, container breakouts could cause catastrophe on the host and other containers running on the host. This type of vulnerability can be mitigated by adhering to the least privilege principle. If a process breaks out of the container, they will be running on the host with the same privileges. In short, do NOT run your container as root! Also, do NOT run your container with docker’s — privileged flag which gives the container access to the host and other containers on the system, bypassing almost all of the normal docker security controls.
Third, malicious images are a huge problem in the docker community. Anybody can post an image to Dockerhub claiming to provide an image for a particular software package. That image could contain remote logging capabilities, command-and-control agents, or network scanning tools to recon your private network. There are some options to protect against this vulnerability such as using a Docker Trusted Registry (Dockerhub behind your corporate firewall), only using official images (clearly identified on Dockerhub), but the best defense is educating your docker users. Developers and operations teams have to be aware of this vulnerability to differentiate between trusted images and untrusted images. It’s also useful to inspect a community Dockerfile and build it yourself since docker images on Dockerhub may not be built using the same Dockerfile linked on the page.
Docker and containers in general are still fairly new and have evolved extremely quickly. This quick evolution leads to inherent concerns about security and performance. Hopefully some of the points made in this post informs you about some docker modes and configuration and forces you to think more deeply about implications and mitigations when operating containers. There’s a wealth of information available on docker’s website about secure operation of containers.
I also encourage you to check out Ben Hall’s katacoda.com where you can learn all about containers in a hands-on virtual lab. Ben gave a talk about Docker Security and Performance at OSCON which inspired this post. Brandon Phillips (founder and CTO of CoreOS) gave a tutorial on Kubernetes (container orchestration and management) and etcd (a distributed key-value store) at OSCON. CoreOS is a container optimized operating system based off of ChromeOS.
