Opinion: What Should I Run in My Containers?

TL;DR Run exactly what your software needs and what you need to own your software.

Tonight a friend of mine asked what I thought about single vs multi-process containers. I’m glad he did. Its a popular topic in the container world, but I haven’t written much about it. Until the last month, I’m not sure I understood it well enough to have a developed opinion on the matter.

One of the best written articles on the multi-process side is at http://phusion.github.io/baseimage-docker/. To paraphrase, when you run a single process in a container that software is running in a broken context.

You just built a container which contains a minimal operating system, and which only runs your app. But the operating system inside the container is not configured correctly. A proper Unix system should run all kinds of important system services. You’re not running them, you’re only running your app.

My initial gut reaction when I read this was negative. The reason is philosophical. My container does not and should not contain an operating system. I use containers to isolate a service from other processes on my systems and to ensure that they do not needlessly create clutter. My thinking was that my system already has an init process, SSHD, and syslogd running. I thought, “And if I need cron running in my container, I’ve gone totally off the rails.” The whole point of using a container instead of a VM was specifically not to run a whole operating system. I was reading the article with my own use-cases in mind.

But then I finished the article, checked out the image, and I’d centered myself. I was jumping all over this for the wrong reasons. The truth is much more complicated than an argument that shapes up like single process containers vs. the world.

Here are a few facts about what software that people will run in containers. Sometimes it will:

• Need these services
• Be pure OOTB never designed for containers (need these services)
• Be customized for containers
• Be custom but never designed for containers
• Be written specifically for use with containers
• Be designed without these dependencies

There is a whole spectrum of software that people will run inside containers. And there is no such thing as a right or wrong way to do so. It is not that black and white. What matters is that the software works, you are able to support it, and that you understand the efficiency trade-offs and failure modes of the system you create.

My whole attitude changed after a bit of thought. That article is great because it teaches people about common Unix system components. It may be dismaying to some, but most people do not understand these tools and how they fit together. Their base-image is pretty good. I’ve used it a few times, but my use-cases allow me to avoid large images.

If you’re curious about how I use Docker…

In Docker in Action I try to use tools like Busybox in my examples because I don’t want people to have to wait all day for downloads.

In prod I build and use purposed, hardened, and minimalistic containers. I start with smallish images and pull out things I don’t need. I kill SUID binaries. I know what UID/GID my services should run as (non-root). I know what capabilities my containers need at runtime and make them as restrictive as possible. I use volumes instead of stdout/stderr. Sometimes I use images and volumes for pushing configuration. I use hand selected resource limits. I use an init process. I use sidekick containers for service health monitoring, service registration, and log forwarding. I push one-shot containers to load balancer cluster nodes that tune the network stack for connection volume. I create cheap deployment stages with DNS search domains.

If I need to get my hands dirty in a shell, I’ll CSSH into the cluster nodes running the target container. I’ll pull a toolbox image in and create a new container with the appropriate volumes and network interfaces attached. I only run short lived containers with any elevated privileges.

If you want to run a full Linux operating system inside of your container, you can almost do that with Docker. If you want to run a single NASM program and create the smallest image possible, go for it. Just remember, these are all running on a system that is already properly configured. Use that system when appropriate.

Rather than worry about how others think you should do it, focus on exactly what it is that you need to do. Get it done first. It is usually okay if you need to iterate and add some functionality.

Docker brings to light and simplifies working with some of the least appreciated parts of Linux. While you get quite a bit with a bare-bones docker run my/image command, you can get so much more if you take the time to research the options just a little bit. Doing so will increase your depth of understanding about the operating system that you are using and help you get specific in how you use it.

I bring this up because there are some features that you would use differently in single vs multi-process containers. For example, if you leverage Linux feature authorization and want to give your wirelog recorder the NET_RAW capability, you wouldn’t want to run that in the same container as your public facing web app. In another example, if you use AppArmor, you wouldn’t want to apply the same profile to both your web application and database. But if they are running in the same container then applying a profile to the container would apply the same profile to both applications. Worse, if you are using SELinux, you could hardly make the case that both applications should have the same labels.

I’m a realist. I know some people with application use-cases just want a base image to work off of and kinda forget the rest. I honestly think that is fine. For that purpose phusion/baseimage-docker is great. Just promise me that whenever possible that you will run your containers as a non-root user.

I’ve drafted Chapter 6 of Docker in Action and should be available through the MEAP in a week or two. That is the chapter you should read if you want to learn about using Docker to isolate your software. It covers users, shared memory, device access, resource limits, Linux feature authorization, and touches on other hardening tools like AppArmor and SELinux. Don’t let the subject matter scare you. I do my best to digest all these complicated topics and present them in a “what you need to know” and “how to do it with Docker” sort of way.