A note on CVE-2019–14271: Running untrusted containers as root is still a bad idea

And building is running

This post addresses CVE-2019–14271 and its consequences for Singularity. In brief, users should still never run untrusted containers with elevated privileges on any container platform. Furthermore, when users build from an existing container, they are implicitly running it. Singularity provides easy, built-in methods to either 1) build containers without elevated privileges or 2) establish a cryptographic chain of trust through self-signed containers if they must be built as root. A more detailed summary appears at the bottom. However we urge all users of containers to review this post in its entirety.

Review of Sylabs’ stance on malicious containers

In February of 2019, a vulnerability in runc and Docker was widely reported (CVE-2019–5736), prompting us at Sylabs to stop and consider what actually constitutes a container vulnerability. This exercise led to the natural conclusion that running untrusted code as root is always a bad idea. It really doesn’t matter if that untrusted code resides inside of a container or directly on your host. Either way, your going to get pwned.

Luckily, Singularity provides some built-in alternatives to downloading random containers from the Internet and running them with privilege. By default, Singularity containers run without privilege. And if you really need to run your containers as root, cryptographic signatures provide a convenient way to verify that your container has been produced by a trusted source.

Deja vu (all over again)

Recent events have led us to revisit this topic. Here’s the story.

Some weeks ago, GitHub user kofalt reported that Singularity v3.0.0 had a bug preventing certain containers from building properly on certain host systems. Longtime community member and Sylabs engineer Dave Trudgian had also encountered this issue a few days prior, and he was in the process of investigating and preparing a patch. Like any developer worth their salt, Dave started by Googling the error. This led him to a similar issue in the Docker GitHub repo, which led, in turn, to this PR containing a patch. But worryingly, the PR also made reference to a security vulnerability (CVE-2019–5739). Sylabs engineers had to determine if this bug was actually a security vulnerability within Singularity too.

In a nutshell, both Docker and Singularity used a similar piece of code during container setup that caused a module of one of the GNU C libraries (libnss) to be loaded from within the container. This caused a mismatch between the host libc and the container NSS module when the host and container distributions were not ABI compatible. For example, this situation occurred when trying to build a Debian Buster container on an Ubuntu 18 host, resulting in a fatal error. But it was also recognized that there could be more serious consequences. A bad actor could place malicious code within one of the libnss_*.so libraries and hide it in an innocent looking container. If the bad actor could trick an unsuspecting user into running their malicious container as root, they could access the host and compromise the system.

In sum, this attack vector is very similar to the one detailed in CVE-2019–5736 which was deemed by the Sylabs team to not be a true vulnerability. And the reason remains the same: it is never safe to run untrusted code (including containerized code) with elevated privileges.

But… building is running!

This is where the library mixing bug takes an interesting turn. Within Singularity, this bug was discovered in the context of building containers. What’s more, if a user is building from a malicious base container, there is no way to determine from the definition file that the host system is being compromised. Consider the following definition file named rando.def:

Bootstrap: library

From: rando-container

%post

echo “Danger Will Rogers!!”

If rando-container above contains a malicious library, then the following command will allow it to write whatever it wants to the host system with root level privileges.

$ sudo singularity build rando.sif rando.def

All that is necessary is the presence of a %post section which will cause the container to be mounted and malicious code hidden in a tainted container to be executed as root.

So why doesn’t Sylabs regard this issue as a security vulnerability?

First it’s important to note that the library mixing bug is not unique. There are many ways that a malicious user could craft a container capable of silently writing to the host system at build time. While it is possible to fix the library mixing bug (as we have with thanks to Cedric Clerget), it is a far more difficult (perhaps impossible) task to prevent a malicious user from creating similar exploits. In fact, it might be as difficult as trying to ensure that users can run any code as root without negative effects.

Second, we believe that classifying the library mixing bug as a vulnerability (and raising an associated CVE) could actually send a damaging message to users. By doing so, Sylabs would signal (wrongly) to the community that we are trying to make it safe to run untrusted containers as root. This has never been a goal of Singularity. Users may see a CVE based on this bug and mistakenly presume that it is safe to run untrusted containers with elevated privileges because no other open CVEs exist. This is a promise that Singularity cannot make.

The fact remains that it is inherently unsafe to run untrusted code (including containers) as root and that building containers is a form of running.

There’s a better way!

The good news is that there are several ways to protect against this kind of attack. Perhaps most obvious is to refrain from using untrusted containers in definition files. You can use the verify command to ensure that a container was signed by a trusted entity before running it or using it as the base during a build.

However, it’s not always possible to verify the integrity of a container before running it or building from it. Perhaps a very interesting/useful container has been uploaded by an unknown entity. Or perhaps it hasn’t been signed at all.

Just as you would cautiously run untrusted containers without root privileges, you can also build from them in an unprivileged manner. Since Singularity 3.0 it’s been possible to use the — remote flag in combination with the build command to build your containers on the remote builder instead of your local system. Singularity 3.3 introduced the — fakeroot option that can be used to build containers in a completely unprivileged manner. Both of these methods prevent a bad actor from tricking you into executing malicious code as root.

In Summary: Docker and Singularity have both recently been affected by a similar bug. While Docker developers classified this bug as a security vulnerability, Sylabs takes the stance that running untrusted code (including containers) as root cannot ultimately be made safe. Sylabs reminds users that building a container implies running the base. Building containers as root is therefore discouraged and the — remote and — fakeroot flags should be used whenever possible. When building containers as root, users should always build from trusted base images.