CVE-2019–5736 AND ITS IMPACT ON SINGULARITY CONTAINERS

This post addresses CVE-2019–5736 and its consequences for Singularity. In brief, users should never run untrusted containers with elevated privileges on any container platform. Singularity provides easy, built-in methods to either 1) run containers without elevated privileges or 2) establish a cryptographic chain of trust through self-signed containers if they must be run as root. A more detailed summary appears at the bottom. However we urge all users of containers to review this post in its entirety.

The open source Singularity project started three years ago to provide an application packaging solution for scientists, allowing them to easily create and deploy reproducible workflows in shared compute environments at extreme scale.

Conceptually, Singularity solved problems associated with root-owned daemons and root-level privileges within containers. Before Singularity, these concerns prevented most system admins and architects (charged with building trusted HPC environments for scientists) from installing container platforms.

The Singularity philosophy includes both software design and accepted practices around the safe deployment of containers, balancing risk with the need for privilege. CVE-2019–5736 serves as a compelling reminder of this tension between functionality and security.

CVE-2019–5736 affects runc, and many other container runtimes. When exploited, it allows malicious containers run by the host system’s root user to run arbitrary code as root on the host. We intend to provide a detailed response not only to this specific vulnerability, but also to clarify Sylabs’ stance on this class of vulnerability in general as it relates to Singularity. Similar to the position adopted by the LXC maintainers, Sylabs does not consider issues like CVE-2019–5736 to be privilege-escalation vulnerabilities in the Singularity container runtime. As stated on the LXC security page :

As privileged containers are considered unsafe, we typically will not consider new container escape exploits to be security issues worthy of a CVE and quick fix. We will however try to mitigate those issues so that accidental damage to the host is prevented.

Sylabs would similarly like to make two clarifications regarding container security:

1. Running an untrusted container as root is analogous to running any other untrusted code as root
2. Running untrusted code (and by extension, untrusted containers) as root is not a security vulnerability, rather it’s antithetical to standard security practices

It is unfortunate that these points seems to go largely unnoticed within much of the container landscape.

CVE-2019–5736 is an attack vector that allows a malicious container to “break out” of its isolation and execute arbitrary code as the host’s root user. The malicious code does this by tampering with the container runtime executable exposed via “/proc/self/exe”, overwriting this binary living on the host with arbitrary code. Critically, in order for this attack vector to be exploited, the malicious container must be executed with root privilege.

Sylabs engineers have obtained and tested the exploit code referenced by Aleksa Sarai in the CVE report; we have confirmed that when running an untrusted container as the root user, it is possible to write arbitrary code to the Singularity runtime binary. Singularity is not affected by this vulnerability when running an untrusted container as an unprivileged user, which is the standard use case for Singularity. While we do not consider this CVE, or others like it to be security vulnerabilities, Sylabs will write patches to mitigate these types of attacks when feasible. With the information we currently have, it is unlikely that Sylabs will write a patch for CVE-2019–5736 in particular, because this would limit Singularity to running on more recent kernels, and prevent many HPC centers from using Singularity. Since users in most of these centers don’t have the ability to run containers as root, the issue is moot.

Because it’s unrealistic to suggest that containers must always run without elevated privileges, Singularity provides a simple, end-to-end method for guaranteeing that a container was signed by a trusted source, and that its contents have remained unmodified . The Singularity Image Format (SIF) allows an entity to cryptographically sign a container, creating a signature block within the SIF file. Critically, users who wish to execute that container with elevated privileges confidently must ensure both of the following criteria are satisfied:

1. That the image has been cryptographically signed and verified against the PGP signature, thus guaranteeing that it is an unmodified copy of the original
2. That the entity who signed the image is a trusted source

The entire chain of cryptographic trust is self-contained within the SIF file, meaning that even if the Singularity Container Library were compromised and malicious actors tampered with popular images, the end user would be aware that the container they downloaded has an invalid signature after running the “verify” command.

Security is not a checkbox that one can tick and forget. It’s an ongoing process that begins with software architecture, and continues all the way through to ongoing security practices. In addition to ensuring that containers are run without elevated privileges where appropriate, and that containers are produced by trusted sources, users must monitor their containers for newly discovered vulnerabilities and update when necessary just as they would with any other software. Sylabs is constantly probing to find and patch vulnerabilities within Singularity, and will continue to do so.

In summary: CVE-2019–5736 exposes a vulnerability that can be exploited in most container systems when a malicious container is executed by the root user. Singularity was developed to allow non-root users to run containers, and in this mode of operation, Singularity is not vulnerable to this or similar attacks. Singularity’s unique image format allows for easy cryptographic signing and verification so that users can judge whether they trust a container before running it with elevated privileges. Whenever any code is executed by the root user, there are security implications, therefore only trusted code and containers should ever be executed with root-level privilege.

Originally published at https://www.sylabs.io on February 16, 2019.