Announcing General Availability of Singularity 3.4.0: Build and Run Encrypted Containers
TL;DR: The generally available release of Singularity 3.4.0 allows you to build and run encrypted containers. Encryption is applied at rest, in transit, and during execution. The software is immediately available here.
The generally available release of Singularity 3.4.0 places emphasis on a single feature: the ability to build and run encrypted containers. We appreciate that some might object to our propensity towards hyperbole, given that seemingly sweeping statement. And that’s precisely what makes this release, frankly, a remarkable one; to quote from the release notes:
The major new feature of this release is the ability to build and run encrypted containers. These containers are encrypted at rest, in transit, and even while running! There is no intermediate decrypted rootfs left around upon termination. Data is decrypted totally in kernel space.
In other words, Singularity containers remain encrypted throughout their entire lifecycle — when they are created, when they are at rest or transferred around, and yes, even when they are in use. Owing to their use of kernel space for data decryption, there is no need to clean up a decrypted rootfs upon termination.
The Single-File Advantage
In some ways, Singularity exploits an unfair advantage: use of a single file to encapsulate the entire container runtime. Sylabs software architect Adam Hughes frames it this way:
From a technical point of view, it’s certainly more complicated in the Open Container Initiative (OCI) case. Individual layers may or may not be encrypted, and/or they might be encrypted with different keys, etc.
The inherent complexity of encryption in the OCI case is well reflected by progress in standards and implementations — the most promising of which today only address but an isolated layer at rest.
In striking contrast, the single-file approach that is employed in the Singularity Image Format (SIF™) has the benefit of not needing to decrypt multiple layers, and put them somewhere before running. Furthermore, Adam surmises that:
Container startup cost should be lower owing to the single-file approach. Because there’s no need to decrypt the rootfs to a filesystem before running, it is reasonable to argue that this single-file approach is also more appealing from a security point of view.
In our estimation encryption over an image’s entire lifecycle, that does not impair its mobility in any way, far outweighs any economies associated with layer-based decompositions — e.g., in cases where a static base image may be layered together with a relatively small amount of sensitive content.
As one of the development leads closest to the implementation of support for encryption, Satish Chebrolu opines that the single-file format, and not having to worry about securing multiple layers, is what makes the approach taken in Singularity highly differentiating.
As you might expect, implementing a comprehensive capability to support encryption over a container’s entire lifecycle is a team sport. Satish initiated the effort and contributed on an ongoing basis. On the development front, his efforts were aided by fellow software engineer Marcelo Magallon, and in-flight testing was contributed by Adam, Dave Godlove, David Trudgian, and Geoffroy Vallee. Finally, Ian Kaneshiro made various contributions to correspondingly enhance the Singularity CLI; based upon even a cursory scan of the release notes, it will become readily evident that encryption supports PEM-encoded RSA keys and passphrases.
Encrypt OCI (Docker) Images Today
As was indicated above, the ability to encrypt OCI images (e.g., Docker images) remains a work in progress at this time. Owing to the high degree of interoperability between Singularity and Docker/OCI however, it is possible to encrypt OCI-format images today with Singularity.
Not only does this interoperability include the ability to acquire and make use of an OCI image via Singularity, it also includes the ability to convert an OCI image into a SIF file. For example,
sudo singularity build — encrypt — passphrase encrypted_lolcow.sif docker://godlovedc/lolcow
results in the creation of the passphrase-encrypted SIF file “encrypted_lolcow.sif” from the OCI-format “lolcow” image obtained from the Docker Hub.
As detailed in the documentation for Singularity, “docker” is but one of a number of choices of available bootstrap agents.
Coming Soon to SingularityPRO
The primary purpose of encryption is to protect the confidentiality of digital data stored on computer systems or transmitted via the internet or any other computer network.
Not too surprisingly then, encryption in a containerization context matters greatly to private-sector organizations in addition to those in the public sector who need to place significant emphasis upon security. It will be welcome news then, for customers of Sylabs’ SingularityPRO, that the next release of this enterprise targeted version of the container platform will include support for encryption as described here — i.e., the ability to build and run encrypted containers.
In addition to support for encryption, the next version of SingularityPRO will also include:
- The Singularity plugin framework
- Support for multistage builds
- Secure UID/GID mappings via user namespaces
- A new “sif” command group
- Support for pushing to and pulling from OCI registries via the “oras” URI scheme
For additional details on these capabilities, please review our blog posts corresponding to the releases of versions 3.2.0 and 3.3.0 of Singularity Community Edition.
As SingularityPRO is now bundled with generally available Singularity Enterprise, Sylabs’ customers will be able to incorporate encryption into their existing and planned workflows that leverage the suite of container services aimed at enabling production deployments of the Singularity solution. Although it is already possible to push encrypted SIF files to, and pull them from, the Container Library via the Singularity CLI, additional enhancements are anticipated. For example, for enhanced usability, encrypted SIF files will soon become visually distinct via the Library’s user interface. This visual distinction will complement that which already exists for cryptographically signed SIF files. Taken together, encrypted, cryptographically verifiable SIF files comprise the most-advanced offering for confidentiality and authenticity in the container ecosystem today.
These same usability enhancements will also appear in the Sylabs hosted, Internet accessible Container Library available online at https://cloud.sylabs.io/.
There’s More to 3.4!
There’s no question that support for encryption comprises the most-significant feature of the Singularity 3.4.0 release. However, we did want to make passing mention here of a seemingly innocuous allusion in the release notes to:
Support multiple-architecture tags in the SCS library
Today, the Remote Builder supports architectures in isolation — i.e., x86_64 OR Arm OR POWER. However, Sylabs’ customers have made it clear that they require simultaneous support for multiple architectures — — i.e., x86_64 AND Arm AND POWER. As of this release of Singularity, the introduction of a new flag accounts for this multiple-architecture possibility. The Remote Builder included with Singularity Enterprise, as well as the Internet accessible Sylabs’ Cloud, will soon have the ability to target multiple architectures in a single deployment of the software. In practice, this means that a container can be created from an architecture different from the one upon which the build command was issued — and yes, that will be supported in Singularity Desktop for macOS.
From enhancements aimed at definition files, to the addition of an “experimental, hidden — fusemount flag to pass a command to mount a libfuse3 based file system within the container,” the release notes offer even more to discover. We encourage you to have a closer look.
Next Steps
The generally available release of Singularity 3.4.0 was just issued by release manager Dave Godlove. You will find it available at the project’s GitHub repo here. We look forward to learning about your experiences with the newfound ability to build and run encrypted Singularity containers.
Bonus: Best Practice Tip
David Trudgian adds:
Ideally, the machine you are running encrypted workloads on should be using LUKS encrypted swap, if it is using swap space at all. Otherwise, it’s possible that application data can be paged out to disk, and exposed there unencrypted — and vulnerable to an attacker with access to the hardware (but no root access to the running system), who can pull the disk and read it on a different machine.
