TL/DR: Singularity 3.2.0 RC1 is now available here! It’s a HUGE release that introduces a plugin framework, plus support for multistage builds and cloud endpoints, and more!
Release manager Dave Godlove made it official late last night: the first release candidate for Singularity 3.2.0 is now available! There’s no need for hyperbole here, as this truly is a release chock-full of impressive new features and functionalities.
Plugin Framework
As this post from February 2018 indicates, plugin support in Singularity has been on our minds for some time. After the successful reimplementation of the Singularity core in a combination of the Go and C programming languages, software engineer Michael Bauer turned his attention to the development of a framework capable of introducing plugin support for Singularity. Once designed, Michael’s prototyping progressed rapidly, and attendees of our inaugural meeting of the Singularity User Group (SUG) were provided with a sneak preview of this very significant addition to the core capabilities of Singularity. Michael’s talk from SUG, which includes a demo, is shared at the end of this post.
Contributions like this are of course impressive in and of themselves, but their value escalates exponentially with each available plugin. Sylabs partner Ellexus is anticipated to be one of the first to leverage this new capability to deliver an integration between Singularity and Ellexus’ tools for system telemetry and I/O profiling. In fact, both Michael and Ellexus CEO Rosemary Francis alluded to integration via the new plugin framework in their talks at SUG.
At the present time, included in this RC1 version of Singularity 3.2.0 is a basic framework for creating and managing plugins. Through a new command group, plugins can be compiled, installed/uninstalled, enabled/disabled, listed, and inspected.
In case it wasn’t obvious, the reason we’re so excited about this brand-new, next-level capability is that it is designed to enable the extensibility and customizability of Singularity in a modular, effective, and efficient way. There have already been numerous discussions, at SUG and elsewhere, regarding this plugin framework. We cannot wait to share in future posts news of the plugins developed by members of the Singularity user, developer, and provider community.
Once again, Michael’s talk from SUG can be found below; it includes a demo that covers many of the commands in this new group.
Multistage Builds
Singularity containers provide the means to package an application along with all of its dependencies and environment — the binaries and libraries in its toolchain, configurations, data — everything except the operating system kernel. By explicitly excluding the kernel, the remaining runtime is encapsulated within a single file based upon the Singularity Image Format (SIF). In general, this means containers are significantly more focused and leaner than, say, virtual machines.
The ultimate means for creating, maintaining, and securing purpose-built containers is through Singularity definition files. Stated simply, def files allow you to articulate the specifics of the build process. Starting with a header, the substance of def files are the remaining sections that address environment settings, runscripts, and much, much more. Previous to this release, implicit in Singularity was the notion of support for only a single stage in def files — in other words, directives embedded in def files would be executed in the context of a single container base.
New to Singularity 3.2.0 is the ability to introduce multiple stages into your definition files. Stated simply, this means that multiple container base images can be leveraged from a single definition file. You could, for example (see the def file below), use a base with golang tools installed to build your application in the first stage of a def file (stage one), and then install your application in a leaner base for use in a subsequent stage (stage two).
Bootstrap: docker
From: golang:1.12.3-alpine3.9Stage: one%post
# prep environment
export PATH="/go/bin:/usr/local/go/bin:$PATH"
export HOME="/root"
cd /root # insert source code, could also be copied from host with %files cat << EOF > hello.go
package mainimport "fmt"
func main() {
fmt.Printf("Hello World!\n")
}
EOF # build
go build -o hello hello.go# Install binary into image without go toolsBootstrap: library
From: alpine:3.9
Stage: two#install binary from stage one
%files from one
/root/hello /bin/hello %runscript
hello
Not only does this result in a purpose-built container that is very lean, through the use of def files with support for multistage builds, this truly is the most-sensible way to create and maintain images for use by Singularity.
Last, but perhaps most importantly, by creating containers from first principles, there exists the potential to address matters relating to security from the outset — as you have, in principle, a very clear picture as to what’s in your container. And once created, your container can be signed with a cryptographically verifiable key.
As you might anticipate, those embracing containerization in the context of embedded systems, IoT, etc., work assiduously to minimize resource utilization — in fact, in most cases, the success of their solutions depends upon it! With support for multistage builds in Singularity, new levels of efficiency and effectiveness are now well within reach.
For the rest of us, it may be time to revisit existing definition files, to see if there are any opportunities to take advantage of this new capability. If you’re new to the concept of definition files in general, this new capability amplifies the value of making the investment to employ them in creating, maintaining and securing the containers you value most.
Our documentation team is developing the content to elucidate use of this new capability; in the interim, the example above should serve to whet your appetite, and get you started. Support for multistage builds, as well as the def file example above, were contributed by Sylabs software engineer Ian Kaneshiro.
Ecosystem Support
With the release of Singularity 3, Sylabs initiated a strategic effort aimed at broadening and deepening the ecosystem in support of Singularity. Immediately, this resulted in an alpha preview of the Sylabs Cloud — a portfolio of services aimed at securing, creating, and sharing images for use with Singularity.
In the 3.2.0 release, management of the cryptographic keys employed to sign and verify Singularity Image Format (SIF) files has been enhanced through three additions to the “key” command group. These command-line enhancements were contributed Sylabs software engineers Joana Chavez and Westley Kurtzer.
As of this version 3.2.0 RC1 offering, the Sylabs Cloud Container LIbrary is regarded as an endpoint for Singularity. To manage support for additional endpoints, the “remote” command group has been added. Armed with this command group, endpoints can now be added/removed, listed, and have their status be known. Endpoints are necessarily authenticated for use via an API token, and a default endpoint can be established. Also contributed by Ian Kaneshiro, endpoint flexibility represents an important milestone in the progression towards on-premise support for Sylabs’ container services ecosystem.
Enhanced Support for the OCI Runtime
The Singularity Container Runtime Interface (CRI) was recently tagged beta. Along with support for an Open Containers Initiative (OCI) runtime in Singularity, Singularity CRI is now natively integrated with Kubernetes for container orchestration. Compliance then, with respect to the OCI runtime specification, remains crucial to the delivery of this Singularity-Kubernetes integration. By encapsulating the ‘raw materials’ needed to create an OCI compliant runtime within its image format, the extensibility of SIF is demonstrated, and its value is preserved.
The existence of the “config.json” file is critical to conveying configuration data for an OCI runtime. Prior to version 3.2.0 RC1 of Singularity, this JSON file was generated dynamically from a descriptor embedded within the SIF file. As of this release however, when building from an OCI source with pre-existing configuration data, the “config.json” is stored within a JSON partition of a SIF file.
Additional Enhancements
At SUG, we introduced preliminary support for the macOS platform through the alpha release of Singularity Desktop. With this latest release candidate, we’ve harmonized the macOS-specific enhancements with the existing Singularity code base. Along with additional flexibility in building Singularity itself for those who’ve made a clone of it from GitHub, this code merging rounds out the additional enhancements included in this release.
Next Steps …
Especially true for release candidates, we are urging the Singularity user, developer, and provider community to clone, build, and test this latest release. We are excited to receive your feedback on it starting with this RC, and look forward to future contributions that can now include plugins for the first time.
Singularity 3.2.0 RC1 is now available via GitHub here. Enjoy!
Michael Bauer’s SUG talk on plugin support is available below and here.
Originally published at www.sylabs.io on April 12, 2019.
