LASTING FIRSTS AT THE 2019 RICE UNIVERSITY OIL AND GAS HPC CONFERENCE
Multiple Firsts
At the Rice University Oil and Gas HPC Conference in Houston earlier this week, there were a number of firsts, and Sylabs factored directly in a few of these.
As a first-time participant in the event, Sylabs was also a first-time sponsor in a brand new category — along with a few other startups, Sylabs was a Launch Pad sponsor . Also for the first time, the RiceU event closed with a half day of workshops; Sylabs presented one of these.
As a sponsoring and workshop participant, awareness regarding the connection between Sylabs and Singularity was increased — as most attendees had much more familiarity with the container technology itself, as opposed to the company developing and delivering solutions around it.
Houston: Do We Have a Disconnect?
Containers in general, and Singularity in particular, were alluded to in a number of the presentations during the technical program at this year’s event.
Easily, the most compelling example of uptake is the anticipated use of Singularity on the Post-K supercomputer in the Japanese national lab known as RIKEN. With Dr. Satoshi Matsuoka as the director of the RIKEN Center for Computational Science, use of containers on supercomputers is hardly surprising, as he has an established track record as a pioneer in this area — with previous deployments that include TSUBAME 3 and ABCI.
Alongside a number of national labs in the United States, high-profile computing facilities such as the systems at Texas Advanced Computing Center (TACC) were also keen to share their uptake of containerization. In his talk, TACC executive director Dan Stanzione related that containerization is key to the Center’s path forward on new deployments such as Frontera, but also factors into the efficient and effective use of existing systems at TACC. In fact, TACC has expended significant effort containerizing the applications and workflows employed in the groundbreaking work conducted on its platforms by a national and international community of researchers. At this point, they have some 4,000 applications containerized via biocontainers and Lmod.
Whereas supercomputer centers from Japan to the United States are making their broad and deep uptake of containerization explicitly known, that was definitely not the case when came it to the well-known brands of the Houston oil patch. Despite having an aggregate performance ranking in the top 10 in their own on premise data centers, these well-known mainstays of the petroleum industry were notable through the apparent of containerization in their existing and planned deployments. Anecdotally, through various conversations during the RiceU event, it is evident that adoption of containers is actually underway. Whereas use of technologies such as Singularity are a ‘no-brainer’ for green-field initiatives that emphasize Deep Learning, refactoring existing HPC applications and workflows (e.g., seismic processing and reservoir modeling) for containerization is progressing at a much slower rate of uptake. We did run across a few notable uptake examples in our conversations, however, and encouraged those savvy members of the oil patch to make known their experiences at the 2020 RiceU event and elsewhere in the exploration geophysics community.
Our Singularity Workshop
The introduction of any new technology can be impeded not only by complete lack of awareness, but also initial perceptions. For many in the Houston oil patch, initial perceptions regarding containerization were based upon use of Docker. Whereas Docker successfully addresses requirements established in those deployments that emphasize microservices based architectures, existing HPC and emerging Deep Learning workloads and workflows emphasize an orientation that is compute intensive — an orientation that demands routine and unencumbered access to compute-enhancing enablers such as GPUs, InfiniBand interconnects, parallel file systems (e.g., GPFS, Lustre and more), etc.
For reasons such as these, Sylabs was delighted to have the opportunity clarify the perception of containerization for over 70 members of the oil and gas community through a workshop delivered at the event. Lead by Sylabs technical writer and part-time geophysicist Ian Lumb, workshop participants also had the opportunity to engage in discussions with Singularity founder and Sylabs CEO Gregory Kurtzer. Along with Sylabs engineering coordinator Dave Godlove, these three members of the development team behind Singularity collectively left the audience with a somewhat altered perspective regarding containerization — one of Singularity containers as the ideal solution for encapsulating the runtimes required not only by Deep Learning, but also by traditional HPC use cases.
By emphasizing Singularity’s stance on integration over isolation, participants left assured that their device-specific requirements could be readily addressed — without the need for heavy lifting via arcane incantations. Arcane incantations for device use, by the way, that also figuratively and literally render ports open that might be of concern from a security perspective. The essential message of a solution for containerization that allows untrusted users to execute untrusted containers in a trusted way via Singularity was not lost on the numerous security conscious workshop participants. That the trust model in the Singularity case is simplified owing to the container’s daemonless execution in userspace, along with cryptographically signable and verifiable images, was also extremely well received by the audience.
Unfortunately, the matter of performance was not a topic that received any attention during this workshop at the RiceU event. However, these days, there is no shortage of benchmarks available from those who have applied Singularity to their most demanding workloads. In all cases, the bottom line is the same: after a negligible penalty at startup, Singularity containers allow applications to execute with close to bare-metal characteristics in terms of performance and latency. Although benchmarking studies involving use cases specific to the oil and gas industry are in short supply at the current time, useful comparables can be found from other industries with analogous workloads. And of course when it comes to Deep Learning, directly relevant benchmarking studies already exist — studies that include use of frameworks capable of supporting distributed processing.
For many in the oil and gas industry, as well as those in the other demographic groups represented at the RiceU event, adoption of Singularity as the means to containerize applications and workflows simply just makes sense — it makes sense because Singularity was designed with compute centric HPC and Enterprise Performance Computing (EPC) requirements from the outset. However, for many, this singularly different approach towards containerization is only now becoming apparent. Thus the value of the extremely collegial Rice University Oil and Gas HPC Conference cannot be understated, as there is no more powerful means for communicating advances in technology such as Singularity than that which occurs by word of mouth, in person and in a welcoming setting.
From those who participated in person in the workshop, to those with whom the team from Sylabs met elsewhere at the RiceU event, we look forward to the better-informed perception regarding containerization; we hope that it will be impactful in both the tactical and strategic efforts undertaken by members of the Houston community writ large. Like Singularity itself, the workshop materials developed by Sylabs’ Eduardo Arango are available online here; this means that you can continue to onboard from a technical perspective at your own pace. Importantly, you are not alone: by engaging with the Singularity user, developer, and provider community, you’re ability to broaden and deepen your technical expertise regarding Singularity is supported going forward. In the end, we hope that your engagement with Singularity will continue to grow, and are extremely grateful to the 2019 RiceU event for providing the ideal venue for mutual introductions.
Originally published at https://archive.sylabs.io on March 7, 2019.
