OSS SPOTLIGHT: Lawrence Livermore National Laboratory and ZFS on Linux
Ever heard of ZFS? If you are a developer, you may have. If you are not working in the circles of IT, maybe not; but that doesn’t mean you are not familiar with it. ZFS is something you may not see or even be aware you are taking advantage of, but ZFS is there. The people who keep your cloud storage running without fail use it. An online service that brings you your favorite movies, television reruns, and exciting new series uses it. Engineers and IT departments that are dedicated to keeping you healthy use it. ZFS is a robust data management tool that was made available through Code.gov, and now finds a happy home in both the private as well as the public sector. And ZFS on Linux, as indexed on Code.gov, is waiting for you to take advantage of it.
ZFS is a file system originally developed and open sourced by Sun Microsystems. ZFS, as a storage management system, eliminates the need for volumes and partitions, thereby bringing thousands of file systems online to draw from a common storage pool, using only as much space as needed. The Lawrence Livermore National Laboratory (LLNL), a research facility answering to the Department of Energy (DOE), originally created and open sourced the ZFS on Linux project. The need for this project was a simple one: LLNL wanted a modified version of this data management system that ran on Linux. To offer scope on how essential ZFS is to LLNL, this lab manages Sequoia, one of the largest supercomputers in the world. Sequoia is comprised of 30,000 hard drives transferring at 1TB/sec while using 500,000 memory cores for processing. At LLNL, over 1000 systems are using ZFS, managing over 100 Petabytes a day. To offer additional scale, a Petabyte is 1,000 Terabytes or 1,000,000 Gigabytes. The amount of data LLNL manages illustrates the draw of ZFS, especially in its resilience to data corruption. ZFS offers enough redundancies in order to deal with potential data loss. With this Linux-version of ZFS, LLNL has experienced only one instance of catastrophic software failure (in 2007), but no data has been lost since implementing ZFS on Linux.
After years of development, LLNL and DOE released ZFS on Linux, now indexed on Code.gov. During its development phase, the open source community participated in its development and maintenance, but the real success of LLNL’s ZFS occurred when the lab made it available to everyone as OSS. Alongside many successful implementations of ZFS on Linux with other agencies and even commercial vendors, LLNL continues to utilize ZFS, always working to develop and improve the platform. Currently, LLNL is developing a variation of ZFS — ZFS+Lustre — to manage the first planned US exascale system, Aurora, capable of a billion-billion calculations per second. Aurora is slated for 2021 at Argonne National Lab. DOE increases the discoverability of ZFS on Linux and many other software projects through DOE CODE, a software services platform and search tool for DOE-funded code managed by DOE’s Office of Scientific and Technical Information. DOE CODE is the mechanism by which DOE provides its inventory of software to Code.gov.
With ZFS on Linux standing as one example of thousands of projects indexed, Code.gov stands by its dedication and impact beyond the D.C. Beltway. Every year the federal government spends $6 Billion on 42,000 software transactions. The Federal Source Code Policy, in requiring agencies to make their source code available to each other for reuse, is an opportunity for the private sector as well as the public. We at Code.gov help agencies implement the policy by harvesting their source code inventories, and offer you access to ZFS on Linux and many other projects. By being the single source for all the federal government’s source code produced since the policy was put in place, Code.gov is helping to reduce the amount of money the government spends on duplicative software transactions while sharing the government’s software with taxpayers, all the while engaging the open source community to help find solutions to some of the government’s most pressing technology issues. This is a success story for LLNL as well as a success story for you. ZFS on Linux is available to you. Code on.
Reach out to us on Twitter or LinkedIn with your questions, and join us in celebrating innovation that begins here. Thank you for joining us on this journey.
