My Hot Picks for 2018 — GPU Acceleration, Hybrid Clouds and Stateless Access to Storage

I like the beginning of a new year. I get to set a new resolution for the next twelve months that invariably I will struggle to keep. But the best part is that as CEO of WekaIO I get to share my views and predictions of the storage industry. As 2017 is now behind us, all eyes are firmly on 2018. With more than two decades working in the storage industry, from my time at XIV and IBM to my current role as co-founder and CEO of WekaIO, I have a bird-eye view of just what lies ahead. From the many conversations I have had with customers over the last year I see three key issues/trends for 2018:-

1. The I/O dilemma created by the condensing of compute with GPUs
2. How to bridge the transition from on-premises to cloud
3. The growing need for stateless access to storage to support IoT projects

The Changing Face of I/O Workloads

Fast GPUs from NVidia and other accelerators (like Intel Nervana™) are greatly condensing compute, creating a big performance bottleneck in their wake. Last year we saw vendors packing eight GPUs in a single server, provide more compute performance than an entire rack of CPUs used to not so long ago. As a result, these servers have much more intense IO requirements, and will require file systems with previously unheard of capabilities. A single client will need to saturate a complete 100gb/sec links (12GB/sec each), and scale this performance across many of nodes. This performance is unattainable by legacy architectures such as NFS lading to GPU or CPU starvation.

Modern machine learning workloads process millions of small files (text samples, voice samples, images), and require high throughput out of many small file accesses. This emerging workload — low-latency high-throughput file performance — requires a new file system design to ensure expensive GPUs are not idling, or projects won’t be able to scale. WekaIO Matrix, a parallel file system designed to leverage NVMe, is the only file system that is suitable to these emerging workloads. Legacy HPC storage parallel file systems like Lustre and GPFS (IBM Spectrum Scale) are great at sustained performance on large files but simply cannot handle the high bandwidth, low latency requirements this small file workload creates.

Storage and the Hybrid Cloud

Every company, small to large is battling how to integrated Cloud into its IT strategy. Increasingly we will see local storage transitioning to the hybrid cloud, where the most value and efficient usage can be extracted from both on-premises infrastructure and the public cloud. When on-premises infrastructure reaches its capacity, the public cloud’s elasticity will prove increasingly useful for “cloud bursting”, which describes the process of extending excess work across to the cloud, and bringing the results back on-premises. The key storage requirement for effective hybrid clouds, is the ability to push snapshots to and from the cloud. In order to achieve efficient utilization from compute, both must work at local speed. Snapshots are the key to identifying and reconciling differences between the on-premises and cloud copies when work is forked to the cloud.

Stateless Access to Storage

Serverless compute and IoT projects are driving a strong and growing need for stateless access to storage. Currently, the de-facto standard for stateless access to storage is the HTTP-based S3 interface, where the most popular system is the AWS S3 object storage system. As IoT applications become more demanding, they will need lower latency than object storage can provide. Many of these serverless projects have a legacy component (such as DevOps toolchain) that will be in production for a long while. It is clearly apparent that we will see more and more demand for a unified single access file system that supports both file systems access protocols (POSIX, NFS, SMB) and S3 and HDFS. Such systems will allow for the running of different toolchains on the same data, eliminating the need to move to other storage systems.

Exciting things are happening in storage, the IT world is evolving and we look forward to 2018 and what news it will bring.