Hi Phill, indeed, the storage is the most difficult aspect of a Kubernetes cluster in my opinion. There are many different solutions. I adopted Ceph, but you could also use GlusterFS or some block storage provided by a cloud provider such as AWS EFS although this last one may cost quite a bit.
The key to Storage is the network. In fact what I came up with turned out to be faster than an SSD although using normal (enterprise) HDDs just because of an upgraded network connection. I managed to achieve reading speeds of 500 MB/s and writing speeds of 800 MB/s.
What you’ll need is a dedicated storage cluster which can be on top of Kubernetes or on bare metal which I preferred. Discussing this topic with a couple of consultants, they suggested Ceph since it offers block storage, s3 and iscsi protocols. And it’s also used by CERN so it must be solid :D
It’s not easy to configure but it’s a beast.
I will write an article on Storage and how I ended up configuring the cluster as soon as I have some spare time. Hopefully in June.
Thanks for your feedback!