ARM Storage Cluster 💪

8 min readApr 22, 2023

This year I embarked on a project to build a Highly Available storage cluster for my home. The requirements were to have a low power cluster that could stay available even if individual nodes were offline for maintenance or hardware failure. This cluster is split into a set of two disk pools, 3 SSDs and 3 HDDs. These pools provide storage for my Kubernetes cluster to run things like Plex, InfluxDB, and a Minecraft server. I still rely on my TrueNAS server for bulk storage that doesn’t need perfect uptime, like my Steam game library.

Parts List

Given the difficulty in getting one Raspberry Pi (much less 6), I chose to use the Libre Computer Renegade SBC (Single Board Computer) for this project. I went with the model with 4GB of ram to have better performance with container overhead and filesystem caching. I didn’t put any thought into the hard drives that I picked, I just grabbed a bunch at the store. Normally I would prefer to get quality drives like the WD Red Plus or Pro series, However the priority on this project was high availability for the entire cluster, not uptime individual nodes. With cheaper consumer drives, I can just pick up replacements at the store and swap out any failed drives.

May as well just subscribe and save on Amazon

This blog is supported through Amazon Affiliate links. When you buy through product links, I may earn commissions to help fund future projects.

History

For the last decade, most of my Homelab has relied on a single TrueNAS server that I built myself. While it has been an amazing server, There has always been a core flaw. It has been a single point of failure, even if there are redundancies built into it. Having to turn it off to upgrade parts, or even reboot it to apply patches meant that everything had to go down and come back up.

One of the first systems I looked at was an Asrockrack barebones kit which could pair a Ryzen 5000 series pro processor with ECC ram. However it was tricky to find these pro processors in stock anywhere [Update: there are now Ryzen AM5 servers that are much better and can use ECC just fine with non-pro chips]. I was also looking into doing a build with Epyc 7002 instead. This was not too much more expensive, but came with a lot more performance (and PCI-e lanes!).

Coming back around to the problem I was trying to solve, I considered having 3 fast TrueNAS servers that used ZFS with GlusterFS for highly available storage. But after planning everything out, I realized it was going to be very expensive, a lot of work, and probably rather overkill. So I thought I’d try to see if I could solve my problem with Raspberry Pis instead. I pivoted my plan to making a cluster of 3 Raspberry Pis with 2 hard drives. One SSD, and one HDD.

After deciding I wanted a cluster of 3, I hit my first roadblock. The fact that the Pi 4 is not in stock anywhere! I tried to test things out on a couple of Raspberry Pi 3’s, however they do not have usb 3 or gigabit ethernet and are pretty slow. After running some benchmarks and tests on a Raspberry Pi 4 2gb model, I found that the Pi 4 could push a single external HDD or a single SSD but not both at once. The Pi 4 can handle the current draw to of each drive on a port, however there is a 1.4A total current limit to all usb ports. Pivoting again to a cluster of 6 nodes (one for each hard drive) would work around that problem, but how would I ever find 5 more Raspberry Pi 4s?

The Renegade

Digging around my apartment, I found a bunch of old SBCs from various projects. These were SBCs like the Nanopi M4, Asus Tinkerboard, and a Le Potato. After doing some googling, I decided to try building a cluster of Libre Computer Renegade computers. This specific SBC has gigabit ethernet, a usb3 port, and has a model with 4gb of ddr4 ram! But more importantly, it has good support for running Armbian. I bought one Renegade to do initial testing with, and benchmarked both of my drives with it.

Benchmarking

It looks cool, therefore it must be fast

Both the Pi 4 and the Renegade were more than happy to deliver high read/write speeds for both the SSD and HDD. I ran one pass of badblocks across each drive, as an initial burn-in test to make sure they were fully working. Then I formatted each of my test drives for ZFS, and ran some benchmarks with aes-128-gcm encryption.

Very early testing with a UPS and a Pi 4 running Rocky Linux

While doing a comparison between the Renegade and Pi 4, I discovered that the Pi does not have any hardware crypto acceleration! Best case, it could get about 107mb/s with aes-128-gcm with all 4 cores fully utilized. This was well short of the max sequential speed for either the hdd or ssd, and was not even considering all the other overhead for a cluster. The Renegade does have hardware crypto, however It did not seem to be working properly out of the box on Armbian when I tested. Luckily, benchmarking openssl was still much faster on the Renegade than the Pi 4, and the Renegade performance was more than enough to saturate a single drive. This made me happy enough to continue with the project, but I should look into this again at some point to improve power efficiency.

Once I was happy with the Renegade for this project and confident that it could do what I wanted, I ordered 5 more to build a complete cluster and started trying out ZFS with GlusterFS. These tests ended up being pretty bad- but that is to be expected, network storage is always going to have more latency than local storage. But what I really didn’t like was that it was both slow and unreliable. After my first benchmarks, I began trying to simulate node failures by shutting down and rebooting individual nodes during tests. GlusterFS became very, very unhappy about this. The healing process would use a lot of CPU while it was working, and it generated a lot of rather confusing logs while it processed things. This didn’t feel like great infrastructure. But the worst part was the experience using the filesystem, as IO operations would get blocked while files were being healed for very long periods of time. When it got into an unhappy state it was difficult to get it back into a healthy state, even from simple error cases like rebooting nodes. This was before I had started trusting it with any real data! I don’t think GlusterFS was meant to be ran at such a small scale, It probably performs more reliably when you have additional servers and more redundancy.

Do you even have a real cluster if you don’t have graphs

I was hoping to have a very simple setup with just ZFS and GlusterFS. Once I realized that I wasn’t going to be happy with GlusterFS in this specific use case, I took a step back to look at my other options. After researching cloud-native storage for Kubernetes, I decided on using Rook (Ceph) as a replacement for both ZFS and GlusterFS. I had wanted to use ZFS because it is very familiar and reliable, however Ceph provides many of the same features like data scrubbing and replication that make me sleep better at night.

Rook

To get rook setup, I had to get all of my Renegade nodes added to my K3S Kubernetes cluster. This was easily solved with an Ansible Playbook to setup each node. Then I just had to deploy the Rook Operator to K3S and let it automatically find the drives and set them up for Ceph. I thought that a ZFS + GlusterFS cluster would have been simpler, but Rook turned out to be great for being easy to set up the infrastructure and also making the storage array easy to use for containers in the cluster. Hopefully that will free up more of my time to improve the horrible job I did with cable management!

I got the 24-port switch last year and somehow it’s already full

Summary

While this arm storage cluster will not be breaking any speed records (Only if I could afford more SFP+ gear!) It has solved my homelab storage problems. The SSD pool is being used for active data like Postgres, InfluxDB, and Minecraft servers. The HDD pool is being used for important but less active data like XCP-ng VM backups. All of the data on these drives are being backed up with Velero to a backup TrueNAS server in another state. Some data like my Steam game library will remain on my primary TrueNAS at home, which will be used for bulk storage that doesn’t need High Availability.

The next project that I plan to tackle is improving my workflow for air-gapped backups. I purchased a bunch of M-discs along with an M-disc burner that I plan to use for periodic snapshots for important data in the event of a serious disaster. It’s hard to judge how effective discs will be over the next 1000 years, but at least it can help me sleep better at night knowing my Factorio save game is etched into a disc to survive for the next millennium.

If you want to talk about cool projects or play on the Minecraft server, feel free to join my personal Discord. If you are more interested in SBCs and using SBCs for gaming, make sure to check out the SBC Gaming Discord or Subreddit!