Lessons Learned from a Decade with Drobo

I’ve had a NAS (Network Attached Storage) device for a very long time. A NAS, for those of you who aren’t total geeks, is typically something you’d find in a small/medium business or a “prosumer” home environment. They’re typically two or more hard drives in an enclosure that provides a way to (ahem) attach some storage to your local network.

You use a NAS when you want to share a bunch of data with a bunch of local computers without the hassle of having to have an always-on computer hosting a file share, which can take up more space, power, and, by their more complicated nature, possibly going wrong. The thinking is the NAS is like a little dedicated computer. You set it up and basically forget it, and it happily stores your stuff and serves it up to the myriad devices you might want to get at your stuff from.

This is all fine and well, and my first NAS, a 1st generation Drobo connected to my router, a 2nd generation Apple Airport Extreme, mostly worked the way I wanted. The Drobo wasn’t quite the speed demon I’d hoped — in fact the performance on the 1st generation of Drobo, with its USB 2.0 connection, was pretty terrible for anything except backups. It was mostly capable of streaming video if I didn’t demand too much of it, and in 2006, when I first entered into the NAS market, my demands were mostly in-line with its capabilities. It was mildly frustrating to set up (mostly due to the limitations of the Airport’s ability to share disks to the network), but it replaced a gigantic old Linux computer running a 1.5TB¹ RAID 5 array that required five times the power, room, and maintenance time, and generated about twenty times the noise and heat as my new solution. I guess I spent a pretty penny on this new solution: the Drobo was about $500 new, and disks to populate it ran another $400 or so, but for my pains I got at little under 2TB¹ of storage, which was huge for the time, and I happily started filling it up.

Like I said, I wasn’t too demanding to start out. My first child was born in 2006 and like any proud parents, my wife and I were taking hundreds of pictures and the occasional video, so a lot of what I was storing were the high-res RAW images from her digital SLR. I’ve also always been a big proponent of having a solid backup for your computer, so I was using the Drobo/Airport combo to back up the computers in the house too. It was slow and prone to error (especially the first version of Time Machine, which was released in 2006 in Mac OS X Leopard), but it mostly worked. I was truly able to “set and forget” as I’d hoped.

There were a few hiccups over the years of course. At one point, changes in the way Drobo allocated space in those early days met up with growing hard drive capacities and I was forced to reformat to get the most out of it. It wasn’t trivial to get everything off the array before reformatting, since I’d grown to depend on the array’s capacious storage, there weren’t a lot of options other than copying everything off onto a couple of larger disks: splitting the now unprotected, unduplicated data across multiple drives, and getting rid of “extraneous” data, like backups, to make everything fit. Despite having a very nice basket, I now had all my eggs in it, and that made me pretty nervous. I was happy when everything was back on the fresh new array with more storage and options than I’d had. It was even a tiny bit more performant, which was good, because it had been an absolute dog before.

At another point, a hard drive failure meant that the Drobo had to “relayout” (that’s re-layout, not relay-out) the information normally spread across four disks across the three remaining good disks instead. This is another nerve-wracking incident in the NAS aficionado’s life as you wait with bated breath for the recovery to take place, dealing with the uncertainty of another disk maybe failing during the process, which will take the whole array with it. On the large and growing array (probably a little under 4TB by this point), it was a days-long event. Once the relayout was complete, inserting a new disk to replace the failed one would result in another days-long relayout. It was at this point I invested in a solid UPS to ensure that during this process a power outage wouldn’t take all my precious data with it.

It’s worth pointing out that this kind of redundancy is the biggest selling point for a NAS: redundancy and fault-tolerance give you a measure of safety. It’s not a complete insurance policy, but it’s part of a larger overall strategy, much like having seatbelts and airbags in your car don’t mean you don’t also need crumple zones and insurance.

So in the end, I had a pretty solid experience with my first Drobo. It had done pretty much what it had promised. A little slower, perhaps (ok, a lot slower), but it had saved oodles of data and reliably recovered from the occasional problem. I was generally happy. I would’ve liked more speed and the ability to grow the array even more (4TB was a hard limit on the older generation, configured the way I had it, and due to the aforementioned egg/basket problem and the expense of new disks, I was disinclined to move the contents again).

In 2010–2011, Data Robotics (now just “Drobo Inc.”), released the DroboFS, a network-based version of the product. Based on the marketing hype, it seemed like the answer to a lot of my gripes about my device. It connected directly to the network instead of needing to be shared out via the (increasingly unstable and underpowered) Airport router via 1GB Ethernet (much, much faster than USB 2.0!) and it supported something called DroboApps that allowed you to run simple Linux-based apps such as media servers directly on the hardware. What’s more, Data Robotics made me an offer I couldn’t refuse by discounting a direct purchase from them. So in 2012, I took the plunge and upgraded to my second Drobo device.

Unfortunately, right from the start I realized the DroboFS wasn’t the panacea I’d hoped for. It was still slow. Not as slow as the glacial 1st gen Drobo, but compared to other NAS’s on the market in 2012, it was still not great. It did have the advantage (with my discount, of course) of being considerably cheaper than a Synology or QNAP alternative, so I didn’t return it, but soldiered on and over a period of weeks moved the content off the 1st gen device onto the DroboFS.

Although things were pretty OK at this point, I began to grow frustrated with the closed nature of the Drobo device. For some reason, one I still don’t understand, Data Robotics chooses to lock down your ability to manage the device to their front-end software, Drobo Dashboard. Out of the box, you can’t, for example, manage the device via a web page, or SSH into it to get an idea of what’s going on. There’s an option in Dashboard to download logs, but they’re encrypted². Only employees of the company can read them.

DroboApps can kind-of resolve some of these issues. I installed OpenSSH and was able to “crack it open” and observe log files and the like on the root Busybox installation that serves as the “brains” of the DroboFS. It wasn’t optimal, but it was better than nothing.

In 2014, after a couple of years of decent operation, I ran into my first issue with the DroboFS that made me think about a replacement. Every once in a while, I’d catch the Drobo performing relayouts. You can tell when this is happening because all the drive status lights blink. It would happen fairly regularly, but there was nothing in the logs that I could find to indicate what was happening. It was at this point that I contacted Data Robotics and had a rather unsatisfying back-and-forth with them because my device was just freshly out of warranty. Eventually a support analyst did take pity on me and tell me that one of my drives was showing early signs of senility and it would be a good idea to replace it, which I did.

This was when the fun began. During the relayout when I replaced the drive, I lost the ability to talk to the Drobo. I could see it, but I couldn’t mount the filesystem (either through the Dashboard or directly, as I’d been doing for years). A second call to Drobo support was equally frustrating, unless I wanted to spend upwards of $100 to buy an extended warranty (aka “DroboCare”). I eventually found an arcane process on the support forums that advised me to remove all the disks, boot the device, sacrifice the contents of another non-array drive to create a new array, upgrade the firmware, shut the unit down, and replace the old disk pack in the exact order it had been in before the problem. Amazingly, this worked.

There’s a funny thing about technology. When it works, you forget it. When it doesn’t work, you fume and fret over it, but, if you get it working again, any initial mistrust you might have fades away quickly. This certainly happened to me with the Drobo. After that incident, I firmly decided my next NAS wouldn’t be a Drobo, but, since it was working fine, I never seemed to get to the point of proactively replacing it. By now the array had grown to an whopping 9TB: two 3TB drives and three 2TB drives with single disk redundancy (3+3+2+2+2–3 = 9).

That brings us to today. Or more accurately, a week ago tomorrow.

I went away for Victoria Day weekend and returned to my apartment to be greeted by my coffee maker, microwave and stove all flashing 12:00 on their clocks. Obviously, there’d been a few power bumps over the weekend.

Now I need to admit my mistake: I moved in 2017 and because of the layout of my new place, my Drobo wound up in a closet, unprotected by a UPS. The two I already have are on my main computer and protecting my entertainment centre, both of which contain a lot more expensive hardware than the Drobo. What I neglected to keep in mind is that the data is actually more precious than the hardware. I kept meaning to buy a dedicated unit for it, but never got around to it.

Ok, I need to admit another mistake of mine. As I mentioned earlier, I’m a huge believer in having a good backup. And I had one. By now, the Drobo houses backups of all my important data, as well as a ton of important but not unique data I didn’t have a second copy of. I’m a belt-and-suspenders kind of guy when it comes to backup, so in order to back that up (all ~6TB of it), I used CrashPlan Home — a cloud-based backup solution that worked really well for me since 2011. Unfortunately, in 2017, right around the time I moved, CrashPlan discontinued the Home service I’d been relying on. It was a pain, but I migrated over to their “Pro” offering, which cost about 1/3 more than the Home version. It has a few additional features over Home, so overall I considered it a wash. It took months to create a new backup from scratch³, but a few months ago, it finally finished.

In March, I retired my seven year old iMac for a new Windows computer. Because of some complexities, I wasn’t able to carry over the backup from the Mac to my new computer, so I had to start the full backup over again. It’s nowhere near done yet, and my old CrashPlan Home backup expired last month. This was my third mistake: I could have paid an additional $10/month for a few months to retain the complete backup from my Mac while my new machine completed its backup, but I was cheap and decided to replace my old machine with the new one, deleting the old backup in the process.

Remember my nervousness at having all my eggs in one basket? Well, here we are again. All my eggs in one basket, and that basket unprotected from power outages, and I’ve just suffered several of those.

A moment of silence, please, to let the enormity of my multiple lapses in judgement sink in.

Still with me? So we’re up-to-date as of last week now. It took me a day or so to realize I didn’t have access to the Drobo. I figured at first it was a problem on the Windows computer’s side. It wouldn’t be the first time that the network connection between the computer and the Drobo would be a source of frustration, so that’s where my thought process took me, especially since the Drobo wasn’t flashing at me or otherwise showing any signs of distress.

It wasn’t until the second day of troubleshooting that I rebooted my computer only to find Drobo Dashboard reporting “Mount Error”. I tried rebooting it, as that sometimes helps, but I got an equally unsettling “Drobo has detected an Internal Problem” error. A search of the support forum, my only source of support for a device Drobo Inc. will no longer support, indicated I should try running a file system integrity check from within Dashboard. The first of these ran for about two days before the activity light, which had been on constantly, winked out. Dashboard still reported “Drive maintenance in progress”. The support post said this might happen, so I tried again. Two days later, the same situation.

Digging further into Drobo support forms, I found this post from 2014 that suggested that if the Drobo was hanging running the filesystem check that I might have more luck by connecting it to a more powerful Linux computer on which I could mount the disk pack via NBD — or Network Block Device. If the description on that page sounds like black magic, that’s because it kind of is.

Obviously, this isn’t a step to be taken lightly. It required a lot of prep work. I had to re-write the startup scripts on the Drobo to start the SSH server before the disk pack was mounted. In order to do this, I had to sacrifice another non-Drobo disk to create a new disk pack that would boot. After doing that, I had to scrounge the internet for the pre-compiled NBD kernel module and server software needed to be run on the Drobo. I had to set up a VM on my computer to act as the NBD client⁴.

Waiting… waiting…

So this is where I am right now, after five days of trying to file check (fsck, in Linux-speak) the Drobo on the Drobo, now I’m trying over the network on a VM running on my desktop. What a future we live in. I’m hopeful that it will run much faster, but with so many variables in play, I’m taking it easy on my computer. I’m leaving it alone and writing this on my laptop, because if this fails, I’m pretty much out of ideas and options.

I do know one thing. Based on this experience, even if the Drobo hardware hasn’t failed me yet, I’m definitely in the market for a new NAS. I’m eyeing the Synology DS418play as a viable alternative. It may have one less drive bay, but everything I’ve read and seen shows that while Drobo insists on closing down and “simplifying” the experience for the user, Synology assumes its users aren’t children that need to be protected from the complexity of their products. I’m quite certain that if I were dealing with filesystem corruption on a Synology array, I’d be working with support resources a little newer than four years old, and I wouldn’t have to unearth old versions of Ubuntu⁴ in order to get access to more powerful diagnostic tools.

This has also taught me a valuable lesson about complacency and backups. I’ve already got a new UPS in my Amazon cart. Regardless of what happens with this restore I never should have had such critical data sitting on commercial power without one. As for my backup headaches, well, let’s just say the next time the choice is between $40 or $50 bucks and having an insurance policy against data loss, I’ll spend the money in a heartbeat.

¹ Remember, this is 2006. Two terrabytes was actually quite a lot of storage at the time. You couldn’t go out and buy a 2TB disk: the top-of-the-line in 2006 was around 750GB, and to get redundancy, you need at least two drives. It’s hard to remember the details, but I’m pretty sure my old DIY array was made up of disks ranging between 80GB and 320GB and the first Drobo probably had one or two 500GB drives which would have been close to the biggest mere mortals could get at the time, at least for consumer drives.

² Encrypted might be a strong word. More like obfuscated. This script can reverse the obfuscation on the older devices, but it doesn’t work on the newer devices or the DroboFS.

³ For the record, CrashPlan did offer a conversion from Home to Pro that would allow you to migrate your backup. It didn’t work for me for a reason I can’t remember, but I’m sure it was my fault, not theirs. At any rate, while the backup did take months, that’s mostly down to the huge amount of data and the limitations of my home internet, and isn’t a reflection on either CrashPlan or the Drobo, even.

⁴ Even that wasn’t straightforward. In the years since this hack was first attempted, NBD development has moved on and the version compiled for the DroboFS (3.7) is no longer compatible with the version that ships with the most recent versions of Ubuntu or other Linux distributions. In order to get it working I had to download Ubuntu 14.04, which shipped with nbd-3.7.