Member-only story
Failed External USB Nightmare with CentOS: Survived!
How I survived an old centos box with a failed external USB drive!
This article is mostly to remind me how to do this again as it is a bizarre and rare situation.
Even though we have migrated to the cloud for over had a decade, some of our legacy infrastructures are still in a data center we manage. One such unwanted, forgotten system is a mail server. We failed to migrate this to the cloud due to the IP reputation issues with cloud providers who recycle their IPs. Honestly, it is mostly because mail server setups are notoriously time-consuming.
So this old mail server had an external USB drive to mirror the primary drive and act as a hot backup of the mail data. It worked fine for 3+ years, but eventually, the consumer-grade external USB drive failed a few days ago.
I noticed a few rsync processes are hung and then tried to access the disk via /mnt/usbdisk, but it locked up my terminal. Eventually, I realized that the USB drive has failed as most commands line ‘blkid’, ‘umount’ etc., were all locking up.
Interestingly, as I was trying to access the failed disk with various tools and including fdisk, I noticed that every attempt had added load to the server. Still, it was weird that the load average was going up, but the top was not showing any specific…
