Summary (tl;dr version):
Fasthosts fessed up, all our sites are up, we’re going to bed…
The Saga Version
Thanks for your many sympathetic messages of support over a rather long and trying weekend here at Dharmachakra. We really appreciated them!
Let’s cut straight to what happened since our last hair-raising installment. After Thursday’s usually routine disk changes went wrong at our server hosts, the team here endured a fruitless weekend being stonewalled by them (“there was an unexpected problem with the repair” — and nothing more…). Finally on Monday morning we got to work with a qualified data engineer and gained access to the server and its previously undamaged data disks. To our and the engineers’ surprise, no data was to be obviously found! We then spent a full day and night running block by block scans of the very large disks to check for data. Our scans revealed that contrary to Fasthosts’ continuing assertions, the disks in question were now totally blank… That’s about as bad as it can get in this business.
We wrote and challenged them, pointing out that even their own log trail of this little fiasco indicated that the data disks should be 100% healthy and intact. Did you blank them by accident, we asked? They wrote back this morning and said they were sorry — someone on Thursday/Friday (when they were doing the initial, planned 3 hour repair) got our disks mixed up with a new pair of blank ones and they spent the entire weekend failing to set up our server for access because of that… You couldn’t make it up.
Thankfully, they still had our real disks. And after a lot of fiddly work by the heroic Will Elworthy through the day and night we have now managed to bring all Triratna websites and services back online. And as a bonus, yesterday we recreated and relaunched 16 of them on our new Cloud service. Our plan now is to have a day of monitoring for unexpected issues, and for Will to have a good break till next week, with Samudradaka continuing the process we have begun in earnest of porting all sites and services to the Cloud. We can relax a bit but the aim will still be to complete that well before the end of November and leave Fasthosts — and bizarre hardware and customer service woes — behind for good.
It’s been a crazy few days of intense focus and on-the-hoof work. We’re really happy it paid off! Thanks again for all your patience throughout. We hope the Dharma resources that are available again will be a support in your own practice.
The Dharmachakra Team