On 31 May, I encountered some errors of my website running in production, from the error log I discovered that the Spaces API access key is not found, this is normally not possible if the access credentials of your live application disappears suddenly.
I signed into my DO account, and navigated to the Spaces tab and just got shocked by the UI.
At the time I realized that all my files are gone except one bugged bucket which should being deleted. I had about 16 buckets there before those storing my production website data. Then I navigated to check if my API key is still there, but it doesn’t.
I created a new support ticket immediately to ask what is going on with my files. And they said an IP performed these actions. The IP is from my colleague’s computer, I asked if he did the action and of course he didn’t. I wrote a program using the access key to do some business logics for my website, and compiled it to my colleague for execution.
Below is my program’s workflow:
- Read the file path from input
- Create a temporary bucket named like “extmp123456789”
- Upload the files to the temp bucket
- Move to real bucket after upload successes
- Delete all files in the temp bucket and delete the bucket itself
The reason why I need to do this is because Spaces will charge for file size that you stored at rate $0.02 per additional GB stored. And their system sucks, if you upload directly to your buckets and the upload fails, there will be some invisible leftover files, you just cannot delete it but they will eat up your storage.
So I explained my situation to the support guy, stating it is impossible that the real buckets being deleted with the temp buckets, and they still not facing to the issue.
Of course I know my colleague and me will not delete all the production data by just a “mistake”.
And this time they finally recognized that this is their mistake, and all of my buckets, production data, are being wiped permanently. They send me two email lately, the first one is a acknowledgement letter to help me understand that my data are gone.
And they gave me $300 credits with expiry date for all my passed away data in the second letter.
Surely I will keep contacting them to see what I can get back other than $300 credits. To anyone who currently using or plan to use DigitalOcean, I believe you have better choices other than this company.
