I Deleted a Database in Production

And I had no backup

Aashish Nehete
Nov 3 · 3 min read

There comes a moment in every developer’s life when they must face this inevitable situation. Really, it’s a rite of passage for our community.

PANIC!
PANIC!

I’m writing this short post to chronicle the events that lead to this dreadful accident along with a few important lessons to learn from this fiasco.

Before we begin, a quick overview of my infrastructure. The infrastructure at WorkIndia is hosted on AWS and some of the terms reference to AWS Services, but this could happen to anyone hosting databases on virtual machines. This particular microservice was hosted on a single virtual machine running the API, a PostgreSQL, and a MongoDB server using Docker containers.

Here’s what happened:

Image for post
Image for post

6:35 PM

I try to SSH into the machine but due to unnaturally high CPU load, my connection is unsuccessful. At this point the usual solution is to reboot or stop/start the instance from the AWS Console panel. BIG MISTAKE!

6:40 PM

Rebooting doesn’t work.

6:40:01 PM

I stop the instance and give it a few minutes to boot a new machine and start the docker containers.

6:42 PM

I see that another instance with the same name has been spawned.

CONFUSED!
CONFUSED!

6:43 PM

Sudden realisation that the service has been configured behind an Auto Scaling Group (ASG). When I stopped the original instance, the ASG couldn’t find a healthy machine and it created a new instance from a very old launch template.

Its okay, at least I have the original instance and all its data. I’ll just change the database host settings in the new machine.

6:43:01 PM

Original instance is automatically terminated by ASG.

Once I stopped the instance, the ASG determined the instance was unhealthy and proceeded to permanently terminate it. This is the default behaviour of an ASG but an instance can be saved from such a fate using a termination policy.

6:44 PM

I realise I have no backup.

PANIC!
PANIC!

6.45 PM

I call in reinforcements, but its too late. The damage is done. All data on both the PostgreSQL and MongoDB was lost.

Recover some data?

My trusted collegues immediately got to work and devised a plan to regenerate most of the lost data using logic from another service. It took them not more than a day, but thankfully the data was recovered, more or less.

Lessons

Infrastrucutre is an error prone business. As an infrastructure engineer, I feel more of my energy is spent on creating fail-safes than actually trying to create a perfectly robust application. Things will fail, its our job to handle the failures gracefully.

Here are a few lessons I’ll take away from the above incident:

Never keep stateless and stateful applications in the same virtual machine.

At our scale, it is unwise to mix stateless applications like API servers with stateful applications like PostgreSQL and MongoDB which have storage requirements. Stateless and stateful applications play by different rules and therefore need to have vastly different failure strategies.

And the most obvious of all: always have a backup.

WorkIndia.in

Stories from the Team Behind www.workindia.in

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store