Making mistakes as a programmer
A few months into my first job as a developer, I was tasked with updating the content on the website at a charity. There was no CMS. I had direct access to the database. At my first attempt, I copied the new text into the content field of the GUI for the database that I was using. I hit save. The entire table went blank. Hundreds of pages of content had disappeared in an instant.
I closed and re-opened the GUI client to make sure this wasn’t some kind of rendering issue. It wasn’t. All the data was gone. I’d only just got this job, my first job in this industry, and I wanted desperately not to lose it. I didn’t want to be the one that had deleted all the content on a pro-bono charity project. I knew the owner of the company would have to be the one making the apology on my behalf and didn’t want to put him through that.
There were no backups. There was no staging environment with a recent mirror of the data. The change would have been instantaneous, so I had to admit my mistake straight away so we could pre-empt the complaints we would get.
“There’s something I need to tell you” I said. “I think I’ve accidentally deleted all the data at the website you told me to update”.
“OK, so what are we going to do about it?” my boss at the time asked.
“I’m not sure” I replied. “Give me an hour to come up with something.”
I racked my brain trying to figure out how to get this content back. The worst case scenario would be to get the client to type it all out again. Since presumably, someone would have had to insert this content in the first place, someone should have the data in their inbox in some form or another. It turns out that everyone who’d worked on the project so far had left.
About half an hour later, I realised that a lot of the site’s content would be in my browser cache. The team had been looking at it recently for general QA, so parts of the website would be in their caches too. I made sure no one visited the website and went through their caches one by one, retrieving the website data as I went. By the end of it, I had managed to restore the website data to how it was before and the client hadn’t even noticed.
I was extremely lucky that I’d happened upon a way to restore the content. There were a number of things I took away from this experience that helped shape how I do things now.
Firstly, when the shit hits the fan it pays to be open and honest about the situation with everyone who will be affected. They’re going to find out anyway so it may as well be from you, as early as possible. That way they can work with you to minimise the damage.
Secondly, always be prepared for the shit to hit the fan. If we’d had backups this situation would have been much easier to recover from. If I’d taken just the one backup before I did the update I would have covered myself adequately. While it’s good to avoid failure in the first place, it’s much more important to make recovery from failure easy.
Lastly, to learn from my mistakes and act on the lessons. Based on this incident I implemented backups for all of the websites that we handled. I stopped using GUIs to access the database and learned to write SQL. Instead of updating our website code by uploading new files using FTP, I instead taught myself to use version control, the unix command line and other tools to help update websites in a less error-prone way. These are table stakes for work as a web developer.
