The worst production bug ever
My worst production bug ever.
After the calm comes the storm.

Yesterday, I felt like I had everything figured out — with a good night’s sleep for once I felt invincible.
Today I tested the theory that sleep is all you need.
Waking up at 10.30am I missed BJJ. But it was ok because I’m sore. I’ve told myself that I’m still in adjusting to the new country so I’m letting myself be more lax.
But things started to go wrong.
First, my database hit the max number of connections. I don’t know what this means exactly but it is bad. It meant that no one could connect to the database.
For the first time in the last two weeks, the app became unusable.
To fix this, I attempted to upgrade the database to a more powerful one to gain access to the support team. Hoping they could shed some light on the matter. Which meant I had to upgrade from the free plan to the $9 a month plan.
Upgrading caused the database to reset. Which meant for the second time the app was unavailable.
This caused the app to crash because it didn’t have a database connection.
I haven’t got around to setting up monitoring or restarting of the app if it crashes so it remained crashed until I realised it was down.
I enjoyed a lovely Pad Thai and papaya salad, unaware of the fire that was consuming Streakoid.
The app was down for over an hour.
But I got back and redeployed and the app went live.
Around this time, I had my user interview with my ex colleague coming up and I wanted to impress him. This is what I left my job for.
I tested the app to prepare for our interview, trying to register a user. It failed. You can’t register at all. You just end up freezing on the registration page.
Thinking it was because of the feature I had introduced earlier that day, I quickly changed the code and deployed a fix before the interview.
I sent my colleague a link to the broken version of Streakoid.
He attempted to register and as expected the app froze. However, he’s a front end dev so he could see that the network request passed.
So he attempted to log in. He was successful.
But then the worst thing that could have possibly happened… Happened.
Streakoid presented my ex colleague with my streaks.
Meaning he could complete my tasks, delete my streaks, and view information that could have been personal.
This was the biggest bug I’d ever had in Streakoid.
All my brain could think of is how much of a dick he must think I am for quitting my job.
Luckily, when he refreshed, he was his account.
I continued the rest of the interview, trying to hide my panic.
When it was over, I immediately dived into the code.
It turns out the API was returning any user when it didn’t find one. Because it didn’t know about his user Streakoid gave him a random user. I was that random user.
I fixed that and deployed a change as quickly as possible.
With only two active users, it’s not the end of the world but even then I have a responsibility to make sure stuff like this never happens.
This happened because of a lack of front end tests. The API functioned actually as it should have. It was how I was using the API, which was the problem.
I then spent the next two hours trying to fix the registration. But I didn’t crack it. I called it quits at 22.00pm because I still had to write the blog post to maintain my 12 day streak.
This was all made worse by the fact I’m running out of deployments for the month. I can’t afford $49 a month at the moment to upgrade. Meaning I can’t afford to keep deploying changes.
It will be difficult to top having the login functionality sign you in as another user.
This has brought a lot of self doubt, stress and anxiety, and it looks like a late night is on the cards. As I still have nonfiction reading and yoga to do.
But these things will happen when you’re building your own software.
Instead of throwing in the towel and panicking, I just need to keep attacking these problems.
These experiences have been painful but valuable. They’ve pointed out that:
It’s time for:
- Monitoring of the server
- Front end API tests for the login and registration flows.
- QA after every release.
Come at me Streakoid, either you’re getting built or you will kill me.
