Respond to bugs like a boss
TL;DR: install Rollbar, win devoted customers.
The day had started out well enough: the coffee pot was full upon arrival, the sun was shining, and I had just deployed our latest and greatest to prod. Then it happened: a confidence mangling collision between an obscure software bug and one of our biggest customers. Ouch.
Within seconds the entire dev team saw the exception hit their inboxes. In the email we had a stack trace, request params, the user’s email address, and all of the HTTP headers. We had everything we needed to turn this devastating failure into a win—but, we had to act fast. We immediately got to work: hotfix branch, bug fix, push to prod, and an email to the customer with something along the lines of:
Hi Jon,
We noticed that you ran into an issue while trying to switch between views on your dashboard. We’ve fixed the bug, feel free to jump back in and try again.
We really appreciate your patience, and please let us know if there is anything more we can do to help.
-Mike
This entire process took about 8 minutes. We didn’t waste time SSH’ing to EC2, we didn’t need to scan the logs, we didn’t have to grep on the request ID to figure out the user ID, then search the database to find the user’s email address. Most importantly, we didn’t have to wait for the customer to tell us something had gone awry. We saw it immediately, fixed it in a matter of minutes, and received the following email in response from our user:
WOW! You guys are ON IT!
Sincerely,
Happy Customer
As a user of something like twenty-thousand websites over the past fifteen years, I can tell you that I have received customer service like this precisely zero times, ever. We built more confidence with our rapid response and personal touch than we ever could have if the view had just worked the way it was supposed to.
Although we are never happy to report that our attempts at building bug-free software have come up short, try as we might, the occasional bug is unavoidable—I will spare you quotations from Djikstra, Knuth, Voltaire, and Homer Simpson supporting this point, but it’s true. Besides, the love that we’ve received from customers by handling live bugs this way has been awesome:
“Seriously?!? How did you even know I had an issue?”
“Love the support!”
“OMG that’s great!”
We are frickin’ rockin’ it on customer service. Go us!
And… while I would love it if our team could hog all the credit for this awesome workflow, the truth is, we’ve had a secret weapon: Rollbar.
For less than a bento box per month we have all of the information that we need to rapidly diagnose and discover issues, prioritize fixes, and ensure that our software quality stays high. Knowing that we have a reliable and immediate feedback loop lets us spend less time obsessing over 100% test coverage and more time building features that our customers actually care about.
Nowadays, we release new features at least once a week and rarely have a significant bug that survives more than a few minutes.
Errors are a fact of life: they are part of a beautiful feedback loop that teaches us how to be better, and we get better a heck of a lot faster when that feedback loop is in realtime.
P.S. I do not work for Rollbar, I pay for Rollbar, and so should you.
