Hindsight is…please don’t make me look
Jeff Bezos admits to Amazon launching their service with a pretty horrific bug. For some (presumably) small amount of time, you could order a negative number of books. Amazon would credit you with the amount of the purchase.
I think all developers can look back on something they shipped with version 1.0 of their software with a bit of horror and a smirk. Some strange dependency. Some complete oversight. Something no one ever thought to test for.
When we first launched DoneDone, we integrated a third-party payment service called Zuora to handle our monthly recurring billing. Back then, we billed customers on a per-project basis — A $15 base fee for three projects, then $2 per each additional project per month, with a cap at $99. Each time a customer created a new project, the app made a connection to Zuora to update the project count so the customer was properly billed at the end of the month.
However, DoneDone also connected to Zuora each time a user logged in. It would then grab an account’s current monthly cost — based on project usage — so the amount could be shown if you were about to add a new project. By the way, showing someone how much they’re going to owe each time they add a project isn’t exactly the best way to increase lifetime customer value either. We’ve since changed our billing strategy completely.
Here’s where we made a horrendously silly mistake. If Zuora was down (which, like all services especially back then, was more than we’d like to admit), our application went down with it. We let the connection exception bubble up. That meant that if you were logged out, you couldn’t log in. If you were logged in, you couldn’t add a new project.
Eventually, we pushed all the data and logic needed to derive the information Zuora was giving us into our own data store and codebase. In reality, this was some simple arithmetic. It was a few lines of innocuous code on our end.
In looking back, I think we had made the initial decision to push off this work to Zuora so we had a single “source of truth” on billing. But, the math was simple. The redundancy of the billing logic wasn’t a huge deal. Certainly, not as much as a hard dependency on a third-party service‘s uptime. We just made a really bad, silly trade-off. A rookie mistake.
Like crediting an account for a negative quantity of items, it’s a mistake you’ll usually make just once.
