Building For Bugs

Long’s Peak — RMNP

I write buggy code and I’m betting you probably do too. Sometimes these are bugs which are easy and painless. Sometimes they require apologies to clients. Sometimes they require migrations. Sometimes they push decisions into product which are long lived like those scars we pick up as children.

The first three categories I listed above aren’t of real interest. For those there’s always a clear path. You have things in place like unit, generative, and integration tests. You have QA and an actively involved client base. In most cases, we’re able to protect us from ourselves.

Bugs which stay with us aren’t an update statement without a “where” clause, or a weird edge case in 2014 because Cairo had four daylights savings times. It’s an honest mistake. A decision that seemed right at the time, but not now. Now you’re backed into a corner and the option people are legitimately floating about is, “what if it’s not a bug, but it’s a feature? Sure, the timestamps are in GMT and the display is in EST, but that’s where most of our clients are! If they have any trouble, they just add or subtract 5 hours, what’s next in the sprint?”

Bugs happen. Product altering, long lasting bugs happen. So what architecture do you have in place for them? How is your system built to avoid bug scarring?

For libraries, dependencies, SDKs, APIs, and even the very OS you’re on, we use versioning. This is an architectural decision that is beautiful in its simplicity. It’s something so integral, that if you don’t do it, fellow programmers will not only laugh at you, they’ll avoid your product like the plague. But so often we forget to bake this in from the beginning.

A good rule of thumb for whether to version something or not is whether it’s something you’re expecting your clients to build off of. The color of the buttons in your UI is most likely not something you’re telling people to build other applications off of, but the naming convention you use for columns in an ETL system sure is. It’s extra work to have to employ versioning for every portion of the application, sure, but the contracts we make with our clients are binding whether you have documentation for it or not.

We think of how to make the changes to our data structures idempotent. We think about how to make our code safe across threads. We think about how to make our distributed systems predictable. We think about how we replay user actions, but when we don’t think of how to make our interactions with our clients resilient in the same way, we regret it. And if you’re thinking that one or two changes to how your product works is no big deal, think back to that decision the next time you’re brought in on a brownfield app. It ended up that way for a reason. The code base doesn’t suck because it’s PHP.

If you end up in a situation where you and your team are thinking that you would have done something different if you were able to start over from scratch, it might be a good sign that it’s time to invest in a get out jail free card and invest in versioning/planning for it to happen again. Then again, scars are a great conversation starter.