Listen to this story
Like most people who’ve played it, I love Tetris. I still remember playing it for the first time on a friend’s Nintendo Game Boy. Not only is Tetris one of the best games of all time, it’s an excellent analogue for technical debt. The impacts of technical debt are something I’m deeply familiar with — I deal with them every day.
I’ll also share a personal story of how my team and I reduced technical debt in some billing code, fixing a $1 million-per-year bug.
Within software companies, product and project managers (PMs) work with software developers to prioritize what code will be written and shipped to customers next. Finishing a Tetris row is like shipping a feature. Shipping a complex feature requires more rows.
Often, the business needs (new features, new products) will lead to trade-offs within the code (hacks, shortcuts) in order to ship on time. Or, changes in the product strategy will be incompatible with its previous design, requiring additional effort to either migrate customers or support both the “new” and “old” logic.
Scenarios like these create technical debt within the product code. A buried gap in Tetris represents technical debt.
All code has technical debt. That’s normal. You can keep playing Tetris with a few gaps.
Too much technical debt will prevent features and bug fixes from shipping in a reasonable amount of time.
This isn’t a problem that can be solved by adding more developers or, more dramatically, replacing your existing developers. It’s called technical debt because, at some point, it needs to be paid down.
Paying down technical debt keeps you competitive. It keeps you in the game.
Similar to running a business, Tetris gets harder the longer you play. Pieces move faster and it becomes harder to keep up.
Similar to running a business, you can never win Tetris. There is no true finish line. You only control how quickly you lose.
Similar to running a business, allowing too many gaps to build up in Tetris will cause you to lose.
The $1 million bug
Not long ago, my team was tasked with updating billing and invoicing logic in our product code to support new pricing plans, a new payment processor, and an improved billing workflow. Some of the details were still being decided by the product team, so we used the spare time to do a deep-dive into the existing code. This improved our understanding of it so we could give accurate estimates of our ability to complete the upcoming changes.
The basic purpose of the code we studied was to review every customer’s account, calculate their bill, and send it over to the invoicing API. It had clearly been written with care and good intentions — it wasn’t so much messy as it was inflexible. It was a monolithic function. There were no tests. There were very few logs. There was barely any documentation. There was some unexplained randomization. It had been written over five years earlier by one of the co-founders. The only changes since then were from an early employee who was no longer at the company.
Was it really a problem? Invoices were going out. The company was making money. There was no indication of an issue. All of this could have dissuaded us from a refactor, but we knew big changes were coming, this function wouldn’t scale to our needs, and we could move faster if this piece were simplified.
We refactored the function within a single sprint and added some much-needed logs. That’s when we discovered what we had actually fixed. Someone from our accounting team stopped by our desks to ask why the number of outbound invoices had unexpectedly increased. The old code had been silently timing out and some customers’ usage wasn’t being tallied for the invoice. That weird randomization? It hid any patterns that might have alerted us to customers who weren’t being billed.
When we ran an estimate, the missing invoices totaled over $1 million per year.
Paying down debt doesn’t always pay off
While this story is completely true, paying down technical debt does not always have such a dramatic effect. We got lucky.
I wish I could offer sage advice on when to pay down technical debt. Unfortunately, the answer is: It’s complicated, and it will always be a balancing act. You can have the cleanest, most well-tested code in the world, but you may also have no paying customers. Conversely, your company could be running some really messy code that delights customers and makes money hand over fist.
Either way, product owners and developers should share an understanding of what technical debt is. They should also know that it cannot be avoided forever. After all, like Tetris, you can never win at software.
Special thanks to Thomas Lubitz for inspiring this article and granting permission to use this analogy, which he presented in a lightning talk while playing Tetris live.