Why Tech Debt Is Slowing Us Down & Eating Profits
A way to improving tech debt management
We all dread the moment that we’re told that the cost to make improvements to our software product will cost more than rebuilding the product from scratch.
Why are we not taking tech debt seriously? The issue is not small.
The global technical debt bill was estimated by Gartner to be $US500 billion in 2010 and predicted to be $US1 trillion by 2015.
Whilst it is common knowledge that tech debt is no good for the long term success of software development, it appears that we do not understand tech debt as well as we think. I’ve also observed this countless times in many of the places which I’ve consulted.
We should not mistaken adopting agility for being an unwritten rule for letting tech debt pile up…
There are a number of frameworks and methodologies, which stem from Agile, which are used widely today. But we should not mistaken adopting agility for being an unwritten rule for letting tech debt pile up until there’s nowhere else to hide it. Philippe Kruchten, PhD, writes that “many agile teams seem to believe that they’re completely immune to technical debt. Although iterations offer the opportunity to reimburse debt in a timely fashion, the opposite often occurs.” Often Agile is misunderstood for moving so rapidly we don’t have time to consider the cost associated with our decisions.
However, the reality is that we don’t give any attention or thought to tech debt until we’ve lost complete control of our project or gone over the project budget. We treat tech debt as either the the “poor cousin” that nobody really could be bothered to give any attention to or that we never take shortcuts and therefore we don’t have any. In my humble opinion, these are some reasons why it is easy for tech debt to get out of control:
- We don’t see and know enough about the negative impacts of tech debt to care.
- Tech debt is hard to “detect”.
- We don’t make time to understand the cost associated with our “shortcuts” (eg. hacking a good-enough solution together to just get the job done for today without thought for tomorrow)
But what if we made those IT project failures more visible? What if we could paint a picture of the probable future outcomes? Would we be more diligent in the way we deliver software and technology products?
The Accrual Of Tech Debt
“Every hack, work around, bad piece of code builds technical debt” — Ben Stopford
For those of you who don’t know, tech debt is the “interest” that we incur when we take “shortcuts” during the development of software and technology products. We are essentially trading speed and effort for time, eg. do it quicker by spending less effort now and incur some time later to fix or “refactor” the taken shortcut in the future.
Ward Cunningham, developed the tech debt (officially, technical debt) metaphor to help us think about the problem of poor software development:
Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.
This is no different to dangers of a line of credit that we have with credit cards. The convenience of tapping the piece of plastic allows us to instantly and effortlessly make a purchases which become difficult to track and manage because there are just so many. Hence, this allows debt to easily accrue. Whilst this debt may be part of a conscious decision, the invisible accumulation of debt may not be. The same thing goes for the several little decisions made along the way in product development in order for teams to continuously and rapidly deliver the high priority items which leads to accrual of tech debt. At some point, the debt is something we will have to repay in one form or another.
In practice, tech debt is accrued in a number of different ways. Here are a handful that come to mind immediately:
- Deferral or sub-optimal implementation of a piece work that just had to get done so it could be shipped asap to the customer
- When work is prioritised, there always seem to be cards that never make it out of the backlog.
- A manager or lead encourages the engineering team to “cut corners” in order to increase the speed of delivery.
- Code reviews are seen as a waste of time and, as a result, the reduced communication allows developers to make decisions without considering the implications.
- One or more developers not being methodical enough in the code they write.
I imagine that those who are involved with building technology products or software were familiar with some if not all the above. There is a much longer list in reality.
The real challenge is having a thorough understanding of tech debt which includes its impacts and consequences so that it can better managed. Otherwise, tech debt will bubble quietly like the lava in a volcano until impede the productivity and other aspects of developing your software product. Let’s take a look at some of these impacts and hence the consequences of tech debt.
Tech Debt Consequences
Cunningham described how “shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite”. Other examples including speeding up the implementation to be the first product to market or picking a familiar technology to get to the goal faster so that further funding can be secured. Hence, tech debt can be good in some cases. But the key point here is that the debt must be “paid back promptly”.
Tech debt can be observed through a variety of symptoms. These can manifest in different ways from defects to a long on-boarding process for developers, several “not-so-fun” tasks, complex working environments and feature development which is more difficult than necessary. After all, no developer wants to spend triple the necessary time doing a task. Thus, all these symptoms lead to slower and sub-optimal software development and has propagating impacts on the rest of the business.
Researchers have identified that tech debt has consequences on team morale, productivity, product quality and project risk. I have also repeatedly witnessed the consequences first hand.
Impacts on Team Morale
David Laribee explained that — “beyond the obvious economic downside, there’s a real psycho-logical cost to technical debt. . . this psychological effect takes a toll on team morale”. For example, product managers, designers and business analyst become demotivated and/or frustrated because they repeatedly design and test new features that never start to get developed because there is a continuous stream of “higher priority” work that keeps coming up. As a result, they become pessimistic about their contributions.
Impacts on Productivity
John Elm describes “code that is not properly maintained is inherently more brittle”. Hence, developers need to spend more time diagnosing and fixing the issues rather than implementing the enhancements that customers and the business are screaming for. Also from speaking with several developer, they all favour working on the “cool stuff” (eg. building new features, experimenting with new technologies, prototyping a new concept, etc) over the mundane refactoring tasks or debugging surprise defects.
Impacts on Product Quality
Everyone can see product quality suffers when your customers see a decline in their experience when using your product. But what they don’t see is the poor product quality that builds up through poor software development. Edith Tom describes that “when code is poorly written such that it is difficult to follow and easy to misunderstand, or when sub-optimally designed solutions didn’t consider all cases, it is a lot easier for defects to arise.”
Impacts on Project Risk
Uncertainty grows exponentially as tech debt accumulates. Timeframes to deliver new features and enhancements are blown out due to the random discoveries of the “unknown knowns” — phrase commonly used by software engineers/developers to describe situations or issues which cannot be foreseen. This increases the risk to shipping the product on time. Also, incremental debt is difficult to track especially when it’s “invisible” (eg. defects are considered visible).
Each of consequence mentioned is coupled in some way to at least one of the others which is why its important that we recognise that tech debt can easily spin out of control. If not managed correct, tech debt has serious implications on developing successful software products. But this problem is addressable. The first step is awareness and the best way to create awareness is to make it visual.
Visualising & Communicating Tech Debt
If we can’t see it, then … doesn’t exist.
It is the norm to hear and speak about tech debt in the abstract across the business and amongst product teams. Investors and stakeholders usually have a vague idea about how it effects their investment, if any. Everyone else talks about it at a surface level and every so often we hear about a software product getting re-written. But there’s too much talking in silos when really we need to be having a shared conversation. To have a shared conversation, we need to have a shared understanding which includes referencing the same details.
So, the first step to having any chance at better managing tech debt is to make it visible. From the “Technical Debt: From Metaphor To Theory and Practice” research, the landscape of tech debt was laid out quite nicely. We can also clearly see here that majority of the tech debt is “invisible” — “out of site, out of mind” classic.
The lack of visibility is a problem for everyone — the product development team, marketing, sales, customer service, stakeholders and investors. For example, the business wants to ship enhancements to Product X in less than 3 weeks but after two weeks during testing the enhancements, the product team stumbles across an unknown issue which takes a week to rectify. Now, the marketing campaigns are delayed, the customer service team receives an increase in enquiries, the sales team looks bad because they overpromised and the stakeholders and investors are raising questions.
Imagine an environment where an investor has visibility to how much debt would be gained by taking a “shortcut” for a tactical decision? Or that a stakeholder has visibility of what this could cost the business in time and money if he/she wants the team to move faster? Taking it a step further, what if we could simulate a series of shortcuts based on a strategic business decision to see what it might cost before we make the decision? What if the investor could see how much was “borrowed” on behalf of the investor to take the shortcuts last quarter in order to deliver on the strategic decision? This data not only allows everyone to have that shared understanding but its credible evidence that can be used to make a truly informed decision, especially in the fast paced environments where we simply don’t have time to work out these numbers each time.
Over the long term, a lack of discipline, lack of visibility and ignorance in software development is more dangerous than a credit card. Like a credit card, if we can’t see the accrued debt, then we’re happier telling ourselves that it doesn’t exist. So let’s start improving the visibility of the tech debt so that we can have an informed discussion about it.
Tackling Tech Debt
Now that we’ve established a shared understanding through visualising tech debt and a shared conversation around it, let’s take a look at the approaches to tackle it.
Secretly, we all know that tackling tech debt is a constant procrastination. New ideas or changes are continually requested and often assigned a higher priority, pushing out the planned tech debt items to tackle. This leaves us little time to consider the short term consequences of our actions so we just action. Hence, the long term consequences usually end up being a distant thought and “low priority”.
“Every action we take has a consequence, sometimes it’s best to consider the consequence before we take action.”-Pkarpicz
Research shows that “every action we take has consequences, sometimes it’s best to consider the consequences before we take action”. We have addressed part of this through increasing awareness and making all tech debt more visible, thus creating a shared understanding about the implications of certain actions taken. But there’s more.
Tech debt is dangerous when used as a blanket statement to justify why something cannot be done or is harder than necessary. Developers and product team members will often make such statements but it doesn’t help the other parties to truly understand the implications and how we will be worse off in different situations. I propose that we also consider the context, purpose and the intended decisions when tackling tech debt. For example, if we cannot articulate how we will be better/worse off by addressing a particular piece of tech debt, should we be spending time on it and is it really debt?
Measuring the number of defects, number of active branches, time-to-build, code coverage %, etc, is start but what does it help us achieve? What does it tell us? What decisions can we make with it? Keeping the number of defects to a minimum sounds good but how do we know what a good number of defects is good. The number of defects will also likely vary depending on the size of your software and the size of your team. Likewise, with the number of active branches. With the code coverage %, one might argue the higher the coverage the better. But we could have 90% code coverage but, from experience, a portion of the tests may not be written adequately and thus defeats the purpose of testing. Meanwhile, you’re coverage still reads 90%. Anyways, I think you get the point. To truly measure tech debt, we need to understand the context, purpose and, most importantly, the decisions we hope to make with the data.
Note that we mentioned earlier that not all tech debt is bad (eg. tech debt that is promptly reconciled over the short term). Hence, the measurement of tech debt and data created around it needs to be sophisticated than a simple count. To be continued in separate article.
The existing methods for tackling tech debt is a start but there is a lot of room for improvement. Let’s take decision matrices for example. They are often used to prioritise which tech debt items to resolve in a future iteration. However, extensive academic research explains the poor quality of using a risk matrix because of extreme rounding errors (aka “range compression”), inconsistencies and the ambiguity. This is no different to a decision matrix, if not worse because of added the ambiguity (eg. urgent, non-urgent, important, not important). “Ambiguity hides problems instead of facilitating the lack of information” says Hubbard.
Tony Cox, PhD, an MIT-trained risk expert, also mentions that the error of “range compression” makes a risk matrix literally “worse than useless”. Here’s an example of the “range compression” problem:
Whilst matrices appear to be doing something (eg. helping to make decisions), unfortunately they are misleading since we are given the wrong results. Hence, we’ve got to find a more reliable way. It is a necessity for any business developing software to be able to quantify their existing and potential tech debt in context of their decisions. Otherwise, how do we measure true success?
Productivity Is King
A survey in January 2018 found that 9 out of 10 companies are “struggling with technical debt, saying it was delaying software delivery”.
As we can see from the above survey, there is clearly a lot of room for improvement as companies do feel their productivity being slowed down by tech debt. It’s also important to remember that we should not be aiming to reduce our tech debt to zero or pay off the debt without understanding the context and why. Hence, we should focusing on finding more appropriate ways to improve tech debt management in addition to understanding the context and purpose of addressing the debt.
Through understanding the implications of tech debt over the short term and long term, we can better manage its impacts on team morale, product quality, productivity and project risk. There’s nothing worse than high churn because you have a poor quality product that nobody wants to work on because the tasks are boring and team members rarely feel like they are contributing to a useful product.
“Every decision has a consequence.” — Damon Darrell
Through knowledge and predictability, we can stay in control. To gain knowledge and predictability we need to be able to quantify, visualise and communicate tech debt in order to better understand the decisions we need to make both before and after the tech debt is incurred. If you knew that the action you took had a 60% chance that it would end in disaster, eg. if a certain decision cost your company double the initial investment just to fix, would you have made a different decision?
I’ll leave you with some questions to ponder whilst your organisation continues to acknowledge the tech debt (which happens to exists in every software product):
- How much is tech debt costing your business?
- How do you know when you’re tech debt is out of control?
- Does the gain outweigh the losses from incurring the tech debt?
- When incurring tech debt for strategic and/or tactical reasons, are you delivering soon enough to give you any chance of making a gain on our debt?
- What’s the debt ratio in your system?
- What is the rate at which your tech debt is accrued?
- What is the level of impact tech debt on your team?
If you would like the learn more about how to improve the way you manage tech debt, get in touch at firstname.lastname@example.org.
Learned something? Click the 👏 to say “thanks!” and help others find this article.
Hold down the clap button if you liked the content!
Clap 30 times!