Technical Debt. Time to pay up.
Alex Katrompas comes from an extensive software engineering background heavily focused on eCommerce, business intelligence, and artificial intelligence systems, where he has dealt with his fair share of technical debt. As Vice President of Technology at Granify, Alex leads a team of engineers, data scientists, and QA specialists in both the United States and Canada. He is currently focused on aligning Granify’s engineering, sales, and product as we work towards our ambitious mission.
“Technical Debt” is a phrase a lot of us hear and use, but it’s likely that it is only partly understood. It’s understood in one sense by the technical teams as the extra work and pain to get new features done. It’s understood by the business as inefficiency and rising costs. What is it really? Very simply, technical debt can be defined as the engineering work you owe yourself because something caused a gap between what you should have done versus what you did do. That gap can take the form of missing features, rigid or poor design, outsourced code that should have been done in-house, etc. The “interest” on this debt is the maintenance and extra work needed to service those decisions. In this post I will explain how to identify debt, identify the major sources of debt, and how to get out of debt.
Do you have technical debt?
How often are you hearing something like “We can’t build this unless we rebuild that?” Each time you want to start a new project or add a feature to your software, do your engineers tell you that you have to build or improve something else? If so that’s debt (and bad design). At some point you skipped something or designed something poorly and now you have to repay it to move forward.
A sign of mounting debt is that your time to complete similar tasks keeps going up. Six months ago adding a new product to your online store took 3 days. Now it takes 5 days. The difference between six months ago and now is that someone is paying off “interest” on debt to keep producing the same level of work.
The biggest red flag of accumulating debt is that you need to implement or expand a software maintenance team. If you have people strictly dedicated to maintenance, keeping the lights on, and bug fixes as opposed to adding new features, and that maintenance team is growing, you probably have a debt problem.
This all comes down to one simple question; Is it taking more and more time and effort to produce the same output? If your answer is yes, you are probably accumulating technical debt.
If you’re building good software, it should take less and less time to produce more and more, not more and more time to produce less and less.
Where does technical debt come from?
Management. It’s important to realize a very harsh reality, technical debt does not usually come from your technical teams. They are the ones that create the debt through their actions, but it’s almost always because they were directed to do so by management (usually inadvertently). Technical teams are like the credit card and management is the guy holding the card ready to buy something he can’t afford. The most common place for producing technical debt is a business decision that places a constraint on the engineering teams which requires they create a debt (not always a bad thing — see below). For example, dates are cut or features are added without allowing time to design and build properly. Imagine a particular piece of code is built quickly and hurriedly cut-and-pasted into multiple places. That code is marked with the to-do comment, “Merge these into a single function/library/object ASAP.” There is your debt and you will immediately start paying interest in higher maintenance costs. This happens all the time but what doesn’t happen all the time is management’s awareness of this very real and expensive cost.
Process. Technical debt may be built into your processes, either knowingly or unknowingly. Many times the act of creating technical debt feels like you got something for free and that immediate gratification feels so good, you make it a process. As an example, a senior leader with whom I worked initiated a software pet-project and “borrowed” an engineer to get it done. The leader wanted “quick-and-dirty.” The two worked together very iteratively and produced the product in a fraction of the time it should have taken. There were no checks and balances, there was no design or thought to what is being built or who would maintain it, there was only “get it done!” Fast forward three years. That “quick-and-dirty” tool is still in production and is still used by senior leadership. The person who built it has long since moved on, and now it’s up to the developers left behind to maintain it. That tool is the pariah of the code base. No one wants to work on it and it takes twice as long to fix or enhance because of poor engineering. That senior leadership is frequently furious and frustrated at the time and expense it takes to update the tool but the connection as to why it’s so frustrating has been lost. Because of that, the quick-and-dirty process that created an initial success was made the standard operating procedure for all development. Living on borrowed productivity and servicing technical debt is now the status quo for those teams and technical debt is simply built into every project, usually unknowingly.
Bad Engineering. Engineers aren’t off the hook; technical debt may simply be from bad development regardless of management decisions. If developers aren’t architecting properly, aren’t documenting, or aren’t adhering to good modern coding standards, they might be building a mountain of debt with every project. One of the most common scenarios where this occurs is with offshore development. The debt accumulated by off-shore development in the form of bad architecture, bad code, poor quality, and lack of documentation can be massive. This is a large topic in its own right, but for now we just need to understand the technical team itself could be the source of debt on its own. This debt is possibly the worst of all, because it’s almost impossible to see from the outside and by the time you realize it’s there, it’s usually too late. This debt almost always takes the form of bad architecture and design, but it’s also almost always wrapped up in bad requirements, bad leadership vision, and bad hiring.
How do you avoid and pay down technical debt?
Understand technical debt is a real thing with a real and measurable cost. If you are a senior leader of a development team, analyzing the technical debt of any project should be standard operating procedure. We regularly talk about the time to complete projects and the man hours needed, but we almost never ask, “What is the debt?” or “What will we have to maintain later?” So first believe debt exists and then go out of your way to look for it. It’s always there. If you don’t see it, there is your red flag, so look harder.
Treat technical debt as a real cost. Do not fool yourself into thinking you can buy-on-credit now and you don’t have to pay later. Every time a date is cut or features are added ask yourself, “What architectural sacrifices are we making to hit our goals?” Then ask, “What will be needed to pay for those sacrifices in 1 month, 6 months, 1 year, and 5 years?” If you are going to go into debt, count the cost as a real cost, factor it into your future development, and make sure you are willing to pay that cost.
Never go into debt if you don’t have to. Technical debt is not always a bad thing. Like monetary debt, sometimes you smartly borrow to invest and then reap a higher reward later. If you have to cut corners to beat a competitor to market, and there is no other way to do it, then borrow against your technology and move the business forward. However, if you are simply cutting corners to make artificial deadlines or because you don’t know better, you need to think twice. Just like with real money, don’t borrow unless you know you need to and you have real plans to pay it back. Then execute on those plans to pay it back.
Make your development manager your “loan officer.” Assuming you have a good development manager, this is the only person that can accurately see your debt and tell you how to stop accumulating it and how to pay it down. It is critical you put the right person in this position and you listen to them. Only this person can see across the teams, see the interactions inside and outside of the teams, and knows all the pressures coming from multiple sides. Have this person do a regular debt analysis and present it to senior leadership. Technical debt analysis should be a required role in a development manager’s job.
Make a point of paying down debt as a matter of course. Never simply service debt forever without paying it down. In every project that is not an emergency, pick a debt point and pay it down. If you’re about to enhance a system, find a debt point related to the enhancement, add 10% to the project and fix it now. Be proactive. Every project touches some debt. Find it and pay it off regularly.
There is always free cheese in the mouse trap. It’s a steal!
Always remember, nothing is free. If you think you are saving money by off-shoring, think again. If you think you saved time by cutting a date, think again. You did not save anything, you borrowed time and money. This doesn’t necessarily have to be a bad thing. That may be the viable strategy to run your business. Just make sure you go into debt with eyes open and with plans to pay back, because one way or the other, you will.