Not A Technicality: Exploring The Total Cost of Code Cost of Ownership

The views expressed here are my own and in no way represent those of my employer, Accenture. Moreover, they do not necessarily represent my behavior given contractual or cultural considerations. This blog is purely intellectual; the ideals expressed here may require strategic consideration. Proceed with caution.

I’m climbing up on the soap box, so prepare your eyes for rolling. We don’t take tech debt seriously enough. If we did we would pay it off, but because it typically affects us at such a small scale, we take it for granted. Rather than clean up the mess, we learn to ignore it. I don’t anticipate anyone experiencing enlightenment with this essay. In many ways, it’s simply a rant. After all, monetary debt seems to be held leisurely and a certain amount might even be considered healthy, depending on one’s growth strategy; why care about a less tangible phantasm? Nevertheless, rant I must.

The Principal

There’s an old saying: “if it ain’t broke, don’t fix it”. Whoever first uttered that was indeed brilliant, but they were no technologist. Always look for improvements. Sure, not all will be cost-effective, but also not all will be cost-prohibitive and some are painfully obviously worth it. For one, stop repeating yourselves. If you aren’t, good for you (and thank you), but you might be in the minority. I’m not suggesting you compress your code into the true minimally repetitive product, but I am suggesting you avoid e.g. 10 lines that are repeated (perhaps up to specific variable reference) 10+ times; at least when you find yourself adding that 11th+ iteration, refactor for goodness sake. From now on I will derisively call this anti-pattern “the scenic route”, as that’s exactly what it is: longer and more nothing at which to look[1].

Another proverb about cat mutilation may be more immediately relevant here. There are indeed many procedures that will produce expected behavior, but regardless of whatever any manifestos[2] say, don’t equate working code with good code. Go ahead and ship it, but don’t pretend it’s your perfected prize possession upon which it is impossible to improve. Once it works and you enjoy some return, reinvest that return into improvements in efficiency: refactor, reduce, standardize, modularize, templatize, etc; whatever keeps you from paying the same price for the same code in the future. If you pay the same for the same code more than twice, you are a sucker.

Once your code is DRY, make sure it is readable. Documentation is important. Good, accurate documentation is invaluable and hard to both generate and maintain. I always promote self-documenting code[3]. It sounds cheesy but you really can tell a story with your code; tell us why you’re creating a counter, or storing the result of an API call, or referencing cached data. It’s not like you pay for any extra bytes of characters in variable names at execution time. This doesn’t have to be perfect, but if everyone feels empowered to improve things, it can easily be improved over time. And there’s no danger of missing stale documentation as code evolves since broken references will always be made obvious (hopefully at build time, but certainly at run time). If you find this too cumbersome (or you just don’t like the approach), choose whatever technique is appropriate for your team. There are plenty of tools that simplify the generation of structured documentation. If your team has proven trustworthy and effective, let them self-organize; if they’re lackadaisical, prescribe and enforce. But don’t just forgo for the sake of your precious deadlines.

And for the love of all you hold dear, lint your code. There are plenty of linters for most popular languages[4], so there’s usually no valid excuse for not running a linter. Preferably, this should be done locally, as part of an SCM pre-commit hook, but it must be done before the code is allowed to be merged to a commonly-developed-upon branch or trunk. I won’t linger on this, as it is not something I think even deserves an argument. Just do it.

From what I’ve encountered in the wild, a major indicator of an unhealthy, poorly-adjusted development team is their approach to commented-out and unreachable code. We typically refer to this, collectively, as “dead” code, but as in the A Song Of Ice and Fire series, “what is dead may never die”. So-called dead code is far more insidious than it seems. Aside from some obvious inefficiencies (e.g. the extra bits it requires to store or transfer), the cost in developer hours is almost certainly[5] some monotonically-increasing function of the number of developers engaged in the project and the dead code itself. Even with notes and warnings in comments, whoever comes across the dead code has to discover for themselves that it is irrelevant. Additionally, no one wants to remove the dead code (despite this brand new thing called source control), for fear it holds some future use, historical or sentimental significance, or out of pure uncertainty for its existence in the first place. After all, if I didn’t write it and it’s not my explicit task to remove it, why bother?

Bother because you are not alone. Bother so the next dev doesn’t have to. It adds up, and whatever diminishing returns (through familiarity) are experienced, they are done so individually, not extending to new members of the team. Don’t worry about losing whatever magic is in the hidden spell; it’s stored in SCM[6]. Don’t worry about pissing off the dev that removed it; in fact, you should ask them why they left it there for you to waste your time on. Don’t claim absolution of responsibility, for you are now complicit. You need not be the one to fix it, and it may not need to be fixed immediately; document and communicate appropriately. If the business stakeholders choose to deprioritize its resolution, at least you exposed the debt; and you can ensure they share or take whatever blame (if nothing else, psychologically) that may arise from default.

Worse than dead code is live wasted resources: those services that are created as proofs of concept or as part of an experiential initiative. A shrewd client once intimated their disbelief that resources needed temporarily would always be fully cleaned up, guaranteed, once created. Frankly, it’s purely a function of time and discipline, but also as frankly, I’ve met few developers, let alone full teams, that enjoy “enough time” or exhibit “enough discipline”. Beyond the obvious “try not to create a bunch of shit willy-nilly” takeaway, I think an appropriate response is to strive to prove this notion wrong: to have the discipline; to make the time (which, speaking of, teardown should be magnitudes faster/easier than instantiation, unless you’ve instantiated a tightly-coupled knot of components).

At the risk of further devolving from rant to afternoon special in which the world is just one positive something away from everyone’s individual ideal forever, we really should do better as an industry/trade. I am not sure there is a perfect, one-size-fits-all approach to establishing discipline, but perhaps the most obvious commonalities are repetition and consistency: do the thing, do it again the same way, continue doing it the same way. As technologists, we live standardized iteration. That we don’t employ it reflexively outside of scripts and state machines is frustrating, but as I am including myself in the “we”, I fully relate. Treat yourself like a cron job and set up a schedule (e.g. in your favorite/company-mandated calendar) for sporadic activities like DRY review or no-longer-needed instance cleanup. Establish a checklist for instantiation that includes tracking considerations; require its completion as part of the task’s acceptance criteria. Certainly automating any or all of this is both possible and arguably preferred, but don’t shirk discipline for lack of automation.

The Interest

If the proper tools, techniques, and rigor are all introduced early, they will be easier to maintain[7]. At the same time, weak links in the toolchain may become more apparent if they emerge early in development, when folks are still elbow deep in code and engineering puzzles and you aren’t staking your business on anything, yet. It provides a level of assurance and collective responsibility that may otherwise be monopolized by a powerful (and precarious) few.

Moreover, and more tangibly, hours planned upfront will save hours scrambled later on, and with much less risk. You who have been summoned to your laptop during a well-communicated and planned-for PTO because someone flubbed resolving a merge conflict and now all the builds are broken; you who have spent days drowning in logs “troubleshooting” an “intermittent bug” only to find someone used the wrong type (“but the tests were all green!”), you feel this. Pay it off before anyone gets hurt, while it’s still all fun & games and red builds don’t make anyone too sad.

The fact of the matter is, while you may not put your most senior developer on every hard problem, you’re probably not putting the newbies and scrubs on them, either. At the same time, most of the problems that take a normal development team the longest to solve are not hard problems. That means you are allocating your workhorses to long hours solving easy problems. The more such easy-but-not-trivial problems you have, the less you can allocate your workhorses to…well, work.

The longer you wait to solve these problems, the longer they take to solve (at least up to some Poisson distribution), etc., etc. Just like interest, it compounds and the rate at which it does so may only be measured empirically, after the fact, when dead code obstructs you, the scenic route becomes tedious, or a cut corner trips you up, however momentarily.

Is That All?

Before I wrap up, a word to the business stakeholders: demand accountability! Request visibility from your development teams and invest in tools thereof if it isn’t as simple as providing view access to existing dashboards (it may be this easy, depending on your toolchain). Set goals on cleaning up tech debt and bake in healthy habit-forming tasks into all new development. Understand that the total cost of code ownership goes beyond KTLO tasks and it can compound over time. Really measure your tech debt in dollar amounts and penalize non-compliance at the business owner level, as you would other monetary inefficiencies. I think a viable option in many cases is to retire and greenfield-replace workloads with sufficiently large sums, but one requires sophisticated awareness in order to make such a significant investment (one might musingly think of the rewrite as “refinancing” your tech debt so it’s easier to pay down).

If your code has low repetition, it will have a smaller footprint and lower overall cost of ownership. The haystack will be smaller when searching for a needling bug. Sweeping changes will be quicker to enact and less error-prone. If your code is self-documenting, it will obviate the flow of logic. This makes it easier to onboard new contributors; less likely for easy-non-trivial problems to occur and quicker to diagnose and resolve them when they do; promotes cleanliness, efficiency, and clarity. An organization that pays down its debts avoids inopportune fire drills and protracted war rooms. It lowers its risk and, over time, its total cost of code ownership.

Technical debt was never a euphemism; it was always an accurate description. The more tech debt you hold, the more you pay. The cost rises very much like compound interest. There’s a potentially wild variation to this “interest rate”, and there are certainly diminishing returns to ensuring one holds absolutely no tech debt, but you can control the principal and you can often significantly influence the compounding rate. Choose a healthy tech debt profile. Beware of too much risk and please pay it off!

Footnotes

[1] Ok, sorry for the shade. If I’m out for a drive or stroll, the scenic route is preferred. We’re talking about the world of business, not a leisurely daydream. Time is money!

[2] I’m personally quite fond of the Agile Manifesto (https://agilemanifesto.org/), but it is not a silver bullet with which to strike down all our enemies (who are werewolves for some reason?). If speed-to-market be your top priority, let me not block your way, but hire someone to clean up the broken china your bulls left when they disrupted all that market.

[3] I don’t always do a great job of implementing this, but it is an ideal I hold dear.

[4] If you’re so inclined to target an esoteric or custom language for whatever reason, then write the linter yourself, damn it!

[5] I’m being extremely fast and loose with the term “almost certainly” especially since I don’t have nearly enough data for even strong anecdotal evidence; where I lack in rigor, I’d like to think I try to make up for in showmanship.

[6] A best practice might be to tag a commit or cut a branch with the dead code and name it as intelligibly as possible; maybe include a link to this tagged commit in the message of the commit that removes it. This assumes git, but the idea extends to other systems.

[7] They do risk, however, becoming taken for granted. E.g. if you always believe all of your code is required to pass a linting step, you may not realize if this one day becomes untrue: something is introduced that falls outside of some file RegEx pattern, and developers are so used to writing really high quality, style-adherent code that they miss a subtle violation never picked up; while the example is certainly contrived, many of us have experienced something similar. As with most things in life, a healthy balance must be struck between trust and skepticism of one’s own tools and techniques.