Two years a cloudgineer: technical debt limbo

Like a couple big project cycles. Then optimism retreats into a clenched tuchus: technical debt. Limbo.

Metaphor for this ish from where the lotus grows.

Here’s one story from my second year a cloudgineer, 2015. We started rebuilding a core product piece — Search — and methodically moving between hosting providers: Websolr > Algolia. Then, shortly after breaking ground, there was a shocking revelation from The Websolr Team.

Oh, massive breaking changes. Well, I guess the migration is now kinda urgent. We have two months. Let us rejoice.

Websolr’s shock deadline burdened us with unintentional tech debt — this wasn’t gonna be about making a mess of things because someone sucks at their job. We were about to make tradeoffs. Card houses for speed.

Unintentional debt is a little bit sad. Our ensuing drama after this email might be familiar to you: we had ruby heads writing hefty amounts of JavaScript (“Don’t worry, we have an Object.keys polyfill”); we integrated a third-party gem to wrap Algolia’s ruby API that was not terribly well-designed. (With time to think now, we’re rewriting a quite simple and flexible Algolia middleware all on our own like grownups.) We made quick assumptions about user experience and focused a little too much on copying LinkedIn vs experimenting and exploring with our stakeholders and crafting a more proximate UX to how scientists need.

Wait, we were supposed to refactor that

Our wonderful palm-size team taking on an industry. Heh, it’s brutal. The brutal of a young startup finding footing: the short-term debt which lays around and metastasizes into “medium-term” debt. Like, paying down the search “short-term” debt took over a year. And it was a year before we even kicked off the refactor to begin with.

I freakin love working for ambitious companies, where the forward rhythm is more like explosive lurches and leaps and every developer is responsible for at least one core feature. (At least.) But the flipside is that it’s a bit maddening to look out the rocket glass down upon an earth-sized backlog of 200+ tickets in tow. It’s a painful mixed-bag exuberant anxiety knowing your users can’t do a simple opt-out of your deluge of product email yet.

That feature is coming, I promise. Lo siento.

I like to imagine at large orgs like LinkedIn or Netflix, with so many human bodies peering into glass, that crunch mode penalties incur only short-term interest which is paid off quickly and there’s a wonderful rhythmic forward motion into new greater futures and features. It’s nothing like our situation. But I know deep down that is not true. The whole industry leans on these narratives and heuristics and ontologies of startup-like plight of wading in this unfulfilledness and incompleteness ad infinitum — you were all startups once that grew too fast because growth.

Ish is in the DNA of the software build process. It’s just like a reality of “oh yeah, shit is actually NEVER gonna be done.” Let’s all relax into that. Slip into that assurance. That’s partially why we do agile.

The ledger of our debt. It’s a spectre of our industry. Living close to the machine, there are prices to pay. Particularly and emotionally, upfront:

“In the regular world, ‘legacy’ has an aura of beneficence…A gift. An enrichment. A patina of age, but good age — venerability.
In computing, however, ‘legacy’ is a curse. A legacy system is a lingering piece of old junk that no one has yet figured out how to throw away. It’s something to be lived with and suffered. The system is unmodifiable, full of bugs, no longer understood. We say it’s ‘brain dead.’ Yet it lives. Yet it runs. Drain on our time and money. Vampire of our happiness. Legacy.”
—Ellen Ullman, Close to the Machine

These limbo places of software engineering: a biiiiig learning of my second year. You’re working on this product and “done” evades you constantly. I mean, I read something about how the Agile methodology pretty much excuses done. Doesn’t doesn’t make sense to when you Iterate ad infinitum.

Did we just deploy too many MVPs back to back?

But you make the calls you have to make.

Ross. You better watch out kid: if you join the ranks of one of the Valley’s going concerns, technical debt and its attendant feels, will be yours.

Again, the hardest thing is never the programming; schizo

Loans are scary. They are a hard thing about building things on the internet that’s not the programming.

Before I was a product engineer I was building fancy marketing pages for Zendesk. In that world it was easier to tear down and rebuild the world. There wasn’t as much co-dependent wiring between the parts. There was one resource type in the db: the post. Overall the CSS and JavaScript was relatively small — I was even able to manage all the needs of the team through a year and a half just by my lonesome. I rewrote chunks of CSS at will. I was rarely hurting because of poor system design choices. Never really had to scale. The WordPress community handled the big issues, the admin interface, infosec, etc...

Now, not.

Joining the product team at Science Exchange was an immediate and accelerated education in accounting.

A fictional account:

A: When I type in this experiment type why doesn’t this lab show first? They seem to have the most positive reviews.
B: Yeah, but the “query” word shows up more times in this other lab’s description.
A: Why are we doing it that way?
B: (crying) I just don’t know anymore…

(The algorithm is better now btw.)

Honestly, the technical challenges of software engineering seem increasingly quaint as time goes on in the face of the cognitive medicine ball of living in the morass of mixed worlds: the limbo between your optimistic vision and truth. I mean, that’s what the reality of debt really is. It’s schizo, fractured; floating mines in your perfect code pond. I mean, I’ll get you some JavaScript; hell, I can get you some JavaScript by 3 0'clock this afternoon. But here’s the type of struggle that feels way bigger than writing perfect acceptance tests:

Presently, our frontend code has 3 CSS flavors (systems or generations, what have you) in two forms of Sass syntax. And only one styleguide for the middle one. It’s 2016. The “new” CSS framework I built in 2014, which I dubbed Iceman for it’s Kind of #338DC8 sexy, made it about 65% to completion. Then, the heuristic storyline, limited capacity and shifting business concerns forced the team to work on other things and optimize for the present. Unlike search’s unexpected derailing, this was intentional debt — we were making “a conscious decision to optimize for the present rather than for the future.” There were meetings. We had to switch gears to work on the things that could get numbers up. There were other awesome things to build!

But I never forgot about you, Iceman! Even as the days turned to months, turned to years (literally). Even while a new framework, Hypothesis, threatens to dethrone you — to correct you. I crooned while you sat (still sit) awaiting appeal: “we’ll get to those other pages soon.”

No we fucking won’t! Because we didn’t.

Coding horror

And then there’s living with the fear of causing debt. [shutters] This fear actually drove my early learning quite a bit — god forbid I should move too quickly, make a bad decision; defer future engineering earmarks for interest payments. I spent some late hours staring at books like Addy’s Learning JavaScript Design Patterns so I wouldn’t reinvent some shitty wheel for our site’s jQuery modules. I was picking up some basic syntaxes in JavaScript and Ruby quick enough; but how to assemble the expressions without complete madness and in a way that made my more senior colleagues LGTM with the smiliest of faces? Without, supreme being forbid, introducing a dreadful anti-pattern?? (And who am I kidding? “Early learning” — as if it never stops.)(And besides that fear of poisoning or diluting or confusing the codebase, even more sinking spiritual doom: tough feels I’m never living up to my contract to enable breakthrough research in innovative ways fast enough.)

Bootcamps should teach an emotional competence section on dealing with technical debt as a web developer. I mean, if you’re smart, learn hard, practice hard, and stay curious as all hell…you’ll write the good codes, I promise. But the background presh from this ghastly limbo requires reckoning:

“Accumulated technical debt becomes a major disincentive to work on a project. It’s a collection of small but annoying things that you have to deal with every time you sit down to write code. But it’s exactly these small annoyances, this sand grinding away in the gears of your workday, that eventually causes you to stop enjoying the project. These small things matter.”
— Jeff Atwood

Achieving balance

In the end we launched the big rushed Search refactor on time. It was much better than before. Faster server response times, a much more comfortable UX for users, and not a single line of jQuery to manage page state (JS on the fleeks). The workmanship was pretty solid, pushed far with cleverness, and we could rejoice despite knowing the plumbing was fragile and tangled in some places: where “our Ember” was too idiosyncratic, where the faceting was constrained by an incomplete implementation of the junctives. Heh, knowing that even the debt wasn’t entirely paid down. But, this time around I remember beginning to face this compounding debt without as much sadness. Maybe I was finally getting used to limbo after two years. Finally equipped after this pretty massive effort. I suspect the more senior peeps felt like they were in familiar terroir. But I like to think maybe we were also getting a little better as a team. Because collectively we did discover and re-discover ways to protect ourselves and be fiducially responsible.

The most simplest practice of which…

For example, at those cut corners in the codes, we did our best to leave clues and confidences in the comments, cairns and beacons pointing toward a path to the black. TODO. NOTE.

Hear dis. Please, make sure you buckshot fire empathetic comments. Just like Duretti Hirpa says or else how we gonna make any sense of this later. Do you feel the same sense of immediate relief when you throw Todos in the code like I do? Treat yoself with todos. It’s like Michael Jordan at the free-throw line hours before the game: imagining with precision how he’s gonna pwn Isaiah later that night. That’s how you pay it down on the playground and draw the and one and be good accountants. If you don’t do this then you’re being an asshole and screwing your colleagues (even the ones who may come after you).

Another piece of advice, make it a team effort and absorb the debt as a team. Because even as rushed decisions return to haunt in the form of bugs and piling promises (or callback hells) to fulfill later, you can sit over lunch and whiteboard and know where you stand, stare down the design flaws, and then know where you have to go.

“Eh, mixed results muddy ranking.”

“This is actually a data problem.”

Yeah all true, but at least when you’re conferencing and plotting you’re mapping (reducing) a road to solvency. Dude, I’m remembering the moments after release of the sharing and brainstorms and they really feel important motivational moments for me. Like, by practicing responsible fiduciariness and inching up into that Prudent/Deliberate vector of Fowler’s quadrant y’all were curing me of some of that fear. I’m happy our team has/had this culture. This process has made me certain every team must find their own ways through the emotional side of tech debt, which Atwood reminds us is real. Else folks might burn out and find illogical ways to blame each other or the ghost who wrote that code, or the PM for not caring about what we care about (total fiction), or the business team for their unceasing demands.


It’s interesting how you can forget some piece of code you wrote last week, but the debt is deeply inscribed, etched on your mental; a growl notification you can’t click away. It’s def one of the many things that will keep you up at night watching Grey’s. Sweet life, sweet life.