Dev Process Lessons From My Life as a Maintainer

Cover of King Crimson’s “In the Court of the Crimson King”
“Where is it? Is it done? Does it work?”

Serving as the chief maintainer of a large codebase with a dozen active contributors has a way of turning you into a steely-eyed skeptic, and a bit of a paranoiac. I repeated these three questions so many times that they began to feel like my mantra.

As a maintainer, I felt that a big part of my role was gatekeeping. That is, to be on guard for bugs and increasing technical debt, and work with contributors to improve their changesets if they don’t pass the threshold. If you were contributing to the codebase, you were likely to hear these three questions. And you’d be surprised how often they would uncover problems.

However, effective gatekeeping doesn’t have to rely on manual review and expertise. Over time, I grew to appreciate a few key process techniques that address these concerns up front, avoiding further recitation of the dreaded mantra from an overworked maintainer:

  • An issue-based, branching workflow keeps changesets locatable, focused, and easy to review.
  • Automated regression and new-feature tests prove that new features work, and that the new stuff didn’t break the old stuff.
  • An automated, continuous build system helps verify quality and completeness without any manual work.

A feature-based, branching workflow keeps changesets locatable, focused, and easy to review.

Nothing answers “Where is it?” more clearly than a dedicated branch in the source code repository. If the code is primarily living on the hard drive of someone’s development machine, not only is it hard to track down those changes, but it sets off all kinds of paranoid thoughts. “Are the files backed up anywhere? Are the changes based on the latest version of our code? Does it work on any other machines?”

Branching workflow image from NuRelm

Beyond simply locating the changes, I’m also keeping an eye out for what’s changing in the codebase and why. If the changes for your bugfix or feature are isolated in a purposeful branch, I can learn a lot at a glance by comparing your branch with the baseline. Which files have changed? Which functions are affected? Are there new dependencies? Most importantly, do the locations of those changes roughly align with what I’d expect to see? If not, is there a provided explanation in the commit log or issue tracking system to put my paranoid mind at ease?

If your changes are tangled up with other changes for unrelated issues, then I have to untangle the web of what’s changing for which purpose, which is time-consuming not just for the me, but inevitably results in a barrage of questions, or a rejected merge request, for you.

Automated regression and new-feature tests prove that the new feature works, and that the new stuff didn’t break the old stuff.

Nothing answers “Does it work?” better than a new, automated test case that verifies that the new feature performs as expected. I can browse the new test cases to learn what the new function is expected to do and within what bounds it’s expected to work. It also helps prevent the new feature from getting broken if I have to deal with any merge conflicts. And with a dozen active contributors, this happens at least once a week.

A maintainer’s paranoid brain is always wondering if all the stuff that used to work is still working. This is where, over time, those automated test cases begin to build real value. Over many years, we built up a suite of over 800 test cases, many of them executing end-to-end flows that exercise several parts of the codebase. When these were passing, we had a high confidence that we hadn’t inadvertently introduced regressions.

Peace of mind soothes the maintainer’s anxiety.

An automated, continuous build system helps verify quality and completeness without any manual work.

Your focused changeset is in a dedicated branch. Great. You wrote new tests to prove that your bugfix or feature is done? Fantastic. Now, to review them, I have to stash my changes, clean my build, switch to your branch, compile it, and run the test suite? Awwwww c’mon.

This is where your automated, continuous build system comes into play. If I can check with my Jenkins CI server to see that your branch builds successfully, passes the old tests, and shows a few appropriate new ones, it saves me a lot of time. Looks good. Let’s merge it.

This branch is building successfully, but we have 32 new failures. That’s a sign that we’ve introduced regressions along with the new feature.

And truth be told, I trust the build machine a lot more than my own. Just like any dev, I’ve got work in progress, not to mention a ton of other development tools on my machine. But if those aren’t codified in the build environment, and reflected on the build machine, then we don’t want those dependencies creeping into the codebase.

Good process helps you move faster and deliver higher quality.

It seems counterintuitive, but it becomes increasingly true as your team and codebase scale.

Although it’s a bit antiquated by modern standards, The Joel Test (written in the year 2000!) is a great place to start if you feel like your process isn’t meeting the challenge, and as always he does a great job of explaining why these techniques matter with anecdotes that will sound familiar. This post about modernizing the Joel Test is a great companion piece, although even it is over 5 years old!

The truth is that modern tools have made these techniques easier than ever to implement, and their use is increasingly widespread, especially in web and distributed application development. However, each organization is at a different point in the process spectrum, and you have to walk before you can run.

And if you’re serving as the maintainer, you’ll be motivated.