A Tale of Two Projects — Lessons in Agility
I have two stories to tell.
The first is about a project run through a very traditional project management and build process, which created certain behaviors in people and on teams.
The second is about a small project I did on my own that was run in a very different way, leading to a very different set of behaviors.
The big takeaway for me is how better processes and practices created an environment where feedback was a critical part of the development process, and how that completely changed the emotional toll taken by this project.
More rest, less worry, more confidence, more courage… all because of feedback loops. Read on for the tales…
The Tale of the Overnight Build
A long time ago (1995 or so), in a city far, far away, I was involved in building the firmware for a new digital consumer device. This was a large, multi-year project with 30 people or so working on it.
This was a pretty large system, having grown organically over a couple of years. It was written in C++, a language with a two-step build process. The compilation phase was typically pretty fast, on the order of a few minutes, but the linking phase, which assembled the final executable, took 8 or more hours.
Because the time needed to make even the smallest change was literally all day, you had only one chance to make that change every day. As a developer, this drastically affected how you worked. You tried to throw every change into your software you could think of into your code, including new features, clean ups, and bug fixes, so you could start your build at the end of the day. Then you’d come in the next day, hoping against hope that your build worked.
Assuming it did, you’d spend most of the next day debugging brand new errors in your code, line by line using a manual debugger (gdb. Any time you had left you’d spend adding more changes to your code you just knew you’d pay for tomorrow.
It was not a pretty way to work as it incentivized you to make as many changes as possible without any opportunity to see if they worked. Risk piled on top of risk, leading to long and frustrating days of trying to debug your system back to life. And things got worse the closer you were to the deadline, leading to mandatory overtime, lost weekends, larger waistlines, and lack of sleep.
The Power of Small Iterations
Working alone, I built a realtime control system for a conveyer belt responsible for wrapping and addressing packages to be mailed to consumers. It wasn’t the code of this system that provided most of the challenge; it was reading from and controlling external devices, like printers, barcode readers, label printers, and a brown paper wrapping machine. There was also a high-speed communications port to automate the operation of this machine. This, as well, was 100% C++.
From the beginning of this project, I made adding functionality and managing dependencies inside of the codebase co-equal priorities. Obviously I needed to add in the features the system needed, but I also knew that creating a tangle of dependencies would slow down testing more and more as the project went on. And, like in the example above, slowing down testing created longer feedback loops, which would cause the changes to get larger, driving up the risk of each change. Nothing good was going to come of this!
I consciously managed these dependencies, which allowed for rapid feedback through targeted unit tests for every line of code. In fact, not only was I writing unit tests for everything, I was following the Agile engineering practice of Test Driven Development (TDD). This let me add functionality to the system and keep the code clean and well factored with confidence, since the tests both proved I was implementing new features correctly and gave me a comprehensive safety net to protect against errors creeping in without me noticing.
The system took me about 8 weeks to complete. The only documentation I had on any part of the hardware was the serial communications spec for how to communicate with the controller for the conveyer belt and various specs for the serial and parallel devices. I TDDed the hell out of this system, writing tests for every serial device and protocol, the real time clock, and so much more.
So, fast forward a couple of weeks. My boss and I showed up at the customer’s site, never having seen the hardware, never having talked to the company building it, and ready to integrate at final delivery.
I walked up to the hardware, plugged in my software, and everything worked the first time! I couldn’t believe it! How in the world was it possible that thousands of lines of code just worked? I was ready to beat my chest and proclaim myself the World’s Best Programmer!!
Alas, this title was not meant to be. Although I had great results, it wasn’t because of me. It was because I used a highly structured and effective development process based on Test Driven Development. Using that let me create an environment where I could decide what to build next, build it, and get feedback about it in just a few seconds. That really quick, very localized feedback loop gave me the courage to write this code, and to walk up to the final hardware with an assumption things would. just. work.
OK, I lied just a little bit. I had misread the spec for the controller protocol and had one long pulse in a serial message misidentified as a short pulse in one command. I corrected my test, watched it fail, changed one line of code, watched my test pass, and then everything worked!
As you might expect, I was feeling pretty good about myself at this point! So good, in fact, that when the customer came to me with some changes I just started hacking them in with him standing over my shoulder. Change after change, hack after hack, we were adding a bunch of great new features. But I had this growing, uneasy feeling…
I politely told the customer we had to stop, and requested that he go to lunch for a bit. What I was noticing about myself is that my heart rate and internal sense of doom was rising with each change we were adding, as I wasn’t following my practice of writing tests then code for these new features. TDD is definitely a discipline, and I was feeling pressured to drop that discipline so we could get these features in!
Once the customer left I went back and added in the tests we skipped and did a bit of necessary refactoring, and I felt that vague ache in my chest vanish and my heart beat return to normal. When he came back, we talked about how the development process had to work to ensure I was building quality for him, and we TDDed our way to even more new features, sans the drama and fear!
Epilogue
OK, so first of all, I readily admit the scale of the two systems was not even close to the same. The first project was orders of magnitude larger than my solo project. But, as we all know, size doesn’t matter! In this case the TDD and careful dependency management techniques scale no matter the system size. As long as dependencies can be managed and code isolated, TDD is still a great strategy for creating rapid feedback, which engenders courage!
My point, if you’ve made it this far, is that having those rapid checkpoints, generating that rapid feedback, gives you courage as a developer to do things that ordinarily would make your heart beat out of your chest. Lack of that feedback loop makes changes scary and risky. Having those tests and having the ability to get that feedback on the order of seconds provides you with the freedom to explore, experiment, and implement sans the drama and fear.
This is the first article in a continuing series of TDD-related content as I build out a TDD course for all new developers at Emerson Digital and Connected Systems. This is the intro to the course, to be followed by actual instruction and practice at TDD. Feel free to read along and send feedback on the evolving course!
WE ARE HIRING — if you’re interested in working in a development team that is so devoted to agile engineering and making the most of its people, please check out our jobs site — https://sensi.emerson.com/en-us/work
