Hot Fix Thursdays Saved My Product

That Time I Was Drowning

It happened late one evening — let’s say it was dark and stormy out to really set the mood. I received an instant message from one of our employees about an urgent issue with my software, a research and analytics platform that was core to our deliverables. It was the fifth message I had received that day.

I sat there frozen, dreading the idea of telling our head of engineering, knowing it would pull them off of a new enhancement we were eager to push out. I also knew we couldn’t ignore the issue. We had been ignoring too many critical issues, accruing technical debt like it was on sale.

I spent the rest of the evening combing through our recent bug reports, customer feedback, and progress on our roadmap. The numbers were shocking. We had failed, big time. It was taking weeks to solve critical bugs and then, in our rush to resolve, we pushed out untested fixes during the workday, causing crashes and mayhem with our users.

The lynchpin, though, was the user feedback. Our employees were using words like “embarrassing” and saying they didn’t bother to report issues because they were not confident a fix was coming. Our sales team detailed the litany of workarounds they had to use when demo’ing the product. Our customers were feeling hoodwinked and discouraged.

I, in turn, was holding fast to our product roadmap and, frankly, kind of hiding my head in the sand. I hated always saying “no”, getting discouraged grunts from engineering, and trying to keep my head above the sea of backlog. I was a firefighter, a nag, and a master of workarounds. I didn’t feel much like a product leader.

I drove home, crawled into bed, and had a dream that my backlog was suffocating me. Something had to change.

The next morning, I called our head of development to share an idea I had cooked up on my way to work.

“What if we could guarantee to users that we would fix their bugs quickly? What if we could guarantee to developers that they wouldn’t be pulled off of their projects all the time? What if we could guarantee a more stable product to our executive team?” I asked.

And so began Hot Fix Thursdays.

The Basic Set Up

Every Thursday, we fixed and released critical bug fixes for our platform. We lived by a few rules to keep it running smoothly.

To earn user trust, we have to be transparent.

We had failed on the communication front before which had exacerbated the problem, so this was key for me. I started by announcing the initiative internally and then publicly to our users. Each week we would announce what was on our fix list. It’s scary committing to a list of work in software, but I knew we needed to.

After release, I would send more communication about what was actually fixes and published release notes to our website. If we couldn’t get to an item, I would alert the user with the issue asap. I would also close our bug report tickets with details of the fix and an offer to answer any of the user’s additional questions. The days of operating in a dark cave were over.

To earn user trust, releases happen every Thursday, no matter what.

Because we had a very manual release process and no automated testing in place, Thursdays were not fun for me or our head of engineering. We had to commit to staying late every single Thursday to push the fixes out. It would have been really easy to skip a week or decide to push out fixes the next morning, while everyone was working. But I knew the first week we didn’t release fixes, we would lose our users. They would assume things were going back to before. I passionately advocated for consistent releases and then tightened my ponytail (as my niece says) and did whatever was necessary to make it happen.

To earn engineer trust, the sprint can’t exceed half a day.

Our engineers were frazzled and tired of being pulled off of core releases eery day. I had to commit to a limited sprint. Scope creep would kill this initiative as fast as it started because we wouldn’t have time to devote to our upcoming new features.

Because I didn’t have developer estimates, I would load the sprint with the top three most important fixes and let the engineer know to stop when they reached three hours of work. We could discuss the progress and, in most cases, move any outstanding items to the following week. After testing and rework, this meant we spent no more than five hours of development time on a release.

To earn user and engineer trust, product management determines urgency.

This might seem obvious, but part of our issue prior was no real prioritization process. It was almost a first-come-first-serve system. The haphazard approach was frustrating to users and developers.

I defined lose criteria and then prioritized the backlog every evening up until the sprint was defined Wednesday evening. A candidate for a hot fix sprint needed to have at least two of the following criteria:

  • reported by a paying client
  • reported by multiple users
  • stopped work with no obvious workaround
  • posed a security threat

That Time I Wasn’t Drowning Anymore

It took us a while to really get good at Hot Fix Thursdays. I would fight the urge to throw everything in the sprint or, worse yet, ask for hot fixes on another day of the week; the developer would fight the urge to keep working until all the tickets were done. It took months for our users to trust we were listening to and solving their issues. It took even longer for us to implement a better release and testing product and to implement key architectural changes so our software was more stable.

But, I can honestly pinpoint Hot Fix Thursdays as the turnaround for our product. Over the next 12 months, user feedback dramatically improved and our revenue for the software doubled.

What’s your experience with urgent fixes? Where could I improve our system?