Beat the Heat: Best Practices for Hotfixes

As we roll into summer and things are heating up, I couldn’t help but think … HOTFIXES. Hotfixes were something I used to dread. If we were coming up on a shaky release, I would hardly be able to sleep fearing the worst thing I could imagine… a hotfix. As we deployed I would cross my fingers and toes hoping that all would be well.

This was the case 90% of the time. However, there were those cringy instances occasionally and I knew that something needed to be done. I learned how to operationalize, prepare for and focus the team during a hotfix. These velocity skills will help you to leverage hotfix tactics strategically making it less scary and just another tool in your developer toolkit.

Defining a Hotfix

How you define a hotfix is critical to operational success. As part of a software development lifecycle or standard operating procedure, your team should have a flexible definition of a hotfix. As well, you should have documented a method for determining what meets the need of hotfix.

A hotfix is generally defined as a patch to a live system due to a bug or vulnerability that meets a certain level of risk and severity. For example, your team might use this as a guideline and define a hotfix as a bug or vulnerability that negatively affects core functionality or user groups with a high severity and greater than average risk. Defining a hotfix becomes even easier when you have definitions for severity and risk. I define a hotfix with severity and risk being on the “defcon” scale; 5 for least severe and 1 for most severe.

With that said, here is my definition of a hotfix. A hotfix is a patch to production code that is released from a branch off that production code only when the severity of the bug or vulnerability is greater than defcon 2 and the risk is greater than defcon 3. Obviously, you then have to decide what constitutes the various defcon levels. I typically think of defcon 1 at a basic level as a critical vulnerability, potential loss of critical data, or user-facing blocker that is encountered greater than 50% of the time.

Bandaids v. Stitches

Patches can be generally defined with some medical terminology. I think of a hotfix as a bandaid. It stops the bleeding and keeps the wound clean. However, typically there is follow-up care that is needed. What caused the defect? Where is the core issue? Did we completely fix the defect or did we just cover it up? Sometimes we apply bandages to avoid the stitches we really needed.

As part of your operational planning for hotfixes, incorporate a retrospective or post mortem for each hotfix. This is where focusing the team on a goal is critical, this is not the time to point fingers. This is a time for discovery, growth, and prioritization. Use this as a learning experience. Many times a hotfix is needed because some process on the team was not optimized.

An ounce of prevention is worth a pound of cure

I personally caused a hotfix by approving a pull request that still needed lead developer review. We learned from that and put in a systematic approval process so that I could only approve code once it had been approved by the lead developer or architect. I say this to point out that by learning from your mistakes and making process improvements you can avoid a lot of hotfixes and patches in the future.

Think about implementing procedures for:

• Having an automated CI/CD process with checks and balances.
• Enable code reviews on every pull request to master.
• Start a peer programming routine so that you learn from your teammates.
• Ensure that you have a code standard that includes commenting methods.
• Add vulnerability scanning in your CI/CD pipeline.
• Have a “Go” or “No Go” meeting at least a day before a deployment.

These little things go a long way to preventing hotfixes and bugs generally.

Hotfix != Chaos

Hotfixes should not be chaotic, they should be just another systematic process in our software development lifecycle. Do not let the heat get to you. Let’s see what we can do to stay calmer than Walter.

Not all fixes need to be hot

As we determined by defining a hotfix, not all fixes will raise to the level of a hotfix. Here is where you rally the troops to think critically and determine if there is a need to stop the normal lifecycle or continue with modification. Most bugs will not rise to the level of a critical need and will likely be in the backlog for a while. So ensure that you are defining the issue appropriately. This blog from BMC is a great quick reference for defining a fix and its severity.

When you look at your definition and determine it is hotfix time, pause briefly and have a quick scrum. You will be glad you did when you determine the core team that will code and test the hotfix. Then, set the boundaries so everyone knows what to expect.

This should include:

• Strictly defining the scope of the fix.
• Setting timelines for code fix, testing, and deployment.
• Determine who is responsible for documenting and entering issues in your bug tracking system.
• The responsible party or liaison with the business/operations teams.

Operationalizing your hotfix practices

You have every opportunity to be successful. You have defined a hotfix, stepped back, and used that definition to inform your decision to move forward. You have even had a scrum and determined the scope and timelines for this special development cycle. So… why do we need to operationalize further?

Would you deploy any code without a process? In today’s agile landscape, unless you are working in the wild west — you have a process for your process. So here’s my grand advice — follow the same development practices just on a smaller and faster scale. That way you don’t have to remember a new set of rules or change what typically works.

Conclusion

Hotfixes do not need to be frightful or painful experiences. By having some common sense barriers and procedures in place, hotfixes can be a learning experience that helps you prevent future emergencies. Stay calm and let the hotfix abide.