Panic Mode, every SaaS/product development team should have one
You kicked off your day, just got your second coffee, when your colleague asks you something about a customer question. The first thing you think is “F**k”, shit is broken and it is bad. If you create digital products, you have been there. It happens.
How you manage this situation is important though. The name Panic Mode suggests chaos. The funny thing is, by having clear rules and a process you will have no chaos at all. Maybe you have a process for it already. We do, let me tell you all about it.
1. Shit hits the fan
Often it starts when a customer notices something, of course it can be found internally while working on the product. Either way, someone in your team gets the signal something is not right, who checks in with the tech or product lead to push the panic mode button.
Let the customer know that you are working on it and keep them updated. Be sure to note their contact information if you have additional questions and to get back to them.
2. Have rules
Okay, so how does everybody on the team knows that shit is hitting the fan? Discuss the rules with your team, define when it is time to panic. This way everyone is on the same page when to call it. And also important, if this happens what should be dropped?
Our questions (if any answer is yes, it is panic time):
- Security risk, access to other environments or leaking data?
- Can the service or a feature not be used?
- Is data being corrupted?
- Are calculations or data shown to the customer incorrect?
- Is conversion at risk?
What we do:
- Drop everything.
- Working on something? Pick it up later.
- Client needs help? Let them know you call back later.
- In a meeting? Drop out.
These are our rules, everybody knows them. This way everybody knows when to call it and what should be done. If you blow of a call or drop out a meeting, your colleague knows why and understands it. Your rules can be different of course. Talk about it to be on the same page.
3. Team updates
When working in a team you need to manage information. If you work in a one man team you can skip this step.
Right away, even before knowing what caused it, we publish a message in the team’s general channel. Let the team know panic mode is on, what is happening (why panic mode is on) and that it is top priority. When people are needed, mention them so they have some information about it and follow up for speed.
Do not only mention development team, also make sure to include the sales/customer service team. They are in direct contact with the customers, therefor it is good for them to be aware. Especially when you expect more support and need them to manage the situation (they also should drop their tasks), include any information they may need to do this.
Give status updates when new information is available, so everybody knows what the current status is. No update? Keep a regular interval to tell what is done and what is next. We do this every hour. This keeps tention from building and let’s people know there is progress.
4. Work on it
I know it is step four already but you can finally start working on it, do your whole debugging ritual. Check out which piece of tech is broken and who you need to fix/debug this.
Use your team for this when you can. Kick off the issue, discuss options and steps. Divide work when possible, if you have multiple things to check out you can work in parellel. Agree on a time to get back together to discuss your findings and talk about the next steps. We usally settle on 30 or 60 minutes.
Maybe this is self explanatory, but create the fix from your main branch and test it locally twice (best to use for eyes/two people). Deploy it right away and do not release anything else, just the fix.
5. Thank your customer
Customers love to get an update about something you fixed. Especially when you handled something important right away. Thanking them for reporting the issue makes them feel like a contributor. An often overlooked process.
The thank you works internally too :) And do not forget the internal update from step 3. Mention the problem is fixed and panic mode is tunred off and optionally mention some steps that should still be done.
6. What happend?
No, you are not done. By pushing the fix right away you bought yourself some time. Now the panic mode is turned off you need to check a few things. Get your product/development team together and discuss the following topics:
- How did this issue happen?
- Are there any other places in the application/code we should look into? Just to make sure it does not have a related issue.
- Anything that should be put on the roadmap or backlog?
- Could this be prevented? If so, which measures could be taken? How could the product development process be improved?
You fixed the issue. You managed panic while keeping calm and control. The team did this together. Maybe you even improved your product development process. That’s great!
What is your Panic Mode-process? Any thoughts or lessons learned? Please let me know, I love to improve the process further.