RigD.io Collaborative Automation for Incident Management
Automatically Open a Slack Channel for PagerDuty Incidents
SlackOps for PagerDuty Part 3
For our third installment of our series to help you master your incident management, we will help you automate creation of Slack channels for your PagerDuty incidents. Following this practice of having a dedicated channel for each incident is a great idea. We find some customers who like to use a “War room” approach. There are two problems with this.
First, When you have more than one issue going on, you will have to fight cross talk, which can lead to distractions for key resources, missing important messages, and general confusion for those who are jumping in late trying to get up to speed.
Second, having more than the required people engaged in an incident in the channel can lead to “unhelpful” commentary, distract non-essential people from the work they should be doing, and potentially lead to “unmanaged” updates getting out.
Creating a dedicated slack channel and bringing in the right people at the right time mitigates these risks and offers a clean record of the incident activity and who worked engaged in resolving it. This is of course an essential part of a good postmortem. So lets jump in and show you how to make this task a simple thing with RigD.
Step 1 It’s easy to open a Slack Channel Manually
We provide two ways to open a dedicated PagerDuty incident channel. The first is through a RigD activity. You can initaite that by typing
Then choose PagerDuty as the work tool and provide the incident number.
The Slack channel will be created with the tool type and incident number in the name to make it easy to find. To be helpful we also add a note to the incident with a deep link to the Slack Channel. Additionally, we add any incident assignees or responders to the channel so they can start to engage right away.
The second way to create the incident channel is from an incident notification if you set up a Service Feed through RigD. Here you simply click the start work button.
Step 2 Use RigD Automation to Open the PagerDuty Incident Slack Channel
Using those manual options for opening the incident channel are not difficult at all and they can work for most. However, there may some circumstances where you or the on call person need to get the team engaged and working the incident before you can get into Slack. To set up this automation we are going to first establish a RigD feed which gets incident events from PagerDuty and delivers them in Slack. Second we are going to use another flow to run our start work activity, which creates the channel. Then we are going to create a RigD trigger that will start the flow based on seeing a specific element in the incident event. If it sounds a little complicated don’t worry, there is a setup guide that will do all the heavy lifting. Start from the PagerDuty help by typing
help with pagerduty
Then choose the Open An Incident Slack Channel button
Now you have thee pieces of information to provide to complete the setup. The first is the name of the PagerDuty service you want to open channels for.
Next provide the type of event to use to create the channel. This guide supports just a simple scenario of opened and acknowledged, you may have other more complex cases you want to explore down the road, such as acknowledged and with a P1 priority.
Finally, we need a channel for posting the notification that the incident channel has been created. Typically this is the same channel where you have your regular incident notifications sent to.
Congrats you’ve just made your incident management process significantly better! Your Feed is set up and new incident events will create a Slack Channel. Give it a try.
Once the incident is opened and then acknowledged the channel is created. The notifications for the incident from the Feed will now provide a button to jump right to that channel. The notification channel will also have a message with the channel details.
When you are all done working on the incident you can easily archive the channel by typing the following in the channel. This also let’s RigD know that the incident is done and to generate a digest of the incident activity.
There were many benefits mentioned earlier in regards to opening a Slack channel per incident. So assuming we want to do that let’s look at the benefit of using RigD vs doing it manually, as we did with our other series parts. Opening the channel through RigD takes just 4 seconds as compared to about 20 seconds when creating the channel manually. As with our previous two examples it’s not a massive amount of time on it’s own, but it adds up. Using our same 7 major incidents a month and $5,600 cost per minute we end up with a savings from using RigD of $10,453. So after performing just three of the simpler tasks of incident management we have reached a total monthly cost without RigD of $54,879 vs just $11,106 and setting all three up takes less than 10 minutes.
Keep an eye out for the next part of our series which covers automating the incident update process. Also please do share your comments or questions so we can keep making incident response easier and faster.