Smarter team communication using ChatOps
Leverage existing DevOps tools, and some new ones, to enable communication in a faster and enhanced fashion.
Stop me if you’ve heard this before…
It’s the middle of the weekend and you’re enjoying a little R&R when you’re unfortunately reminded that you’re on call for support by email notifications going off on your phone. An alert has gone out to the whole department with just a message indicating health checks for production systems have not passed and in all likelihood are offline.
You break away from your weekend activities, log onto your laptop, and start to investigate the issue. In the meantime, a slew of response emails are sent out and look something like this:
Emails, texts, and any number of status calls fly by and you find yourself switching between numerous tasks:
- Diagnosing and Fixing
- Hunting down and waiting for the right people to be available for their specialized task
- Generating reports and sending updates
- Deploying fixes
… all for any number of people requesting they be done.
While you’re fixing, explaining, updating, and deploying you’re also losing precious seconds, minutes and even hours trying to recover from what is amounting to be a costly outage. Think about this: for every time a person is forced to context switch in order to perform necessary collaboration with a team member they’ve cost [themselves] as much as 40 percent of [their] productive time.
There has to be a better way. Not just for recovery incidents, but for general collaboration. You’ve invested so much in your applications, tools, and development practices (such as DevOps), why not take it one step further and integrate the power of automation into your team’s communication paradigm? When this is done properly, you can centralize your collaboration space, automation tools and processes to make them accessible to everyone on your team. Thus saving frustration, time and cost. This practice is called ChatOps.
ChatOps?
The goal of ChatOps is two fold — offer more transparency and receive faster feedback from our teams. We‘re provided a one stop shop for collaboration, issue resolution, process workflows, historical records and even training.
At the core of ChatOps, you have a collection of bots, or application that executes automated tasks, working symbiotically with a messaging platform to listen for events that occur either within an infrastructure or chat. The trick is doing so in an organized and beneficial way.
Looking back at that alert, instead of triggering a mass email to IT_ALL, that same trigger can be utilized in a different way. A bot subscribing to the event can turn around and notify the Prod Support channel. Alerting only the people who have subscribed to the channel and need to worry about and address the issue. At the same time, generate a status report and (here’s what you won’t get in email blasts) a solicitation for action to recovery in that very same medium. Tasks like recycling application pools, increasing server capacity, or reverting to previous working versions of your code base are common remedies for these situations and can be invoked immediately by typing simple commands or executing on their own.
For scenarios that require more investigation, teammates can then collaborate to fix the issue within the platform and not have to worry about updating a late joining member, annoying others with 50 email dings, receiving updates in real time, and most importantly fix the issue in as little time as possible.
The uses here are limitless. Let’s look at another scenario that’s less frantic than production recovery — Every week your team is responsible for generating reports for the Business Analysis team.
This involves:
- Taking site metrics
- Gathering user data
- Running it through some graphics libraries to display the information in a reader friendly way.
Each of these tasks alone are easy enough but take an hour each week to compile and process. Not a big deal, right?
Well, it adds up. Over the course of a year you’ve clocked well over a work week’s worth of time just generating this same report over and over again. What is the solution? Write a bot to do this for you. Or better yet, enable the BA team to request these reports and discuss them among themselves via the chat channel.
So instead of people needing information and you being their bottleneck, they can now request the report on demand, several times a week if they’d like, and your team has one less context switch. That is the goal of ChatOps. Bringing teams information and collaboration in a smart and efficient way.
What does this bot system look like?
In the past, there were and still are great solutions that provided ‘out of the box’ functionality to common use cases and have excellent communities supporting them. However, in order to have a bot fleet you would need to host and maintain a bot framework like you would any other application. This means in addition to the bots you write yourself, hosting resources, installing updates, and maintaining another platform which only adds complexity to your systems when you’re trying to remove it.
This is why I’m a huge fan of Microservice frameworks like AWS Lambda and Azure Functions that allow us to quickly develop bot functionality while host it on a low-cost Serverless environment. In a follow-up article I’ll provide a deep dive on what this configuration would look like, but at a high level all you have to do is write the functionality your aiming for, integrate with your chat application via commands or Webhooks, and you’re good to go. Even better, since you’re using tools within your cloud provider’s environment, you won’t have to worry about that additional layer of integration.
Where to get started with ChatOps
First and foremost, find a good collaborative chat application that supports team channels. When GitHub coined the phrase, they said ChatOps is about “[Putting] tools in the middle of the conversation”. That conversation needs to take place where everyone on the team can access it easily. As I’ve said, getting out of emails is one of the biggest benefits here as you leave behind that extra noise, nuisance and broken communication.
“Put tools in the middle of the conversation”
— Jesse Newland, GitHub
Find your pain
Find those tasks that you have to execute over and over. and follow the same processes. Then, try to automate them so you can complete the work in a few keystrokes with a bot command. Don’t go for a moonshot with your first bot using AI libraries and massive infrastructure scripts, but rather pick a small task and automate it. DevOps processes are perfect candidates for this. Once you have those bots in place, build on them. You’ll find that functionality compounds quickly and before you know it you’ll have a network of self serving bots that are available to you and your team.
