ChatOps - сonversations, put to work

Published in

DevOops World … and the Universe

13 min readMay 13, 2016

More than two years passed since Jesse Newland presented “ChatOps at Github”- youtube video at RubyConf 2013 about using Hubot for automation and deployment at Github. Actually, that was a beginning of ChatOps history. For those two years there were lots of topics discussing ChatOps — at different conferences, Reddit, Hacker News and other technology related platforms. Since then it’s becoming more useful. So what’s the reason for this, and what actually ChatOps is?

ChatOps, a term widely credited to GitHub, is all about conversation-driven development. By bringing your tools into your conversations and using a chat-bot modified to work with main plugins and scripts, teams can automate tasks and collaborate, working better, cheaper and faster. But actually, ChatOps is a little bit more. ChatOps is a collaboration model that connects people, tools, process, and automation into a transparent workflow using chat conversation. Conversation-driven development is not new, probably those who worked with IRC remember the same structure — private channels, a bot that responds to some keywords and executes some scripts. So why ChatOps is becoming so popular nowadays? Simple reason for that: ChatOps is its digital-age manifestation: a combination of the oldest form of collaboration with the newest technology. ChatOps is a rather new operational paradigm where work that is already happening in the background today is brought into a common chat-room. In doing so, you are unifying the communication about what work should get done with actual history of the work being done.

From a keyword to production

Let’s start from the beginning — team conversation. As you know, the majority of companies and development teams already using some chats for communication. Talking about chat-clients you firstly think about the most popular ones that satisfy development needs: Slack, HipChat, FlowDock, Campfire. But those clients won’t be so wide-spread without chat bots that can make your work easier. Lots of teams already using some bot notifications, TODO lists, Github/Bitbucket integration and so on. That’s the part that already done and working. Let’s break this down a little more. There are the chat bots made for ChatOps — and they’re all open source.

Image is taken from Geshan Manandhar presentation

Hubot: GitHub’s bot written in CoffeeScript and Node.js (docs)
Lita: Written in Ruby
Err: Written in Python

The bot part is pretty clear, but how does it help me to deploy it to production? This is where StackStorm comes alive:

StackStorm is a powerful automation tool that wires together all of your apps, services and workflows. It’s extendable, flexible, and built with love for DevOps and ChatOps.

But the question comes to mind — why do we need StackStorm if we can use the bots directly? Yes, we can, but run into barriers. Among them:

Tying oneself too closely to a single chat vendor
Needing a little more smarts in processing what is happening before alerting, and often interrupting, the humans
Spending significant effort to integrate and then update integrations with underlying systems
Requiring human sign off and wanting that sign off to address an entire workflow, as opposed to a single step action

In short — the reason that StackStorm is getting rapid adoption for ChatOps is that underneath your ChatOps — you need an event-driven automation platform. It integrates with your chat client from one side and development tools (like Ansible, Docker, Jenkins etc.) from another side. So when you are asking the chat-bot in Slack to launch an AWS instance, it sends this to StackStorm platform, and via AWS pack it launches directly on your AWS account. As simple as this picture:

From DevOps to ChatOps

This part is named so because ChatOps is a layer above DevOps. If you have your environment up and running, your company has a run-books for every event, than ChatOps would be the next step for you. I’ll try to describe how to add this layer to a running environment, so I won’t jump into details of DevOps, cause you probably already using this practices. Typical use case for ChatOps could be anything related to troubleshooting or automation. Currently, my test environment consists of:

AWS account with couple of VPC, and some running instances
A local environment for Development and testing.
Github, Docker, Ansible
Slack team chat, with couple of bots (TODO, notification etc.)

As you may see, I have some working environment on this step, and it may even represent some small company that uses the same tools as I described. To use ChatOps, we need some additional tools above the existing system.

Slack hubot plugin
An Ubuntu 14.04 LTS server (vagrant may be used)
StackStorm
StackStorm AWS/Linux/Core packs

Installing Slack hubot plugin

If you already have Slack up and running, just go to “Browse Apps” and install Hubot. After installation, you can add some custom configuration like bot name (In my case his name is stanley, and it’s a default name for StackStorm), bot icon and so on. Also, you will be given HUBOT_SLACK_TOKEN. This API token will be used later on with StackStorm integration. Ah, and don’t forget to invite your bot to the channel you will operate in.

Installing StackStorm

Let’s install StackStorm on Ubuntu 14.04 LTS server on AWS account. While the system can operate with lower specs, these are the recommendations for the best experience while testing or deploying StackStorm:

Testing

Dual CPU system
1GB of RAM
Recommended EC2: t2.medium

Production

Quad core CPU system
>16GB RAM
Recommended EC2: m4.xlarge

This is the reference deployment image made by StackStorm

It’s all the StackStorm components that need to be installed for normal operation. Let’s start with the basics:

Install Dependencies (MongoDB, RabbitMQ, and PostgreSQL)

sudo apt-get update
sudo apt-get install -y mongodb-server rabbitmq-server postgresql

Setup repositories

curl -s https://packagecloud.io/install/repositories/StackStorm/stable/script.deb.sh | sudo bash

The following script will detect your platform and architecture and setup the repo accordingly. It will also install the GPG key for repo signing.

Install StackStorm components

sudo apt-get install -y st2 st2mistral

In case you are not running RabbitMQ, MongoDB or PostgreSQL on the same box, please adjust the settings:

RabbitMQ connection at /etc/st2/st2.conf and /etc/mistral/mistral.conf
MongoDB at /etc/st2/st2.conf
PostgreSQL at /etc/mistral/mistral.conf

Setup Mistral Database

# Create Mistral DB in PostgreSQL
cat << EHD | sudo -u postgres psql
CREATE ROLE mistral WITH CREATEDB LOGIN ENCRYPTED PASSWORD 'StackStorm';
CREATE DATABASE mistral OWNER mistral;
EHD# Setup Mistral DB tables, etc.
/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf upgrade head
# Register mistral actions
/opt/stackstorm/mistral/bin/mistral-db-manage --config-file /etc/mistral/mistral.conf populate

Configure SSH and SUDO

To run local and remote shell actions, StackStorm uses a special system user (default stanley). For remote Linux actions, SSH is used. You will need to add this user to the servers you want StackStorm to operate with. Create a system user:

sudo useradd stanley
sudo mkdir -p /home/stanley/.ssh
sudo chmod 0700 /home/stanley/.ssh

Then, generate ssh keys

sudo ssh-keygen -f /home/stanley/.ssh/stanley_rsa -P ""

Authorize key-based access

sudo sh -c 'cat /home/stanley/.ssh/stanley_rsa.pub >> /home/stanley/.ssh/authorized_keys'
sudo chown -R stanley:stanley /home/stanley/.ssh

We need to enable passwordless sudo, for being able to run sudo commands normally

sudo sh -c 'echo "stanley    ALL=(ALL)       NOPASSWD: SETENV: ALL" >> /etc/sudoers.d/st2'
sudo chmod 0440 /etc/sudoers.d/st2

Make sure Defaults requiretty is disabled in /etc/sudoers. It was disabled by default on Ubuntu, but if you using another OS, check it

sudo sed -i -r "s/^Defaults\s+\+requiretty/# Defaults +requiretty/g" /etc/sudoers

In case you’re using another user, or already generated ssh key file, adjust configuration in /etc/st2/st2.conf

[system_user]
user = stanley
ssh_key_file = /home/stanley/.ssh/stanley_rsa

Start Services

sudo st2ctl start
st2ctl reload

and verify that st2 is running

st2 --version

Configure Authentication

Create a user with a password:

sudo apt-get install -y apache2-utils
echo "Ch@ngeMe" | sudo htpasswd -i /etc/st2/htpasswd st2admin

where Ch@ngeMe is a password, while st2admin is a default login for StackStorm. Enable and configure auth in /etc/st2/st2.conf:

[auth]
enable = True
backend = flat_file
backend_kwargs = {"file_path": "/etc/st2/htpasswd"}

Restart the st2api service:

sudo st2ctl restart-component st2api

Authenticate (you will be prompted for password) and export the token for st2 CLI, and check that it works:

st2 auth st2admin

Here is a shortcut to authenticate and export the token. If StackStorm server is operated by a single user, and access to it is restricted, you may add this string to .bashrc for authenticating every time you login, but it’s not security recommended.

export ST2_AUTH_TOKEN=$(st2 auth st2admin -p Ch@ngeMe -t)

Check that it works — you will see the current actions that installed on your StackStorm server.

st2 action list

Installing WebUI

NGINX is used to serve WebUI static files, redirect HTTP to HTTPS, provide SSL termination for HTTPS, and reverse-proxy st2auth and st2api API endpoints. StackStorm depends on Nginx version >=1.7.5; since Ubuntu 14 has an older version in the package repositories, you will have to include the official Nginx repository into the source list:

sudo apt-key adv --fetch-keys http://nginx.org/keys/nginx_signing.key
sudo sh -c "cat <<EOT > /etc/apt/sources.list.d/nginx.list
deb http://nginx.org/packages/ubuntu/ trusty nginx
deb-src http://nginx.org/packages/ubuntu/ trusty nginx
EOT"
sudo apt-get update

Install st2web and nginx

sudo apt-get install -y st2web nginx

Generate self-signed certificate or place your existing certificate under /etc/ssl/st2 (Replace with your data)

sudo mkdir -p /etc/ssl/st2
sudo openssl req -x509 -newkey rsa:2048 -keyout /etc/ssl/st2/st2.key -out /etc/ssl/st2/st2.crt \
-days XXX -nodes -subj "/C=Ukraine/L=Kyiv Alto/O=7insyde/OU=Information \
Technology/CN=$(hostname)"

Remove default site, if present. It’s not by default, but to be sure, check it

sudo rm /etc/nginx/sites-enabled/default

Copy and enable StackStorm’s supplied config file. If you modify ports, or url paths in the nginx configuration, make the corresponding changes in st2web configuration at /opt/stackstorm/static/webui/config.js.

sudo cp /usr/share/doc/st2/conf/nginx/st2.conf /etc/nginx/sites-available/
sudo ln -s /etc/nginx/sites-available/st2.conf /etc/nginx/sites-enabled/st2.conf
sudo service nginx restart

Now you are able to connect to WebUI with your password that you generated earlier.

Installing ChatOps

The easiest way to enable StackStorm ChatOps is to use the st2chatops package. It requires node.js to be installed:

curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
sudo apt-get install -y nodejs

Install st2chatops package:

sudo apt-get install -y st2chatops

Review and edit /opt/stackstorm/chatops/st2chatops.env configuration file to point it to your StackStorm installation and Chat Service you are using. By default st2api and st2auth are expected to be on the same host. If that is not the case, please update ST2_API and ST2_AUTH_URL variables or just point to correct host with ST2_HOSTNAME variable. Use ST2_WEBUI_URL if an external address of your StackStorm host is different. Since we are using Slack, go to Slack web admin interface, and copy the authentication token into HUBOT_SLACK_TOKEN. If you are using a different Chat Service, set corresponding environment variables under Chat service adapter settings: Slack, HipChat, Yammer, Flowdock, IRC , XMPP.

Reload the StackStorm configuration and start the service:

sudo st2ctl reload
sudo service st2chatops start

Installing Packs and Configuring Rules

As I said earlier, StackStorm is a platform that integrates with different tools. StackStorm packs are connection modules between StackStorm and a 3rd party development software. On the current step, you probably have only a couple of packs installed. You can check it using WebUI in the Actions tab. But for normal integration with Slack, AWS, and some other services we need more packets. Let’s install them from official repo:

st2 run packs.install packs=st2
st2 run packs.install packs=chatops
st2 run packs.install packs=aws

It should like similar to this:

On this step, we have a connection between Slack bot and Stackstorm server, but we didn’t set up any commands for the bot (it uses default ones) as well as didn’t configure any notification rules. If you try to communicate with the bot ( ! is used to access the bot by default — can be changed via st2chatops.env config file) you will see something like:

The problem is that I didn’t receive any notification from the bot, it just gives me the link to WebUI, with the current action. That’s because we don’t have any st2 rules applied. If you check WebUI tab “Rules”, you’ll see that it's empty (“st2 rule list” works as well in the console). We need to create a rule to receive some notifications from the bot. The rule should be a trigger, and when this trigger executed it should post the result to the Slack channel. The trigger could be found in default core.st2.generic.actiontrigger, and action is chatops.post_result which is part of chatops pack. In my case it matches the criteria action_name by the next rule:

(st2|packs|core|aws|linux).*

It means that notification will be sent in case if action name is one of the above.

If you’ve done everything properly, the bot will start to respond with the action result from the server. But you still don’t have enough bot commands to check it correctly, just try to do with “! pack info aws”

Creating Aliases

You’d probably already have a question like: “Why do I have 7 packs installed but only 10 useless commands on Hubot”. And yes, that’s some frustrating thing I’ve also faced. The reason for that is every Hubot command is represented with its alias. It’s used to connect Hubot command with the pack action. The problem is, those packs that you’ve downloaded do not have any aliases preconfigured. And those aliases that you see when typing !help on Slack are default ones for some actions like packs and st2. The packs by default located at /opt/stackstorm/packs/. You may check it yourself that most of the packs do not have aliases folder inside. When you reload st2ctl service, it looks for aliases folder inside the pack and adds it by default. To manage it manually, you can use:

st2 action-alias {list,get,create,update,delete}

My recommendation is to delete aliases that you’re not planning to use. The procedure of deleting/creating is quite simple, and should be done every time you add or delete an alias.

st2 action-alias delete ALIAS
st2ctl reload --register-aliases
service st2chatops restart

In our case, I don’t need most of the packs aliases, and some st2. So I deleted them and wrote new ones. Let’s create some aliases for the Linux pack. Each alias is a yaml file, so first of all, create a folder and give it a correct permissions:

mkdir -p /opt/stackstorm/packs/linux/aliases
chown root:st2packs /opt/stackstorm/packs/linux/aliases -R

Now you can add some yaml files to this folder, in my case:

-rw-r — r — 1 root st2packs 159 May 11 09:18 linux_cp.yaml
-rw-r — r — 1 root st2packs 122 May 11 08:53 linux_dig.yaml
-rw-r — r — 1 root st2packs 152 May 10 13:41 linux_loadavg.yaml
-rw-r — r — 1 root st2packs 166 May 11 09:18 linux_mv.yaml
-rw-r — r — 1 root st2packs 143 May 10 14:39 linux_netstat.yaml
-rw-r — r — 1 root st2packs 155 May 10 14:04 linux_process.yaml
-rw-r — r — 1 root st2packs 160 May 11 09:22 linux_rm.yaml
-rw-r — r — 1 root st2packs 189 May 10 13:45 linux_service.yaml
-rw-r — r — 1 root st2packs 141 May 10 13:54 linux_vmstat.yaml

Aliases yaml file has quite a simple format:

name (the name of the aliases)
action_ref (which command it will execute)
description (the brief description of the command)
formats (command format in chat)
result (a result format in chat)

Here’s an example of some Linux and AWS yaml files:

 — -
name: “aws_get_instances”
action_ref: “aws.ec2_get_only_instances”
description: “Get Instances from AWS EC2 account”
formats:
 — “aws get instances”
result:
 format: |
 Found {{ execution.result.result | length }} results:{~}
 {% for instance in execution.result.result -%}
 * {{ instance.tags.Name }}
 > {{ instance.id }}
 > {{ instance.instance_type }}
 > {{ instance.ip_address }}
 > {{ instance.private_ip_address }}
 > {{ instance.public_dns_name }}
 > {{ instance.root_device_name }}
 > {{ instance.vpc_id }}
 {%+ endfor %}

As you may see, result format is actually a template where we can operate with loops, variables and so on.

 — -
name: “service”
action_ref: “linux.service”
description: “Stops, Starts, or Restarts a Linux service”
formats:
 — “linux service {{hosts=localhost}} {{sudo=true}} {{service}} {{act}}”

And here how linux service alias would like, from Slack:

Or aws_get_volumes for example:

Just remember that every bot command should be represented with an alias stored in the packs folder. So if you want your bot to work with a hundred commands be ready to create hundred of yaml files. I’ve already created some basic linux/aws yaml files, feel free to use them. They can be found here: https://github.com/7insyde/chatops

Summary

Twenty minutes ago, when you’ve started reading this article, you had a default DevOps environment you used to. Now, you have a basic ChatOps system up and running. Despite it’s only the beginning of the journey, and we still have lots of things to improve, we are operating on the next level, and it is ChatOps. Later on, you may configure

additional aliases
install more packs that you need to
create more complex triggers and sensors
improve security with role-based access control

and I hope you will like it. StackStorm as part of ChatOps opens a great way to operate in your environment communicating with developers, engineers, testers, managers being on the same channel with you. It defines a security policy that only a bot have access to the environment while L1/L2 support just operates the bot. It solves lots of problems including “console sharing” and those situations like: “John, come here and look at my screen…”

I want to end this article with the words of Evan Powell — founding CEO of StackStorm:

ChatOps is a huge movement in DevOps — and crossing into the enterprise. An analyst recently told me that 2016 is a the year of ChatOps and I tend to believe him.