Practical ChatOps: infrastructure on demand
In previous articles we covered one of the ChatOps solutions based on StackStorm and Errbot, and how to integrate those in Slack for a greater good. Now it is time to solve some real world problem with the help of our already deployed Bot.
We often encounter a situation when we need a clear environment to quickly test some commands, tools, installations or automation recipes. Usually spinning some Docker container helps, but once in while there are no free resources on a local machine to get “that thing” up and running right now (assumed closing browsers or the IDE is not an option). It would be very convenient if we could request AWS EC2 instance with predefined parameters from the Bot, right?
Booster packs
StackExchange is an online “marketplace” for StackStorm integrations called packs. Many interesting things could be found out there. So let’s grab some pack for playing with AWS. After quick search we found that aws_boto3 pack should do what we need. Do you remember that we could ask the Bot to install packs into ST2 for us? Time to use this feature. Type command as direct message to the Bot:
!st2 pack install aws_boto3Some time will pass and you will get notification that the pack was installed. Unfortunately, this is not enough. We need to provide permissions for StackStorm to launch instances in AWS account.
Stairway to the Cloud
If you have already used EC2 instance to deploy our setup, this is as simple as creating IAM role with appropriate access level and assign that role to your instance. So all calls via Boto3 library will automagically use credentials from the instance metadata. It is the most secure and production-ready way to authenticate your services in AWS. However, when testing on a laptop, you will need to supply AWS API access key ID and secret access key via AWS CLI inside the stackstorm container. Following our previous examples:
cd st2-docker
docker-compose exec stackstorm bash
apt-get update
apt-get install python-pip
pip install awscli
aws configureThe last command will interactively ask you the keys and then save them. To test it works, simply try to get your VPCs (assumed you have granted an access):
aws ec2 describe-vpcsExpect to get JSON output listing at least default VPC.
Make sure an IAM role (or user with given API key) has permissions for at least calling
RunInstancesandDescribeInstancesmethods.
Now we can test if a creation of EC2 instance works from StackStorm. Issue this command to spin up a virtual machine with Ubuntu 18.04 LTS:
st2 run aws_boto3.create_instance ImageId=ami-0b425589c7bb7663dAfter a minute or so you will get a response from ST2 like this:
id: 5b7ac373cda4cc01327a437c
action.ref: aws_boto3.create_instance
parameters:
ImageId: ami-0b425589c7bb7663d
status: succeeded
result_task: add_tags
result:
exit_code: 0
result:
ResponseMetadata:
HTTPHeaders:
content-type: text/xml;charset=UTF-8
date: Mon, 20 Aug 2018 13:35:20 GMT
server: AmazonEC2
transfer-encoding: chunked
vary: Accept-Encoding
HTTPStatusCode: 200
RequestId: 08ecdb75-915f-4b9a-a83b-8dba82f20b7f
RetryAttempts: 0
stderr: ''
stdout: ''
start_timestamp: Mon, 20 Aug 2018 13:34:43 UTC
end_timestamp: Mon, 20 Aug 2018 13:35:22 UTCThat indicates that the last action in the action chain create_instance was successful. Get the execution details for the whole chain by referencing execution ID from the top of the last command output:
st2 execution get 5b7ac373cda4cc01327a437c --detailWe should see the instance attributes as JSON in the result property. Find instance ID there. We don’t really need that instance anymore, it was just to test our ability to create it, so better just to terminate it for now:
aws ec2 terminate-instances --instance-ids i-0f671854c166f4a0d
{
"TerminatingInstances": [
{
"InstanceId": "i-0f671854c166f4a0d",
"CurrentState": {
"Code": 32,
"Name": "shutting-down"
},
"PreviousState": {
"Code": 16,
"Name": "running"
}
}
]
}Now log out from the container.
How robots learn things
It is time to create an action alias for our Bot to be aware how to execute the create_instance workflow on our behalf. But before writing an alias, we need a place where to put it. Let’s create our very own pack first! The st2-docker project provides a special directory packs.dev, where we can develop custom packs. Let’s call our pack infra:
mkdir packs.dev/infra/aliasesIn that new directory create a file named pack.yaml in the infra directory with the following content:
---
ref: infra
name: infra
description: Infrastructure on demand
keywords:
- AWS
- EC2version: 0.1.0
author: Humble Me
email: me@example.com
contributors:
- Humble Me <me@example.com>
Now we have a pack to put out stuff in. With that we are able to create the alias. Write this into the file infra/aliases/create_instance.yaml:
---
name: "create_instance"
pack: "infra"
action_ref: "aws_boto3.create_instance"
description: "Create EC2 instance with GNU/Linux (Ubuntu 18 by default)"
formats:
- "please create instance( from {{ImageId=ami-0b425589c7bb7663d}})?( with {{KeyName}} key)?"
result:
format: |-
{%- set instance = execution.result.tasks[0].result.result.Instances[0] %}
Thanks for your patience!
I have created new instance for you: `{{instance.InstanceId}}` (`{{instance.PrivateIpAddress}}`).After that, enter stackstorm container once more and execute:
st2ctl reload --register-aliasesThis command loads our new alias from infra pack. Need to check if it became available:
st2 pack get infra
+-------------+--------------------------+
| Property | Value |
+-------------+--------------------------+
| name | infra |
| version | 0.1.0 |
| author | Humble Me |
| email | me@example.com |
| keywords | [ |
| | "AWS", |
| | "EC2" |
| | ] |
| description | Infrastructure on demand |
+-------------+--------------------------+st2 action-alias list --pack infra --width 20
+----------------------+-------+----------------------+---------+
| ref | pack | description | enabled |
+----------------------+-------+----------------------+---------+
| infra.create_instanc | infra | Create EC2 instance | True |
| e | | with GNU/Linux | |
| | | (Ubuntu 18 by | |
| | | default) | |
+----------------------+-------+----------------------+---------+st2 action-alias match 'please create instance' --width 40
+-----------------+------------------------------------------+
| name | description |
+-----------------+------------------------------------------+
| create_instance | Create EC2 instance with GNU/Linux |
| | (Ubuntu 18 by default) |
+-----------------+------------------------------------------+
Let’s see if we could run via the Bot. Check the response on !st2help message:
Looks wonderful so far. By issuing the following command for our Bot, we could achieve the same result:
!st2 please create instanceOr if you’d like to use different AMI or key pair:
!st2 please create instance from ami-xxxxxxxx with my-aws keyAt the end, the result should look like this:
You may notice that the bot reports each action status while executing the workflow.
It is useful for debugging, but not very convenient for real usage. We’ll deal with that later.
Knock-knock. Who’s where?
All what we did looks cool. But when we would have many users interacting with the Bot, how could we know who uses our precious resources? The Bot should know who’s talking to it and make some decisions what to do (or not to do) based on that information.
Lucky for us, there is a way to get the Slack user name via special action_context.parent.api_user Jinja variable available in the action parameter value template. Using this data we could get full user profile using Slack API and process such information.
Let’s grab another pack: stackstorm-slack. It will help us to send queries to Slack from StackStorm.
!st2 pack install slackEasy, isn’t it? Not so fast… Now we need to provide ST2 credentials for talking to Slack, because for now only Errbot was granted with that ability. It should be done within the stackstorm container:
docker-compose exec stackstorm bash
cd /opt/stackstorm/configs/
cat >slack.yaml <<EOF
action_token: xoxb-123456789098-QwErTyUiOpaSdFgHjKlZxCvB
admin:
admin_token: ''
attempts: 1
auto_join_channels: []
organization: ''
set_active: true
post_message_action:
channel: ''
icon_emoji: ':panda_face:'
username: ''
webhook_url: ''
sensor:
strip_formatting: false
token: ''
EOF
st2ctl reload --register-configsPut your bot’s API token as a value of the action_token YAML key. Now we would be able to make some action in Slack on behalf of the Bot.
Flow like a river
Let’s make a very simple, but powerful, addition to our instance creation process. We will assign an owner to each newly created machine. To achieve that, it is no longer sufficient to use just the alias, we need to create our own workflow, that would call Slack action to retrieve a user info and then create_instance with additional Owner tag containing the user’s email.
This requires a little bit more YAML “code” to write, so just grab the repository of infra pack and re-register it in the stackstorm container.
Take a look at the files to understand what’s going under the hood, there are some explanatory comments.
cd packs.dev
rm -rf infra
git clone https://github.com/vutny/stackstorm-infra.git infra
cd ..
docker-compose exec stackstorm bash
st2ctl reload --register-actions --register-aliasesThe command syntax has not been changed, but now the Bot knows who is talking to it and will tag AWS EC2 instances accordingly. Also, notification messages about task completion are suppressed, so you will get only the final message with the new instance info.
Making a difference
Of course, this is very basic “hello world” type of tutorial about how to reuse and create your own packs with workflows in StackStorm and expose automation via chat bot in Slack. Many things that matters for you and your organization could be implemented on top. Explore ST2 documentation (it is great), then add other parameters, failure detection, access configuration, instances life-cycle management — those are only limited by your imagination and real business demands.
Thank you for your attention!
