“Serverless” Slack bot on AWS (vs Azure). Getting notified instantly.

Published in

DevOops World … and the Universe

15 min readOct 23, 2017

ChatOps, a term widely credited to GitHub, is all about conversation-driven development. By bringing your tools into your conversations and using a chat bot modified to work with key plugins and scripts, teams can automate tasks and collaborate, working better, cheaper and faster.
~ Eric Sigler

When you read articles and definitions like the one above it tempts you to try it out, sounds like a cool practice, doesn’t it? I’m no exception, I just wanted to get my hands dirty and get familiar with ChatOps tools, specifically bots for the messengers we use daily. Would be very handy to throw a message to corporate messengers such as Slack or Skype For Business and get the job done.

I’ve already tried playing with simple bots, though never wrote about that. This time we will focus specifically on Slack bots and notification services.

Why Slack?

Well, I use both SFB and Slack. Here I have both Windows and MacOS machines, and not so long ago I had a problem with memory leak in SFB (looks like it’s fixed with one of the latest updates) on my Mac. Still it has left an unpleasant residue.

Statistics, companies that use Slack and SFB:

Skype for business is mostly used as enterprise solutions by big companies. Of course this is not the key feature why we’ve gone for Slack. But a number of integrations is:

SFB is good when we talk about integration with corporate mail (Exchange/Office 265) and other MS Office products, Slack beats SFB with lots and lots of integrations with different applications, components, software. After all, there are many articles and people talking about Slack and its bots, I haven’t heard much about SFB bots however there is a framework (in preview). Might dig in there one day, but for now lets just stick with Slack.

For the beginning, I expect my bot to do the following:

Phase 1: React to commands in Slack channel, do some basic operations;
Phase2: Send messages to slack channel upon alarm from monitoring or some other notifications services.

An idea and cloud provider selection

My plan was to configure slack bot to run on some serverless platform, for example AWS Lambda. Since this service was the first that came to mind I decided to continue with AWS (and I also work with this provider the most), and later on try it on some others.

So based on my idea and info I found, for better visualization we can represent the whole process on a diagram:

As you can see on this diagram I’ve split the process into 2 phases.

Getting started

In the Phase 1 we could make use of Slack slash commands, that is also a good way to trigger some bot actions. In my case I’ve gone for another way: “bot” keyword.

The almost whole process for building this phase is well described in this tutorial on slack website, also found a good article by Rigel Di Scala that was very helpful, and I used his code and idea for the further development. Oh yes, I forgot to mention, I wanted to have my event handler Lambda function written on Python, just because I’m more comfortable with Python rather than other languages.

Configuring slack side of the bot

There’s a reference above to this article with detailed description how to set up slack side of the bot. In short:

Go to “Manage apps” in Slack client;
Click “Build” and “Start building”;
Select bot name and Slack WorkSpace;
“Create app”, scroll to “Bots” section and hit Add a Bot User”;
Specify display name and default name for your bot;
Make sure “Always online” is enabled;
“Add bot user”
On the left pane select “OAuth & Permissions”;
Click “Install App To workspace” -> Authorize
Note down “Bot User OAuth Access Token” you will need it later;
Scroll down to “Scopes” section, Select Permission Scopes add “Post to specific channels in Slack” permission;
Re-install your app with new settings, select required channel;
In “Event Subscriptions” section on the left pane find section “Subscribe to Bot Events” and add “message.channels” event to make your bot act upon a message in a specific channel;
Go back to “Basic information” and note down “Verification Token” we gonna need it later;
For the Phase 2 we gonna need incoming webhook URL, so let’s find it in “Incoming webhooks” under “Webhook URL” and make a note of it.
Keep bot settings open in another tab of your browser, we will getting back here every once in a while when setting up AWS side of the bot.

Let me share a couple of screenshots for better visualization.

Bot permissions to send messages to Slack channel

Remark: We are only interested in sending and receiving messages to slack channels, in this scenario bot will not be handling messages sent in PM.

Phase 1: Teaching bot how to walk and talk.

At this stage our bot is a tabula rasa, newborn baby, and, as it happens with babies, we obviously need to teach him how to do some cool features for us.

For example, let’s teach “him” how to show, start, stop, restart AWS EC2 instances. And just for fun lets also add a functionality to get smart in replies when we send bot a messages with too many exclamation marks.

At this stage we have our “superbot” configured and its available in our slack channel. So far AWS side configuration is really simple and intuitive. There we have to create and configure a couple of components.

API Gateway.
Lambda Function (Python 3.6)
Scheduled CloudWatch event. I’ll get back to this one later.

Here (Lambda) we are going to have our Python code with slack event handler. We don’t need to import any Python modules that are not installed on AWS, so we will edit our code in built-in editor on Lambda.

Lambda function

Create a new blank Python 3.6 function. Give it a name, something like bot_event handler. Name handler “index.handler”
Add the following environment variables

BOT_TOKEN : Bot User OAuth Access Token taken from ”OAuth & Permissions” slack app settings page. We use this to authenticate our bot in slack. (Outgoing)
slack_token : Bot verification key taken from “Basic Information” slack app settings page. Required to verify that requests are actually coming from Slack. (Incomming)
aws_access_key_id : Key ID of AWS account with sufficient roles
aws_secret_access_key : Secret key of above mentioned AWS user
region_name : AWS region, where we want our bot to do some cool stuff for us.

Remark: Here you might want to use AWS KMS to encrypt/decrypt values of environment variables. For this function I use clear text values but for another one, later in this blog post, I’m going to make use of AWS KMS.

As for the function code (index.py) I wrote the following handler, it’s permanently available on my team’s GitHub repo

Bot_Event_Handler.py

Let’s pay attention to some moment in this code.

Importing modules. The one we focus on here is BOTO3, that allows us to interact with AWS services.

We parse JSON payload sent from Slack to our Lambda function through API Gateway resource. We need to have the first ‘ if ‘ (Line#27) to pass event subscription URL verification on slack setting page. We are going to deal with that a little bit later, just don’t give up reading.

Line#61: Here we do kinda important verification:

Message is sent to slack channel by real user (not a slack bot). Just for fun you can try to remove this one and enjoy watching a bot talking to himself in infinite loop in your Slack channel.
Message starts with ‘bot’ keywords
Token received in JSON payload matches slack_token environment variable.
If at least on of these conditions returns False, message is ignored and not processed by bot event handler. Remark: If bot doesn’t return anything to you channel doesn’t mean it doesn’t “read” what you’ve sent to Slack channel. Lambda function is invoked on all messages with no exceptions and all further verifications are happening inside a handler code. Use already mentioned slash commands if you want to save some $.

Line#68: Just for fun, we create a tuple of possible messages bot will send as a reply if we send him a message with more then one exclamation mark in a row. As a possible option here we can store all variations of reply messages in DynamoDB at Amazon and randomly pull one from there.

If we look a little up in the code we have defined a function “verb” that uses BOTO3 module to do some actions with AWS EC2 instance (Start/Stop/Restart) and returns result based on what’s happened inside the function.

The rest are showing EC2 instance (BOTO3), parsing, pattern matching, firing up messages back to slack (appending BOT_TOKEN env variable), and all other stuff required to cover all bugs discovered during an anti-humanistic testing and testing by dividing by zero, etc. 🙂

API Gateway

This one is going to be very simple.

Create new API, let’s say named “bot-trigger”
Create new resource, I named mine “superbot”
Create POST method withing early created resource. Integration type should be “Lambda Function”. Select Lambda region, then just start typing name on newly created function and select the one that popped up, Save -> OK.
Don’t forget to deploy API to a new (if this is a first time you are deploying API) stage once you are done configuring it’s resources and methods.
Go to “Stages” -> “name_of_the_stage” and note down Invoke URL, we gonna need it in a next step.

And some screenshots:

Finishing slack configuration

Done configuring AWS side (at this stage we think so at this moment), one minor thing is still left to make the whole process work.

Lets get back to Slack app settings page, go to “Event Subscriptions”, enable it and paste Invoke URL we just created in AWS Api Gateway.

What happens under the hood here is Slack fires up HTTP POST request to a URL we entered. It sends verification token and randomly generated “challenge”, it expects to get value of “challenge” back in a body of HTTPS reply. This verification message looks like this.

{
"token": "Jhj5dZrVaK7ZwHHjRyZWjbDl",
"challenge": "3eZbrw1aBm2rZgRNFdxV2595E9CY3gmdALWMmHkvFXO7tYXAYM8P",
"type": "url_verification"
}

Remember we have mentioned “challenge” in Line#27 of our code, so now we understand why we need that.

Keep lambda function warm or Postman Always Rings Twice

Assuming we passed verification of subscription URL and finally we should be good to go with all the setup and bot is ready to accept commands, furthermore do some actions and send us answers. Lets send a first message, and let screenshots talk for us:

Just like “Postman Always Rings Twice” our COLD bot always replies twice. The next message we send to bot got reaction from bot only once. And this is not what we really like and what we really want to have due to a number of reasons.

After some research I found it to be a Lambda Function warmup problem. Such Lambda behavior is well described in this blog post by Sam Corcos.

So I set rule in CloudWatch Events that pings my function every 15 minutes, and in this way always keeps it warm.

Problem solved and my little bot finally works as expected.

Phase1: Finally done, bot, what can you do?

Let me send my bot some messages and post screenshots here. Show AWS instances in a region specified in environment variables of Lambda Function

Ok we can see an instance is stopped, let’s start it up.

Oops, I forgot a syntax of “Start” command, and AWS could not find an instance with instaceID = “instance”. At least we can see that exceptions handling worked fine. Trying over

Lets see if it has really started up

OK, start command worked. Should we try “dividing by zero and sending command “restart” with no arguments?

Lets fire up some non existing command.

And at the end that “getting smart” functionality.

Phase 2: Monitoring notifications to Slack channel

So far we’ve been working on handling messages, sent to Slack channel, by our bot. We can see it works, accepts some basic commands, and this little bot can post results back to our Slack channel.

If bot can already post results back to us, why not use the same bot to push notifications from monitoring systems. Let’s teach our bot some more, well technically we are not teaching bot any more but just adding notification sent to Slack channel as a Bot user.

Assuming we’ve configured alarm in CloudWatch monitoring. For example, CPU utilization alarm with a threshold of 70%. Also we have an S3 Bucket, to store some files and we want real time monitoring of those objects. In this case I want to get notified when:

CPU utilization of specific EC2 instance is higher than 70%
File is added or removed from S3 bucket.

Earlier in this blog post, in section “Configuring slack side of the bot” I mentioned you should make a note of “Webhook URL”, so we need that one now.

Another lambda function

I created another function that handles notification delivery. First of all, I should mention I’ve taken an AWS Lambda blueprint Using “cloudwatch-alarm-to-slack-python3” as a base function, and then modified it according to my needs.

Remember we were talking about KMS earlier, lets use it to encrypt sensitive data such as web hook ULR stored in environment variables. How-to is well described in above mentioned blueprint’s comment block.

1. Navigate to https://<your-team-domain>.slack.com/services/new2. Search for and select "Incoming WebHooks".3. Choose the default channel where messages will be sent and click "Add Incoming WebHooks Integration".4. Copy the webhook URL from the setup instructions and use it in the next section.To encrypt your secrets use the following steps:1. Create or use an existing KMS Key - http://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html2. Click the "Enable Encryption Helpers" checkbox3. Paste <SLACK_CHANNEL> into the slackChannel environment variableNote: The Slack channel does not contain private info, so do NOT click encrypt4. Paste <SLACK_HOOK_URL> into the kmsEncryptedHookUrl environment variable and click encryptNote: You must exclude the protocol from the URL (e.g. "hooks.slack.com/services/abc123").5. Give your function's role permission for the kms:Decrypt action.Example:{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1443036478000",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": [
                "<your KMS key ARN>"
            ]
        }
    ]
}

So lets set our environment variables and make sure web hook URL is encrypted.

This is the code of our Lambda function, again it’s permanently available on my team’s GitHub repo

Notifications_to_Slack.py

Notification configuration

We need to configure AWS components (CloudWatch, S3) to sent notification to this Lambda function.

Create SNS topic that has newly created Lambda function as a subіscriber;

Configure CloudWatch alarm to send notification to this SNS topic;

Configure S3 Events notification

Phase 2: Results

At this stage we are done. Let me share some screenshots to show how it works.

I've loaded CPU on my instance for testing purposes and here's what we got in Slack Channel:

Some time later, I stopped a "CPU killer" process and got this in Slack

Noticed those colors? I’m sending text as a Slack message attachment, just because there I can apply colors and some cool formatting, see details here.

Lets add some files to S3 bucket.

Ok, I uploaded two files and got 2 messages. I might need to change it to send one message with a list of added files, but for testing purposes I consider this result a success. Remove one of the files:

AWS Lambda vs Azure Function

Everything works fine, but it's pretty tied up with Amazon Web Services. I wanted to try having my bot brains on a Microsoft serverless service called Azure Function.

I've set it up and wrote Azure Function. It works slightly different than Lambda so I had to modify my code a little bit.

Pros

Azure Function has an HTTP(S) endpoint, we dont need to set up anything like API gateway there and can throw request right to our function's endpoint.
User Interface. It's just my personal opinion, I felt pretty comfortable there as for the first time. Having "test" pane and function output pane on the same page is very handy.

Cons

Unfortunately there are a lot… To begin with, my code modified for Azure Function didn't work, due to a number of reasons. Of course it has no Boto3 python module, and it shouldn't be there actually, why would they care about AWS specific module, so first I tried to install it using this method. And here first of cons popped up.

Very and very slow Powershell Kudu console. I had to use CMD instead, it's a little bit better but still…

Once I got it installed I tried to import it and faced another problem:

Slow import of python modules (boto3). This could actually be caused by the next one from the list of cons:
Python on Azure Function is in preview so far;

It's OK, I can let it take some more time as long as I still can import module.
Still I could not run my function. Now it generated another errors and the next disadvantage of Azure Function I should have noticed at the very beginning (If I was smart enough to read documentation before trying).

Python 2.7 is default for Azure Function;

Good thing here is that we can install another version. I’ve installed 3.6.1 described in this article. Now I’m able to run my function with no errors.

Changed event subscription URL in Slack app settings, pointed it to AZ Function. And I got another error.

I’m sure it does respond with the value of challenge. I spent couple of hours here, testing function’s output. At the end it turned out that Slack doesn’t wait forever for challenge from URL. Azure function didn’t respond in a timely manner. Disabling import of Boto3 module solved the problem, verification passed. Obviously I had to enable it again once I got event subscription URL verified.
Another disadvantage of AZ Function

Not possible to encrypt sensitive data in environment variables.

OK, we finally got function working and URL verified. Writing some message to Slack channel hoping bot will do its job.

As you can see bot replied 4 times, first reply we received in around 30–40 seconds after we sent slack message and last reply with 5 min delay. We should be only getting one reply.

Then I played with setting of this function in host.json and function.json. I ended up blaming slow response of Azure Function, therefore removed import of all custom modules and made code really simple, doing one task: return reversed text. Here's the result:

Then I made the code even simpler: Return "Working on it…" message

I tried to send POST to AZ function HTTP endpoint with correct payload using CURL and function was triggered only once, at the same time Function was triggered 4 times when we fire it up from Slack.

Here I gave up, closed Azure Portal Console and changed URL back to API Gateway on AWS. If anyone, who read this article has any ideas how to pass through this, please leave it in a comment and I’ll definitely give it a try.

So here we go, goal achieved! We have a SuperBot that sits in Slack channel with us, sends us important notification and ready to do some basic stuff for us right in Slack channel.

Hope this blog post helped some people to understand what can we get from chat bot and where to start, feel free to experiment and bend it however you want, there are many variations of bot usage and many programming languages to help you set it up according to your needs.

And remember☝️🏼 bot is the only "person" in the world who is never going to ignore you in a messenger (as long as AWS and Slack still on the track😉).