Generating Slack Notifications from AWS CloudWatch Alarms

Published in

Analytics Vidhya

8 min readSep 1, 2020

In my foray into cloud computing, I have decided to try out different things and document them in an article. My first article will be focused on sending CloudWatch notifications to a Slack workspace using the Slack Webhook and AWS Lambda function. The AWS Services that will be used to achieve this include an EC2 instance, CloudWatch, AWS SNS(Simple Notification Service) , Lambda & AWS Systems Manager Parameter Store.

AWS CloudWatch

Amazon CloudWatch is a tool that captures all the information necessary to track the performance of all applications hosted in the AWS cloud. CloudWatch tracks measures like latency, request count, application logs, CPU usage, & other custom metrics. CloudWatch logs are captured in near-real-time and CloudWatch Alarms can be used to watch your metrics against a specified threshold and proactively make decisions like trigger an EC2 instance to auto-shutdown when the CPU usage crosses a particular threshold.

ARCHITECTURE

The EC2 instance will periodically send its CPU usage to CloudWatch. Once the CPU usage crosses a particular threshold (40%), the CloudWatch alarm goes from “OK” to “In Alarm”. When this happens, a message is sent to an SNS topic which in turn triggers a lambda function that retrieves the pre-saved slack webhook URL from SSM Parameter Store and sends a notification to the slack workspace.

BUILDING THE APPLICATION

Below are the steps needed to make this happen:

Create a Slack app: First, we need to need to create a Slack app at https://api.slack.com/apps/new and set a development workspace for the application. When the Slack app is created, the next step is to activate incoming webhooks. This will enable us to post messages from external sources into our Slack workspace. After activating incoming webhooks, we then add a new webhook to the workspace by clicking the button at the bottom of the page. You will get a prompt in your slack channel once the integration is complete. Once the integration is successful, we copy the webhook URL at the bottom of the page and store it in a SSM parameter store as we proceed.

2. Create an SNS topic for our application: Next, we are going to create an SNS topic that will be responsible for receiving the cloud watch messages when the CPU usage of our EC2 instance crosses a certain threshold. On your AWS console, navigate to Amazon SNS and create a new topic. Enter the topic name and click “create topic” as the default settings would suffice for what we are hoping to achieve.

Alternatively, you can create an Amazon SNS topic from the command line by using the command below

aws sns create-topic --name High_CPU_Usage

3. Create an EC2 instance: Next we are going to launch an EC2 instance to generate metrics that will be reported to CloudWatch. We are going to launch a free tier eligible LINUX instance of type t2.nano, 1 vCPU, and 0.5gb of memory. Since this instance has only one instance, it shouldn't be difficult to stress the CPU so we can generate a high alarm message on CloudWatch based on the set threshold. We take note of the instance id because we are going to need the instance id to create a CloudWatch alarm.

make sure your instance is running then click on monitoring, and enable detailed monitoring. we enable detailed monitoring for convenience purposes, this will allow us to track metrics generated from the instance every minute. After enabling detailed monitoring, we then proceed to create the CloudWatch alarm.

4. Create a CloudWatch Alarm: Next we are going to create a CloudWatch alarm to receive metrics from the running instance. On the AWS console, search for Cloudwatch, click alarms on the left panel, and create an alarm. Select metrics, select EC2 and per-instance metrics, and paste in the ID of the EC2 instance we just created and check the CPU utilization checkbox.

in the “Specify Metric and Conditions” section, we set the Alarm name to HighCpu and set the threshold type to “Static” and define the condition as greater than 40. we then select the SNS topic we created earlier as the recipient of the alarm notifications when we cross that particular threshold.

when you create the alarm, you will notice that it has a status if “Insufficient Data”. This is because it has not collected enough data from the running instance. Alternatively, you can create a CloudWatch alarm with the command below from the command line interface.

aws cloudwatch put-metric-alarm \
    --alarm-name HighCpu \
    --metric-name CPUUtilization \
    --namespace AWS/EC2 \
    --statistic Average \
    --period 60 \
    --evaluation-periods 1 \
    --threshold 40 \
    --comparison-operator GreaterThanThreshold \
    --dimensions Name=InstanceId,Value=i-0fa30e34dddb555a9 \
    --alarm-actions arn:aws:sns:us-east-1:123456789012:high-cpu-alarm \
    --unit Percent

5. Create a System Manager Parameter: Our slack webhook URL is unique to our workspace and we do not want just anybody to have access to it. so we are going to store it in the AWS parameter store like a credential and retrieve it whenever we need it. on the AWS console, we navigate to AWS systems manager and click on the parameter store. We then create a new parameter to hold the value of the Slack webhook URL. set the name of the parameter to slackwebhookurl, select the type as “securestring” to encrypt the passwords, and paste the URL in the value section and click create parameter.

Alternatively, you can create an SSM parameter with the command below from the command line interface where $WEBHOOK_URL is the variable that houses the webhook.

aws ssm put-parameter --cli-input-json '{"Type": "SecureString", "KeyId": "alias/aws/ssm", "Name": "SlackWebHookURL", "Value": "'"$WEBHOOK_URL"'"}'

6. Create the Lambda Function: The next step is to create the lambda function that will push the high alarm notifications to slack. before we create a Lambda function, we need to create an IAM execution role for our function. we navigate to the IAM page from the AWS console and we click on roles. Click on create role, select “AWS service” as the type of entity, and choose Lambda as the service. We need to grant Lambda full access to SSM and also a basic execution role.

Now, we can go ahead and create the function. Navigate to AWS Lambda and click on create function. Assign a name to your function, select the python runtime, and also select the role that we just created.

Now we can go ahead and create the function.

The next step is to add a trigger to our function. Triggers are services that execute the functions they are attached when a specified condition is met. We will select the SNS topic that receives messages from CloudWatch as our trigger. note that you can add more than one trigger to a Lambda function.

The Lambda function is going to receive the message from the SNS topic in JSON format. Then we would extract entities like Alarm name, New State Value, and reason for the current state in a variable. we’d then create a slack message template that contains information from the entities above. Next, we would retrieve the webhook URL to our workspace and make a request to the API. See code snippet below.

import json
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopenimport boto3#Create a SSM Client to access parameter store
ssm = boto3.client('ssm')#define the function
def lambda_handler(event,context):
    #retrieve message from event when lamda is triggered from SNS
    print(json.dumps(event))
    
    message = json.loads(event['Records'][0]['Sns']['Message'])
    print(json.dumps(message))
    
    '''
    Retrieve Json vriables from message
    AlarmName is the name of the cloudwatch alarm tht was set
    NewStateValue is the state of the alarm when lambda is triggered which means it has 
                gone from OK to Alarm
    NewStateReason is the reason for the change in state
    '''
    
    alarm_name = message['AlarmName']
    new_state = message['NewStateValue']
    reason = message['NewStateReason']
    
    #Create format for slack message
    slack_message = {
        'text' = f':fire: {alarm_name} state is now {new_state}: {reason} from Nesq\n'
                 f'```\n{message}```'
                    }
    #retrieve webhook url from parameter store
    webhook_url = ssm.get_parameter(Name='slackwebhookurl', WithDecryption=True)
    
    #make  request to the API
    
    req = Request(webhook_url['Parameter']['Value'],
                    json.dumps(slack_message).encode('utf-8'))
    
    try:
        response = urlopen(req)
        response.read()
        print("Messge posted to Slack")
    except HTTPError as e:
        print(f'Request failed: {e.code} {e.reason})
    except URLError as e:
        print(f'Server Connection failed:  {e.reason})

Testing our application

Now we would SSH into our Linux instance. Navigate to the EC2 page from the AWS console and connect to our using a standalone client or the browser-based SSH connection. I personally prefer the browser-based SSH connection because it is fast and seamless.

Now, we are going to use the Linux stress utility to stress the CPU which will concurrently increase CPU usage. Stress is a Simple command-line utility used to conduct CPU memory and disk tests. Now, we have to install some extra Linux packages in our instance in order to make this utility available. Afterwhich we would install stress.

# Install Extra Packages for Enterprise Linux
sudo amazon-linux-extras install epel# Install stress
sudo yum install -y stress

Now, we’re going to beat up the CPU for 5 mins with the code below; This will push our CPU to 100% usage. we would then monitor the CloudWatch alarm as the CPU is being stressed.

# Beat it up for 5 mins
stress --cpu 2 --timeout 300s

Below is chart of our CPU Usage and the Alarm state changing from “OK” to “In Alarm”

Now we can check cloud logs to see if our function has been successfully executed.

Below is the Notification on slack when Lambda was triggered.

CONCLUSION

Organizations that use several AWS services in their day to day activities can leverage this app to generate real-time custom notifications from CloudWatch on slack based on whatever conditions you set.

Code Repository:

https://github.com/ToluClassics/Slack-AWS-Integration

References:

https://linuxacademy.com/