Running your application in the cloud can help you avoid dealing with many hassles. For example, you don’t need to worry about the infrastructure setup, operating system patches, etc. But if something goes wrong, either caused by the bugs of your own application, or the cloud provider’s service outage, the worst thing that could happen is that you don’t know it at all until all your customers’ phone calls flood to your support team.
The AWS CloudWatch Alarms allows you to watch CloudWatch metrics and to receive notifications when the metrics fall outside of the levels (high or low thresholds) that you configure. Monitoring everything is one of the principles for the devops team.
Where do you want to receive the alarm notifications? Email? Text Message? If your team is using Slack, a slack channel might be the best place to receive the notifications. Your whole team can monitor the alarms, discuss and take actions right there in your daily workspace.
How it works?
Alarms watch metrics and execute actions by publishing notifications to Amazon SNS topics that has a Lambda function subscribed to it, the Lambda function is invoked with the payload of the published message. See here. The lambda function then can post a message to a slack channel using the slack incoming webhooks API.
Follow the instructions here to setup the Incoming Webhooks. Incoming Webhooks are a simple way to post messages from apps into Slack. Creating an Incoming Webhook gives you a unique URL to which you send a JSON payload with the message text and some options.
When you use the Webhooks URL to post a message, there is no authentication on the Slack side. The URL itself is the secret for the incoming webhooks API. You don’t want to share the url to anyone except the lambda function.
We need to setup SAM CLI to develop this application. See here.
In the template file, we defined a parameter SlackChannelParameter for the slack channel name so that we can post messages to a specific slack channel. This parameter will be passed to lambda function as environment variable.
We also defined a parameter ProductNameParameter so that we can deploy this application for each of our product teams. Each of our products will have it’s own slack channel and SNS topic.
The slack webhook url is a secret url to the slack webhook API. This string should not be exposed to any code or user except the lambda function that will use it to post the slack message. We store the webhook url in the AWS systems manager Parameter Store as a secure string that is encrypted by a KMS key declared in the template file. In the template, we give the lambda function access of the parameter store and the KMS key.
Here is the template file
Here is the index.js
create aws s3 storage bucket
aws s3 mb s3://cloudwatchalarmtoslack-deploy-package
build the sam application
sam build -m package.json
package the application and save to the s3 bucket
sam package — output-template-file cloudwatchalarmtoslack-deploy-template.yaml — s3-bucket ‘cloudwatchalarmtoslack-deploy-package’
deploy the application to aws cloudformation for product istrada
sam deploy — template-file ./cloudwatchalarmtoslack-deploy-template.yaml — stack-name cloudwatchalarmtoslack-istrada — capabilities CAPABILITY_NAMED_IAM — parameter-overrides SlackChannelParameter=istrada-devops ProductNameParameter=istrada UsernameParameter=liuhongbo EnvironmentParameter=prod
After the application is deployed to cloudformation successfully , go to AWS console parameter store and add a parameter with path set to /prod/cloudwatchalarmtoslack/istrada/hookurl as a secure string encrypted with the key cloudwatch-alarm-to-slack-encryptkey-istrada.
In order to test the application, we can use aws cli to manually publish a message to the topic.
First, create a file message.txt with content:
Find your topic arn in your cloudformation stack’s resources list.
then publish the message,
aws sns publish — topic arn:aws:sns:us-east-1:746123053018:istrada-cloudwatch-alarm — message file://message.txt
You should see the message in your slack channel.