Slack Integration with Cloud Custodian

Cloud Custodian is an open-source tool that you can use to manage your cloud resources. It allows users to define a policy written in YAML to ensure a secure, well-managed, and cost-optimized infrastructure. Cloud Custodian can be run locally, on an instance, or Serverless in AWS Lambda.

This article focuses mainly on integrating slack with Cloud Custodian but we will also cover the basics of Cloud Custodian first. So if you have no prior experience with Cloud Custodian, don’t worry, I got you covered. By the end of this article, you will be able to create a policy with Cloud Custodian as well as send alerts from Custodian to slack.

This article is for someone:

• New to Cloud Custodian
• Having prior experience with Cloud Custodian

This article will cover:

• Installing and Setting up Cloud Custodian
• Creating a policy in Cloud Custodian
• Slack integration and c7n-mailer
• Creating custom slack templates

Getting started

First of all, install Cloud Custodian on your system. You can follow the instructions from here or use the below command.

pip install c7n

Now make sure you have AWS CLI installed as Custodian uses your AWS creds while running policy. Configure your AWS credentials file before moving forward.

Open the AWS credentials file in your favorite editor.

vi ~/.aws/credentials

Add the following details and save the file.

[default]
aws_access_key_id=YOUR_ACCESS_KEY
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY
region=DEFAULT_REGION

Creating Custodian Policy

A Cloud Custodian policy typically consists of the following:

• The type of resource to run the policy against
• Filters to narrow down the set of resources
• Actions to take on the filtered set of resources

Let’s dive in and create our first Cloud Custodian policy to get all gp2EBS volumes.

Now run the policy using the following command:

custodian run gp2_ebs_volumes.yaml -s out/

The above command will run the policy specified (here gp2_ebs_volumes.yaml) and will store the policy output in out directory.

The contents of the policy output will look like this:

If you are getting any error make sure the IAM User whose credentials you are using have appropriate permissions.

Now, if you open the resources.json file, you can see the list of EBS volumes filtered on the basis of filters you specified in the policy.

Slack Integration

Now, we have the resources we need but how we can send notifications to slack. For this, we need to add notify action in the policy. The updated policy will look like this.

We will look into the changes we made in the policy later but first, we need to understand what notify action does and how Cloud Custodian handles it. Cloud Custodian provides a tool c7n-mailer, which we will be using for sending notifications to slack.

c7n-mailer

Architecture

Working

c7n-mailer creates a Lambda Function and a CloudWatch Event Rule in your AWS account. The CloudWatch Event Rule is scheduled for 5 minutes (default) and is added as a trigger to the lambda function. In other words, after every 5 minutes, the Event Rule triggers the lambda function. The lambda function reads the SQS queue(created while setting up c7n-mailer, we will look into this later in the article) for the available message, processes the message, and sends the notification to the Slack channel.

Now you must be wondering what kind of message does the lambda function reads from the queue and how is that message is sent to the queue. Don’t worry it’s nothing new we only have to go back where we left and look into the notify action we added in the policy. If you look carefully in the policy you will find that we have specified the following in the notify action.

slack_template: The slack message template to be used when sending notifications to slack.

slack_message_color: Severity (Danger | Warning | Good)

violation_desc: A message specifying policy violation details.

action_desc: A message specifying details regarding the action taken.

to: Slack Webhook Incoming URL

transport: Transport type and Queue URL.

When we run the policy, the notify action will send a message to the SQS queue containing filtered resources details as well as other required details. This message will be read by the lambda function (when triggered) which in turn will process the message and finally send the notification to slack using the Webhook URL you provided in the policy.

So till now, we have understood how c7n-mailer works, the only thing left is setting up c7n-mailer and running the updated gp2_ebs_volumes.yaml policy file.

Setting Up c7n-mailer

• Install c7n-mailer using the following command.

pip install c7n-mailer

• Create an SQS queue cloud-custodian-mailer-queue in your account.
• Create a mailer.yamlfile.

Note that we have added a role arn in the mailer.yaml file, this role will be used by the lambda function deployed by the c7n-mailer so make sure this role has appropriate permissions related to SQS, Lambda, and CloudWatch.

Run the following command:

c7n-mailer --config mailer.yaml --update-lambda

This will deploy the cloud-custodian-mailerlambda function and the cloud-custodian-mailerCloudwatch event rule in your AWS account.

cloud-custodian-mailer lambda function

cloud-custodian-mailer event rule

As you can see from the screenshot, the CloudWatch Event rule is added as a trigger for the lambda function.

The only thing left now is running our updated script again and waiting for the slack notification.

The notification will look like below.

The layout of the message is according to the slack template provided by us in the notify action. We have specified slack_default slack template, you can check out the code for slack_default template from here.

Custom Slack Templates

To customize the slack message received, we have to create a new template according to our needs and specify this template in notify action. The template is written in Jinja2.

actions:
    - type: notify
      slack_template: your_custom_slack_template_name

By looking at the default template you can get a rough idea about how things are working, you can refer to this slack guide on Secondary message attachments to get full insights about various elements present in the template and design your custom template.

But the template will eventually render data, so where we can find the data which we can use to render data in the template. If you recall, when we ran the policy, a resources.json file is created, and if you look at the slack_default template, you can see how it is using the resources array which we have also seen in the resources.json file. But how is that resources data accessible to our slack template? It’s easy, the message sent by the policy to the SQS queue contains the resources data, the notify action internally adds the resources data to the SQS message along with other details. So, if you want to see what other details you can render in your slack template, you can log the SQS message from your cloud-custodian-mailer function and add the respective fields in the template.

I have created a simple custom template to render a few other details along with the way they are presented in the slack message. You can do all the creativity you want, I have kept it simple.

Custom Slack Template

Finally, we must update the cloud-custodian-mailer lambda function to add the custom slack template. Create a templates directory, copy the template file inside it, and run the following command.

c7n-mailer --config mailer.yaml -t templates --update-lambda

Run the policy again and wait for the slack notification. The updated message will look like this.

This concludes our tutorial on how to integrate slack with Cloud Custodian and creating custom slack templates.

If you find this article useful please share it with your friends and colleagues.

Let’s connect on LinkedIn, Github, Twitter.