Manage AWS CloudWatch Log Group Retention using Automation

This Blog has moved from Medium to blogs.tensult.com. All the latest content will be available there. Subscribe to our newsletter to stay updated.

What is CloudWatch Logs? Amazon CloudWatch logs lets you monitor, store and access your log files from Amazon EC2 instances, AWS CloudTrail, Lambda functions, VPC flow logs, or other resources.

CloudWatch logs provides several features such as searching by keywords, patterns and date ranges etc and cloudWatch logs is classified as “hot storage” as we would be using it for debugging live issues. Hence it provides lot of the features. It is much costlier than other storage options such as S3. So in order to get most out of CloudWatch log keeping the cost under control, it would be ideal to move the older logs to S3 for a long term retention, you can learn more about this from this blog.

CloudWatch organises logs in a log group and when a new log group is created, it’s retention period is set to Never expire, which means logs will be retained forever.

What is accomplished by this automation?

When a new CloudWatch log group is created, this will set a desirable retention time for that log group. After that retention time all log stream(s) data of log group will be deleted automatically.

Prerequisites

• AWS account
• IAM user of that AWS account (It is best practice for everything to be done by the IAM user, not from root account)
• IAM user should be authorised to access services for creating this automation task.

How this automation works?

When a new CloudWatch log group is created, a CloudWatch event rule triggers a lambda function. Then the lambda function sets a desirable retention time for the CloudWatch log group.

Create IAM role

Here we are creating IAM role for an AWS service called Lambda. By using this role Lambda can access the other AWS resources.

Create an IAM role and add the following policy into that. To know how to create IAM role and attach policy for a service please refer to this document.

Create Lambda function

Create an AWS Lambda function and place this code in that. While creating a lambda function runtime it should be on Node.js 8.10 and choose previously created role in existing role. To know how to create AWS lambda function refer to this blog.

Create CloudWatch event rule

• Go to services → CloudWatch → Rules → click on Create rule .
• Event Source → choose Event Pattern → select CloudWatch Logs in Service Name , AWS API Call via CloudTrail in Event Type (If you don’t have Trail setup in CloudTrail, do first. To get help refer this document), CreateLogGroup in Specific operation(s)→ Targets → select Lambda function → select previously created lambda function → click on Configure details .
• Give Name, Description, State should be enabled → click on Create rule .

Conclusion

Now that we have learnt how to set CloudWatch log groups retention time automatically. If you have observed carefully, after deploying this automation into AWS account, it will only set retention time for newly created log groups. But what about already existing log groups? We need to set retention time for them also right? How do we do that? You can use our open sourced NodeJS script to set the retention for all the log groups at once. Let us know if you have faced any issues while running this script.

And stay tuned, for my next blogs..