How to Save AWS Infra Cost by Setting Cloudwatch Log Retention Programmatically

If you forget to set AWS CloudWatch Log retention then your log data will stay and accumulate indefinitely. Because the default configuration for log retention is Never expire.

You maybe notice that your AWS CloudWatch cost is increasing over time and you have too many Log Groups to set retention manually — you still can do it, but there is a better solution.

The better solution is described here, which provisions AWS Lambda, and EventBridge with CloudFormation.

However, let me provide you with a simpler version — a simple script that you can run in your terminal over CLI in two steps:

• Step 1: Copy the following code into your local file (let’s call it retention.py)

# This function set a desirable retention period for the existing/newly created CloudWatch log groups across
# different AWS regions
#
# For the original code/idea please refer to https://github.com/aws-samples/amazon-cloudwatch-retention-period-setter

import logging
import os

import boto3
from botocore.config import Config

# Appropriate logging
LOGGER = logging.getLogger()
LOGGER.setLevel(logging.INFO)
logging.getLogger('boto3').setLevel(logging.CRITICAL)
logging.getLogger('botocore').setLevel(logging.CRITICAL)

# Specify the regions to scan its log groups and set retention period for
# valid values for REGIONS_TO_SCAN variable? Possible values are: i.e., ap-southeast-1, us-east-1, us-east-2, us-west-1
regions = [item.strip() for item in os.environ['REGIONS_TO_SCAN'].split(",") if item]

# Specify the retention period in days. Possible values are: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545,
# 731, 1827,  2192, 2557, 2922, 3288, and 3653.
RETENTION_PERIOD_IN_DAYS = int(os.environ.get('RETENTION_DAYS', 30))
VALID_RETENTION_PERIOD_VALUES = [1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922,
                                 3288, 3653]

# Making sure we configure our boto3 client with a different Retry Configuration
custom_config = Config(
    retries={
        'max_attempts': 10,
        'mode': 'adaptive'
    }
)


def main():
    # log your environment variables
    LOGGER.info(f"Regions to be scanned = {regions}")
    LOGGER.info(f"Retention period to be set = {RETENTION_PERIOD_IN_DAYS}")

    # test retention period for a valid value
    if RETENTION_PERIOD_IN_DAYS not in VALID_RETENTION_PERIOD_VALUES:
        return {'statusCode': 200, 'body': '`RETENTION_PERIOD_IN_DAYS` is set to `' + str(
            RETENTION_PERIOD_IN_DAYS) + '`. Valid values are  1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, '
                                        '545, 731, 1827, and 3653.'}

    # test the regions
    if not regions:
        return {'statusCode': 200, 'body': 'No regions found in `REGIONS_TO_SCAN` variable. Have you configured it?'}

    # Iterate through each region, setting boto3 client accordingly
    for aws_region in regions:
        client = boto3.client('logs', region_name=aws_region, config=custom_config)
        response = client.describe_log_groups()
        nextToken = response.get('nextToken', None)
        retention = response['logGroups']

        while (nextToken is not None):
            response = client.describe_log_groups(nextToken=nextToken)
            nextToken = response.get('nextToken', None)
            retention = retention + response['logGroups']

        for group in retention:
            if 'retentionInDays' in group.keys():
                LOGGER.info(f"Retention is already set for {group['logGroupName']} LogGroup, {group['retentionInDays']} in {aws_region}")
            else:
                LOGGER.info(f"Retention is not set for {group['logGroupName']} LogGroup,in {aws_region}")

                setretention = client.put_retention_policy(
                    logGroupName=group['logGroupName'],
                    retentionInDays=RETENTION_PERIOD_IN_DAYS
                )
                LOGGER.info(f"PutRetention result {setretention}")

    return {'statusCode': 200, 'body': 'Process completed.'}


if __name__ == '__main__':
    main()

• Step 2: Run it with appropriate environment variables:

export AWS_PROFILE=<your aws profile>
export REGIONS_TO_SCAN=ap-southeast-1,us-east-1
export RETENTION_DAYS=7
python retention.py

That’s it! Now all your Log Groups have retention. Happy cost reduction!

Conclusion

In the blog, you learned how to save AWS infra costs by programmatically setting CloudWatch log group retention.

Disclaimer: We appreciate and would like to provide credit to Mohamed Wali for this idea.