Exporting cloudwatch logs to S3 through Lambda before retention period.

We recently had a requirement where Cloudwatch streams were to be transferred to S3 after ’x’ days and post the export task, the logs were supposed to be deleted from the cloudwatch console.

Approach: Lambda will daily trigger the script at 12:01 am and will transfer the logs of the whole day (00:00–23:59) for ’n’ days before the same date script is running on. For instance, if the script is running on 18–02–2019, it will transfer all the logs of 12–02–2019 (00:00–23:59).

The deletion part is taken care by the retention policy. To make sure the logs are put to S3 before the retention period, you can keep the log retention period > log-export-date.

API’s and libraries used:

Permissions required:

  • Lambda_basic_execution: This role had full permission of cloudwatch and S3
  • S3 bucket policy: Bucket policy to specifically grant cloudwatch permission to GetBucketAcl and PutObject in S3.

Code flow:

create_export_task() of CloudWatchLogs library of boto was extensively used for creating the export operation to S3. It creates an export task, which allows you to efficiently export data from a log group to an Amazon S3 bucket. This is an asynchronous call. If all the required information is provided, this operation initiates an export task and responds with the ID of the task

Datetime was used to calculate today’s date and then figure out which day’s logs are to be transferred today.

Math library was used to floor up the floating integer returned as timestamp and make the parameter acceptable by ToTime and FromTime within the create_export_task.

Stepwise implementation

  1. Create a lambda role with the following permissions:
Lambda role with following policies attached

2. Add the following bucket policy in the target destination bucket to specifically grant cloudwatch permission to GetBucketAcl and PutObject in S3:

{
“Version”: “2012–10–17”,
“Statement”: [
{
“Effect”: “Allow”,
“Principal”: {
“Service”: “logs.us-east-2.amazonaws.com
},
“Action”: “s3:GetBucketAcl”,
“Resource”: “arn:aws:s3:::bucket-name”
},
{
“Effect”: “Allow”,
“Principal”: {
“Service”: “logs.us-east-2.amazonaws.com
},
“Action”: “s3:PutObject”,
“Resource”: “arn:aws:s3:::bucket-name/*”,
“Condition”: {
“StringEquals”: {
“s3:x-amz-acl”: “bucket-owner-full-control”
}
}
}
]
}

3. Switch to lambda console now, choose the author from scratch option and create a new function with python 3.7 as the runtime environment, choose the existing role we created, which is, lambda_basic _execution in this case.

4. You will land on to screen like this.

Lambda console

5. Choose cloudwatch events as a trigger, you’ll then redirected to configure the triggers

Add the log group name you want to set us trigger

Choose the desired log group, you can add multiple log groups if required.

6. Choose cloudwatch event for running the cron, I wanted the cron to run at 12:01 am every day, so I have configured it that way, you can change the cron expression based on your requirements with the help of this. One thing to remember is the cron expression for aws rule has 6 characters, unlike others which have 5 only.

Cron values for every character

7. Add the following code inline in chosen lambda function, for the handler, put file-name.function-name, the name python file I made is retention.py, and within the code, I have the main function as lambda_handler, so handler for me is retention.lambda_handler.

Please note that fromTime and to fields accept the timestamp in milliseconds format and in the normal timestamp, the number of seconds that have elapsed since 00:00:00 Thursday, 1 January 1970. So to convert it in milliseconds, I’m multiplying it by 1000.

Also, increase the default timeout settings to 13 seconds*number of log-groups, in case you’re adding more log groups to array group_name[]. We are using this to pause the execution for a while as create_export_task has a limitation which says that:

Each account can only have one active (RUNNING or PENDING ) export task at a time.

So after execution of the first log group, the code waits for 10 secs for the first task to mark it as completed and then move to another log group.

You can also add a retention policy based on your use-case from the console. Since it is a one-time task and cron is not required for this, I have added this from the console itself. I have added a function as well to calculate the retention period based on the user for ‘nDays’.

Pop-up when you click on ‘Never Expire’.

Edit 1: In case you want to iterate through all the log groups present in cloudwatch, add this function to fetch all the group names:

def group_names():
groupnames = []
paginator = log_file.get_paginator('describe_log_groups')
response_iterator = paginator.paginate()
for response in response_iterator:
listOfResponse=response["logGroups"]
for result in listOfResponse:
groupnames.append(result["logGroupName"])
return groupnames

And Replace group_name = group_names() under def lambda_handler(event, context).

Do not forget to increase the lambda time-out settings.

Save and run the function to test it and let the lambda spark. :)