Forwards logs from AWS S3 to AWS CloudWatch real time

AWS CloudWatch is a monitoring service for AWS resources, you can use it to collect logs, metrics and set alarms or automatically react to changes in your AWS resources. However, there are some AWS Services logs that are send to S3 by default.

• Load Balancer Access Logs
• Cloudfront Logs
• WAF Logs

To leverage on AWS CloudWatch capability you can actually forward logs real time from S3 to CloudWatch using the configuration below.

Forward Logs from S3 to CloudWatch Architecture

The configuration for sending the logs involves the following steps:

1. Create IAM Role with the relevant permission to access S3 and write logs to cloudwatch.
2. Create the cloudwatch log group.
3. Create the lambda function with the necessary logic to extract the logs and send it to cloudwatch in batches.
4. Attach the IAM Role to the lambda function.
5. Create the event notification and attach it to the S3 bucket.

In the example below, i will be forwarding Application Load Balancer Access logs to cloudwatch. The entire setup will be using Terraform Infrastructure as Code. You can easily achieve the entire setup without manually configuring anything via the console. The code can be found in my github repository, all you need to do is to update the dev.tfvars environment variables and run the command listed in the README.md.

Create IAM Role with the relevant permission to access S3 and write logs to cloudwatch.

Create a file iam-role.tf , this will include all the necessary permission lambda execution requires. We will need to grant the lambda permission to find the log group and create log stream to put the logs into it. The lambda must be given permission to GetObject from S3.

Create the cloudwatch log group

Create a file cloudwatch.tf , this is the cloudwatch log group where all the logs will be forward into.

Create the lambda function with the necessary logic to extract the logs and send it to cloudwatch in batches

Create a folder forward-logs-cloudwatch-lambda and createindex.js file in the directory.

As the lambda code include several components, i will be adding comments in the codes. Do refer to the explanation in the code to understand the purpose of each component. We will be using the following AWS APIs:

• s3.getObject
• cloudWatchLogs.describeLogGroups
• cloudWatchLogs.createLogGroup — Optional as we already pre-create in the previous step
• cloudWatchLogs.describeLogStreams — Use to retrieve the sequence token
• cloudWatchLogs.createLogStream — Create a new log stream when the existing stream session expired
• cloudWatchLogs.putLogEvents — Add Logs into the stream with the sequence token as parameter

Attach the IAM Role to the lambda function

Next, we will be creating lambda.tf to run the lambda function and attach the IAM Role to the lambda function.

The first portion of the terraform code is to reference the Node JS lambda code earlier and create the function.

Create the event notification and attach it to the S3 bucket

In the same lambda.tf file, we will now give S3 permission to execute the lambda function and add the lambda function as event notification in the S3 bucket. We will configure the event when new object created in S3 it will trigger the lambda function — this will help to forward the logs in real time.

Conclusion

With this 5 steps, you can now forward your S3 logs to Cloudwatch and leverage on cloudwatch function to better manage your logs. You can refer to the entire terraform code based in my github account. Do comment below or reach out to me via LinkedIn if you need clarification on the implementation.