Forwarding CloudWatch Logs to Logentries using CloudFormation

Photo by Robert Larsson on Unsplash

The Logentries documentation describes how to setup CloudWatch Logs forwarding via the AWS Console. This process is tedious and time-consuming if you need to forward many log groups, so let’s save time by using CloudFormation!

Note: Logentries is now called Rapid7 InsightOps, but that’s a mouthful, so I’ll refer to the service by its former name.

High-level architecture

CloudWatch Logs can stream log data to other AWS services by attaching a subscription filter to a log group, with a filter pattern determining which logs are sent to downstream services (and which aren’t). We’ll use this feature to selectively stream data to a Lambda function which will upload the logs to Logentries.

This diagram depicts how log data moves from the originating AWS resource, through CloudWatch Logs to Lambda, and finally, Logentries.

Preparing the deployment package

Download the official Logentries Lambda function from GitHub. This is the code which will send our CloudWatch Logs to Logentries. Technically, this can be used as-is however, to reduce the size of the deployment package, we’ll remove the unneeded files.

https://github.com/rapid7/r7insight_lambdaCW

Create a new archive containing only r7insight_lambdaCW.py and the certifi directory, then upload this to S3. Let’s assume we have access to a bucket named my-deployment-packages.

aws s3 cp r7insight_lambdaCW.zip s3://my-deployment-packages/logentries/r7insight_lambdaCW.zip

Our deployment package is in S3 and ready to be referenced by a CloudFormation template. Excellent!

The CloudFormation template

Like all good cooking programmes, here’s one I prepared earlier.

This template declares four AWS resources:

1. a Lambda function, which will execute the code in our deployment package;
2. a CloudWatch Logs subscription filter, which calls the Lambda when new logs match its filter pattern;
3. a Lambda permission, which allows the subscription filter to invoke the function;
4. and an IAM Role, which Lambda will assume to execute the function

…and expects two parameters:

1. the name of a CloudWatch Logs log group, to which the subscription filter will be attached;
2. and a Logentries log token, which will be used by the Lambda function.

To keep things simple, I’ve hard-coded the filter pattern to the empty-string (which will match all logs) and the Logentries region to eu. Of course, these values can be changed or passed as parameters too; adjust to taste!

We’ve a deployment package and an associated CloudFormation template, now it’s time to deploy it!

Stack deployment

Suppose we are running an ECS service named my-ecs-service. CloudWatch Logs will create a default log group for this service named /ecs/my-ecs-service — this is the LogGroupName we need to provide to our CloudFormation template! We also need to provide a LogentriesLogToken, which can be obtained by creating a new log in Logentries.

We have our parameters and we're ready to deploy our CloudFormation stack.

aws cloudformation deploy \
  --template-file logentries.cloudformation.yml \
  --stack-name my-ecs-service-logentries \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides ‘LogentriesLogToken=1a2b345c-d6e7–890f-1a23–45b67c8d9012’ ‘LogGroupName=/ecs/my-ecs-service’

Voilà! We have a CloudWatch Logs subscription filter which executes a Lambda function when new logs are written. The function sends the logs to Logentries, accompanied by the provided log token.

Repeat for each desired log group or, for bonus points, orchestrate multiple deployments using a configuration management tool such as Ansible, or group the deployments into one template using CloudFormation nested stacks.

Bon appétit!