Send Lambda Functions usage metrics to Amazon CloudWatch for tuning

This Blog has moved from Medium to blogs.tensult.com. All the latest content will be available there. Subscribe to our newsletter to stay updated.

In this blog post, I will describe how one can publish a custom metric to the CloudWatch for better monitoring of your Lambda functions cost.

What are CloudWatch metrics?

Metrics are data about the performance of your systems. By default, several services provide free metrics to resources like Amazon EC2 instances, Amazon EBS volumes, Amazon RDS DB instances and AWS Lambda functions. You can enable detailed monitoring on some of the resources like EC2 instances or publish your own application metrics. Amazon CloudWatch loads all metrics in your account for search, graphing and alarms. To know more, follow this documentation.

Why do we need Lambda Usage metrics?

Lambda functions send several metrics to CloudWatch, out of these there is only one metric related to usage which is Duration metric. But this metric is not sufficient for tuning the Lambda functions for better performance and cost optimisation. We can tune a Lambda function by changing the Memory allocated to it; when we change the memory of a Lambda function then Lambda service allocates corresponding CPU capacity to the function so higher the memory, higher will be the CPU. But it is not always true that higher the memory implies higher the cost as it is possible that Lambda executes much faster with high memory so overall cost might be lesser. It is not easy to determine the sweet spot, which memory gives optimal performance for the cost so we need to tune the function with different memory configurations. So we need to send some custom metrics to analyse how is our tuning impacting the cost and performance.

Lambda Usage Graph from CloudWatch

How does this automation work?

When a new CloudWatch log group is created, a CloudWatch event rule triggers Lambda-A(lets assume lambda called “A”). Lambda-A checks the data which is coming as event data. If the data is of Lambda log group then it puts subscription filter on the log group. Whenever the log group gets data in every lambda execution, the data is sent to Lambda-B. Lambda-B generates custom metrics from the log data and sends to CloudWatch.

So, now you have some basic understanding about this automation. Lets deploy it in AWS..

Prerequisites

• AWS account
• IAM user of that AWS account (AWS says that it is a best practice as everything has to be done by the IAM user, not from the root account)
• IAM user should be authorised to access services for creating this automation task.

Create IAM role

Here I am creating an IAM role and inline policy in the role for AWS Lambda service. By using this role, Lambda can access other AWS resources. Use the following JSON to create the inline policy.

To know how to create IAM role and attach policy for a particular AWS service, follow this documentation.

Create Lambda functions

Here I am creating two Lambda functions called Lambda-A and Lambda-B. You can give whatever names you want. While creating Lambda functions use the above role for both. Now use this code in Lambda-A and this for Lambda-B. Set environment variable(put destinationLambdaFunctionName in Key and Lambda-B in Value) in Lambda-A. Give the following permission to Lambda-B by using AWS CLI. For CLI reference, follow this documentation.

aws lambda add-permission --function-name "Lambda-B" --statement-id "Lambda-B" --principal "logs.REGION.amazonaws.com" --action "lambda:InvokeFunction" --source-arn "arn:aws:logs:REGION:ACCOUNT_ID:log-group:/aws/lambda/*:*" --source-account "ACCOUNT_ID"

Note: Replace REGION and ACCOUNT_ID with yours. Region is where you are setting up this whole automation and should be same everywhere.

To know how to create Lambda function in AWS, follow this documentation.

Create CloudWatch event rule

• Go to services → CloudWatch → Rules → click on Create rule
• Event Source → choose Event Pattern → select CloudWatch Logs in Service Name , AWS API Call via CloudTrail in Event Type (If you don’t have Trail setup in CloudTrail, do first. To get help refer this document), CreateLogGroup in Specific operation(s)→ Targets → select Lambda function → select Lambda-A previously created→ click on Configure details .
• Give Name, Description, State should be enabled → click on Create rule.

Conclusion

So now, you have finished the deployment of the whole automation. If you have observed carefully after deploying this automation, only metric data is passing from newly created Lambda log group to Lambda function. If you want to send custom metric for other Lambda then you have to set subscription filter on that Lambda log group. For existing log groups use this script and run this script using following commands.

Note: You need to install NodeJS and NPM in your local machine to run the script.

npm install aws-sdk
node FILE_NAME --profile PROFILE_NAME --region REGION --logGroupPrefix LOG_GROUP_PREFIX --filterPattern FILTER_PATTERN --filterName FILTER_NAME --destinationArn DESTINATION_ARN

The first command is for using npm to install aws-sdk dependency for the script. Second command is actually doing the execution of the script. In the command replace FILE_NAME, PROFILE_NAME, REGION, LOG_GROUP_PREFIX, FILTER_PATTERN, FILTER_NAME, DESTINATION_ARN with your data.

• FILE_NAME : Your script file name.
• PROFILE_NAME : It is AWS profile name which is configured in your local machine.
• REGION : AWS region. E.g. pass ap-south-1
• LOG_GROUP_PREFIX : Prefix of log group. E.g. pass /aws/lambda/ as string to do for only lambda log group.
• FILTER_PATTERN : Filter value that log group contains.
• DESTINATION_ARN : The ARN where you want to send data. E.g. your Lambda function ARN.

Now you can see that the custom metric is publishing automatically into CloudWatch. This is one use case of sending custom metric to CloudWatch. Apart from this there are many other useful cases where you can use custom metrics. So let me know if you have any other use cases.