AWS CloudWatch- Infrequent Access with Lambda and Serverless | Save upto 50% on CW cost

Jump to TLDR for implementation later in the page, check github repo for full implementation.

if not enjoy the full story below:

AWS can be expensive, everyone knows that. However, what’s truly interesting is understanding how to keep costs manageable. Having worked with AWS for many years, I’ve learned a host of strategies to maintain a tight budget, and I’m excited to share a new method with you.

Here at Speridian Technologies I have learned the importance of a robust monitoring and observability architecture within AWS. It’s not just about gathering data — it’s about gaining actionable insights that can guide informed decision-making. But here’s the rub: sophisticated monitoring, particularly with AWS CloudWatch, can become prohibitively expensive, especially when you’re tracking a large number of resources or when your logging requirements are high.

The costs can sneak up on you. A few cents for a metric here, a few more for an alarm there, and before you know it, you’ve got a runaway bill that’s as opaque as it is oversized. Until now, the strategy has been to be judicious with what you monitor — every log, metric, and dashboard must justify its existence. But that often meant trade-offs between cost and visibility.

That’s where the newly launched AWS feature of CloudWatch Infrequent Access comes in. This game-changing service allows users to continue to benefit from CloudWatch’s detailed monitoring and observability tools while mitigating cost concerns. AWS CloudWatch Infrequent Access is designed for logging data that you need to keep but access less frequently. It’s a more cost-effective solution for long-term storage of operational data, without sacrificing the ability to access this information when you need it. It has its limitation and you have to decide between what to keep in standard and what to push towards infrequent access.

Here’s how it works: Instead of persistently storing all data in a higher-cost tier, you can now classify your logs as “infrequent access” based on your access patterns. This data is then stored at a significantly reduced cost, and while access times may be slightly slower along with some other standard features, you can review the table here, it’s a trade-off that can lead to substantial savings without compromising on the depth of insight.

Imagine reducing your CloudWatch costs by up to 50% for logs that you access less than once a month. For many businesses, this can translate into thousands of dollars saved annually, making the choice between financial efficiency and operational excellence a dilemma of the past.

Official blog post and many other article online only talks announcement of this feature and manual way of doing it using online console. But we all know, this is not how architecture is deployed in real life. Below I am sharing a quick way on how to set infrequent access for serverless architectures deployed using AWS CloudFormation and SAM.

TLDR;

Find full code here on github

When we define a lambda in our cloudformation template like this.

Resources:
  NiceJob:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ../src/job
      Handler: nice-job.handler
      Runtime: nodejs16.x
      ...
      ...

AWS, under the hood creates log group for us in “STANDARD” class. Which is of course expensive for what it’s worth.

Below is how you can define lambda in cloudformation with INFREQUENT_ACCESS log group of cloudwatch.

Resources:
  NiceJob:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ../src/job
      Handler: nice-job.handler
      Runtime: nodejs16.x
      Timeout: 30
      Policies:
       - AWSLambdaExecute
      Role:
        Fn::GetAtt:
          - NiceJobLambdaExecRole
          - Arn       
      LoggingConfig:
        LogGroup: !Ref NiceJobLogGroup
  NiceJobLambdaExecRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: NiceJobLambdaLogging
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: !GetAtt NiceJobLogGroup.Arn
  NiceJobLogGroup:
      Type: AWS::Logs::LogGroup
      Properties:
        LogGroupName: !Sub "/aws/lambda/${AWS::StackName}/NiceJob"
        LogGroupClass: INFREQUENT_ACCESS

Over here, we are basically creating log group ourselves and assigning a class to it, we also create execution role along with it. This is very simple and straightforward solution which can potentially save thousands of dollars every month.

AWS continues to evolve, and with features like CloudWatch Infrequent Access, it’s empowering users to find the right balance between cost and capability. By making the most of this new feature, you can significantly lower your AWS bill while still maintaining the monitoring standards necessary for your operations.

Let me know what you think of it.