Monitoring AWS RDS Free Storage With AWS Cloudwatch

Background

The primary objectives of DevOps comprises Monitoring along with Culture, Automation and Sharing responsibilities. AWS has always played a vital role in flourishing the DevOps philosophy. Databases are the crucial part of every web-based applications. Storage requirement for databases grows on an incremental basis everyday. In this post, I will be diving into how we can leverage the technologies provided by AWS to address these needs.

Introduction

In AWS, We use RDS for hosting databases; cloudwatch for monitoring, SNS to send notification and Lambda as a Compute service for Serverless applications.

As Shown in Above diagram, We will be setting up a Free Storage metric in Cloudwatch and create an alarm against it. The alarm will publish message to SNS Topic when there is state change in cloudwatch Alarm. The SNS Topic will then send an email to the assignee who is responsible for that Free Storage (probably the one who created and manages the resources). The SNS Topic will also invoke the lambda function which inturn send the notification to Slack using incoming web-hook so that all the people subscribing to Alerts in that particular channel in slack related to RDS get notified.

Cloudformation Templates

We will be using AWS Cloudformation to provision and manage these resources. Cloudformation is a Management tool for Developers and DevOps to manage resources using Infrastructure as Code (IAC).

Creating a Cloudwatch Alarm

The following AWS Cloudformation template creates a AWS Cloudwatch alarm setting threshold of 2 GB. We can only apply bytes as metrics and also have Threshold value in bytes (i.e. 2GB = 2147483648 bytes).

AWSTemplateFormatVersion: '2010-09-09'
Description: CloudWatch Alarm for FreeStorage Monitoring in RDS
  
Resources:
  RDSFreeStorageAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmActions:
        - !ImportValue 'RDS-Alert-SNS-Topic'
      AlarmDescription: 'RDS Free Storage Alarm for quiz-development-RDS'
      ComparisonOperator: LessThanThreshold
      DatapointsToAlarm: 1
      Dimensions:
      - Name: DBInstanceIdentifier
        Value: quiz-development-RDS
      EvaluationPeriods: 1
      MetricName: FreeStorageSpace
      Namespace: AWS/RDS
      OKActions:
        - !ImportValue 'RDS-Alert-SNS-Topic'
      Period: 60
      Statistic: Average
      Threshold: 2147483648
      TreatMissingData: ignore
      Unit: Bytes

We have set the actions i.e. either when the alarm is OK state (when there is enough free storage than the threshold 2GB) or when the alarm is Alarm state (when there is not enough free storage than the threshold 2GB). The action publishes a message to Simple Notification Service (SNS).

Create a SNS Topic

This cloudformation template creates the SNS Topic that sends the email to the assignee who can be a DevOps or a System Administrator accountable for the scenario. The template also invokes a Lambda function by creating permission to invoke lambda. The Lambda function sends alerts to a channel in Slack.

AWSTemplateFormatVersion: '2010-09-09'
Description: SNS topic for sending Alerts for FreeStorage of RDS databases
Resources:
  RDSFreeStorageSNSTopic:
    Type: AWS::SNS::Topic
    Properties: 
      DisplayName: 'RDS Free Storage Monitoring'
      Subscription: 
      - Endpoint: !ImportValue 'RDS-FreeStorage-notification-lambda'
        Protocol: lambda
      - Endpoint: "krishnachalise418@gmail.com"
        Protocol: email
  RDSFreeStorageSNSTopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Properties:
      Topics:
        - !Ref RDSFreeStorageSNSTopic
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              AWS: '*'
            Action:
              - "SNS:Publish"
            Resource: '*'
  LambdaTriggerPermission:
    Type: AWS::Lambda::Permission
    Properties: 
      Action: 'lambda:InvokeFunction'
      FunctionName: !ImportValue 'RDS-FreeStorage-notification-lambda-name'
      Principal: "sns.amazonaws.com"
      SourceArn: !Ref RDSFreeStorageSNSTopic
Outputs:
  RDSFreeStorageSNSTopicArn:
    Value: !Ref RDSFreeStorageSNSTopic
    Export:
      Name: !Sub '${AWS::StackName}'

Creating a Lambda to send notification to Slack

We will be using SAM template for deploying AWS Lambda. SAM templates are custom cloudformation templates to deploy ServerLess Applications. You can find the source code for Lambda written in Golang over in this gist.

when invoked, The lambda reads and parses the message sends message to a channel in Slack to alert the subscribers for the scenario.

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Resources:
  LambdaExecution:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: build/
      Handler: main
      Role: !GetAtt LambdaExecutionRole.Arn
      Runtime: go1.x
      Timeout: 10
      MemorySize: 128
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties: 
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
        - Effect: "Allow"
          Principal:
            Service:
            - "lambda.amazonaws.com"
          Action: 
          - "sts:AssumeRole"
      Path: "/"
      ManagedPolicyArns:
      - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
Outputs:
  RDSFreeStorageLambdaArn:
    Value: !GetAtt LambdaExecution.Arn
    Export:
      Name: !Sub '${AWS::StackName}'
  RDSFreeStorageLambdaFunctionName:
    Value: !Ref LambdaExecution
    Export:
      Name: !Sub '${AWS::StackName}-name'

Deploying the Cloudformation Stacks

Deploying Lambda

go build -o build/main src/main.go
sam package \
  --template-file template.yml \
  --s3-bucket <some-bucket-name> \
  --output-template-file output.yml
sam deploy \
  --template-file output.yml \
  --stack-name RDS-FreeStorage-notification-lambda \
  --capabilities CAPABILITY_IAM

Deploying SNS

aws cloudformation \
  create-stack \
  --stack-name RDS-Alert-SNS-Topic \
  --template-body file://template.yml

Deploying Cloudwatch Alarm

aws cloudformation \
  create-stack \
  --stack-name quiz-development-RDS-alarm-RDSFreeStorageAlarm \
  --template-body file://template.yml

You will receive verification email to confirm the subscription. After subscribing for the email, you will receive emails when free storage is less than the threshold. You will also receive the message in Slack which looks like this: