Tracking DynamoDB Storage History with CloudWatch
Amazon DynamoDB uses Amazon CloudWatch, which collects and processes raw data from DynamoDB into readable, near real-time metrics. These statistics are retained for a period of time, so that you can access historical information for a better perspective on how your web application or service is performing. By default, DynamoDB metric data is sent to CloudWatch automatically.
However, there is one metric which many customers care about and is not currently tracked by DynamoDB on CloudWatch, that is your table size history. This blog post shares a simple process on how to track your tables size over time, which can be easily adapted to track several other metrics such as number of items stored over time.
The easiest way to understand your tables current storage capacity is to call the
DescribeTable command which returns your table size in bytes:
TableSizeBytes: The total size of the specified table, in bytes. DynamoDB updates this value approximately every six hours. Recent changes might not be reflected in this value. For this reason, we will update our CloudWatch metric every six hours.
EventBridge receives an event, an indicator of a change in environment, and applies a rule to route the event to a target. Rules match events to targets based on either the structure of the event, called an event pattern, or on a schedule. For example, trigger the event every six hours, similar to a cron job.
Then for our EventBridge target, we will use a Lambda function.
Lambda is a compute service that lets you run code without provisioning or managing servers. Lambda runs your function only when needed and scales automatically, from a few requests per day to thousands per second. You pay only for the compute time that you consume — there is no charge when your code is not running.
Lambda will be invoked by our EventBridge trigger and the code within our Lambda will do a few simple things:
1. Call DynamoDB
ListTables to retrieve all the tables within the region
2. Call DynamoDB
DescribeTable for each table returned in the list request and retrieve the
TableSizeBytes we previously mentioned.
3. Write our metric to CloudWatch using
For our Lambda function we will be using a Python runtime and Boto3 code.
CloudWatch enables you to publish, monitor, and manage various metrics, as well as configure alarm actions based on data from metrics. As mentioned, we will use
PutMetricData to write our custom metric to CloudWatch every six hours. This will allow us to view our data growth/shrinkage over time using the CloudWatch Web Console our by reading the metrics using CloudWatch API’s.
When we call
ListTables by default DynamoDB only returns up to 100 tables, so for that reason we include pagination to ensure we read all the tables in the region for your account.
Once we have all of the tables names and sizes, we then push those metrics to CloudWatch:
CloudWatch metrics are explained in more detail here, but essentially we create a namespace called
DynamoDB Table Size and we track our metric Table Size using the metrics DynamoDB returned us. We use the DynamoDB table name as a Dimension as it is unique for the region. We then use the table size in bytes as the Value and set our Unit to
Below is a Sample image of how the metrics look on the CloudWatch Web Console. My tables had received no traffic on the day the image was taken, thus the straight lines.
Metrics should be set up like below when viewing, Statistic=Average and Period=6 hours:
Install using this sample CDK application
If you would like to try this out in your own account, feel free to use the below CDK project which will create the resources for you. The install instructions are included in the README.
Please ensure you test any code which you run in your account before running in production.