Using CloudFormation to Import your DynamoDB Table from S3

Amazon DynamoDB recently released the ability to import your table data from Amazon Simple Storage Service (S3). DynamoDB import from S3 helps you to bulk import terabytes of data from S3 into a new DynamoDB table with no code or servers required. Supported file formats are CSV, DynamoDB JSON, or Amazon ION. Imports can be initiated from the AWS Management Console, the AWS Command Line Interface (AWS CLI), AWS SDK or Amazon Cloudformation which we will showcase in this blog.

More details for the newly released feature can be found in this blog by Aman Dhingra and Robert McCauley.

In this example we use a small CSV with 100 rows. It contains 3 columns, pk which is a guid, sk which is a date and message which is random text. Download sample data here, foo.csv. Be sure to unzip it before uploading to your S3 bucket.

foo.csv

We add the CSV file to an S3 bucket, which we later define in our CFN template. I done this from the command line on my local Mac:

aws s3 cp foo.csv s3://myS3Bucket/myS3Prefix/foo.csv

Once we have the data on S3, we can create and run the CFN template. Below you will find the sample template in both YAML and JSON formats:

Create table with CSV data in S3 (YAML)

Create table with CSV data in S3 (JSON)

Again, from the command line of my local machine I executed the Cloudformation template import.yaml

aws cloudformation create-stack \
--stack-name "importstack" \
--template-body file://import.yaml

Cloudformation took just under 3 minutes to create the table including the time to import the data. Below you can see a preview of the 100 items in the DynamoDB Console.

DynamoDB Web Console

Some things of note when importing a DynamoDB Table from S3:

1. When importing from CSV files, all columns other than the hash and range keys of your base table and secondary indexes are imported as DynamoDB strings.
2. DynamoDB will show Item Count and Table Size as 0 initially. This is because DynamoDB uses the DescribeTable API to populate these metrics which is only updated approximately every 6 hours.
3. If you use DynamoDB Streams for your table, imported records from S3 will not be emitted to it as Streams are enabled as soon as the import is complete.
4. AWS::DynamoDB::GlobalTable does not support import from S3.
5. Troubleshooting is described in detail here.