Moving data from DynamoDB to Cloudant or CouchDB

Introducing the dynamodbexport npm module

Glynn Bird
Mar 24, 2017 · 3 min read
{
"temperature": {
"N": "8391"
},
"time": {
"S": "2017-03-09T01:38:11+0000"
},
"id": {
"S": "1489023491"
}
}
{
"temperature": 8391,
"time": "2017-03-09T01:38:11+0000",
"id": "1489023491"
}

Getting the data out

The AWS SDK provides a comprehensive toolkit for interacting with AWS services. You need to to “scan” a whole DynamoDB table, performing a chain of API calls to pull back records in batches until you’ve consumed them all. I’ve written a script to do this for you: dynamodbexport.

The aptly-named dynamodbexport npm module moves data from DynamoDB to IBM Cloudant or Apache CouchDB™.
$ npm install -g dynamodbexport
$ export AWS_ACCESS_KEY_ID="OGIIWJGNWNIITJHWTHSO"
$ export AWS_SECRET_ACCESS_KEY="YRPHIIIWJJJYwKLGV28JJuiuwnjiiqq06AS"
$ dynamodbexport --table iot --region us-east-1
{"temperature":30730,"time":"2017-03-09T02:21:48+0000","id":"1489026108"}
{"temperature":17072,"time":"2017-03-09T02:15:22+0000","id":"1489025722"}
{"temperature":18177,"time":"2017-03-08T21:27:23+0000","id":"1489008443"}
Export complete { iterations: 1, records: 3, time: 0.145 }

Importing into CouchDB/Cloudant

I already have a tool to import data into CouchDB: couchimport, which you can install in a similar way:

$ npm install -g couchimport
$ export COUCH_URL="https://MYUSER:MYPASS@MYHOST.cloudant.com"
$ dynamodbexport --table iot --region us-east-1 | couchimport --db iot --type jsonl

It works for local databases too

The dynamodbexport tool also works for local DynamoDB databases. Just leave out the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, and it will assume a local database.

$ dynamodbexport --table iot | couchimport --db iot

IBM CODAIT

Things we made with data at IBM’s Center for Open Source Data and AI Technologies.

Glynn Bird

Written by

Developer @ IBM. https://glynnbird.com

IBM CODAIT

Things we made with data at IBM’s Center for Open Source Data and AI Technologies.