Moving data from DynamoDB to Cloudant or CouchDB

Introducing the dynamodbexport npm module

Glynn Bird
Mar 24, 2017 · 3 min read

If you have data in an Amazon DynamoDB service and want to move it to IBM Cloudant or Apache CouchDB, how would you go about it? First of all, DynamoDB has a peculiar form of JSON. A single temperature measurement would be expressed like this:

{
"temperature": {
"N": "8391"
},
"time": {
"S": "2017-03-09T01:38:11+0000"
},
"id": {
"S": "1489023491"
}
}

Instead of the more straightforward JSON:

{
"temperature": 8391,
"time": "2017-03-09T01:38:11+0000",
"id": "1489023491"
}

Cloudant and CouchDB can store any JSON documents with nested objects of arbitrary complexity, whereas DynamoDB stores a series of key values at the top of the JSON tree.

Getting the data out

Image for post
Image for post
The aptly-named dynamodbexport npm module moves data from DynamoDB to IBM Cloudant or Apache CouchDB™.

First, install the tool:

$ npm install -g dynamodbexport

Define a couple of environment variables with your Amazon API credentials:

$ export AWS_ACCESS_KEY_ID="OGIIWJGNWNIITJHWTHSO"
$ export AWS_SECRET_ACCESS_KEY="YRPHIIIWJJJYwKLGV28JJuiuwnjiiqq06AS"

Then simply run dynamodbexport, supplying the name of the table to export and the AWS region it is hosted in:

$ dynamodbexport --table iot --region us-east-1
{"temperature":30730,"time":"2017-03-09T02:21:48+0000","id":"1489026108"}
{"temperature":17072,"time":"2017-03-09T02:15:22+0000","id":"1489025722"}
{"temperature":18177,"time":"2017-03-08T21:27:23+0000","id":"1489008443"}
Export complete { iterations: 1, records: 3, time: 0.145 }

The tool makes as many API calls as it needs to extract the data, converting the JSON to a more compact form as it goes.

Importing into CouchDB/Cloudant

$ npm install -g couchimport

Set an environment variable with your target Cloudant/CouchDB service’s URL:

$ export COUCH_URL="https://MYUSER:MYPASS@MYHOST.cloudant.com"

Then run both the dynamodbexport and couchimport commands together, piping the output of the former into the latter:

$ dynamodbexport --table iot --region us-east-1 | couchimport --db iot --type jsonl

The --type jsonl parameter tells couchimport that it is to expect one JSON document per line, and --db iot defines the name of the target database. (Make sure your target database already exists, since couchimport does not create new databases.)

It’s that simple! You can now use Cloudant’s awesome MapReduce tools to aggregate the data or replicate it to other devices.

It works for local databases too

Moving data from local DynamoDB to local CouchDB is as simple as:

$ dynamodbexport --table iot | couchimport --db iot

You’ll find more details on command-line usage and programmatic access for dynamodbexport on npm. And please ♡ this article if you’d like to recommend it to other Medium readers.

Center for Open Source Data and AI Technologies

Things we made with data at IBM’s Center for Open Source…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store