Export data from DynamoDB

Export from DynamoDB to CSV

In this blog post, I’ll explain the export data from a DynamoDB table to a CSV file.

DynamoDB is Amazon AWS’s managed NoSQL database service.

For really big datasets, running a continuous (and parallel) scan might be time consuming and fragile process (imagine it dying in the middle).

Fortunately, the easiest way to achieve what you wanted by using below code and run any task/script after generating CSV.

DynamoDB data is exported in two steps. First, we convert DynamoDB data to JSON, then as second step, export that json to CSV.

Step 1: Export DynamoDB to JSON file

const AWS = require("aws-sdk");
AWS.config.update({ region: "us-east-1" });
const fs = require("fs");
const TABLE_NAME = "Write your table name";
const docClient = new AWS.DynamoDB.DocumentClient({
  sslEnabled: false,
  paramValidation: false,
  convertResponseTypes: false,
  convertEmptyValues: true,
});
let jsonExport = async ()=>{
  let queryParams = {
    TableName: TABLE_NAME,
  };
  let result = [];
  let items;
  do {
        items = await docClient.scan(queryParams).promise();
        items.Items.forEach((item) => result.push(item));
        queryParams.ExclusiveStartKey = items.LastEvaluatedKey;
    } while (typeof items.LastEvaluatedKey != "undefined") {
       await fs.writeFileSync("prodUser.json",JSON.stringify(result, null, 4));
       console.info("Available csv records:", result.length);
    }
}
jsonExport()
  .then((res) => {
    console.log(res);
  })
  .catch((err) => {
    console.log(err);
  });

Reference: Exporting DynamoDB table data to Amazon S3 — Amazon DynamoDB

Step 2: Convert JSON file to CSV

Once we have converted JSON data, we need to give specific attributes in the database in the fields array.Otherwise you do not need to give options object if you want all the attributes in the database.

let export_data = require("./prodUser.json");
const { Parser } = require("json2csv");
let fs = require("fs")
const fields = [
  "id",
  "full_name",
  "mobile",
  "bank_details",
] //particular attributes coming
const options = {
  fields,
};
try {
  const parser = new Parser(options);//If you want all the attributes in the database, it is enough to remove the name options.
  const csv = parser.parse(export_data);
   fs.writeFileSync("file.csv", JSON.stringify(csv, null, 4));
   console.info("Available csv records:", csv.length);
  } catch (err) {
  console.error(err);
}

Thank you for reading!