Get CSV from S3 and convert it to JSON with Node.js
1 min readOct 17, 2018
In my project, I needed to write some code that downloads a CSV file from S3 and converts it to JSON format. I’m just sharing what I did and making some brief notes for each of the APIs.
Environment
- node: 8.10.0
- aws-sdk: 2.211.1
- csvtojson: 2.0.8
Assumed Knowledge
- How to get/set AWS credential information
- How to use and define those credentials in .env file
require('dotenv').config();
const AWS = require('aws-sdk');
const csv = require('csvtojson');const S3 = new AWS.S3();const params = {
Bucket: 'bucket name',
Key: 'csv file name'
};async function csvToJSON() { // get csv file and create stream
const stream = S3.getObject(params).createReadStream(); // convert csv file (stream) to JSON format data
const json = await csv().fromStream(stream);
console.log(json);
};csvToJSON();
getObject(): Retrieve the objects from S3
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property
getObject().createReadStream(): Pipe the objects from S3 to Node.js Stream object
csv().fromStream(readableStream): Convert to JSON from CSV readableStream.
Conclusion
Actually, I did not know what exactly the Stream object in Node.js is. The task gave me a chance to think about it deeper. Also, this is my first post on Medium. I will share what I thought, did and experienced. I hope my post helps someone. Thank you.
