JSON to CSV batch converter

1. Intro

Dedicating to testing APIs I faced some scenarios when we needed to parse many JSON files and convert them to cumulative CSV.

There are plenty of tools on the market, and the npm registry contains some good examples, but we needed to have a solution that will process the JSON files in a batch. With the solutions available now we have some limitations like:

1. we have to have already the array of objects,

or

2. we can not do batch processing and parsing of the JSONs files,

or

3. we can not flat the JSON before parsing to CSV.

Therefore I decided to create a small npm package called “jtocsv” to handle the stated above problem :).

The package is quite simple and you can install it locally or globally. What it actually does — it reads the JSON files from a folder path, parses and flattens the data, and combines the data in the output CSV file. It provides flexibility through command-line arguments along with the possibility to use the package inline within your script. Column headings will be automatically generated based on the keys of the JSON documents. Nested documents will have a ‘.’ appended between the keys.

The main idea is — to read all the JSON files from the provided folder, parse them, flatten them, and form an array of objects, which will be utilized to convert to CSV, and saved with the provided arguments path.

As it flattens the JSON files and sometimes the API responses can have different amounts of keys — the script will parse the largest JSON in the folder first to have the most complete CSV headers.

It is easy to use — work as expected without much customization)

It is fast

It is scalable as it works with big files using Streams

2. Command-Line Options

path: (Required) Path to the folder containing JSON files (objects or JSON arrays).

path: (Required) Path to the output CSV file.

3. Examples

Read all JSON files and save the parsed data to CSV file:

Installation:

npm i -D jtocsv

Use via the command line:

jtocsv /path/to/jsons /path/to/output.csv

Use as inline script:

import { saveJSONSToCSV } from 'jtocsv/parser.js'
await saveJSONSToCSV('json_path', 'csv_path')

There are error handlers for different scenarios. For example, if there is one not JSON among the others it will not form the CSV line just for this JSON, but will process all the rest, etc.

I have some plans to improve it and vice-versa functionality for converting CSV of the same form back into the original array of JSON documents, where the column headings will be used as the JSON document keys and all lines must have the same exact number of CSV values.

I believe there are many ways to improve the process. Feel free to share your experience in the comments!

Thanks for reading and happy coding/testing ❤️

There is a link for the source code.

GitHub - Valiantsin2021/jtocsv

4. Contributing

Feel free to write your opinions, and all Contributions are welcome!