Exporting Firestore Collection as CSV into Cloud Storage on Demand, the easy way
Imagine we have a Firestore collection that needs to exported into CSV for whatever reason, and we want to easily download the CSV on demand anytime anywhere.
Usually we would need to develop a few pre-requisites before writing code to download the CSV, such as:
- Developing UI to download CSV.
- Write a cloud function API to return the CSV, then calling it on your preferred client.
- Handle authentication & permissions on UI / API cloud functions.
If you don’t want to build UI or API to download the CSV, this article will guide on how get CSV files from your Firestore collection as quickly as possible. The only code to write is to generate CSV and dump it into Cloud Storage for Firebase. If you are a fan of diagrams, here’s one that illustrates the entire process. We will go through each process in this article.
Step 1: Write a Pub/Sub Trigger
Let’s create a Cloud Function that will read a collection from Firestore, then convert it to CSV and upload to Cloud Storage:
The cloud function above is a Pub/Sub Trigger function that can be executed via sending a message to the topic generate-application-csv. You can trigger it via Google Cloud Platform > Pub/Sub. This function uses Json2CSV to convert firestore docs into CSV, which is then uploaded to your Cloud Storage bucket.
Step 2: Deploy the Function
Deploy the function via Firebase CLI . Once it’s deployed you’re all done with all the coding you need!
firebase deploy --only functions:generateApplicationCsvStep 3: Trigger the Pub/Sub Function to generate CSV
To trigger the function head over to Google Cloud Platform > Pub/Sub. (Be sure to select the same project as your Firebase project). You should see the topic generate-application-csv as what we have defined in our cloud function:
Click the topic then choose Publish Message, and enter any message. This message field is not used in our cloud function. Click Publish to trigger the cloud function.
🎉 Final Step: Download the CSV from Cloud Storage via Firebase Console
Go to Storage > Firebase Console, you should see your files there! To download, check the CSV and click open.
🎁 Bonus: Creating CSVs as a batch job
With this approach you can extend this functionality to create batch jobs that convert Firestore collections into CSV. You can use scheduled functions to create a fixed schedule when CSVs are generated, or with Pub/Sub you can flexibly schedule/pause/stop your batch job with Cloud Scheduler.
Bottom Line/Silver Lining
With this setup, you don’t have to worry about security issues that may arise from building UIs/APIs, apart from securing your Cloud Storage againts unauthorized reads, of course! This is because the setup lives inside the Google Cloud Platform and Firebase environment, and you are using their tools to your convenience.
As always, this is just one of the many ways to generate a CSV from Firestore, and should be taken with a grain of salt. Please try it out and see whether it fits your needs to quickly get a CSV from Firestore.
Also, as a reminder, pulling huge collections out of Firestore can also incur additional reads to your quota.
If you find this useful, please do give this article some much needed 👏👏👏
