Migrating Collections from MongoDB to Cloud Firestore

Image credits: Google Firestore Logo, Mongo MongoDB logo

Introduction

Cloud Firestore is a flexible, scalable and cloud-hosted NoSQL database for mobile, web, and server development from Google Cloud. Cloud Firestore also offers seamless integration with other Firebase and Google Cloud products, including Cloud Functions. Firestore stores data as documents organized into collections. With Cloud Firestore you get automatic multi-region data replication, strong consistency guarantees, atomic batch operations, and real transaction support. Developers choose Firestore for client applications needing data on the fly and automatically synced between client applications using realtime listeners.

This article explains how to migrate collections from MongoDB to Firestore in 3 easy steps. If you want to try these steps the sample files are provided in this repository.

Prerequisites

1. MongoDB database where your source data to be migrated resides
2. Mongo Shell to interact with MongoDB
3. Install Mongo database tools specifically you will need mongoexport
4. Install jq to process JSON data
5. Install nodejs and firestore-export-import npm package
6. Target Cloud Firestore to import data to. Add a Firebase Service Account and get the service account key as explained here. This key is a JSON file with the following data that includes your private key so that you can talk to the Firestore DB.

{
  "type": "",
  "project_id": "",
  "private_key_id": "",
  "private_key": "",
  "client_email": "",
  "client_id": "",
  "auth_uri": "",
  "token_uri": "",
  "auth_provider_x509_cert_url": "",
  "client_x509_cert_url": ""
}

How does the migration work?

We will understand this with an example.

Let us say you store the a JSON as a donuts collection in a MongoDB.

[
	{
		"type": "donut",
		"name": "Cake",
		"ppu": 0.55,
		"batters":
			{
				"batter":
					[
						{ "code": "1001", "type": "Regular" },
						{ "code": "1002", "type": "Chocolate" },
						{ "code": "1003", "type": "Blueberry" },
						{ "code": "1004", "type": "Devil's Food" }
					]
			},
		"topping":
			[
				{ "code": "5001", "type": "None" },
				{ "code": "5002", "type": "Glazed" },
				{ "code": "5005", "type": "Sugar", "unhealthy": true  },
				{ "code": "5007", "type": "Powdered Sugar", "unhealthy": true },
				{ "code": "5006", "type": "Chocolate with Sprinkles" },
				{ "code": "5003", "type": "Chocolate" },
				{ "code": "5004", "type": "Maple" }
			]
	},
...
]

NOTE: You can add this data to your MongoDB as a donuts collection if you want to try this yourself. The data is available here. mongoimport --uri $MONGODB_URI --collection donuts --type json donuts.json --jsonArray.

MongoDB stores this data by assigning an ObjectId for each object in the collection as below

> db.donuts.find();
{ "_id" : ObjectId("61d59cae348ca76a1703bd38"), "type" : "donut", "name" : "Raised", "ppu" : 
0.55, "batters" : { "batter" : [ { "code" : "1001", "type" : "Regular" } ] }, "topping" : [ {
 "code" : "5001", "type" : "None" }, { "code" : "5002", "type" : "Glazed" }, { "code" : "5005
", "type" : "Sugar", "unhealthy" : true }, { "code" : "5003", "type" : "Chocolate" }, { "code
" : "5004", "type" : "Maple" } ] }
....
....

Firestore requires this data in a different format as shown below.

{
  "61d59cae348ca76a1703bd38": {
    "ppu": 0.55,
    "topping": [
      {
        "code": "5001",
        "type": "None"
      },
      {
        "type": "Glazed",
        "code": "5002"
      },
      {
        "type": "Sugar",
        "code": "5005",
        "unhealthy": true
      },
      {
        "type": "Chocolate",
        "code": "5003"
      },
      {
        "code": "5004",
        "type": "Maple"
      }
    ],
    "name": "Raised",
    "type": "donut",
    "batters": {
      "batter": [
        {
          "code": "1001",
          "type": "Regular"
        }
      ]
    },
    "__collections__": {}
  },
  ....

When you migrate data from MongoDB to Firestore, we need to transform the data format to a format compatible with Firestore before pushing to the target. The process explained here assumes that you want to retain the Object Ids from the source MongoDB as Ids to refer the documents in the target Firestore database collection.

The steps are:

1. Export a collection from the source MongoDB as JSON
2. Transform the data format
3. Import the collection into target Cloud Firestore

Export a collection from the source MongoDB

• Set the MONGODB_URI environment variable

export MONGODB_URI=mongodb://IPADDRESS:27017/test

• Export donuts collection to a JSON array

mongoexport --uri=$MONGODB_URI --collection=donuts -o donuts_mongo_export.json --jsonArray

The resultant file should look like this.

Transform data format

• Use jq to transform the data format to be compatible with Firestore

jq 'map( {(._id."$oid") : . }) | [.[]| map_values(del(.__v, ._id))]| add' donuts_mongo_export.json > donuts_firestore_import.json

This transformation does the following:

• Assigns ObjectId as the key and content as the data
• Removes _id and __v
• Consolidates objects in the array into a single object

The resultant transformed file should look like this.

Import the collection into target Firestore

• Use firestore-import to upload the transformed data into the Cloud Firestore. You will need the service account key json file in this step.

firestore-import -a <<PathToKey.json>> -b donuts_firestore_import.json —nodePath donuts -y

Once imported here is how the data appears on the Firestore console.

Congratulations!! You have learnt to migrate collections from MongoDB to Cloud Firestore. Hope this short tutorial helped you.