Firebase Data Manager — Part 1

Trendyol is the one of the biggest e-commerce platforms based in Turkey. Customers can easily find any products on different categories like clothing, FMCG, electronics, cosmetics and furniture in Trendyol app. Addititionally Trendyol also offers services such as cargo, insurance, and a second-hand product market.

In Trendyol, all decisions should support with data. From time to time the domains might need to take an immediate action. Thus, it is crucial to check metrics like gross merchandise volume, unit sold product, active shopper count, and new buyers per platform instantaneously.

We fulfill this need by designing a dashboard in the following architecture. After our scheduled Airflow dags are taking data from Bigquery and Vertica , they write corresponded parts to the Firebase. Our frontend structure reads the data from Firebase and show the metrics to the other colleguages.

What is Firebase?

Firebase is the platform that is able to provide the backend for a mobile or web app, including realtime database, user authentication, static hosting, and more.

The flexibility of Firebase to display database updates in “real time” is one of the main reasons why we prefer it. Thus, you can maintain the appearance of your app’s display without having any delay related to relevant data.

Unlike relational databases, Firestore stores data in the form of documents, which are subsequently sorted into collections. A document has fields with matching values, similar to a key-value pair. Also a field can be composed of several categories such as string, number, array, boolean, etc. A document’s data can be arranged using arrays or nested objects called maps.

Why do we need Firebase Data Manager?

In our designed that we mentioned above, we manage react components by using firebase collection and document structure.We managed our documents and collection on Firebase manually for a while.

The problem is that when we accidentally deleted a document, it was not possible to recover it and we had to create it manually again.Likewise, we had to create repetitive react components and nested documents one by one. It was really a burden to our team.We have integrated our system with Gitlab to keep our Firebase structure historically and to make the editing process easier.

Firebase manager system is formed by adapting the document and collection structure of Google Firebase into the file structure.Collections are a combination of documents.Documents consist of fields.Documents can also contain collections. Since it is very similar to basic folder logic, we have created a nested folder structure.

We considered these files that contain field_data.json as documents in our system.If there is no field_data.json in a folder, our system considers these folders as collections.

Let’s take a look to our example. You can see this “Dashboards” collection. Inside this dashboard collection we have “International” document. Since “International” is a document, it has field_data.json. In the next layer, we have subpages as a collection. This collection has more than one documents and as you can see the attributes of these documents can be easily found in field_data.json files.

As an example of field_data.json file, we are adding attributes related with our react components to this json file:

{
    "name": "BasicTable",
    "props": {
        "title": "Top Products",
        "select-label": "Products",
        "scroll": {
            "y": 500,
            "x": 900
        },
        "extra-info": "Today's top sold products",
        "columns-props": [
            {
                "render-type": "image",
                "title": "Content",
                "key": "content",
                "align": "center"
            },
            {
                "title": "Name",
                "render-type": "text",
                "key": "name"
            },
            {
                "title": "Reel Ciro",
                "render-type": "text",
                "key": "reelCiro",
                "value-format": {
                    "currency-type": "EUR",
                    "pattern": "$0,0",
                    "type": "currency"
                }
            }
        ]
    }
}

Operations

There are several operations that we should handle in our manager like all other database: create, update, delete documents and collections.

The set_document_firestore function is our smallest atomic function. It takes collection path, document name and finally data as input. It updates or creates the document in the collection path according to the data it receives as a parameter.

def set_document_firestore(collection_path,document,data): 
	try:
		firestore_client.collection(collection_path).document(document).set(data) 
	except firebase_admin.exceptions.FirebaseError as e:
		print("Error: {}".format(e))
		exit(1)

We use the iterate_path function to process nested documents. We update the collections in each document easily.

def iterate_path(collection_obj):
	for document in collection_obj['document_list']:
		set_document_firestore(collection_obj['path'], document['name'],document['fields'])
		for collection in document['collection_list']:
			iterate_path(collection)

For deleting operation, we are using this code snippet:

def delete_document(collection_path,document_name):
	db = firestore.client() 
	db.collection(collection_path).document(document_name).delete()

In the next part we will explain how we integrate our system by using CI/CD mechanisms on Gitlab and manage thousands of changes just with one merge request. You can reach the second part of our article.