Introducing CQRS in Tajawal

In a few minutes reading, I am going to introduce our experience in Tajawal.com for one of our challenging parts, the DB performance. And how it evolved until we had the convenient solution that gives us the ability to help the business to go forward.

Introduction (Problem Statement)

I was working in one of our back office applications that helps our customer service agents, fulfillment, fraud and other internal teams to manage/fulfill the bookings whether it is coming from online customers or through our agents.

And some of our mission as a team towards our agents:

1. Giving the flexibility to search with different criteria/patterns to retrieve the correct customer data.
2. Performance, the agent doesn’ t have to wait on the call too much time waiting for the data to come.

And as our business was growing fast (and still :)), the number of bookings was growing and the DB queries started to be slower.

Note that: our DB is a MongoDB and the search engine/index DB we are using is Elasticsearch (ES).

Fixing the Current Situation

Indexes:

A database index is a data structure that improves the speed of data retrieval operations. — Wikipedia

Most of the persistent databases like MySQL or MongoDB are providing the ability to have an index, so, we built our index with single and compound indexes that can fulfill the agents' queries.

With the variety of our customers, we have different criteria for searching the bookings and we were limited to 64 indexes per collection.

Async Calls:

The client application doesn’t have to wait for all the data to come in a single response, instead, we can release three requests:

1. The current page of data (listing).
2. The pagination and data stats.
3. Filters data.

So, if we get one request hanging, we still can get the rest.

Timeouts:

We set the timeout that matches nginx timeout so the query process doesn’t stay there forever and block the other requests to be fulfilled.

Creating The Search Engine

In parallel, we started to look for the optimal solution for the search operation and we built the search engine with Solr that later migrated to ES.

Syncing:

Now, we need to have all the data in MongoDB to be in ES and there is one of two ways to sync the data between two data sources:

1. Scheduled Jobs: it is running every one minute to get the last updated or created bookings and push them to ES.
   Note that, one minute is the maximum time of a particular booking until it shows up to the agent and this has to be agreed on with all the application stakeholders.
2. Event Based: whenever you update or create a new booking, you push it to the message broker that syncs it to ES. This logic is injected in your application DB models.
   There is a centralized solution for it by using Change Streams that allow applications to access real-time data changes.

Adoption:

After having the ES DB ready to serve the customer, now, how do we fit it into our application?

We had a new implementation for ES search operation that follows the application DB contract/interface so it helps us switching between implementations. Read more about Strategy.

And switching between implementations is based on configurations:

1. Application Level Based: route all the agents' queries to one of the data sources.
2. User Level Based: route specific user to one of the data sources and this overriding the application level configuration.

This approach helped us to migrate some agents (along with the QA team) to ES in order to verify the new release and giving us the green light to migrate all the other agents to ES and also, the ability to rollback if we had to.

Monitoring:

Building the real-time dashboard that has some comparisons between both sources and at some time frames. During the migration from Solr to ES, we had the dashboard for three data sources. And this is integrated with the alerting system.

Result:

We were able to provide our agents with more search options along with the performance that of course helped them to serve quality and speed to our Tajawal’s customers.

So, What is CQRS?

CQRS stands for Command Query Responsibility Segregation. At its heart is the notion that you can use a different model to update information than the model you use to read information. — Martin Fowler

We have a different reading model that helped us to utilize the ES capabilities to deliver a better experience to our agents through its high performance.

If you are interested to know more about CQRS pattern, check the following references:

• https://martinfowler.com/bliki/CQRS.html
• https://microservices.io/patterns/data/cqrs.html

Conclusion

CQRS pattern might inspire you to solve a performance problem but in this article, I wanted to take you through the evolution of fixing the problem and how we approached it.