Processing Data Reliably Using Event-Driven Architecture

Just following the trend or is it beneficial?

On a mature platform like Haraj, the need to process various data has become commonplace to provide a better experience for users. With many active users every day, the amount of data that must be processed is substantial. This presents a challenge for us to ensure our application can process this data reliably, quickly, and accurately.

For example, to ensure that no daily transaction data is missed or corrupted, we don’t rely solely on the integration process between the payment gateway and our application when a transaction occurs. Many issues can arise during this process, such as internet connection loss, payment gateway server downtime, and errors in our application code. On the other hand, we cannot afford to lose a single transaction as it could have a fatal impact on the business.

Photo by Arno Senoner on Unsplash

To address this, we developed an application that runs daily, downloading all transaction data from the previous day. This transaction data is then processed one by one and stored in the database. This way, we can ensure that every transaction is captured and not corrupted.

However, a problem arises with this method. The download and processing of transaction data are also not immune to similar issues, right? This is where we decided to use event-driven architecture for such applications.

With this architecture, here are some benefits we can obtain:

1. We can easily repeat the processing of specific data when our application encounters issues at certain times. For example, if our application runs on 2024–07–11 and let's say according to the rules, it processes transaction data from the previous day, 2024–07–10, and it encounters an issue. With event-driven architecture, we can easily rerun the process even on 2024–07–14, by sending a message with the date 2024–07–10, to the queue.
2. We can process data faster and more efficiently in parallel. If needed and feasible, we can run more than one instance of the application to process data in parallel, speeding up the data processing.
3. We can ensure data processing is done sequentially. A common example is when processing transaction data, updating search indexes, and processing logs. With event-driven architecture using a queue that supports the first-in-first-out (FIFO) feature, we can ensure that data is processed in the correct order, avoiding fatal errors. For instance, in search index updates, a user sends a new post to our system, then 2 minutes later updates the post, and 1 minute later deletes it. The correct sequence of events is new post, update post, delete post. If this order is reversed, the search results provided by our system will be incorrect, such as the post appearing even though it has been deleted.

Common Design

Here’s an example of the most common event-driven architecture design we implement at Haraj. Using AWS infrastructure, the AWS services that support event-driven architecture and are frequently used include EventBridge, SQS, Lambda, and ECS Fargate.

1. EventBridge: We use EventBridge to trigger Lambda Functions on a scheduled basis.
2. Lambda: The Lambda Function sends the required messages to SQS.
3. SQS: The queue stores messages that will be consumed by workers.
4. ECS Fargate / Lambda: Our developed applications process data in the system. For long-running data processing tasks, we use ECS Fargate, while for quick and less critical data processing, we use Lambda.

By leveraging event-driven architecture, we can create a more reliable and responsive system for processing data. This design not only enhances the scalability and flexibility of applications but also enables more efficient workload management.

By implementing queuing mechanisms and asynchronous processing, we can ensure that each event is handled timely and accurately, reducing the risk of data loss or delays.

Ultimately, adopting event-driven architecture opens new opportunities for innovation and optimization in various application scenarios, delivering better user experiences and optimal system performance.