Modern software architecture: Event-driven design meets Event-sourcing
TL, DR: In today’s software landscape, making the right architectural decisions is crucial for ensuring performance, scalability, maintainability, and overall success. Among the numerous patterns available, Event-Driven Architecture (EDA) and Event Sourcing (ES) stand out as two of the most popular choices for complex software systems. While it’s possible to employ EDA without ES or vice versa, their combination can be surprisingly effective.
Event-driven architecture contrasts with traditional request-driven systems, where components explicitly call each other’s methods in a tightly coupled manner or API calls directly in a synchronous way. In event-driven architecture, components are loosely-coupled and communicate indirectly through events, promoting flexibility, scalability, and modularity.
Event sourcing, another powerful design pattern that also employs event, emphasizes the importance of maintaining a chronological record of events for better auditing, analysis, and historical tracking. The main idea of event sourcing is eventually consistency.
Let’s look into each of them and see how can we combine them to build a scalable architecture tailored to our use case, that’s serving millions of customers from thousands of merchants effectively.
Event-Driven Architecture explained
Event-driven architecture focuses on the flow and processing of events within a system. Events, representing significant occurrences or state changes, serve as the backbone of communication between different components. In an event-driven system, components — such as microservices or functions — communicate asynchronously through the generation, detection, and consumption of events. This approach promotes loose coupling, allowing for flexibility, scalability, and responsiveness to dynamic changes. EDA is particularly well-suited for distributed systems, where components can operate independently, reacting to events in real-time without direct, synchronous interactions.
Common use cases of EDA:
Scalable asynchronous processing, i.e. workers
In scenarios with an extensive number of tasks and the flexibility of asynchronous processing, this configuration proves highly scalable. In Service A, Component X generates messages, publishing them to the message broker (i.e. event broker). Subsequently, these messages are subscribed to and processed by many consumers (referred to as workers) as required. The event’s origins may include cron jobs, user interactions, and similar sources.
In a traditional system, tasks are typically processed sequentially or with the aid of multi-threading in supported stacks. Nevertheless, even with multi-threading support, there is a finite limit as only vertical scaling becomes a viable option.
Message queueing
This is an variant of the above asynchronous processing but the workers are now laid in another service. This enables service B to scale seamlessly as the volume of events increases. The events that arrive in service B might come from different services.
This can be used in an architecture in which service B is responsible for processing high number of small tasks at the same time, for example, to deliver massive number of push notifications or sms notifications to mobile devices.
Cross domain communication
This is a more advanced version of EAD where multiple components of the system might be interested in the same message, or might not. So the complexity here would be more on the broker end where the events are routed by different bindings to different services.
The above diagram illustrates a sample setup where events from service A are routed to service B and C using different queues. Please note that I have employed the RabbitMQ terms in here: federation, exchange and v-host. However, this setup can be done also using other message brokers.
Event Sourcing with CQRS
In a traditional database-centric approach, we typically store the current state of an entity. In contrast, event sourcing shifts the focus from storing the current state of an application to capturing and storing the changes to that state as a sequence of immutable events. These events represent specific occurrences or transactions within the system and are stored in an event log or event store.
Key Concepts
Events: Immutable records representing state changes in the system. Events capture what happened, not the current state.
Event Store: A specialized data store that maintains a chronological sequence of events. This allows for easy reconstruction of the system’s state at any point in time by replaying the events.
Aggregates: Units of consistency that group related events. Aggregates are responsible for enforcing business rules and maintaining data integrity.
Very often, event sourcing is used in pair with CQRS. This separates the read model (query model) and the write model (command model) for the concerns separation. There are 2 additional important concepts of event sourcing when it comes to this combination:
Read Model: A separate model designed specifically for querying and presenting data. It’s optimized for read operations and is constructed based on the events generated by the write model. The read model is denormalized and tailored to the specific needs of the queries it serves. From the same source of events, different views of the read model can be built, providing different perspectives of the business domain.
Projection: The process of transforming and updating the read model based on the events stored in the event store.
In this setup, events are populated to record each state of the aggregate arising from the business flow and then stored in the event store. Subsequently, the projection retrieves these events from the event store and begins transforming them into various read tables. These tables can then be presented in different views, offering distinct visualizations for business users. It’s essential to highlight that while pulling events is one option, projection can alternatively employ the pub/sub pattern by utilizing a message broker to subscribe to new events.
This setups enhances system resilience by capturing a chronological sequence of events, providing a comprehensive audit trail for thorough traceability. This approach hence allows for effective temporal querying, enabling the reconstruction of the application’s state at any given point. Furthermore, it brings up flexibility in adapting to evolving business requirements by separating data model changes from the historical event log.
With the combination with CQRS, the mechanism promotes loose coupling between command and query responsibilities, offering resilience and flexibility.
Putting the Event Driven Architecture and Event Sourcing together
As described above, both EDA and ES provide a very powerful way for scalability and collaboration. EDA emphasizes loosely coupled components that communicate through events, while ES captures and persists the history of state changes. So if we take the events populated by ES and use them for EDA, that would be a natural combination.
Events that are populated can contain not only state of the change but also details of the change. Those events then can be carried to and consumed by another services/components within the EDA.
The above diagram shows how we can combine ES and EDA together in a very straightforward way. The events from event store can be published and consumed by different services and components through message broker.
The advantages of combining EDA and ES
Combining Event-Driven Architecture (EDA) with Event Sourcing offers significant advantages. One of the primary benefits is the enhanced responsiveness of the system. This setup allows for rapid adaptation to real-time changes, making it highly efficient in dynamic environments.
Another key advantage is the comprehensive historical record of state changes, facilitating detailed historical analysis and ensuring robust audit trails for compliance. Additionally, its immutable and replayable nature enables real-time analytics, scalability, and seamless adaptation to evolving analytical requirements. Events can also be streamed to data analytic pipelines or replicated to datalake for further processing.
Additionally, the flexibility and scalability inherent in this combination are noteworthy. Systems can evolve effortlessly to meet growing business needs, ensuring long-term sustainability and adaptability.
Lastly, the improved resilience of the system is also a noteworthy benefit. This setup offers robust mechanisms to recover from disruptions, ensuring continuity and reliability even in challenging circumstances.
Navigating the challenges
Combining Event Sourcing and Event-Driven Architecture (EDA) introduces challenges in coordinating distributed systems, ensuring scalable and performant event storage, and managing schema evolution. Firstly, coordinating the interactions of multiple microservices in a distributed environment poses complexities in maintaining proper event sequencing and system consistency. Careful design of event flows and coordination mechanisms is necessary to prevent issues like event duplication or out-of-order processing across services.
Secondly, the scalability and performance of event storage are critical challenges. Event Sourcing relies on immutable events, and EDA relies on efficient event communication between services. Ensuring the durability and reliability of event storage infrastructure becomes paramount for maintaining system responsiveness, especially in distributed environments where events are produced and consumed across the network.
Thirdly, addressing schema evolution and versioning is crucial. As systems evolve, data schemas associated with events may change. Balancing backward and forward compatibility while preserving the integrity of historical events poses a challenge. Implementing effective versioning strategies is essential to prevent issues with event consumers that may be processing events based on different schema versions. Overcoming these challenges requires thoughtful design and technological choices to create a resilient and scalable event-driven architecture based on Event Sourcing and EDA.
One last thing (but not least), operation of this setup is not an easy task. Monitoring and managing a system that combines EDA and ES require new operational practices. Ensuring the availability, reliability, and performance of the system in a production environment can be challenging. Operating message broker also requires specific expertise. We need to understand pros, cons, and good practices of the message broker that we use.
Final thoughts
Despite these challenges, the fusion of Event Driven Architecture and Event Sourcing is a powerful approach that, when managed well, offers a dynamic and robust architecture. It is essential to address these challenges during the design and implementation phases to fully leverage the potential of this combined architectural strategy.
