What Exactly Is This Event Bus, What Does It Solve? Outbox pattern And DotNetCore.CAP

Merve Topal

Published in

adessoTurkey

10 min readMar 14, 2023

What Exactly Is This Event Bus, What Does It Solve?

Outbox pattern And DotNetCore.CAP

Hello,

I am excited and happy because this is my first post here,

Let’s start..

You know why we’re moving from monolith to distributed architecture in almost every project today, so I’m skipping this part. One of the most important issues to be considered in this transition process is how the communication between services should be. We did not have such a problem in monolithic structures because the whole project was running on a single content, but when it comes to microservice architecture, how the data traffic between services will become an important issue.

Here we will focus on what is event bus, what is facilitator, how it is implemented, which patterns are applied for data consistency. There are many different subtleties that make asynchronous communication between microservices much easier or much more difficult.

First of all, the event bus…

The event bus is a mechanism that enables different services (publisher-subscriber) to communicate with each other without being aware of each other. A service may send an Event to the event bus without knowing who or how many people will receive it. Services can also listen for events on an event bus without knowing who is sending the events. In this way, services can communicate without being dependent on each other.

Event buses are useful when you don’t want services to be interdependent. Instead of a service with many references to other services you can send events to an event bus and not have to worry about who will take care of them. This will make it much easier to develop an application and break it down into separate parts. Thus, when a service in the ecosystem is down, all system or other services are not affected and the application life cycle continues. In addition, thanks to the event bus, you will not lose the data that the crashed service will process by applying various patterns, and the service can process these data when it is up. And of course, while all this is happening, our main point is that the application lifecycle continues.

What about the Outbox pattern?

If our topic is event bus, naturally distributed systems, and asynchronous communication between services, it may not always be possible to use more than one external service together and in a single transaction. The reliability of our application is also reduced when we develop by ignoring integrity. Now, we will examine together how we can use different services transactionally through the Outbox design pattern and what the Outbox design pattern is.

One of the biggest challenges of microservice architectures is to ensure data consistency. Some data is transferred from one service to another via asynchronous events to ensure consistency.

Yes, in such an atmosphere, situations such as disconnection, error, and unexpected physical or software interruptions in the process occur before the messages are sent during the inter-service (or service and event bus) communication processes reach their destination. By preventing the loss of the message, we will definitely encounter simple but critical points that come into play to resend the message when the unfavorable situations pass. This is a possible scenario in distributed architectures such as microservice architecture.

The problem arises when two situations occur:

You are saving the data to your database, but an error occurred while publishing the event to your messenger. You post an event to your message broker that something has happened on your system, but then it fails when trying to save it to your database. In either case, there is no consistency between what your database saves and what you publish to the message broker. What you want in this case is a single atomic transaction that can save the data to your database and broadcast the event. If one fails, the other is rolled back.

This is where Outbox is a hugely effective pattern.

Keeping the communication data between services in the Outbox in the asynchronous process, protects the integrity by ensuring that it is not affected by the communication and data transfer problems that may occur due to possible malfunctions, system interruptions, or disconnections.

It actually works with a very simple logic. Solution: A change is made to the database. Then, it stores the operation to be executed as a task in the database. In other words, in one transaction, we both do the operation we want to do in the database and leave a mark for the action we will take after this transaction. We also scan tasks periodically with a background application. If a new task is found, we process it via the background service and mark it as complete. This is how we can summarize the Outbox design pattern.

Now, let’s consider the event bus and the Outbox pattern together.

Let’s assume that a new event will be published to an event bus as a result of the operations performed on a request to a service. If this operation is performed at the time of need, we must make sure that the sent message reaches the event bus. If a problem occurs, all operations performed by this service must be reversed for consistency. But is this true? After all, it is quite normal for services to experience temporary problems in a distributed architecture.

It should be able to keep the events to be processed until the related service is fixed and be able to process the events from where it left off when the service stands up. Many event bus technologies can provide us with this situation. But what if something happens to the event bus, not to the services that will process the event, and we can’t send the event? This means that we lost the event, which leads to data inconsistency between services.

In such a case, instead of directly communicating with the event bus, the related event is recorded in a table that will correspond to the Outbox and kept waiting. Then, with a background service that scans this table at regular intervals, the events registered there are tried to be transmitted to the event bus.

Thus, in terms of architectural design, possible data loss risks will be prevented, and it will be guaranteed that the message will reach the target at least once, depending on whether the event bus is working or not during the communication process between the services. As a result, the services will become a little more loosely coupled.

It is understood from here that the Outbox pattern can be used if you want to guarantee at least one transmission for the message to be sent to the event bus.

So far, we have talked about the use of the event bus and the Outbox pattern and its advantages. At this point, I would like to talk about a technology that offers these approaches together.

DotNetCore CAP. Let’s see what’s inside.

DotNetCore.CAP

CAP is a lightweight, easy, and efficient .NET based, open-source library. It provides ease in terms of manageability and has the event bus function, since the producer-consumer structure in distributed systems does not fully guarantee reliability.

In systems that take the microservice approach, we usually need to use events to integrate each service. In this process, simple use of the message queue does not guarantee reliability.

CAP is a library that has adopted the Local Message Table approach that works integrated with the existing database in order to resolve the exceptions that will occur when the services communicate with each other in the microservice architecture. It can ensure that event messages are not lost under any circumstances.

Here, the Outbox pattern comes to mind. In other words, an event bus solution with an Outbox pattern feature in itself.

CAP is one of the libraries that has successfully implemented the transactional Outbox pattern for .NET Core.

As you can see in the picture, this is how it promotes its NuGet Package.

CAP provides a simpler way to publish events and implement subscriptions. You don’t need to inherit or implement any interface during the subscription and submit process.

One of the strengths of asynchronous communication is reliability, where errors in one part of the system do not propagate or cause the entire system to crash. To ensure the reliability of the messages, the messages are stored in the CAP and strategies such as retries are used to achieve the eventual consistency of data between services.

Compared to other service buses or event buses, CAP has its own characteristics. It does not require users to implement or inherit any interface when sending or processing messages. It has very high flexibility, so CAP is very simple to use, very easy, and light.

CAP has a modular design and is highly scalable. You have many options to choose from, such as message queues, storage, serialization, etc., where many elements of the system can be replaced with custom applications.

Now, let’s take a look at some of the conveniences provided by CAP:

FailedRetryInterval

If the message transfer fails during the message sending process, the CAP will try to send the message again. This configuration option is used to configure the interval between each retry.

FailedRetryCount

When the value for maximum number of retries is reached, the retry will stop, and the maximum number of retries will be changed by setting this parameter.

CollectorCleaningInterval →Expired messages are deleted.

SucceedMessageExpiredAfter →The expiration time of the success message (in seconds).

When the message is successfully sent or consumed, it will be removed from the database when the duration reaches SucceedMessageExpiredAfter seconds. You can set the expiration time by specifying this value.

FailedMessageExpiredAfter →The expiration time of the failed message (in seconds).

If the message fails when sent or consumed, it will be removed from the database storage when the time reaches FailedMessageExpiredAfter seconds. You can set the expiration time by specifying this value.

The subject of message transmission to the consumer side, in the CAP in transaction approach:

At Most Once

Situations where you are guaranteed to receive all messages once or not at all are covered by the Once at Most approach.

At Least Once

Situations where you are guaranteed to receive all messages once, or several times in the event of an error, are those covered by the at least once approach.

Supported structures:

CAP uses persistent storage media to store event messages in the database. CAP uses this approach to deal with message loss in all environments or network problems.

It gives you an idea of how the CAP works when implementing the outbox template.

CAP, which is very simple to use, supports RabbitMQ, Kafka and Azure Service Bus, so you can use any message broker service you want, or you can seamlessly transition on demand.

On the database side; SQL Server, MySQL, PostgreSQL, and MongoDB are among the options it supports.

And it also has entity framework support. At the end of the article, you can review my project where I used Entity Framework and Unit of Work pattern with CAP.

Each event is kept at the database level, preventing data loss in case of possible errors or interruptions. In cases where communication is restored, it resends messages that have not been sent to the message queue system and that have not expired, thus allowing development in isolation from the message queuing system used and even from errors or interruptions.

After starting the CAP, two tables are created by default in the database, their default names are cap.Published and cap.Received.

Message publishing on CAP

After integrating the DotNetCore.CAP NuGet package into your project, you can publish your message with the name you want with the “ICapPublisher” interface.

await _capPublisher.PublishAsync<object>(“producer.MyMessage”,MyObject);

Again, since you can use the capPublisher object that you inject through the context object, you can perform your database operations and publish messages in the same transaction.

Message subscribing in CAP

In order to get the message published by the producer on the Consumer side, it is sufficient to pass [CapSubscribe(“producer.MyMessage “)] on the consumer in the relevant project. It is important that you capture it with the name of the message you publish here.

Creating subscriber groups

CAP can distribute an incoming message to more than one group, allowing it to be listened to by more services or controllers.

[CapSubscribe(“producer.MyMessage”, Group = “group1”)]

Creating a dashboard

CAP has a dashboard feature so that we can easily see the sent and received messages in real-time. For dashboard integration, you can download the DotNetCore.CAP.Dashboard NuGet package to the application and easily integrate it into your project in the startup file. You can see successful and unsuccessful messages on a dashboard screen under the address you matched in Startup; for example, /cap-dashboard. (It looks very similar to the Hangfire dashboard.)