Outbox pattern — a quick overview

Today we’re going to talk about a classic problem that may occur every time one considers building a decoupled system following the event-driven architecture. What should happen if emitting an event is not successful?

Of course, you could log the event’s body just before sending it. However, what about obfuscated personal data? Well, it could be taken from the database. But what should one do if the database access on production environment is restricted? These kind of questions should not be asked since that kind of problems should have their fairly simple solution — even if it is an event schema mismatch or the message broker being temporarily unavailable.

Photo by Paula Hayes on Unsplash

Communication in event-driven architecture

At first, let’s focus on asynchronous communication. Dividing a project into a couple of microservices can increase teams’ autonomy, shorten the pipelines and come up with smaller, more domain-focused deployments units (although some of these benefits can be also achieved by developing a modular monolith).

The independence factor can be boosted if the services communicate asynchronously — the fact that service A emitted event Vn+1 does not mean that the message will be lost when service B is not prepared for consuming it. The event will be temporarily skipped and eventually processed once the new version of service B is deployed.

Here’s how more and more teams achieve independence and event backups while avoiding bottlenecks introduced by synchronous processing at the same time. However, communication or performance problems may occur also between the service itself and the event broker. Here’s where the outbox pattern comes in handy.

Outbox pattern

The pattern consists of a few principles:

1. The event publisher guarantees that the event will be sent to the event broker.
2. After processing an event or http request, the service stores the event intent in the database. You can find an example migration script in the snippet below.
3. A separate transaction periodically looks for the events that have not been sent and tries to emit them. Once it succeeds, the event status is updated.

CREATE TABLE `outbox_event_intent` (
`id` bigint NOT NULL AUTO_INCREMENT,
`created_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
`modified_at` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`version` bigint NOT NULL,
`sent_at` TIMESTAMP DEFAULT NULL,
`body` LONGTEXT NOT NULL,
`status` VARCHAR(255) NOT NULL,
`idempotency_key` VARCHAR(255) NOT NULL,
`partition_key` VARCHAR(255) NOT NULL,
`type_id` VARCHAR(255) NOT NULL,
`topic` VARCHAR(255) NOT NULL
);

Benefits

First of all, it stresses the publisher’s responsibility for sending the events. If for any reason the event does not end up on the broker, one can be 100% sure that the intent is saved provided the service’s client got a response indicating that request was processed successfully.

1. Having all of the intents stored with their unique identifiers makes it easy to implement an endpoint for resending any of the events. This identifier in our case is called idempotencyKey and is used by event consumers in order to make sure that every event is process at most once.
2. The intents saved in the database do not need to follow event retention policies established on the broker. As a result, the intent storage is also an event back up.

Any drawbacks?

1. Saving the intent should be possibly close to the business process. It means that every microservice that applies the outbox pattern is also responsible for storing the intents. As a result, although the main principles introduced by the outbox pattern can be hidden behind a layer of abstraction, writing a bit of boilerplate code seems to be inevitable.
2. If sending the event used to be a part of a synchronous process, you might need to redesign it a bit. Why? Well, it may happen that the actions taken after sending the event relies on it being sent. After introducing the outbox pattern, no one knows when exactly will the event end up on the topic. As a result, the actions depending on the event being sent should be moved to a newly created listener that will perform them once it’s on the topic. It might sound tricky, but the diagrams below should make it more clear.

Alternative approach

Although introducing the outbox pattern may sound like a great idea, a completely natural behaviour is trying not to repeat yourself not to repeat yourself. As a result, it’s likely that you will consider centralising the solution. Let’s focus on two ideas that would make it possible.

1. A centralised database with a predefined schema and a separate service that processes the intents.
2. A separate service exposing a REST API that accepts the intents, stores it in its own database and then processes them. Basically it’s about the outbox pattern encapsulated inside of a separate deployment unit.

Let’s focus on the first one at the beginning. Implementing it would mean that every service that communicates with the event bus would also need to connect to two databases. Well, it seems that we’ve just replaced a bit of boilerplate code with… complexity. The amount of code may differ depending on the programming language and used frameworks but the fact that resolving one problem originates another one in undeniable.

Also, a huge disadvantage of both approaches is the fact that they violate one of the outbox pattern principles — storing an intent has to happen in the same transaction as other side effects that result in storing new data. Obviously, everything can be achieved by proper distributed transaction management, but again — boilerplate code will be replaced with complexity.

Example implementation

Having said that, the solution based on coming up with a proper abstraction layer covering the implementation details (entities and database schema) seems to be the safest and the simplest one. Since we all know that talk is cheap, let’s look at the code snippets written in Java.

public void save(String topic, String partitionKey, String idempotencyKey, Object event)
        throws JsonProcessingException {
    log.info("Saving event intent - topic: {}, partitionKey: {}, idempotencyKey: {}", topic, partitionKey,
            idempotencyKey);

    outboxRepository.save(EventIntent.builder()
            .status(EventStatus.NEW)
            .idempotencyKey(idempotencyKey)
            .partitionKey(partitionKey)
            .typeId(event.getClass().getName())
            .body(objectMapper.writeValueAsString(event))
            .topic(topic)
            .build()
    );
}

Here’s the first part of the process, which is translating the event and its parameters to an event intent that’s eventually stored.

public void send() {
    log.info("About to start processing event intents. Number of batches: {}, batch size: {}", numberOfBatches,
            batchSize);

    IntStream.range(0, numberOfBatches).forEach(i -> {
        List<EventIntent> eventIntents = outboxRepository.findBatchForUpdate(batchSize);

        for (EventIntent intent : eventIntents) {
            log.info("About to send an event with idempotencyKey: {}", intent.getIdempotencyKey());
            try {
                kafkaTemplate.send(toProducerRecord(intent)).get(10, TimeUnit.SECONDS);
                intent.setStatus(EventStatus.SENT);
                intent.setSentAt(ZonedDateTime.now());
            } catch (Exception e) {
                log.error("Sending an event with idempotencyKey: {} failed. Exception message: {}",
                        intent.getIdempotencyKey(), e.getMessage());
                intent.setStatus(EventStatus.FAILED);
            }

            outboxRepository.save(intent);
        }
    });
}

The snippet above contains a function that should be called periodically so that all unsent intents are processed. The processing is divided into batches so that the time interval, batch size and number of batches could be configured for each service separately.

Both snippets placed above are the part of the same KafkaOutbox class, cover the main Outbox Pattern principles and are the part of the common library used in every service that communicates with the event broker (Kafka in this specific scenario). Keep in mind that because of that the boilerplate code may differ when it comes to technical details.

First of all, the migration script needs to be copied to the repository. Secondly, every service needs to declare it’s own repository interface for storing the intents. Then, a new instance of KafkaOutbox class with a proper repository injected needs to be created and put into the application context. Finally, the job for processing the intents has to be declared.

In this article I tried to cover the pattern in as abstract way as possible so that technical implementation details are not a blocker for anyone who is not familiar with Java. However, the challenges that might result in a different implementation are listed in the paragraph above.

Happy outboxing!