Outbox Pattern for reliable data exchange between Microservices
This story talks about how efficiently we can exchange data between microservices in event-driven architecture. How outbox pattern helps in achieve this and what problem does it solve and what else it can not solve.
If you’ve built a couple of microservices, you’ll probably agree that the hardest part about them is data, microservices don’t exist in isolation and very often they need to propagate data and data changes amongst each other.
Receiving a request, saving it into a database, and finally publishing a message can be trickier than expected. A naive implementation can either lose messages or, even worse, post incorrect messages.
User Service → This service will have a list of registered users.
NewsLetter Service → This service will have newsletters configured for users based on the subscriptions. Users can subscribe too. It had a document data store.
Kafka Streams → In the above use case three events
New User Singedup,News Letter Subscription, andEmail Updated.User Service DB → The underlying DB is Postgres. When a new user signs up the user data is stored in local store. When any user requests for email update the email is updated in the User table.
What is the problem here?
When a microservice changes the state for which it is the system of record and then signals to subscribers via an event that it has changed its state, how do we ensure that subscribers receive the event and are therefore consistent with the producer?
The system of record may become inconsistent with downstream consumers because after writing changes to an entity, we may fail to publish the corresponding message. Our connection to the message-oriented middleware (MoM) may fail, or MoM may fail.
The new record is saved to the data store, but the event is not raised so subscribing systems become inconsistent.
Let us understand the use case above. The User Service manages the user data, this service is a single source of truth for the user. When new users signup with user service, it has to save the local copy and let the NewsLetter service know that the new user has signed up. The NewsLetter Service will assign default newsletters to the user. The user can manage them later.
The User Service can let the other service in both synchronous and asynchronous ways.
Synchronous Approach
The service can communicate through Rest , RPC or any other synchronous way. This might create some undesired coupling, though, the sending service must know which other services to invoke and where to find them. It also must be prepared for these services temporarily not being available. In addition to that, we should add capabilities like request routing, retries, circuit breakers, and much more.
The general issue of any synchronous approach is that one service cannot really function without the other services which it invokes. While buffering and retrying might help in cases where other services only need to be notified of certain events, this is not the case if a service actually needs to query other services for information. Another downside of such a Synchronous approach is that it lacks re-playability, i.e. the possibility for new consumers to arrive after events have been sent and still be able to consume the entire event stream from the beginning.
Asynchronous Approach
Both the above problems can be addressed through an Asynchronous approach. i.e propagating events from User Servicethrough a durable message log such as Apache Kafka. By subscribing to these event streams, each service will be notified about the data change of other services. It can react to these events and, if needed, create a local representation of that data in its own data store, using a representation tailored towards its own needs. For instance, such a view might be denormalized to efficiently support specific access patterns, or it may only contain a subset of the original data that’s relevant to the consuming service.
Durable logs also support re-playability, i.e. new consumers can be added as needed, enabling use cases you might not have had in mind originally, and without touching the source service. Once the User Service events are in a Kafka topic (Kafka’s topic’s retention policy settings can be used to ensure that events remain in a topic as long as it's needed for the given use cases and business requirements), new consumers can subscribe, process the topic from the very beginning and materialize a view of all the data in a microservice’s database, search index, data warehouse, etc.
Distributed Transactions may seem like an answer but possess two issues. First, we are probably using a backing store and message-oriented middleware from different vendors or OSS projects that don’t support the same distributed transaction protocol. Second, distributed transactions don’t scale well.
We might naively try to fix this by sending the message first, then updating the backing store if that succeeds. But this won’t necessarily work either, as we might fail to write to the database.
The new record is posted to downstream systems, but the local database call is rejected, and so the upstream system is now inconsistent. A phantom send.
This lead to the Dual Writes Issue.
Dual Writes lead to Data inconsistency
Dual writes frequently cause issues in distributed, event-driven applications. A dual write occurs when an application has to change data in two different systems, such as when an application needs to persist data in the database and send a Kafka message to notify other systems. If one of these two operations fails, you might end up with inconsistent data. Dual writes can be hard to detect and fix.
Dual Writes can be fixed using Outbox Pattern.
Outbox Pattern
When you apply the Outbox pattern, you split the communication between your microservice and the message broker into two parts. The key element is that your service provides an outbox within its database.
It is similar to Gmail's outbox, the email will be persisted until it delivers to the intended recipient.
Typically outbox is a table that resides at the source system. In our case, it is User Service.
The contents of the outbox table
There is no such rule to have an outbox structure in the below format. Please re-design as per your requirement. I have used this format for the demo.
Now that the outbox table is set up, we have to define a process in the eco-system to read data from the outbox table in another service. The other service can have any kind of database (RDBMS/Doc Store). Below are approaches that we can implement.
- You can implement a service that polls the outbox table and sends a new message to your message broker whenever it finds a new record.
- You can use a tool like Debezium to monitor the logs of your database and let it send a message for each new record in the outbox table to your message broker. This approach is called Change Data Capture (CDC).
I prefer approach 2 because of its capabilities in enterprise use cases and less maintainance headache.
Outbox Pattern With CDC using Debezium In Action
Technologies used
Java 11
Spring Boot 2 and Spring Cloud.
Gradle
Debezium Kafka Connector.
Postgres 12.
Mongo DB.
Apache Kafka.
Docker
The infra is set up in the docker file.
We have a multi-module spring boot project with Gradle as a build tool.
- User Service → This service lies with all user details and a single source of truth for user data. It has Postgres as a datastore. This service has an outbox table. This service has Postgres schema and seed data that creates tables and some data during the start-up of Postgres instance through docker.
- Outbox Transformer → This is a utility module that connects to Debezium Kafka Connector. It has a Custom Outbox Transformer, where the transformer controls the events emitted by database changelogs. We are not using the default transformer we wrote our own that satisfies our use case.
- NewsLetter Service → This is the target service for outbox log events. It has MongoDB as a datastore.
Please click on the GitHub link for the complete source code.
Docker Compose file
Custom Outbox Transformer
At Line 22 → The conditional logic, Debezium captures all the changes for the table. Here, I am restricting for only Insert/Creation of a new record.
At LIne 27 → Get the value from Data change event_name and used that as Kafka's topic to publish the message.
To get more idea on Debezium’s CDC message format. Please read that in my earlier story about CDC.
Debezium Kafka Connector
The outbox type is defined for the custom transformer that we wrote.
curl -X POST http://localhost:8083/connectors/ \
-H 'content-type: application/json' \
-d '{
"name": "student-outbox-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "postgres",
"database.port": "5432",
"database.user": "postgres",
"database.password": "postgres",
"database.dbname": "user_DB",
"database.server.name": "pg-outbox-server",
"tombstones.on.delete": "false",
"table.whitelist": "public.outbox",
"transforms": "outbox",
"transforms.outbox.type": "com.eresh.outbox.OutboxTransformer"
}
}'Demo
Build Steps
First, build and prepare docker image for outbox-transformer. The Docker will add this generated jar file of outbox-transformer to Debezium connect. As we are using Custom Outbox Transformer we need to add this to the build path of the Debezium connector.
FROM debezium/connect
ENV DEBEZIUM_DIR=$KAFKA_CONNECT_PLUGINS_DIR/debezium-transformer
RUN mkdir $DEBEZIUM_DIR
COPY build/libs/outbox-transformer-0.0.1-SNAPSHOT.jar $DEBEZIUM_DIRBuild
$ cd outbox-transformer
$ ./gradlew clean build
$ docker build -t outbox-transformer .The response of the docker build
[+] Building 41.9s (8/8) FINISHED
--------
=> [internal] load build definition from Dockerfile
=> [2/3] RUN mkdir /kafka/connect/debezium-transformer 0.7s
=> [3/3] COPY build/libs/outbox-transformer-0.0.1-SNAPSHOT.jar /kafka/connect/debezium-transformer 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:e4bebe4b973e13ce083f64e3743c3f002d5ba83812dc30e350425f9d6d50f195 0.0s
=> => naming to docker.io/library/outbox-transformerNow we have added our custom transformer to the Debezium connector.
Invoke Docker Compose file
$ cd .. (Go to root directory that's where docker-compose.yaml file resides. Then run below command
$ docker-compose up -d
I haven’t used any Data migration tools like Liquibase or flyway to create tables. I used docker to create tables, that execute during Postgres initialization. If you see there are two tables already created.
Create topics and Debezium Connector configuration. For Simplicity, I have created a shell script file for that job for us.
$ ./init.sh
--------------
-e
-------- Creating Kafka Topics Initiated --------
-e
-------- Creating Kafka Topics new_user --------
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic new_user.
-e
-------- Creation Kafka Topics new_user Completed --------
-e
-------- Creating Kafka Topics news_letter_subscription --------
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic news_letter_subscription.
-e
-------- Creation Kafka Topics news_letter_subscription Completed --------
-e
-------- Creating Kafka Topics email_changed --------
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic email_changed.
-e
-------- Creation Kafka Topics email_changed Completed --------
-e
-------- Creating Outbox Kafka Connector For Debezium --------
{"name":"student-outbox-connector","config":{"connector.class":"io.debezium.connector.postgresql.PostgresConnector","tasks.max":"1","database.hostname":"postgres","database.port":"5432","database.user":"postgres","database.password":"postgres","database.dbname":"user_DB","database.server.name":"pg-outbox-server","tombstones.on.delete":"false","table.whitelist":"public.outbox","transforms":"outbox","transforms.outbox.type":"com.eresh.outbox.OutboxTransformer","name":"student-outbox-connector"},"tasks":[],"type":"source"}-e
-------- Creating Outbox Kafka Connector For Debezium is Completed --------We have done all infra setups. It’s time to start the services.
- Start User Service as Spring Boot application. You can customize environment variables as per your needs.
- Start NewsLetter Service as a Spring Boot application. You can customize environment variables as per your needs. (For this you need to install MongoDB in your local. I used Community Edition 4.2 version).
Use Case 1, Create New User
Let’s Create a new user. During the creation of the user the User Service will save the user in its local copy of USERS Tableand another copy in OUTBOX TABLE with event_name as new_user . Then, the CDC of Debezium Kafka Connect will publish the newly created event on the topic. This topic is subscribed by News Letter Service and creates two default News Letters to the users Best Sellers, Weekly Frequency and Hot keywords, Monthly Frequency .
$ curl -X POST \
'http://localhost:9000/api/user' \
-H 'content-type: application/json' \
-d '{
"fullName": "Medium Demo",
"email": "medium.demos@gmail.com",
"mobileNumber": "+919876543214",
"gender" : "Male"
}'
{"id":"a2beaf38-4569-44e6-8d37-bf54553f7b69","fullName":"Medium Demo","email":"medium.demos@gmail.com","mobileNumber":"+919876543214","gender":"Male","createdAt":"2021-08-19T16:35:29.719768"}Verify-in User Service Tables
A new record is created in User table and Outbox table with event_name as new_user .
Verify NewsLetter Service
Use Case 2, Users can create Subscriptions
During this use case, there is no action on User Service . But I invoked the News Letter Service through User Service (Please bear with me on this). This will add a new row in outbox table with event_name as news_letter_subscription .
$ curl --location --request PUT 'http://localhost:9000/api/user/subscribe' \
--header 'Content-Type: application/json' \
--data-raw '{
"userId": "a2beaf38-4569-44e6-8d37-bf54553f7b69",
"subscriptions": [
{
"frequency": "Monthly",
"newsLetter": "Top Brands"
}
]
}'A new row has been inserted in the outbox table with event_name as news_letter_subscription .
Verify NewsLetter Service
A new record is inserted in the document with the subscription details.
Use Case 3, Update email of the user
This will update the USER table and insert a new row to OUTBOX table event_name as email_changed. The subscriber listening to the topic in News Letter Service will update all the subscriptions for users with the modified email.
$ curl --location --request PUT 'http://localhost:9000/api/user/email/a2beaf38-4569-44e6-8d37-bf54553f7b69' \
--header 'Content-Type: application/json' \
--data-raw '{
"email" : "medium.demo@medium.com"
}'
{"id":"a2beaf38-4569-44e6-8d37-bf54553f7b69","fullName":"Medium Demo","email":"medium.demo@medium.com","mobileNumber":"+919876543214","gender":"Male","createdAt":"2021-08-19T16:35:29.802427"}Verify-in User Service Tables
The User the table is updated and a new row is added in the outbox with event_name as email_changed .
Verify-in NewsLetter Service
Learnings
We have learned about Outbox Pattern using Change Data Capture of Debezium. The outbox pattern is a great way for propagating data amongst different microservices.
By only modifying a single resource — the source service’s own database — it avoids any potential inconsistencies of altering multiple resources at the same time which don’t share one common transactional context (the database and Apache Kafka). By writing to the database first, the source service has instant “read your own writes” semantics, which is important for a consistent user experience, allowing query methods invoked following to a write to instantly reflect any data changes.
At the same time, the pattern enables asynchronous event propagation to other microservices. Apache Kafka acts as a highly scalable and reliable backbone for the messaging amongst the services. Given the right topic retention settings, new consumers may come up long after an event has been originally produced, and build up their own local state based on the event history.
Advantages of Outbox Pattern
- The big advantage of this pattern is its relative simplicity, compared to alternative solutions
- Outbox Pattern gives us at least one Successful transactional commit.
- Ability to audit and replay events.
- Avoid Complex application logic.
- The tailored data format is conducive to consumers and reduces integration issues. We can use Avro and schema can be evolved.
Challenges or Problems that can’t be solved by Outbox Pattern
The Message Relay might publish a message more than once. It might, for example, crash after publishing a message but before recording the fact that it has done so. When it restarts, it will then publish the message again. As a result, a message consumer must be idempotent, perhaps by tracking the IDs of the messages that it has already processed. Fortunately, since Message Consumers usually need to be idempotent (because a message broker can deliver messages more than once) this is typically not a problem.
