Golang: Event-driven architecture

ihippik© drawn with love

Introduction

Let’s imagine that we have written an online shop using a microservice architecture.
Services are divided according to business opportunities:

• Product
• Customer
• Orders
• Payment

Each service has its own repository (PostgreSQL)
In my example, the database is one for the entire store, but a scheme of the same name has been created for each service.
Since we know that the asynchronous type of communication between services improves their availability, we select it.
Using an event bus is best suited here.
That is every time we change, create or delete our domain model, we publish the corresponding event and those services that are interested in subscribing to them.
I will use the already familiar NATS Streaming.
This approach has a lot of advantages, but there are also not pleasant moments. One of them is how to store data in services consistent?!

Our problem

Let’s explain the essence of this problem in more detail.
Suppose a client makes an order in our online store and for this, we need to perform the following sequence of actions:

• Register a new client (microservice ‘client’)
• Reserve goods on order at the warehouse (microservice ‘warehouse’)
• Create an order with this product (microservice ‘order’)
• There are also a few nuances. Withdraw money from the customer’s card (microservice ‘payment’)

To complete the order, the domain models of our 4 microservices must be updated, and they must do this manually.
Let me remind you that we made our life a bit more complicated and decided to communicate between services asynchronously, through an event bus. And we had difficulty storing consistent data.
How can we guarantee that we will create an order and make payment for it only after a successful reservation of the goods?
In the case of a monolithic service, or case of synchronous interaction between services, we could execute it in one transaction.
We will have to publish events as part of a local transaction. Unsuccessful development options will not be considered now,
since this is not the topic of this article, I will only say that in such cases countermeasures are usually provided.

Event publishing

We will dwell in more detail on the publication of messages.
Each database has its own implementation.

In PostgreSQL, this is Write-Ahead Log.

WAL — a mechanism to help recover data after a crash.
The principle here is that before your data is stored in the database itself, it is saved in this log, which can later restore everything.
We will also need logical decoding , which is the basis for logical replication in PostgreSQL.
Logical decoding just uses the WAL mechanism and the publication/subscription model with one or more subscribers.
Subscribers receive data from the publications to which they are subscribed.
Logical decoding usually begins by taking a snapshot of the data in the published database and copying it to the subscriber.
After this, changes on the publication side are transmitted to the subscriber in real-time when they occur. That is what we need!
Logical decoding data output is implemented not at the DB core, but through the wal2json plugin.
You can find instructions for installing it via the link or use the ready-to-use docker image from debezium.

WAL-Listener service

And now about the main thing.
I wanted to tell you about the service that I especially wrote to implement all this magic — WAL-Listener
In Golang, logical replication is supported by the excellent pgx driver.
Look at it in more detail. It is very fast and has many useful features, and my library is based on it.
What can we do with it? Here is the procedure that will help you use it in your project.

• Configure the service in such a way that it will listen to the tables and actions you need (insert, update, delete).
  In the case of adding and changing, you will receive a data map, and in the case of deletion, you will receive a primary key of the record.
• Your microservice changes the domain model data and WAL-Listener publish these events in NATS Streaming server.
  The name of the topic you can subscribe to will be formed according to the following principle: prefix + db_name + db_schema
• Service that needs to know about these changes, subscribes to these events and disposes of them at its discretion.
  If your client is not available, you can receive all events after reconnecting.

There are also a few nuances.
DDL, the truncate operation and sequences are not replicated, an empty message will come instead.
Also, the primary key must be used on your tables, otherwise, there will be problems with the update and delete events.

P. S. Well, for fans of storing logic in the database, there is the opportunity to do everything simply. You can write a trigger and use the NOTIFY command.
You define a trigger event for any of your tables and publish any data you need.
And already on the client subscribe to these events. Almost all drivers for PostgreSQL can do this. Of course, this method is not suitable for something serious.