Organizing Kafka events by partitioning.
In this article, you will read about how we leverage Kafka partitioning to resolve problems in practice and what we learn from that.
Birth of a new service
At Qonto, we offer a feature that allows a user to exchange funds instantly between two accounts. This type of transfer is known as a book transfer.
As Qonto is growing, the number of transfers is increasing. We decided to create a new payment system based on event-driven design to ensure the scalability and robustness of these internal transfers.
This new service is called “BookTransfers”. When the user requests a book transfer, it will orchestrate an authorization request and a settlement request to our Core Banking System. We have to wait for the events published by the Core banking System to confirm the transfer is processed.
As Kafka is a technology broadly used at Qonto to implement an event-driven solution. We chose Kafka as the message broker to help us in sending and receiving events.
While designing those events, as for any other payment system, we had to pay attention to our way to implement producer and consumer. How to ensure that an event will be consumed? How to make a consumer idempotent in case an event is published or consumed twice? How to handle or prevent inconsistency? How to choose a partition when publishing an event? In this article, we’ll focus on this last question and explain how we answered it using the Kafka partitioning strategy.
The partitioning strategy is a Kafka key concept that allows us to ensure the partition on which an event will be published. To set this strategy, Kafka provides a key called “partition key”.
This key allows Kafka to select the partition on which an event will be published. This parameter is set by the producer while publishing an event. By default, it’s a random ID, which means that the partition used is chosen randomly. However, this parameter can be set manually.
Why does partitioning matter?
To implement the confirmation step generated by our Core Banking System, we receive within a very short time two events for each debit and credit side of a single transfer as illustrated below.
- Event A: Debit confirmation event
- Event B: Credit confirmation event
At the reception, each event will update a transfer status in the database.
We need then to ask ourselves a few questions before starting to implement our consumers and producers.
A. Do we want to make sure that Event A is always processed before Event B? Here, we only need to make sure we receive both events for a single transfer but we don’t have to receive them in a specific order. We are looking for eventual consistency in BookTransfers.
Each confirmation event received will update the status of the book transfer, regardless of whether they’re debit or credit.
When both events are received, the BookTransfers service will notify the acceptance or rejection of the movement.
B. Do we allow events A and B to be consumed in parallel by separate consumers? Here, the answer is no. Since we want to avoid using a DB transaction to update the BookTransfers status, consuming both events in the very same time window would expose us to a race condition issue.
Here, since each consumer processes events synchronously, we want to look at Kafka partition key.
As we know a single partition will be assigned to a single consumer, publishing events A and B on the same partition should ensure one worker will consume both of those events.
Now we need to select our partition key and sort correctly our events.
We know this partition key should be a common parameter between the credit and debit events.
When checking our potential candidates, we select creditor IBAN. Both events have actually the same creditor information.
However, we understood that the creditor IBAN wouldn’t be a great candidate because we can’t control how often a customer generates a transfer. Two customers could generate a different number of book transfers per month, so we would then lose load balance on partition events.
This key should be common to both events, but since we want to keep this load balance, it must be generated randomly.
The next candidate that should meet this requirement is what we call a movement_id. It’s a Base64 ID identifying a transfer on all our backend services. Both sides share the same movement_id.
Since it meets all the requirements, we choose this candidate to be our partition key.
Conclusion
This experience was a great learning on Kafka partitioning. Besides being useful while creating a new system based on event-driven, the question of which to use key should be asked almost every time you create a topic. Most of the time, you may not need to change it, but it’s good to know what possibilities this feature is offering to you.
In the end, it’s your use-case that will dictate whether you need to change the topic’s partition key and you need to use it to decide which key to use.
About Qonto
Qonto is a finance solution designed for SMEs and freelancers founded in 2016 by Steve Anavi and Alexandre Prot. Since our launch in July 2017, Qonto has made business financing easy for more than 250,000 companies.
Business owners save time thanks to Qonto’s streamlined account set-up, an intuitive day-to-day user experience with unlimited transaction history, accounting exports, and a practical expense management feature.
They stay in control while being able to give their teams more autonomy via real-time notifications and a user-rights management system.
They benefit from improved cash-flow visibility by means of smart dashboards, transaction auto-tagging, and cash-flow monitoring tools.
They also enjoy stellar customer support at a fair and transparent price.
Interested in joining a challenging and game-changing company? Consult our job offers!
