How we ensure eventual consistency in a distributed system

Previously, we shared our take on communication inside a Service-Oriented Architecture. Today, we’d like to talk about a common pitfall in this kind of system, and the way we solved it, adding a new tool in our toolbox.

Consider the following code. We’re asked to invite a user, so:

1. we open a transaction;
2. we save an invitation in the database;
3. we send an email to the invitee to complete their invitation.

fun inviteUser(request: InvitationRequest) {
   transaction { // (1)
      val invitation = db.save(buildInvitation(request)) // (2)
      emailQueue.enqueue(buildEmail(invitation)) // (3)
   }
}

This code seems perfectly fine: should the queue be unavailable for some reason, (3) will fail, rollbacking the whole transaction, and we’ll be able to try again later. No harm done, right?

Well, not exactly. There are multiple ways this code can go wrong. We might lose the database connection and be unable to commit. Or the machine could simply crash right after the 3rd step.

Even if the 3rd step fails, rollbacking might not be the most suitable solution. There’s a well known problem in distributed systems called the Two Generals Problem. Simply said, due to the unreliability of the network, if the 3rd step throws there’s no way to know for sure if the message was enqueued or not. The network might have failed while delivering the final — positive — answer from the queue. We’ll never know.

In case of network error, we never knows for sure if the call succeeded or not.

So in the end, we might have rolled back an invitation, leaving no trace in our database, while the invitee actually received an invitation email. In this case the harm would be minor, we wouldn’t have caused major troubles to our client.

It would be a much bigger problem if we had enqueued a money transfer.

So let’s just move the enqueuing out of the transaction, right?

fun inviteUser(request: InvitationRequest) {
   val invitation = transaction { // (1)
      db.save(buildInvitation(request)) // (2)
   }
   emailQueue.enqueue(buildEmail(invitation)) // (3)
}

Nice try, but now if (3) fails, we might have the opposite problem: an invitation in database which will never be used, because the email was never sent to the invitee. So that’s not really better.

To solve this common problem, we built a tool based on persisted post-commit tasks.

fun inviteUser(request: InvitationRequest) {
   transaction { // (1)
      // (2)
      val invitation = db.save(buildInvitation(request)) 
      // (3)
      postCommit.addTask(SendEmailTask(buildEmail(invitation))) 
   }
}

This tool does 3 things.

1. Do not run the task inside the transaction. Instead, serialize and persist the task in the database as part of the transaction. This way, either everything goes well and we have both the invitation and the task in the database, or nothing.
2. Register a post-commit hook on the transaction. A post-commit hook is simply some code that will run only after the transaction has successfully been commited. This hook retrieves the task from the database, deserializes it, and executes it. If it succeeds, simply delete it.
3. Should it fail, have a job that regularly retries persisted post-commit tasks. This is very important to prevent the pitfall of the previous solution. The queue might be unavailable during the post-commit hook, or the machine could simply crash in between. So be prepared to retry.

And that’s it. 🎉

After a few years of usage, here are some tips based on errors we made.

• You may have seen that the Two Generals problem is still there: if the “enqueue” in the post-commit throws, but the queue actually received the message, the job will retry it, sending the same message twice. Indeed, and it should be stressed that this problem cannot be solved. But thanks to this tool, it can easily be mitigated: make sure to save the ID of the message to be sent in the task. This way, if the task is executed twice, the 2 messages will have the same ID, making it easy to deduplicate them on the consumer side. If you’re lucky, the deduplication might even be supported by the middleware itself. And if this task is a direct call to another service, this ID will help make it idempotent as well.
• Even if you can’t really prevent duplicated messages — and it’s OK — , you can try to reduce the occurrences. In the job, don’t fetch tasks that are too recent. Indeed, if the task was created less than a few milliseconds or seconds ago, the post-commit hook might be currently executing it, and you’ll increase drastically the chance of executing tasks twice.
• Some tasks will fail forever, this is unavoidable. If you don’t want an infinite stream of error logs, save a count of retries on the persisted task and stop retrying at some point. In our case, just like for handling DLQs, we developed a similar UI to deal with those abandoned tasks. We can ignore them, or retry them with a different content for example.
• The framework you use to register post-commit hooks is likely to execute them synchronously right after committing the transaction. If you’re handling an HTTPS request, it means you won’t answer the request before the post-commit hook completes, and that’s probably not what you want. It’s a good idea to dispatch the post-commit hook to another thread.
• This tool doesn’t work well if you need the result of the task synchronously inside the transaction, since the task is actually executed after the transaction. The only answer we have here is to design the system so that you never need to rely on the result of tasks. For example in the case above, if we needed the result of sending the email, we would instead listen for events from the mailing service, and react to these events instead.
• Apply this tool consistently. For example, in our case we already had common libraries providing clients for everything that needs to be sent to the network: emitting an event on Kafka, adding a message to ActiveMQ or doing an HTTPS call. We used this tool in each of them, enabled by default. Bypassing it is actually voluntarily hard, as we rarely see a good reason to do so.

With this tool applied all across our core banking system, we never worry about inconsistencies after crashes or rollbacks. The fact it works transparently and by default brings a real peace of mind, and even newcomers can produce rock-solid consistent code on their first day.

Thank you for reading this far, I hope this will be as useful to you as it has been to us, and if it sounds like the kind of system you want to work on, come say hi.

Originally published at https://memo.bank/en/magazine.