Retry Requests Fearlessly with Idempotence
A practical guide to being fearless in the presence of duplicate service requests
Idemfu*kingwhat?
Every single day of our lives is unique — most of the things that happen are likely not going to happen again in the same manner. They happen exactly once. Unfortunately, that isn’t the case with software systems and requests between them. Why should that matter? Imagine purchasing a new laptop and because of some duplicate service requests being charged twice. That doesn’t sound fun, does it?
In theory, exactly-once request delivery is impossible — the pitfalls and design challenges are illustrated by the classic Two Generals Problem. However, in practice, we rarely care about delivering our requests exactly once. We normally go for the second-best thing — at-least-once delivery where we keep sending the same service request until we are certain that it is received (it might be received multiple times though). But hey, we don’t want to get charged twice, right? We can avoid that by making sure that handling the same request multiple times doesn’t change the result beyond the initial application. In fact, that’s the essence of idempotence.
Now that you have received a duplicate service request, how do you handle idempotence? Let’s explore your options with an example of a bank account. The code examples are written in C# because its syntax is fairly simple and should not be challenging to understand for the majority of developers.
Maybe you don’t need to do anything after all!
Some requests are idempotent naturally and you don’t have to worry about retrying them at all. Take a simple bank account aggregate for instance:
To open a new bank account you can send an OpenAccountRequest like this:
OpenAccountRequest is idempotent as it is. Whenever you receive a request, you can check if an account with the same ID exists in our database. If the account already exists — you know that you received a duplicate request, therefore you can ignore it. Moreover, if your database guarantees account IDs to be unique you can simply check for the duplicate key exception instead:
It’s all fun and games until you try to withdraw or deposit money. If you send the same deposit request twice because of a network error — you deposit twice the money even though you initially intended to deposit once. Twice the money — twice the fun!
Do you know the current state?
One way to handle duplicate requests is to ask for the current state of the aggregate. Strictly speaking, this solution is typically considered for a bit broader problem of optimistic concurrency than idempotence and it doesn’t identify duplicate requests. However, if you ask to provide the account balance in your DepositToAccountRequest, you will be capable to at least protect yourself against duplicates:
Since you can’t identify duplicate requests you can’t be certain whether you received a duplicate request or the client simply wasn’t aware that the balance had been changed. Therefore you would most likely reject all invalid state requests and ask the client to get the latest state and try again (hence no try-catch block ignoring InvalidStateException).
This solution is fairly easy to implement and it doesn’t introduce technical details into our account aggregate. However, it is not very flexible since for other requests to be protected in the same way providing account balance may not be enough. Not to mention it could fail in highly concurrent scenarios, i.e., while you retry a successful withdrawal request someone else deposits the same amount of money making you withdraw twice the amount. To mitigate those issues you can artificially generate a state by versioning your account. Whenever you change the account aggregate you must increase its version:
Unlike with account balance, you can include account version into any of your requests. Highly concurrent scenarios are covered as well since you are only allowed to increase the version. And if you are using Event Sourcing pattern, most likely you already have a version in your aggregates.
It is important to consider the limitations of this approach. To send a request, you need to know the current state of the aggregate. And if the aggregate is changed rather frequently, using the system becomes quite difficult and inefficient since a lot of requests get rejected because of outdated state. But hey, at least you are not that scared of duplicates anymore, right?
Just put a database transaction all over this!
Another, very common solution requires an ACID compliant database. First, you need to add unique IDs to your requests which shouldn’t be changed when retrying:
Then you have to create another database table that is going to store the history of request IDs. The table should guarantee IDs to be unique. When you receive a request you need to insert the request ID into the new database table right before persisting your aggregate state within a database transaction. If the request ID is already stored you know that it’s a duplicate request. Unlike with account balance or version, now you are actually making requests idempotent.
Can we optimize this? You probably don’t need to store every request ID since the dawn of time. Can you assume that you never receive duplicate requests after a certain amount of time? If that’s the case — you can have a process that periodically removes IDs from the history table making it very lightweight.
What if a transaction isn’t an option?
Sometimes you don’t have an ACID compliant database making transactions not an option anymore. What you can do instead is store request IDs directly into your aggregate:
This doesn’t protect you against duplicates in highly concurrent scenarios, however, if you don’t send your request retries concurrently (in most cases you shouldn’t) you don’t have to worry about that.
If the aggregate is being changed frequently storing every request ID might increase its size by quite a lot. But if you only care about the most recent requests (like with database table before), you can optimize this by storing only a certain number of the most recent operations.
Once again, if you are one of the Event Sourcing folks, the life sometimes is just a tiny bit better to you. Since you are already storing your aggregates as event streams, you can add a request ID to each of the resulting events. Whenever you rebuild your aggregate you can rebuild the request history as well and validate a new request against it:
There’s one more option if you are using Event Sourcing. It is based upon being able to generate deterministic UUIDs defined in RFC 4122. If your event store can guarantee unique event IDs you can use a request ID to deterministically generate resulting event IDs. Now the only thing that is left is to catch a duplicate key exception since the rest is being handled by your event store:
Wow, you are still reading? It’s time to summarize!
There’s no right way to deal with idempotence — it depends solely on your business and technical requirements. Sometimes you might receive service requests that are idempotent naturally. At other times, if your aggregates don’t change very often, it could make sense to provide the current aggregate state in your requests as a way of dealing with duplicate requests. Maybe you have an ACID compliant database? If that’s the case, you could potentially go with database transactions since it is fairly simple to implement, can be used with all of your requests, and doesn’t leak technical details into domain. Do you use Event Sourcing pattern with an event store that guarantees unique event IDs (i.e. SQL Server)? Then you could avoid transactions by generating deterministic event IDs. What if your event store doesn’t guarantee unique event IDs (i.e. Greg Young’s Event Store)? You could store request IDs in your aggregates instead. Not to mention that likely there are other ways used in certain scenarios that I have not covered.
Thank you for reading. I’d love to hear how you handle idempotent requests.
