The Ack problem — Part 9
TL;DR, and other topics for a future series
What are the key takeaways from all this?
- Databases are downstream API calls: your SQL and DB calls must be idempotency-aware if you want your APIs to be idempotent;
- Message consumers are API servers in disguise: the idempotency requirements and solutions are the same;
- Because your SQL (or equivalent) must be idempotency-aware, no amount of generic infrastructure/middleware/library/framework/etc will make your code idempotent, unless of course this generic code also implements your DB and downstream calls, i.e. becomes your service layer altogether;
- Your favorite message broker assumes that your message consumer is idempotent. You will receive duplicate messages, and your core logic must be ready to deal with them. If you claim to have solved this in a generic manner, or that you have an advanced messaging lib that implements this for you, please check the previous point.
What is next?
All the discussion so far was done under one key assumption: the services are not forgetful. This is made possible because idempotency tokens (and the like) are saved in databases and other long-term persistent storage.
We will discuss something I call “the Backup problem” in a future series. But here is a sneak preview.
Backups have been introduced to avoid data loss during disaster recovery procedures (DR). But backups are “point in time” copies, meaning that when we restore these backups we actually “go back in time”: to the last backup time (also known as the recovery point objective, or RPO). In other words, if you make backup, this means that you accept losing some data. That is the backup paradox.
Typically when this occurs, we just resume from an earlier point in time and we apologize to our customers that some transactions will need to be repeated. However, micro-services make this much much MUCH harder. Indeed, database backups are typically made per DB instance, and we love having each of our services running on their own, different DB instance. This means that each service has a backup from a different “point in time”.
Which means that one service may “remember” a specific idempotency token, but some other services, either upstream or downstream, may not. This may produce “interesting” effects.
But as I said, this will be the topic of a next series!
