My last post on exactly-once in Apache Kafka provoked a couple of replies from smart people. Never one to let someone else get the last “Well actually…” I thought I’d give a quick rundown.
In reverse chronological order…
Henry Robinson: Tinkerbell Consensus
Henry Robinson says that Exactly-once or not, atomic broadcast is still impossible in Kafka — or anywhere and says that I am proposing “Tinkerbell consensus”.
Now, I’m sure you see the flaw in this: “Bell” is Tink’s last name so this should really be “Tinker Bell consensus”. And Tinker Bell is awesome so let’s not use her name in disparagement! :-)
Henry and I discussed this post on Twitter and I think the root of this is a misunderstanding, likely my fault. I think Henry and I would agree on the following points:
- FLP is a theorem
- Atomic Broadcast is equivalent to consensus
- Atomic Broadcast and consensus algorithms that are correct (safe) must sacrifice liveness under some conditions.
- If you believe that consensus is a successful practical distributed systems building block despite the liveness/correctness tradeoffs you should believe that Atomic Broadcast can be too
I think the crux of the disagreement is that Henry thinks I’m saying that “if we believe in the possibility of consensus it is possible”. Leap and the net will appear! Hence the Tinker Bell thing.
I can see how you’d read it that way, but that is not what I am trying to say.
The paragraph Henry critiques begins “So how does this play out in practice? Well, as a practical matter…”. I am trying to address the question of whether a real-world system can provide Atomic Broadcast as a practical abstraction. I think this is really the critical question. When people say “Exactly once is impossible” this is what they are actually asserting.
In an earlier draft of the post I did this by doing a deep dive into how to reconcile the fact that, on one hand, we have FLP claiming consensus is impossible, and on the other hand, consensus algorithms are perhaps the single most successful distributed systems building block in the world and are at the heart of systems that power a non-trivial part of the economy. This is a very interesting topic, but the problem is that it takes you very far afield of the question we’re actually trying to answer and loses most people.
What I realized is that I don’t need to make an argument for the practical possibility or impossibility of atomic broadcast. Why? Because Kafka is already a consistent log and the new capabilities are just piggybacking on that in their implementation. The new features don’t change the correctness/liveness tradeoffs of Kafka. Hence if you think consensus is impossible as a real-world systems building block then you already believe Kafka is impossible. If you believe consensus is a practical building block for real-world systems then given the implementation we described, and the fact that it boils down to features around a consistent log, you should believe the same can be true for Atomic Broadcast using Kafka.
This, by the way, is exactly our goal. We think Atomic Broadcast is a hugely powerful abstraction with a very close relationship to event-driven systems, and that being able to provide this in a practical, operable way that can run at company scale will change how companies build systems and services. We hope to make this as important as a building block within organizations as consensus has been in distributed systems internals.
Flavio Junquera: No Consensus in Exactly Once
Flavio gives a really good deep dive into the problem and makes several good points.
First he addresses the meaning of the word “delivery” in exactly-once delivery and asserts that it must be an application defined thing and can’t be purely supplied by the message transport layer. I think this is an important point. This is inline with the use of the word delivery in the distributed systems literature but seems to be a point of much confusion.
Next he addresses the need for transactions or other abstractions allowing atomicity in processing beyond just idempotence. This is a very important but very subtle point and is the motivation for the feature in Kafka.
Finally he points out that “exactly once” need not require consensus or be equivalent to full atomic broadcast depending on how you define it. This is a very good point and one I hadn’t thought of.
Informally atomic broadcast is the property that all messages that are sent are delivered, in order, to all receiving processes. In a system like Kafka I think the informal phrase “exactly once” is very much equivalent to atomic broadcast. However Flavio makes a good point that if you remove properties that Kafka already has such as supporting a total order this could be weakened. So with Flavio’s weaker definition consensus is sufficient but not necessary.
Obviously if you accept Flavio’s argument for a weaker definition of exactly once, that can only strengthen the argument against its impossibility.
Tyler Treat: You Cannot Have Exactly-Once Delivery Redux
Tyler discusses the possibility of exactly-once delivery. Tyler has written before about this topic. He’s a smart guy and a great writer. I think much of this discussion hinges on the meaning of the word “delivery” which I think both Tyler and I would agree is ambiguous. I think Tyler and many other people tend to interpret a message as “delivered” if the bytes are sent over the network. Obviously we can’t ensure this happens only one time if we also want to ensure that all messages are delivered. However this is not the only interpretation. The distributed systems literature has typically used the word to be an application defined criteria, hence the appearance in many definitions of Atomic Broadcast. Flavio’s blog also had a good discussion of this.
Tyler prefers the term “Exactly Once Processing” which I agree is a better.
As Tyler points out the feature set we’ve added isn’t magic pixie dust. You have to use the abstractions Kafka provides (either the lower-level transactions or higher level processing apis) to get the guarantees. Tyler wonders how easy that will be. I think that is the real question: building simple, intuitive abstractions that people can successful use to implement correct applications is pretty much the core challenge in distributed systems design.
We think the factoring of the problem into two pieces makes this far far easier:
- Processing using the streams api
- Connectors using the connect api
This makes guarantees for processing quite transparent (you just set a config to enable it). The only real restriction is that you must maintain state using the provided facilities.
Connectors require more thought about the interaction between Kafka and the external system but these can be made reusable. So if one person has implemented a Kafka=>Mysql connector and tested it everyone else can just use that.
Anyhow, I appreciate the intelligent critiques, and look forward to deeper feedback as people start to use the new capabilities to build applications.
