The Goal
A simple google search on ‘kafka’ brings up good old Franz’ Wiki only on third place. Why is that and what does that mean in the large scheme of things? Who is this Apache (Kafka) guy who has surpassed Franz according to the modern measuring stick of relevance ranking?
I can’t leave my house
Or answer the phone
I’m going down again
But I’m not alone
Most people outside the IT industry have no clue that technologies like Apache Kafka underlie and allow for the accelerating acceleration of change that technology fuels. Most people inside the IT industry are somewhat aware of Apache Kafka having such an impact. The latter crowd consists of two groups; a group of outsiders who mostly know Kafka by name and blog posts, in other words from a bird-eye perspective — and discuss and judge it accordingly, and a group of insiders who spend most of their waking hours thinking, coding, trouble-shooting, sweating, smiling, and sometimes crying because of it… Some find time to write a blog post.
Settling at last
Accounts of the soul
This for the trash
That paid in full
The benefits and (competitive) advantages that Apache Kafka as a streaming platform offers can hardly be over-stated. But, as with all (or most, and did I miss something?) good things in life, large benefits do not come cheap. There is no free lunch some people say. Actually, lunch comes at a rather high price — meaning a lot of effort in studying, coding and sleepless nights sometimes followed by aha moments: a steep — and definitely quite long — learning curve.
At times Apache Kafka may puzzle a developer more than ‘die Verwandlung’ — oddly translated as ‘The Metamorphosis’ — does to a student of literature trying to penetrate the deeper layers of meaning attributed to the transformation of Gregor Samsa. But with the prospect of streaming billions of messages a day, what would you expect?
As for the fall, it
Began long ago
Can’t stop the rain
Can’t stop the snow
If you are still part of that unhappy group of outsiders and want to enter the group of enthusiastic insiders, you are part of the audience we are trying to target.
The we-form in which this series will be written is more similar to what Fernando Pessoa called ‘heteronyms’ than that we are dealing with an author with a multiple personality disorder (colloquially often confused with Schizophrenia). Nevertheless, ‘our’ mind is a non-linear one, and this clearly shines through in both the verbose writing and verbal communication. There is — at this moment in time — only one physical entity writing these words.
I sit in my chair
I look at the street
The neighbor returns
My smile of defeat
Using Kafka requires a different way of thinking, a radically different perspective on software architectures. You will need to transform yourself into a true IT-polymath, combining skills and expertise from a large number of technical fields. With such a vast number of new concepts (with corresponding terminology) as brokers, topics, partitions, producers, consumers, consumer groups, offsets, ordering guarantees, replicas, leaders, Avro, tombstone messages, (K)streams, (K)tables, and fancy sounding configuration settings like min.insync.replicas, cleanup.policy, max.compaction.lag.ms, unclean.leader.election.enable, where to start?
Start where you are.
I move with the leaves
I shine with the chrome
I’m almost alive
I’m almost at home
One of the goals of this series is to give you both a starting point and a complete and realistic walkthrough on what it takes to develop and host a stream-native service architecture in ‘the cloud’. And it has to be said, it is just not an easy thing to do. Rewards are plenty, even for the unfortunate ones who cherish the artificial honor of financial compensation.
Most existing blog posts — didactic and entertaining as they sometimes might be— often discuss narrow and isolated issues without an over-arching narrative. To give you a more complete picture, we will discuss both Kafka’s internals and its stack, and — to approximate the real world — the complete deployment of a containerized micro-service architecture in AWS.
As you might have noticed this series of blog posts is also a tribute to dear Leonard Cohen. It is therefore drenched with quotes and lyrics. In times when people rather run than meditate, his wise words and calm and soothing voice will hopefully have a beneficial effect on most of the readers. The beautiful words are his and the bad jokes are mine.
No one to follow
And nothing to teach
Except that the goal
Falls short of the reach
We need to cover a lot of technical ground the coming months. Just hold tight, be forgiving to the shortcomings of our didactic skills and graph drawing faculties, be inspired and most of all enjoy the ride.
‘Talk is cheap, show me the code’. Let’s cut the crap and get started in the next post.
