Apache Kafka in a Nutshell

Architecture, Use Cases, and a Getting Started guide — rolled into one

So, you’ve heard of this Kafka thing that’s popping up all over the shop. From green-sprout startups to greedy multi-nationals, in all sorts of weird and wonderful contexts. It appears that someone has figured out a Swiss Army knife piece of über-tech that works for microservices, event-streaming, CQRS, and just about anything else that makes the headlines.

As someone who’s been working with Apache Kafka since 2015, helping several clients build complex event-driven and microservices-style applications, I’ve heard my fair share of questions. First, it was I who was doing most of the asking. Then I found myself on the receiving end. At the time of writing, I’m trending among the top-ten writers in Quora — a feat I owe mostly to the mistakes I’ve made along the way, which have made me a little more aware. With the shameless chest-beating out of the way, I thought I’d put together an FAQ-style post that breaks Kafka down, focusing on the aspects that matter.

What is Kafka?

Apache Kafka is an open-source event streaming platform that was incubated out of LinkedIn, circa 2011. Originally used to pump (lots of) messages through LinkedIn’s veins, it had since been released into the wild. And here we are.