Apache Kafka Architecture

Apache Kafka is a powerful distributed stream processing platform originally developed by LinkedIn, written in Scala and Java. In this article, I walk through the foundational concepts, design and architecture, concluding with hands-on coding examples using Node.js and the KafkaJS library.

Introduction to Kafka

Kafka enables the processing of real-time data streams in a distributed and scalable manner. It’s widely used in systems requiring reliable communication between components, especially for event-driven architectures and microservices.

However, these words sounds very markety and I like to deconstruct things to its basic first principles so that is what I’m going to do. This guide outlines the essential components, explains the core concepts like brokers, producers, consumers, and topics, and dives into more complex abstractions like partitions, consumer groups, and Kafka’s distributed nature.

Kafka Core Components

Here we discuss the fundamental components of Kafka.

Topics and Messages

Messages in Kafka are organized into logical categories called topics. Each topic is append-only and immutable (ie you cannot go back and edit a message). New data is added sequentially at the end. Disks love that.