Anatomy of an Event Streaming Platform — Part 2

This post in the series discusses the fundamental concepts and the architecture of event storage layer of an event streaming platform

Photo by JOSHUA COLEMAN on Unsplash

Responsibilities of the storage layer

The grouping of events

Event streams and topics in Kafka

Partitions

Brokers and clusters

Partitions/shards

This example Kafka topic has four partitions P1–P4. Two different producer clients are publishing, independently from each other, new events to the topic by writing events over the network to the topic’s partitions. Source
Partitions in an Azure Event Hub
New events are appended to the end of the partition

Why partitioning?

Offsets

Data retention policy

Where to next?

References

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Dunith Dhanushka

Editor of Event-driven Utopia(eventdrivenutopia.com). Technologist, Writer, Senior Developer Advocate at Redpanda. Event-driven Architecture, DataInMotion