Kafka the afterthoughts: message encoding and schema management
Published in
9 min readJan 24, 2020
Apache Kafka is a distributed streaming platform originally developed at Linkedin and later got open-sourced. It’s designed to handle high-throughput of data feed and generally used for a broad spectrum of applications that could be grouped in 2 classes:
- Real-time data pipeline for asynchronous communication between multiple services
- Real-time streaming applications that perform on the fly data transformations
This article is the start of a blog post series where I share my notes about working with Apache Kafka. In this first part, I will not dive in and explore all Kafka capabilities and how it works. I will rather share notes and thoughts, from my journey, about data management and usage of Apache Avro with Apache Kafka.
Topics Include:
- Why schema management is challenging?
- Why choose Apache Avro for encoding your data?
- How does Avro play along with Kafka?
Target Audience:
- You have a basic knowledge of Kafka
- Looking for a better understanding of your technical choices
- Looking for quick tips to start right with Kafka