Kafka the afterthoughts: message encoding and schema management

Apache Kafka is a distributed streaming platform originally developed at Linkedin and later got open-sourced. It’s designed to handle high-throughput of data feed and generally used for a broad spectrum of applications that could be grouped in 2 classes:

• Real-time data pipeline for asynchronous communication between multiple services
• Real-time streaming applications that perform on the fly data transformations

This article is the start of a blog post series where I share my notes about working with Apache Kafka. In this first part, I will not dive in and explore all Kafka capabilities and how it works. I will rather share notes and thoughts, from my journey, about data management and usage of Apache Avro with Apache Kafka.

Topics Include:

• Why schema management is challenging?
• Why choose Apache Avro for encoding your data?
• How does Avro play along with Kafka?

Target Audience:

• You have a basic knowledge of Kafka
• Looking for a better understanding of your technical choices
• Looking for quick tips to start right with Kafka