Setting up Kafka Mirror Maker for Kafka Topic Replication

Apache Kafka Mirror Maker 2 assists to replicate topics across Kafka clusters

Image Courtesy: Unsplash by Jovis Aloor

In this article, we’ll demonstrate how to use Apache Mirror Maker to replicate Kafka Topics across two Kafka clusters.

Set Up

For demonstration, we’ll have the following set up:

1. We’ll set up two Kafka clusters in two separate Kubernetes clusters (with AKS). You can refer to this article to set up the Kafka cluster in AKS. If you don’t want to use AKS/Kubernetes, you can also use local Kafka clusters
2. We’ll use the Kafka built-in kafka-mirror-maker.sh script to run Mirror Maker. However, in this article, we’ll run the script as a Kubernetes deployment

Configuring Mirror Maker

To start with, let us begin with the following folder structure:

First, let us configure the connect-mirror-maker.properties file. This file contains the details about the Kafka bootstrap servers, replication flows between clusters, and replication factors. The following shows the sample…