MirrorMaker2 and topics configuration
During migration to the new mirror maker 2 I’ve noticed, that topics used 10 times more space, then on current mirror maker (mm1).
The reason was that retention.bytes value was set to default 10485760000. I’ve changed it to the same value as in mm1: 107374182.
In some time I’ve noticed that the retention.bytes value was set to default value again. So, It seems someone alters it.
Googling + ChatGpt gave no results.
I started reading Kafka Connect logs, and noticed this:
[2023–07–13 10:11:25,339] INFO syncing topic configs took 87 ms (org.apache.kafka.connect.mirror.Scheduler)on one of the nodes.
I made an experiment:
docker exec -it kafka /opt/bitnami/kafka/bin/kafka-configs.sh --bootstrap-server k1:9092,k2:9092,k3:9092 --entity-type topics --entity-name logs --alter --add-config retention.bytes=107374182
docker exec -it kafka /opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server k1:9092,k2:9092,k3:9092 --describe --topic logs | grep retand noticed, that retention.bytes is altered in the same time as that log appears.
Google said that this scheduler by default syncs topic settings. retention.bytes was never set on the source cluster, so scheduler just removed new value.
Disabling the sync service in mm2-msc.json fixed the problem:
...
"sync.topic.acls.enabled": false,
"sync.topic.configs.enabled": false,
...