Setup Kafka with Debezium using Strimzi in Kubernetes

Sincy Sebastian
May 30, 2019 · 5 min read

Kubernetes automates the distribution and scheduling of application containers across a cluster in an efficient way. Strimzi gives an easy way to run Apache Kafka on Kubernetes or Openshift and Debezium provides custom Kafka connectors.

This blog will demonstrate the steps to deploy Debezium in Kubernetes using Strimzi. We will be using the example from Debezium (https://github.com/debezium/debezium-examples/tree/master/unwrap-smt). The scenario we are going to use is to receive events from MySQL database(source) using Debezium connector and save them in Elastic search(sink), using Strimzi in Kubernetes development or test environment.

Image for post
Image for post
Flow Diagram

Prerequisites:

Note: Replace the text styled in italics with your corresponding values.

Step1: Strimzi artifacts

Download the artifacts from strimzi repository ( https://github.com/strimzi/strimzi-kafka-operator/releases) and extract the folder — strimzi-0.11.4.zip.

Image for post
Image for post

Step2: Deploy Strimzi Cluster operator

1)Replace the files with the your namespace: I used default.

2)And apply the operator

Image for post
Image for post

Step3: Deploy Kafka cluster operator

This will deploys Kafka and Zookeeper with necessary configurations.We will be deploying a non-persistent cluster which is for development and testing and not for production.

And check the pods and services using kubectl command.

Image for post
Image for post

Step4: Test the Kafka set up

Producer : Use the producer utility in the Kafka pod , and send message to the bootstrap service (here it is my-cluster-kafka-bootstrap) using the below command and enter messages.

Consumer: Use the consumer in the Kafka pod to receive messages. In another terminal execute the below command and see the messages

To find out the topics in the Kafka cluster use the below command.(Zookeeper is accessible through localhost:2181 and my-cluster-zookeeper-client is not accessible) from outside.

Step5: Deploy Kafka Connect with custom connectors

5.1) Add the required connectors to Strimzi Kafka connect image.In our case, add Debezium MySQL connector and confluent Elastic search connector to Strimzi Kafka Connect image.

  • Create a DockerFile with the below content and save it .

And place the connector plugins in your local machine.

Image for post
Image for post
Folder structure

5.2)Build the docker file

Inspect the docker image plugin directory and check the added connector plugins:

5.3)Tag and Push to Docker hub

Create an account in Docker Hub if you don’t have. Now in command line execute the below commands to log in and push the new image.

Image for post
Image for post

Tag and Push the image

Refer the docker tutorials for more details on build and push.

5.2)Modify and Deploy KafkaConnect in the cluster

Take the yaml file from Strimzi example folder (examples/kafka-connect/kafka-connect.yaml) and add your new image .

and apply

Check the Pods and Services

Execute the below command to list the available connector plugins

Step 6: Create a MySQL test DB and expose as a service

Step7: Start the Connectors

Take file source.json from the Debezium repository

https://raw.githubusercontent.com/debezium/debezium-examples/master/unwrap-smt/source.json

and modify the field value database.history.kafka.bootstrap.servers.

Take es-sink.json from https://github.com/debezium/debezium-examples/blob/master/unwrap-smt/es-sink.json and modify the elastic search end point with yours.

Now execute the below commands

  • View the connectors
  • View all the topics
  • View the messages from mysql connector in Kafka the topic “customers”
  • View the records in Elastic Search

Step8: Modify MySQL records and validate in Elastic search

  • Login to MySQL terminal and run
  • And see the changes are reflected in Elastic search

References

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch

Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore

Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store