Kafka Connect

Streaming MySQL changes to Elasticsearch using Kafka Connect

How can we use Kafka Connect to stream changes in a MySQL database to an index in Elasticsearch

Photo by Jon Flobrant on Unsplash

In this article, we’ll explain the process of streaming changes from the “movies” table in MySQL (“moviesdb” database) to the “movies” index in Elasticsearch using Kafka Connect.

To establish this connection between MySQL, Kafka Connect, and Elasticsearch, we’ll make use of connectors. Specifically, we’ll utilize the Debezium Connector for MySQL to link MySQL with Kafka Connect. Subsequently, we’ll use the Elasticsearch Service Sink Connector to connect Kafka Connect with Elasticsearch.

Here is the project overview:

Let’s get started!

Prerequisites

If you would like to follow along, you must have Docker installed on your machine.

Create Project Folder

In a terminal, navigate to your workspace and create a folder called kafka-connect-mysql-to-elasticsearch. Inside, it…