Kafka Connect, Postgres, and Amazon S3 — Part 1

A pipeline that moves data from a source to a sink can be created using Kafka Connect, Postgres (Source), and Amazon S3 (Sink).

Introduction

I am going to demonstrate how to use Kafka connect to build an E2E pipeline using Postgres as the source connector and S3 Bucket as the sink connector

Part 1(Current Page) : Producer Postgres records to Kafka

Part 2 : Consumer Kafka topic records to s3 bucket

Pipeline Architecture

• Stream Flow

What is Kafka Connect?

Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka® and other data systems. It makes it simple to quickly define connectors that move large data sets in and out of Kafka.

Kafka Connect can ingest entire databases or collect metrics from all your application servers into Kafka topics, making the data available for stream…