How to connect Kafka to MongoDB Source
This is a story about how I connected to a MongoDB database in my local through Kafka using confluent.
For the uninitiated, the cloud and Big Data is a bewildering place. There are so many tools available nowadays, there seems to be always an alternate route to whatever your plan is. Especially in the corporate world.
To make sense out of the cloud, I attended a boot camp in my office where we learned the basics of Kafka, Spark, Alluxio and Presto, to name a few. As a part of the bootcamp, we were required to create a kafka connector for the mongodb database. The end goal was, whenever there would be any change in the database, the same would be written into Kafka topic and would be visible to any Kafka consumer, listening at that topic.
I had to refer multiple websites and YouTube videos, before I could finally get the whole thing up and running in the wee hours of the morning. So, in order to save you the trouble, here is a handy, all in one place guide to get there.
Setting up MongoDB
Basically what the guys at Kafka — MongoDB connector are trying to tell us is that, if you have a single…
