How Debezium & Kafka Streams Can Help You Write CDC Solution

I had worked on multiple applications for enterprises where legacy applications have been migrated to the modern technology stack. Starting this year I am working on one such application.

Way to design such an application is hard because one cannot simply take the entire big complex application and migrated it to a new one, it’s always better to do it part by part (baby steps). The new application is divided into two parts- read-side and write-side. Considering a part to migrate, the read side of the application is being migrated at first. The consumers will read the data from the new application however, all the write-side operations will still be performed using the legacy application. However, once we have received the certainty that the new application is working as anticipated, the other side of the legacy application will migrate to the new technology stack.

With such decisions in mind, it becomes extremely important for the new application to sync the data from the old one and provide the exact results back to the consumer. For this innovative platform, a technology shift in regards to the database is being made though, the older application works with Oracle Database. Now having a CDC (change data capture) solution that puts all the changes happening inside the old database into this new database platform becomes crucial. What could be a better way than to use Kafka as a messaging platform and process it using Kafka Streams?