Streaming Data out of the Monolith: Building a Highly Reliable CDC Stack

Yuancheng Peng
Jun 7, 2018 · 8 min read

Introduction

CDC and Debezium

A simple architecture of a CDC stack with debezium MySQL connector

High Availability

Deduplication architecture

CDC stack with deduplicator for high availability

Deduplicator key-value store

Exactly-once

Example code of “exactly once” with kafka transaction
CDC stack with 2 debezium connectors. Duplicated messages are filtered out by the deduplicator and the overall service behaves as only one single CDC stream.

Monitoring

Performance

Operations

Conclusion

BlaBlaCar Tech

The Tech team of carpooling unicorn BlaBlaCar

Thanks to Maxime Fouilleul, Thomas Lamirault, Adrien Loison, Clément Vasseur, Nicola-Marie O'Donovan, and Olivier Bonnet.

Yuancheng Peng

Written by

Software Engineer at BlaBlaCar

BlaBlaCar Tech

The Tech team of carpooling unicorn BlaBlaCar