Distributed Systems for Backend Engineers · Distributed Systems in Practice
Change Data Capture
Stream database changes to other systems reliably.
CDC reads a database's replication log (PostgreSQL WAL logical decoding, MySQL binlog) and publishes row changes as events — e.g., Debezium to Kafka. Uses: keeping search indexes and caches in sync, feeding data warehouses, implementing the outbox pattern without polling, and migrating between databases. Consider schema evolution, ordering per key, snapshotting initial data, and the coupling of consumers to table structures (prefer outbox tables as the contract).
CDC use cases
| Use case | CDC source | Sink |
|---|---|---|
| Search index sync | products table | OpenSearch |
| Cache invalidation | orders table | Redis delete |
| Analytics | All tables | Warehouse (BigQuery/Snowflake) |
| Outbox relay | outbox table | Kafka topics per event type |
| Zero-downtime migration | Old DB | New DB |
# Debezium PostgreSQL connector (excerpt) with outbox event router
connector.class=io.debezium.connector.postgresql.PostgresConnector
plugin.name=pgoutput
table.include.list=public.outbox
transforms=outbox
transforms.outbox.type=io.debezium.transforms.outbox.EventRouter
transforms.outbox.route.by.field=aggregate_type
Common pitfalls
- Consumers coupled to internal table schemas.
- Replication slots growing unbounded when connectors stop (disk full).
- No initial snapshot strategy.
Key points
- Read the replication log
- Prefer outbox tables as contracts
- Monitor replication slots
Further reading: Debezium documentation
Practice questions
Why can a stopped CDC connector fill up a PostgreSQL disk?
Answer
The replication slot retains WAL until the consumer confirms it; if the connector stops, WAL accumulates indefinitely.
Why use an outbox table instead of CDC on business tables for integration events?
Answer
The outbox is an explicit, stable event contract; CDC on business tables couples consumers to internal schemas that change with refactoring.
Open in the app to mark lessons complete, take notes and track your progress.