DATA ENGINEERING · QUESTION 93
What is Change Data Capture, and how is it used in data engineering?
On this page
Change Data Capture (CDC) identifies inserts, updates, and deletes in a source system and publishes those changes for downstream processing. It can reduce the need for repeated full-table scans and support fresher replicas or analytics. A robust CDC design tracks ordering, offsets, schema changes, deletes, and replay so consumers can recover without losing or applying changes incorrectly.
LET'S MAKE IT SIMPLE
First, the idea in plain English
Instead of copying an entire database table every time, CDC sends the changes that occurred since the last successful read. A warehouse or search index can then update only affected records. The consumer must still know which change came first and how to recover if it misses part of the stream.
A USEFUL COMPARISON
Picture it this way
A newspaper can print the entire directory each day, or issue a change sheet listing new, updated, and removed entries. CDC is the change sheet, but the reader needs a reliable page number and ordering to apply it correctly.
FOLLOW THE JOURNEY
Capture source changes and apply them with ordered checkpoints
Trace database changes through a log or polling connector to a downstream target.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Recovery prevents gaps after failures.
Source inserts, updates, and deletes flow through a CDC connector with ordered offsets to a consumer that applies changes and checkpoints progress.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Recovery prevents gaps after failures.
UNDER THE HOOD
How it works, step by step
Capture changes
Read transaction logs or poll a source using a defined cursor.
Why it matters: The method affects freshness and source-system load.
Preserve event metadata
Record key, operation, ordering position, and schema version.
Why it matters: Consumers need enough context to apply updates safely.
Apply downstream
Upsert or delete target rows idempotently.
Why it matters: Retries and replays should not corrupt the target.
Checkpoint and recover
Persist offsets and support replay from a known position.
Why it matters: Recovery prevents gaps after failures.
OUTSIDE THE TEXTBOOK
A practical example
A product catalog changes in an operational database. A CDC connector reads committed changes and publishes keyed events; a search index consumer applies updates and deletes. If the consumer restarts, it resumes from its last checkpoint and can replay events safely.
GO A LITTLE DEEPER
Technical details that matter
- Log-based CDC often captures committed transactions with lower query overhead than polling, but depends on source-log retention and connector behavior.
- Initial snapshots need coordination with ongoing changes to avoid gaps or duplicates.
- Schema evolution and transaction ordering across tables require explicit consumer rules.
MAKE A GOOD CHOICE
When is this useful?
- Near-real-time warehouse ingestion.
- Keeping search indexes or caches synchronized with source records.
- Replicating operational changes without repeated full scans.
Scheduled incremental queries can be simpler when freshness requirements are loose and the source has reliable update timestamps.
WATCH OUT FOR
Common misunderstandings
- Assuming CDC events arrive exactly once and in business order.
- Ignoring delete events and tombstones.
- Failing to coordinate the initial snapshot with the live change stream.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- CDC depends on source log access and retention.
- Schema changes can break consumers.
- Replay and duplicate handling remain necessary.
WHAT TO REMEMBER
Key takeaways
- 1CDC propagates row-level changes rather than repeatedly copying all data.
- 2Track keys, operation, ordering, schema, and offsets.
- 3Design for snapshots, replay, deletes, and idempotent consumers.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
How do you handle duplicate CDC events?
Use stable source keys and offsets, then make target application idempotent so replay does not duplicate logical state.
How do you combine a snapshot with CDC?
Coordinate a consistent snapshot boundary with log position, then apply subsequent changes from that position and validate completeness.
SAY IT OUT LOUD
How to explain this in an interview
“CDC captures inserts, updates, and deletes from a source and sends them downstream, often using transaction logs. It supports fresher data without full-table scans. I would preserve keys, ordering metadata, schema versions, and offsets, then make consumers idempotent and replayable. Initial snapshots, deletes, and schema changes need explicit handling.”
Connect this question to the roadmap topic it supports.