DATA ENGINEERING · QUESTION 27
How do you handle late and duplicate events in a data pipeline?
On this page
I define event identity and event time, then make processing idempotent so retries do not create duplicate effects. For late events, I set an allowed lateness or correction policy and update affected aggregates or partitions when appropriate. The right design depends on whether outputs are append-only or revisable, how long corrections remain possible, and how consumers learn that data changed.
LET'S MAKE IT SIMPLE
First, the idea in plain English
Events do not always arrive once, in order, or immediately. A network retry can send the same record again, and an offline device may upload an old event later. If a pipeline assumes a perfect stream, counts and totals can become wrong. Reliable processing needs a way to recognize repeated events and a rule for how old information changes results that were already published.
A USEFUL COMPARISON
Picture it this way
Imagine a school recording attendance slips. A photocopied slip should not count the same student twice, and a late slip may require correcting yesterday’s record. The office needs an identity for each attendance event and a policy for reopening a completed day. Data pipelines face similar corrections at much larger scale.
FOLLOW THE JOURNEY
Deduplicate by identity and revise outputs by policy
Trace duplicate detection and late-event handling through a reliable pipeline.
A stable event key and durable deduplication state are required across retries. A time window alone may not catch duplicates that arrive much later.
Incoming events are validated and keyed by stable event ID; duplicates are ignored or merged, on-time records update results, and late records follow a correction or rejection policy.
How to read this visual
A stable event key and durable deduplication state are required across retries. A time window alone may not catch duplicates that arrive much later.
UNDER THE HOOD
How it works, step by step
Define event identity
Choose a stable source event ID or a carefully specified business key.
Why it matters: The pipeline needs to recognize the same logical event across retries.
Make writes idempotent
Use upserts, deduplication state, or transactional processing so replays have safe effects.
Why it matters: Retries are normal in distributed systems and should not double-count.
Set a lateness policy
Define event-time windows, correction periods, and behavior after the cutoff.
Why it matters: Consumers need to know whether past results can still change.
Expose and audit corrections
Track changed partitions or versions and communicate updates to downstream consumers.
Why it matters: Correcting a table is not enough if cached or aggregated outputs remain stale.
OUTSIDE THE TEXTBOOK
A practical example
A delivery event may be resent after a network timeout and arrive again when a device reconnects. The pipeline uses a stable event ID to avoid counting the same scan twice. If a scan arrives after the daily delivery summary was published, the system updates the affected order and partition within its correction window and records that the summary changed.
GO A LITTLE DEEPER
Technical details that matter
- Exactly-once processing is usually an end-to-end property built from idempotency, transactional boundaries, and source-offset management.
- Event time and watermarks help streaming processors decide when a window is sufficiently complete, but later events may still require corrections or be dropped.
- Deduplication state needs a retention period; keeping it forever costs storage, while expiring it too early allows old duplicates through.
- Upserts and slowly changing dimensions support correction, but downstream aggregates and caches must also be refreshed consistently.
MAKE A GOOD CHOICE
When is this useful?
- Event ingestion from unreliable networks or at-least-once message systems.
- Financial, order, and telemetry pipelines where duplicate counts matter.
- Streaming analytics where late records can revise recent windows.
If sources provide strong ordering and unique IDs, upstream guarantees can reduce downstream work, but consumers should still define retry behavior. For immutable audit logs, retain events and build corrected views rather than deleting history.
WATCH OUT FOR
Common misunderstandings
- Assuming the broker or orchestrator guarantees no duplicate side effects.
- Deduplicating only by timestamp when separate events can share a timestamp.
- Discarding late events without assessing business impact or making the policy visible.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Deduplication requires stable identity and retained state.
- Late corrections can invalidate downstream reports or aggregates.
- A finite lateness window trades completeness for bounded operational state and finalization delay.
WHAT TO REMEMBER
Key takeaways
- 1Expect duplicates and retries in distributed pipelines.
- 2Use a stable event identity and idempotent writes.
- 3Define event-time lateness and correction rules explicitly.
- 4Propagate corrections to downstream consumers and keep an audit trail.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
What does idempotent processing mean?
Processing the same logical event repeatedly has the same final effect as processing it once, often through stable keys and upsert or transactional behavior.
How do you choose a deduplication key?
Prefer a stable source event ID. A composite business key can work only if its uniqueness and reuse semantics are clearly defined.
What is a watermark?
A watermark is an estimate of event-time progress that helps a streaming processor decide when to close a window. Late data after that point needs an explicit policy.
SAY IT OUT LOUD
How to explain this in an interview
“I assume events can be retried, duplicated, and delayed. I use a stable event identity and make writes idempotent so a replay does not double-count. For late events, I define an event-time correction window and decide whether to update, quarantine, or reject records outside it. I would also track revisions and ensure downstream aggregates know when published results have changed.”
Connect this question to the roadmap topic it supports.