DATA ENGINEERING · QUESTION 10
What is a data pipeline, and how would you design a reliable one?
On this page
A data pipeline moves data from sources through ingestion and transformation to storage or downstream consumers. A reliable design defines data contracts, validates inputs, makes retries safe through idempotent writes, handles duplicates and late records, and tracks each run. Orchestration manages dependencies; monitoring and alerting expose failures and quality issues. Recovery and backfills should be designed rather than improvised after an outage.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A pipeline is a repeatable path that carries data from where it is produced to where it is used. It may run on a schedule or respond to new events. Reliability means more than completing a task once: it should detect bad inputs, recover from temporary failures, avoid corrupting results when rerun, and tell operators what happened. Users should also know what data is fresh and what guarantees it has.
A USEFUL COMPARISON
Picture it this way
Think of a delivery route with checkpoints: parcels are received, checked, sorted, and delivered, while exceptions are recorded for retry. A data pipeline has similar stages, but replaying a delivery can create duplicates unless each write is designed to be safe.
FOLLOW THE JOURNEY
A pipeline needs quality checks and a recovery path
Show ingestion, validation, transformation, storage, orchestration, and operational feedback.
Orchestration and monitoring span the pipeline rather than representing one mandatory product or a single processing step.
Sources flow through ingestion and validation to transformations and storage, while orchestration tracks dependencies and monitoring reports failures and quality.
How to read this visual
Orchestration and monitoring span the pipeline rather than representing one mandatory product or a single processing step.
UNDER THE HOOD
How it works, step by step
Define the contract
Specify schema, freshness, ownership, and expected quality.
Why it matters: Consumers need a clear expectation and changes need a safe path.
Ingest and validate
Capture source data, record offsets or watermarks, and reject or quarantine invalid records.
Why it matters: Bad inputs should not silently contaminate downstream results.
Transform idempotently
Make rerunning a batch or event safe, using keys, merges, or deduplication.
Why it matters: Retries are inevitable; repeating work should not duplicate outputs.
Orchestrate and observe
Track dependencies, run state, freshness, failures, and quality metrics.
Why it matters: Operators must know what succeeded and what needs intervention.
Recover deliberately
Retry transient failures and support controlled backfills for corrected data.
Why it matters: Recovery should preserve correctness and explain any reprocessing.
OUTSIDE THE TEXTBOOK
A practical example
A nightly orders pipeline reads changes since a recorded watermark, validates required fields, deduplicates by order ID and update time, and merges records into a warehouse table. If the job fails halfway, it can rerun safely. A late correction can be backfilled, and a freshness alert tells analysts when the table is behind.
GO A LITTLE DEEPER
Technical details that matter
- Idempotency means repeating an operation has the same intended effect as running it once; it is often implemented with stable keys and upserts.
- Event-time processing uses when an event occurred, while processing-time uses when the system handled it. Late arrivals require a policy for revising prior results.
- Retries should distinguish transient errors from invalid data and use bounded backoff to avoid hammering dependencies.
- Data quality checks should cover schema, completeness, uniqueness, freshness, and domain-specific rules.
MAKE A GOOD CHOICE
When is this useful?
- Scheduled ingestion from operational systems into analytics storage.
- Streaming or batch processing where failures and replay are expected.
- Shared datasets that need freshness, quality, and ownership guarantees.
For a one-off migration, a full orchestration platform may be unnecessary. Preserve repeatability and validation even if the implementation is a small script.
WATCH OUT FOR
Common misunderstandings
- Blindly retrying a non-idempotent job and creating duplicates.
- Dropping late data without documenting the policy.
- Alerting only on task failure while stale or malformed data continues downstream.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Exactly-once effects across distributed systems are difficult; designs commonly combine at-least-once delivery with deduplication.
- Backfills can consume substantial compute and affect downstream consumers.
- Quality rules can reject legitimate new data if change management is weak.
WHAT TO REMEMBER
Key takeaways
- 1Treat schemas, freshness, and quality as part of the pipeline contract.
- 2Make retries and reprocessing safe.
- 3Track runs and data outcomes, not only process status.
- 4Plan for duplicates, late data, and controlled backfills.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
What does idempotent mean for a pipeline?
Repeating a run produces the same intended result rather than duplicate side effects, often through stable keys and merge/upsert operations.
How do you handle late-arriving events?
Define event-time windows and a correction policy, retain enough state for updates, and make downstream consumers aware of revised results.
What should an alert report?
Identify the affected dataset or run, the failure or quality signal, the freshness impact, and a link or instruction for safe recovery.
SAY IT OUT LOUD
How to explain this in an interview
“A data pipeline moves data through ingestion, validation, transformation, and delivery to consumers. I would define the schema and freshness contract, make writes idempotent so retries are safe, and handle duplicates and late records explicitly. Orchestration tracks dependencies, while monitoring checks both run health and data quality. I would also design backfills and recovery paths before the pipeline becomes business-critical.”
Connect this question to the roadmap topic it supports.