DATA ENGINEER ROADMAP · STAGE 03 OF 04
Pipelines and orchestration
Coordinate data work across dependencies, schedules, changing inputs, and late-arriving records.
Design a pipeline whose tasks can be monitored, retried, and safely run more than once.
CORE SKILLS
Key concepts in pipelines and orchestration
Batch and streaming patterns
Select processing patterns based on freshness needs, event volume, and operational complexity.
Topics include
- Scheduled batch processing
- Event streams and consumers
- Latency and completeness trade-offs
Orchestration and dependencies
Represent pipeline steps and dependencies so execution order, failures, and reruns are visible.
Topics include
- DAGs and task boundaries
- Scheduling and backfills
- Retries and timeout policies
Incremental processing
Process new or changed records without rebuilding every historical result on each run.
Topics include
- Watermarks and checkpoints
- Late and out-of-order data
- Deduplication and merge strategies
Idempotency and recovery
Make retries safe by ensuring that repeating an operation does not create unintended duplicate effects.
Topics include
- Idempotent writes
- Atomic outputs
- Replay and recovery plans
APPLY WHAT YOU LEARN
Practical project
Design an orders pipeline that handles updates and late-arriving events, with a safe backfill plan and clear retry behavior.
CHECK YOUR UNDERSTANDING
You are ready to move on when
- Can reason about duplicate delivery and partial failure.
- Can explain how to recover and validate the rebuilt output.