MLOPS & DEPLOYMENT · QUESTION 67
How do you orchestrate a reliable machine-learning pipeline?
On this page
I represent an ML pipeline as explicit stages with dependencies, inputs, outputs, and acceptance criteria: ingest, validate, transform, train, evaluate, and publish. Each stage should be observable and safe to retry, with versioned artifacts and clear failure handling. Orchestration manages execution and scheduling; it does not replace data quality, idempotency, or decisions about model promotion.
LET'S MAKE IT SIMPLE
First, the idea in plain English
Training often involves several tasks that must happen in order. A pipeline orchestrator schedules them, passes outputs to later steps, retries transient failures, and shows where a run stopped. The pipeline still needs to define what makes data and models acceptable; a green orchestration run only means the tasks completed.
A USEFUL COMPARISON
Picture it this way
A film production has scheduled stages, dependencies, and deliverables: editing cannot finish before footage exists. A coordinator keeps the work moving, but does not decide whether the final film is good. An ML orchestrator coordinates computation while quality gates assess the result.
FOLLOW THE JOURNEY
Orchestrate stages with explicit dependencies and quality gates
Trace an ML run from data validation through model evaluation and artifact publication.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Successful orchestration is not the same as model readiness.
A directed pipeline connects ingestion, validation, feature preparation, training, evaluation, and publishing, with retries and failure branches for each stage.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Successful orchestration is not the same as model readiness.
UNDER THE HOOD
How it works, step by step
Define stage contracts
Specify inputs, outputs, dependencies, and resource needs.
Why it matters: Explicit contracts make retries and debugging safer.
Validate before expensive work
Check data and configuration before training begins.
Why it matters: Early gates avoid wasted computation.
Run and record stages
Execute tasks with traceable artifacts and bounded retries.
Why it matters: A run should be inspectable and recoverable.
Gate publication
Publish only candidates that pass model acceptance criteria.
Why it matters: Successful orchestration is not the same as model readiness.
OUTSIDE THE TEXTBOOK
A practical example
A scheduled forecast pipeline validates a new sales partition, builds point-in-time features, trains a candidate, and compares it with the current model. A transient storage timeout retries the affected task; a data-quality failure stops the run and alerts its owner. A model candidate is registered only after its evaluation gate passes.
GO A LITTLE DEEPER
Technical details that matter
- Idempotent tasks should produce the same logical output when retried with the same inputs.
- Use immutable artifact references between stages rather than “latest” paths.
- Separate orchestration retries from data backfills and model-promotion policies.
MAKE A GOOD CHOICE
When is this useful?
- Multi-step training, evaluation, and batch-scoring workflows.
- Scheduled or event-triggered jobs with dependencies and recovery requirements.
- Tracking lineage and failures across shared ML infrastructure.
A short one-step training script may not need a workflow engine. Use simpler scheduling until dependencies, observability, or retry needs justify orchestration.
WATCH OUT FOR
Common misunderstandings
- Retrying non-idempotent writes and duplicating outputs.
- Using mutable “latest” artifacts across stages.
- Automatically publishing any model after a successful training task.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Orchestrators add infrastructure, permissions, and versioning work.
- Retries can amplify load or hide persistent failures.
- Complex DAGs can become difficult to reason about without clear ownership.
WHAT TO REMEMBER
Key takeaways
- 1Represent dependencies and stage contracts explicitly.
- 2Make tasks observable and safe to retry.
- 3Keep quality and promotion gates separate from job completion.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
What makes a pipeline task idempotent?
Re-running it with the same logical inputs does not create duplicate or inconsistent effects, often by using stable output keys and transactional writes.
How do you handle a failed stage?
Classify transient versus permanent failures, bound retries, preserve run context, alert an owner, and resume only from safe versioned outputs.
SAY IT OUT LOUD
How to explain this in an interview
“I define explicit pipeline stages, dependencies, and artifact contracts from ingestion through evaluation. Tasks should be observable and safe to retry, with validation before expensive training. The orchestrator schedules and recovers work, but quality gates decide whether a candidate is acceptable. I would publish immutable artifacts and keep training success separate from production promotion.”
Connect this question to the roadmap topic it supports.