MLOPS & DEPLOYMENT · QUESTION 20
What should you monitor for a model in production?
On this page
Monitor the prediction service and the model separately. Service signals include availability, latency, errors, and resource use; model signals include input quality, distribution changes, prediction behavior, and task performance when labels arrive. Define baselines and alert thresholds around decisions the team can take. A drift alert is a reason to investigate, not automatic proof that the model needs retraining.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A model can be online and still be making poor predictions. It can also be accurate in principle while the service is too slow or broken for users. Production monitoring therefore checks whether the application works and whether its predictions remain useful. Some quality measures need verified outcomes, which may arrive late; teams should distinguish those from quicker but indirect warning signals.
A USEFUL COMPARISON
Picture it this way
A delivery company tracks both whether its vans are running and whether packages reach the right homes on time. Engine health alone does not measure delivery success. Likewise, model uptime is not model quality, and a change in incoming packages does not automatically mean deliveries are failing.
FOLLOW THE JOURNEY
Separate service health, input signals, and outcomes
Show three monitoring layers and how they feed investigation rather than automatic retraining.
Input shifts and prediction changes are leading signals; actual task outcomes provide stronger performance evidence when labels are trustworthy and available.
Production requests flow through a model service; service metrics, input and prediction signals, and delayed outcome metrics feed dashboards and an investigation workflow.
How to read this visual
Input shifts and prediction changes are leading signals; actual task outcomes provide stronger performance evidence when labels are trustworthy and available.
UNDER THE HOOD
How it works, step by step
Define what can fail
List service, data, model, and downstream decision failures that matter.
Why it matters: Monitoring should connect to an operational response.
Instrument the request path
Record suitable latency, errors, input checks, model version, and prediction metadata.
Why it matters: Without context, teams cannot diagnose which component changed.
Compare to a baseline
Track distributions, slices, and outcomes against meaningful reference periods.
Why it matters: A change is only useful to interpret when the comparison is appropriate.
Investigate before acting
Check upstream changes, data quality, label delay, and user impact before retraining or rollback.
Why it matters: Not every alert indicates model failure, and automatic action can make it worse.
OUTSIDE THE TEXTBOOK
A practical example
A demand forecast service tracks API latency and error rate, the freshness and ranges of incoming sales data, prediction distributions by region, and forecast error once actual sales settle. A sudden input change triggers a check for an upstream schema update. The team only trains a candidate if the investigation shows a real quality problem and can validate the result.
GO A LITTLE DEEPER
Technical details that matter
- Data drift measures changes in inputs; concept or performance drift concerns whether the relationship to outcomes or task quality has changed.
- Slice metrics can uncover failures hidden by an aggregate score, but slices need adequate sample sizes and privacy-aware handling.
- Alert thresholds should account for seasonality, baseline variance, delayed labels, and the cost of missed or noisy alerts.
- Logging sensitive features or raw prompts may create privacy and retention risks; collect only what is needed and control access.
MAKE A GOOD CHOICE
When is this useful?
- Operating online prediction APIs or scheduled batch scoring jobs.
- Detecting upstream data failures before they silently affect decisions.
- Providing evidence for investigation, retraining, or rollback decisions.
A short-lived prototype may need only basic service logs and manual outcome review. As user impact and automation grow, add model-specific signals, alert ownership, and documented incident procedures.
WATCH OUT FOR
Common misunderstandings
- Using uptime as a proxy for model quality.
- Automatically retraining whenever an input distribution shifts.
- Collecting raw sensitive inputs without a clear need, retention policy, and access controls.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Ground-truth labels may be unavailable, delayed, or biased by the model’s own decisions.
- Statistical alerts can be noisy in small or seasonal datasets.
- Monitoring cannot prevent every failure; teams still need ownership and recovery procedures.
WHAT TO REMEMBER
Key takeaways
- 1Track service reliability and model behavior independently.
- 2Use input and prediction changes as signals, not proof of quality decline.
- 3Measure outcomes when trustworthy labels become available.
- 4Design alerts to support a clear, privacy-aware response.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
What is the difference between data drift and concept drift?
Data drift is a change in input distribution. Concept drift is a change in the relationship between inputs and outcomes; it can reduce predictive quality even when inputs look similar.
How do you monitor when labels are delayed?
Track data quality, service behavior, and carefully chosen leading indicators, then calculate task metrics when labels arrive. Treat early signals as investigation prompts.
Should drift automatically trigger retraining?
Usually not. First determine whether the change is real and harmful; any candidate model still needs evaluation and a controlled release.
SAY IT OUT LOUD
How to explain this in an interview
“I would monitor service health such as latency and errors, data quality and distributions, prediction behavior, and task performance once labels are available. These signals answer different questions: a live endpoint may still produce poor results, and input drift is not automatically a failure. I would set useful baselines, inspect affected segments, and make alerts trigger investigation rather than blind retraining.”
Connect this question to the roadmap topic it supports.