MLOPS & DEPLOYMENT · QUESTION 74
How do you explain and monitor model predictions in production?
On this page
The explanation method should match the question: global analysis describes overall model behavior, while local methods describe factors associated with one prediction. I verify explanations for stability and faithfulness, document limitations, and avoid presenting correlations as causes. In production, capture the model version and relevant inputs needed for review while applying privacy and retention controls.
LET'S MAKE IT SIMPLE
First, the idea in plain English
People may need to understand why a model produced a result, whether they are debugging an error or reviewing a consequential decision. Some methods describe which inputs influenced a prediction, but those explanations are approximations and do not necessarily reveal a causal reason. The intended audience and decision determine what explanation is useful.
A USEFUL COMPARISON
Picture it this way
A doctor may explain which test results contributed to a risk assessment, while also noting that a test association is not the same as a cause. Model explanations similarly help inspect behavior but should not be presented as complete access to the model’s internal reasoning.
FOLLOW THE JOURNEY
Match explanation scope to the decision and validate its limits
Compare global and local explanations and place review in a monitored production flow.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Production investigation needs context and governance.
A versioned model produces a prediction and local explanation; aggregate analysis describes global behavior, while a reviewer checks stability, faithfulness, and privacy.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Production investigation needs context and governance.
UNDER THE HOOD
How it works, step by step
Define explanation need
Clarify who needs an explanation and what decision it supports.
Why it matters: Different audiences need different levels of detail.
Choose scope
Use global analysis for overall patterns or local analysis for a case.
Why it matters: Explanation methods answer different questions.
Validate method
Test stability, faithfulness, and sensitivity to input changes.
Why it matters: A plausible explanation can still be misleading.
Record and review
Link explanations to model versions and protect sensitive data.
Why it matters: Production investigation needs context and governance.
OUTSIDE THE TEXTBOOK
A practical example
A support team reviews why an escalation model prioritized a ticket. It sees a local explanation highlighting recent repeat contacts and account status, then checks whether the explanation is stable under small changes and whether those fields are appropriate. The team does not tell customers that these factors caused the outcome without further evidence.
GO A LITTLE DEEPER
Technical details that matter
- Permutation importance and SHAP-style methods provide different views and assumptions.
- Correlated features can make attribution ambiguous even when predictions are stable.
- Explanation logs can expose sensitive inputs and should follow least-privilege access and retention rules.
MAKE A GOOD CHOICE
When is this useful?
- Debugging unexpected predictions.
- Supporting analyst review and model governance.
- Understanding broad feature behavior and potential data-quality issues.
Use inherently interpretable models when their performance and task fit are sufficient. For causal questions, use causal analysis rather than feature attribution alone.
WATCH OUT FOR
Common misunderstandings
- Calling feature attribution a causal explanation.
- Assuming one explanation method is faithful for every model and input.
- Logging full sensitive inputs indefinitely for later explanation.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Explanations can be unstable or method-dependent.
- Complex models may not have a single human-readable rationale.
- A valid local explanation may not describe overall model behavior.
WHAT TO REMEMBER
Key takeaways
- 1Distinguish global from local explanations.
- 2Validate stability and faithfulness before relying on an explanation.
- 3Protect explanation data and avoid causal claims without evidence.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
What is the difference between global and local explanation?
Global methods summarize overall model behavior; local methods describe factors associated with one prediction. Neither alone answers every debugging or governance question.
Do feature attributions show causation?
No. They describe model associations under a method’s assumptions; causal claims require appropriate causal evidence.
SAY IT OUT LOUD
How to explain this in an interview
“I first clarify whether the need is to understand overall behavior or one prediction. I choose a suitable method, test whether its explanation is stable and faithful, and explain its limits rather than implying causation. In production I link reviews to the model version and protect the input data used to create explanations.”
Connect this question to the roadmap topic it supports.