MACHINE LEARNING · QUESTION 56
What is probability calibration, and why does it matter?
On this page
A classifier is calibrated when predictions assigned probability p are positive about p of the time over a suitable population. Calibration matters when decisions use probabilities, risk estimates, or expected costs rather than only ranking. A model can rank cases well but be overconfident or underconfident. Assess calibration on representative data and use methods such as Platt scaling or isotonic regression when appropriate.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A model might assign a 70% chance to many cases. If it is calibrated, roughly seven in ten similar cases should be positive over time. Calibration is different from whether the model ranks cases correctly: a model can order cases well but give probabilities that are too high or too low.
A USEFUL COMPARISON
Picture it this way
A weather forecast that says “70% chance of rain” should be right about seven times in ten across comparable forecasts. The forecast can still rank rainy days above dry ones while systematically exaggerating the chance.
FOLLOW THE JOURNEY
Compare predicted probabilities with observed frequencies
Explain reliability diagrams and the difference between ranking and calibration.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; A calibrator can change ranking or become stale under population shift.
Predicted probabilities are grouped into bins and plotted against observed positive frequency; a perfectly calibrated model follows the diagonal.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; A calibrator can change ranking or become stale under population shift.
UNDER THE HOOD
How it works, step by step
Collect predictions
Save scores and verified outcomes on held-out examples.
Why it matters: Calibration compares forecasts with observed events.
Group probabilities
Aggregate predictions into bins or use a smooth calibration estimator.
Why it matters: Groups reveal systematic over- or underconfidence.
Fit a calibrator
Use separate calibration data with a method suited to sample size.
Why it matters: Post-processing can adjust probabilities without retraining the base model.
Validate decisions
Check calibration and task performance on fresh representative data.
Why it matters: A calibrator can change ranking or become stale under population shift.
OUTSIDE THE TEXTBOOK
A practical example
A lending workflow uses predicted default probabilities to estimate expected loss. The team checks whether cases assigned similar risk probabilities have matching observed rates, then considers calibration using held-out data. It evaluates by important segments and refreshes the check when the applicant population or model changes.
GO A LITTLE DEEPER
Technical details that matter
- Platt scaling fits a logistic mapping; isotonic regression fits a flexible monotonic mapping and generally needs more calibration data.
- Reliability diagrams depend on binning and sample sizes; confidence intervals help interpret sparse bins.
- Calibration can shift when base rates or data distributions change.
MAKE A GOOD CHOICE
When is this useful?
- Decision systems using expected cost or risk thresholds.
- Communicating probabilities to downstream services or people.
- Comparing confidence quality separately from ranking quality.
If only ranking matters, calibration may be less important, though threshold decisions still need evaluation. A decision-specific score can sometimes replace probability estimates.
WATCH OUT FOR
Common misunderstandings
- Assuming a model score is a calibrated probability.
- Using the same data to fit and assess a calibrator.
- Reporting one overall calibration result when subgroup base rates differ.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Calibration estimates need enough representative labeled examples.
- Post-hoc calibration can become stale as populations change.
- Improved calibration does not guarantee better ranking or decisions.
WHAT TO REMEMBER
Key takeaways
- 1Calibration means predicted probabilities match observed frequencies.
- 2Ranking and probability accuracy are distinct properties.
- 3Fit and assess calibration using separate representative data.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
How do you measure calibration?
Use reliability diagrams and quantitative measures such as expected calibration error with awareness of binning limitations; also inspect relevant segments.
Does calibration improve AUC?
A monotonic calibrator may preserve ranking, so it need not improve AUC. Its purpose is to improve probability interpretation.
SAY IT OUT LOUD
How to explain this in an interview
“Calibration asks whether predicted probabilities match observed frequencies. A model can rank cases well but be overconfident, which matters when decisions use expected costs or risk. I would check a reliability diagram on representative holdout data and consider a calibrator fitted on separate data. I would also assess important segments and monitor changes in population prevalence.”
Connect this question to the roadmap topic it supports.