MLOPS & DEPLOYMENT · QUESTION 72
How do you A/B test a machine-learning model in production?
On this page
An A/B test randomly assigns eligible users or requests to a control and candidate model, then compares preselected outcome metrics over an appropriate period. I define the unit of randomization, guardrail metrics, sample size, and stopping rules in advance. Check assignment integrity and segment effects, and account for delayed outcomes. Offline gains do not guarantee online user benefit.
LET'S MAKE IT SIMPLE
First, the idea in plain English
Offline evaluation uses saved examples, but a model can change how people interact with a product. An online experiment compares the current system with a candidate under real use. Random assignment helps distinguish the model’s effect from other changes, provided the test is designed and monitored carefully.
A USEFUL COMPARISON
Picture it this way
A store tests two checkout layouts by showing each to randomly selected shoppers and comparing completion and complaints. If both groups differ in important ways or the test stops early after a lucky result, the comparison may mislead. A model experiment needs the same care with assignment and outcomes.
FOLLOW THE JOURNEY
Randomize exposure and compare outcomes with guardrails
Show control and candidate arms, assignment, primary metric, and safety checks.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Averages and early results can hide harm or noise.
Eligible traffic is randomly assigned to control and candidate model groups; outcomes and guardrail metrics are measured and compared under predefined analysis rules.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Averages and early results can hide harm or noise.
UNDER THE HOOD
How it works, step by step
Define hypothesis
Choose primary outcome, guardrails, population, and minimum worthwhile effect.
Why it matters: Predefined criteria reduce selective interpretation.
Assign consistently
Randomize at a unit that avoids contamination, such as user or account.
Why it matters: Assignment should create comparable groups.
Monitor integrity
Check traffic allocation, errors, latency, and safety metrics.
Why it matters: A broken experiment can expose users or invalidate evidence.
Analyze and decide
Include uncertainty, delayed outcomes, and segment checks.
Why it matters: Averages and early results can hide harm or noise.
OUTSIDE THE TEXTBOOK
A practical example
A recommendation model is assigned by user ID so one person consistently sees the same variant. The team measures purchases as a primary outcome and page latency and complaint rate as guardrails. It checks assignment balance and waits through the planned measurement window before deciding whether to expand.
GO A LITTLE DEEPER
Technical details that matter
- Choose the randomization unit to match interference risks; requests from the same user may not be independent.
- A/A tests can check assignment and measurement systems before comparing models.
- Sequential monitoring needs valid stopping methods; repeatedly checking a conventional significance test can inflate false positives.
MAKE A GOOD CHOICE
When is this useful?
- Testing whether a model changes user or business outcomes.
- Comparing ranking or recommendation candidates under real traffic.
- Validating that offline improvements translate to product value.
Use shadow evaluation when user-facing behavior must not change, recognizing that shadow runs cannot measure user response to the candidate.
WATCH OUT FOR
Common misunderstandings
- Changing prompts, UI, and model together without separating effects.
- Stopping when results first look positive.
- Ignoring guardrail harm or delayed outcomes.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Experiments can expose users to a worse candidate.
- Rare outcomes need large samples or long durations.
- Interference and noncompliance can weaken causal interpretation.
WHAT TO REMEMBER
Key takeaways
- 1Randomize at an appropriate unit and define metrics in advance.
- 2Monitor experiment integrity and guardrails.
- 3Use a valid analysis plan and account for delayed outcomes.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
Why randomize by user rather than request?
User-level assignment avoids one person switching variants and experiencing inconsistent behavior; the correct unit depends on interference and product design.
What is a guardrail metric?
A metric such as latency, error rate, or complaints that should not worsen beyond an acceptable limit while optimizing the primary outcome.
SAY IT OUT LOUD
How to explain this in an interview
“I would define the hypothesis, primary outcome, guardrails, sample, and stopping plan before launch. Users or another appropriate unit are randomly assigned to control or candidate, and I check allocation and service health. Then I compare outcomes with uncertainty and segment checks, accounting for delayed labels. A positive offline score alone is not evidence of online benefit.”
Connect this question to the roadmap topic it supports.