MLOPS & DEPLOYMENT · QUESTION 68
When would you use batch versus online model inference?
On this page
Batch inference scores many records together on a schedule and is suitable when results can be prepared ahead of use; it favors throughput and simpler recovery. Online inference computes a prediction in response to a request when latency or fresh inputs matter. The choice depends on freshness, request volume, latency targets, and serving cost, and many products use both.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A recommendation can be calculated overnight and shown the next day, or generated when a user opens a page. The first is batch scoring; the second is online inference. Batch work can process large groups efficiently, while online prediction offers fresher results but must meet service-level expectations for each request.
A USEFUL COMPARISON
Picture it this way
A bakery can prepare tomorrow’s standard orders in one large batch, or make a custom sandwich when a customer arrives. Batch preparation is efficient when orders are predictable; on-demand work is useful when the customer’s current choice matters.
FOLLOW THE JOURNEY
Match inference mode to freshness and latency requirements
Compare scheduled batch scoring with request-time online prediction.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Predictions need traceability and freshness controls.
A batch job reads a dataset and writes predictions for later use; an online request retrieves current features, runs the model, and returns a prediction within a latency budget.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; Predictions need traceability and freshness controls.
UNDER THE HOOD
How it works, step by step
Set freshness and latency
Identify when a prediction is needed and how current its inputs must be.
Why it matters: The product requirement selects the serving mode.
Choose execution path
Schedule bulk scoring or run prediction within a live request.
Why it matters: Each mode has different throughput and availability needs.
Plan feature access
Prepare batch joins or low-latency online lookups.
Why it matters: Feature retrieval can dominate inference cost.
Monitor and refresh
Track job completion or request latency and data/model versions.
Why it matters: Predictions need traceability and freshness controls.
OUTSIDE THE TEXTBOOK
A practical example
A bank can batch-score customer segments overnight for a campaign list, but score a card transaction online for an immediate risk decision. The batch job can be retried by partition; the online service needs low latency, high availability, and a fallback policy.
GO A LITTLE DEEPER
Technical details that matter
- Batch inference can exploit vectorized computation and retry by partition.
- Online serving may require model loading, feature retrieval, autoscaling, and strict timeout budgets.
- Hybrid systems need clear rules for which prediction version takes precedence.
MAKE A GOOD CHOICE
When is this useful?
- Batch for offline reports, precomputed recommendations, and scheduled risk lists.
- Online for interactive decisions and immediate user feedback.
- Hybrid use when some predictions can be cached but exceptions need fresh scoring.
A streaming pipeline can continuously update predictions between fully scheduled batch and per-request inference.
WATCH OUT FOR
Common misunderstandings
- Using online serving when freshness requirements do not justify its operational cost.
- Using stale batch scores for decisions that require current state.
- Ignoring model-version differences between batch and online paths.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Batch results become stale between runs.
- Online systems are sensitive to serving outages and tail latency.
- Operating both paths can produce inconsistent predictions.
WHAT TO REMEMBER
Key takeaways
- 1Batch optimizes grouped throughput; online inference serves request-time predictions.
- 2Freshness and latency requirements drive the choice.
- 3Version and monitor both inference paths if a hybrid is used.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
How do you make batch scoring recoverable?
Partition work, use stable input and model versions, make writes idempotent, and record completed partitions so retries do not duplicate results.
What is the main online-serving bottleneck?
It varies: model compute, feature lookup, network calls, or queueing may dominate. Trace end-to-end latency before optimizing.
SAY IT OUT LOUD
How to explain this in an interview
“Batch inference scores records together ahead of use and works when scheduled freshness is sufficient. Online inference responds to a live request and fits low-latency or current-state decisions, but adds serving and availability requirements. I would choose from product freshness and latency needs, then account for feature access, retry behavior, and model-version consistency.”
Connect this question to the roadmap topic it supports.