MLOPS & DEPLOYMENT · QUESTION 69
How would you serve a machine-learning model at scale?
On this page
I would define the request contract, throughput, latency percentiles, availability, and model size first. Package a versioned model behind a stateless service when possible, load features efficiently, and scale replicas based on measured demand. Add batching only if latency permits, enforce timeouts and input limits, and monitor model and service metrics separately. Capacity tests should use realistic request shapes.
LET'S MAKE IT SIMPLE
First, the idea in plain English
Serving means making a trained model available to applications that need predictions. At scale, many requests may arrive at once, and the model has to meet response-time and reliability targets. Model computation is only one part: loading features, serializing requests, queueing, and returning results all consume time.
A USEFUL COMPARISON
Picture it this way
A busy clinic needs enough staff, appointment flow, and records access, not just one skilled doctor. A prediction service similarly needs capacity, request handling, and reliable data access around the model itself.
FOLLOW THE JOURNEY
Scale the full prediction request path
Trace traffic management, feature lookup, inference, and monitoring.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; A benchmark that omits features or tail traffic can mislead.
Clients pass a gateway and load balancer to stateless model-serving replicas; replicas retrieve features, run inference, and emit latency, error, and model metrics.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; A benchmark that omits features or tail traffic can mislead.
UNDER THE HOOD
How it works, step by step
Define service targets
Set throughput, tail latency, availability, payload size, and model constraints.
Why it matters: Capacity decisions need measurable requirements.
Package inference path
Version model, preprocessing, dependencies, and request schema together.
Why it matters: A model artifact alone is not a functioning service.
Scale and protect
Use bounded queues, autoscaling, timeouts, and load shedding where suitable.
Why it matters: Overload should fail predictably rather than cascade.
Measure real workload
Test realistic traffic and monitor service plus prediction behavior.
Why it matters: A benchmark that omits features or tail traffic can mislead.
OUTSIDE THE TEXTBOOK
A practical example
A recommendation API packages model and preprocessing in a container, uses stateless replicas behind a load balancer, and retrieves user features from a low-latency store. Load tests include cold starts and large requests. Autoscaling uses queue and latency signals, while a safe fallback handles feature-store timeouts.
GO A LITTLE DEEPER
Technical details that matter
- Model loading and memory may limit replica density; quantization and shared model servers can change the trade-off.
- Dynamic batching improves throughput but can increase per-request waiting time.
- Tail latency often reflects queueing and dependencies, so optimize p95/p99 rather than only average inference time.
MAKE A GOOD CHOICE
When is this useful?
- Interactive prediction APIs with defined latency and availability targets.
- High-throughput scoring services with variable traffic.
- Serving models that depend on online features or external tools.
Use batch inference if live response is not required. Managed model platforms can reduce operations but should still be load-tested against the workload.
WATCH OUT FOR
Common misunderstandings
- Scaling only on CPU while GPU memory or feature lookup is the real bottleneck.
- Benchmarking only average latency.
- Allowing unbounded requests or queues to exhaust resources.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Replicas and accelerators can be expensive.
- Autoscaling reacts with delay and may not handle sudden spikes.
- External feature and network dependencies limit end-to-end availability.
WHAT TO REMEMBER
Key takeaways
- 1Define service objectives before choosing infrastructure.
- 2Optimize the complete request path and tail latency.
- 3Use bounded capacity controls and realistic load tests.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
How does batching affect latency?
It can improve hardware utilization and throughput, but requests may wait for a batch to fill. Tune batch size and wait time against latency objectives.
How do you handle overload?
Use bounded queues, admission control, timeouts, load shedding, and a defined fallback rather than allowing resource exhaustion.
SAY IT OUT LOUD
How to explain this in an interview
“I first clarify throughput, tail latency, availability, and model constraints. Then I package the model with preprocessing and schema, scale serving replicas, and measure feature lookup and inference as one request path. I would use bounded queues, timeouts, and realistic load tests, monitor p95 or p99 latency, and define overload behavior and fallbacks.”
Connect this question to the roadmap topic it supports.