LLMS & GENERATIVE AI · QUESTION 45
How would you reduce LLM inference cost and latency?
On this page
I would first measure end-to-end latency and cost by request type, separating model time from retrieval, tools, and network overhead. Then I would reduce unnecessary input and output tokens, choose the smallest model that meets quality needs, cache safe repeatable work, and use batching or streaming where suitable. Every optimization needs evaluation because shorter or cheaper calls can reduce answer quality.
LET'S MAKE IT SIMPLE
First, the idea in plain English
An LLM application spends time and money across more than the model call: it may search documents, call tools, and transfer large prompts. Measuring each part shows where the delay or expense comes from. Teams can then trim redundant work or choose a more efficient path while checking that users still get the answer quality they need.
A USEFUL COMPARISON
Picture it this way
A delivery route can be improved by measuring where time is spent: loading, traffic, or the final handoff. Sending a smaller vehicle is not always better if it cannot carry the delivery. Similarly, optimize the actual bottleneck while protecting the quality and reliability requirements.
FOLLOW THE JOURNEY
Measure bottlenecks before choosing an optimization
Decompose request cost and latency and connect optimizations to quality checks.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; An optimization is not useful if it breaks the task.
A request trace separates retrieval, model input and generation, tools, and network time; targeted optimizations are evaluated against quality and reliability.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; An optimization is not useful if it breaks the task.
UNDER THE HOOD
How it works, step by step
Measure baseline
Trace latency, token usage, tool calls, and cost by request class.
Why it matters: Optimization should target a measured bottleneck.
Reduce avoidable work
Trim redundant context and constrain unnecessary output or repeated calls.
Why it matters: Less work can lower cost and response time.
Choose efficient execution
Compare models, caching, batching, streaming, and retrieval settings.
Why it matters: Different controls affect quality and latency differently.
Validate trade-offs
Re-run quality, safety, and service tests against the baseline.
Why it matters: An optimization is not useful if it breaks the task.
OUTSIDE THE TEXTBOOK
A practical example
A knowledge assistant’s trace shows that it sends the entire chat and many document chunks to the model. The team first retrieves fewer, better-ranked passages and keeps only relevant history. It measures the new latency and citation accuracy, then considers a smaller model for straightforward questions while routing harder cases to a stronger one.
GO A LITTLE DEEPER
Technical details that matter
- Time to first token and total generation time are distinct user-perceived metrics.
- Caching is safe only when keys include relevant context, permissions, model version, and freshness requirements.
- Batching may improve throughput but increase queue time; streaming improves perceived latency but not necessarily total computation.
MAKE A GOOD CHOICE
When is this useful?
- Reducing spend for high-volume inference workloads.
- Improving response times in interactive applications.
- Sizing model tiers and routing policies against quality requirements.
If model inference is not the bottleneck, optimize retrieval, serialization, or downstream tools instead of changing models.
WATCH OUT FOR
Common misunderstandings
- Optimizing token count without checking answer quality or citation coverage.
- Caching personalized outputs without correct identity and freshness boundaries.
- Comparing providers with different workloads or latency percentiles.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Smaller models may fail on complex tasks.
- Caching can serve stale or cross-user data if keys are poorly designed.
- Batching and routing add orchestration complexity.
WHAT TO REMEMBER
Key takeaways
- 1Profile complete request paths before optimizing.
- 2Reduce tokens and choose models according to task complexity.
- 3Re-evaluate quality, privacy, and latency after every change.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
What is the difference between time to first token and total latency?
Time to first token measures how long before streaming begins; total latency measures completion. Streaming can improve perceived responsiveness without reducing total work.
When is response caching safe?
When the output is repeatable and the cache key accounts for user permissions, relevant context, model and prompt versions, and freshness requirements.
SAY IT OUT LOUD
How to explain this in an interview
“I would start with request traces and separate model time, tokens, retrieval, tools, and network overhead. Then I would target the largest bottleneck: reduce redundant context, limit output, route simple work to a smaller model, or cache safe repeatable results. I would compare latency and cost with task quality, safety, and privacy before rollout.”
Connect this question to the roadmap topic it supports.