LLMS & GENERATIVE AI · QUESTION 31
When would you use RAG versus fine-tuning?
On this page
Use Retrieval-Augmented Generation (RAG) when an application needs changeable or private facts retrieved at answer time, with sources that can be inspected. Fine-tune when you need to change a model’s learned behavior, format, or task performance using examples. They solve different problems and can be combined; neither is a reliable substitute for the other.
LET'S MAKE IT SIMPLE
First, the idea in plain English
RAG gives a model useful documents while it answers, so the knowledge can be updated by changing the index. Fine-tuning changes model parameters using additional training examples, which can teach a repeated style or task behavior. Choosing between them starts with identifying whether the gap is missing information or the way the model responds.
A USEFUL COMPARISON
Picture it this way
RAG is like bringing reference books into an open-book exam; fine-tuning is like practicing a procedure until it becomes familiar. The comparison is approximate: a model does not learn like a person, and fine-tuning does not guarantee precise storage of facts.
FOLLOW THE JOURNEY
Choose based on whether knowledge or behavior needs to change
Compare retrieval-time context with training-time parameter updates.
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; A hybrid system may be appropriate when both gaps exist.
A decision flow sends changing or private facts to retrieval, stable response behavior to fine-tuning, and systems needing both to a combined approach.
How to read this visual
This simplified visual highlights the main relationships. In practice, the exact design depends on the requirements; A hybrid system may be appropriate when both gaps exist.
UNDER THE HOOD
How it works, step by step
Define the gap
Identify missing facts, response behavior, or both.
Why it matters: Different gaps need different interventions.
Use retrieval for knowledge
Find current evidence and pass it with the question.
Why it matters: Sources can change without retraining the model.
Fine-tune for behavior
Train on representative examples of the desired task.
Why it matters: Repeated patterns can be learned into model parameters.
Evaluate the choice
Compare quality, cost, update needs, and failure modes.
Why it matters: A hybrid system may be appropriate when both gaps exist.
OUTSIDE THE TEXTBOOK
A practical example
A company policy assistant should usually retrieve current handbook sections and cite them, rather than trying to memorize policy through fine-tuning. If the same assistant must consistently return a strict structured format, fine-tuning may help with that behavior. The team still evaluates retrieval and output quality separately.
GO A LITTLE DEEPER
Technical details that matter
- RAG adds indexing, retrieval, ranking, access-control, and prompt-construction components.
- Fine-tuning requires curated training data and model-training infrastructure, and updates may be slower to roll out.
- Compare both approaches on a representative evaluation set; prompt-only baselines can also be enough for simple tasks.
MAKE A GOOD CHOICE
When is this useful?
- RAG for current product documentation, internal knowledge, and cited answers.
- Fine-tuning for stable formatting, classification, or task-specific response behavior.
Start with better prompting, structured APIs, or conventional search when those meet the requirement. Use a hybrid only when evidence and behavior both need improvement.
WATCH OUT FOR
Common misunderstandings
- Fine-tuning a model to memorize frequently changing facts.
- Assuming RAG guarantees that the model uses retrieved evidence correctly.
- Treating RAG and fine-tuning as mutually exclusive.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- RAG is limited by document quality and retrieval relevance.
- Fine-tuned behavior can regress and does not inherently provide citations.
- Both approaches require evaluation and ongoing operational ownership.
WHAT TO REMEMBER
Key takeaways
- 1RAG supplies knowledge at inference time; fine-tuning changes model behavior.
- 2Use the method that addresses the measured gap.
- 3A combined approach can be useful but increases complexity.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
Can RAG and fine-tuning be combined?
Yes. Fine-tuning can shape stable response behavior while retrieval supplies current evidence. Evaluate the combined pipeline because interactions can introduce new failures.
Does fine-tuning add new facts reliably?
It may influence responses, but it is not a dependable, inspectable knowledge store. For facts that change or require citations, retrieval is usually more appropriate.
SAY IT OUT LOUD
How to explain this in an interview
“I choose based on the problem. RAG is a good fit when the model needs current, private, or cited information because documents can be updated without retraining. Fine-tuning is more appropriate for stable task behavior or formatting that examples can teach. They can be combined, but I would compare options against representative examples and include operational costs and failure modes.”
Connect this question to the roadmap topic it supports.