LLMS & GENERATIVE AI · QUESTION 02
What is Retrieval-Augmented Generation (RAG), and when should you use it?
On this page
Retrieval-Augmented Generation (RAG) retrieves relevant material from an external knowledge source and supplies it to a language model alongside the user’s question. A typical system chunks and indexes documents, retrieves likely matches at query time, and asks the model to answer using that context. RAG is useful when answers should reflect private, changing, or cited information, but retrieval quality and source quality still limit the result.
LET'S MAKE IT SIMPLE
First, the idea in plain English
A language model may know general patterns but not the latest company handbook or a private engineering guide. RAG gives the application a way to look up relevant passages before asking the model to answer. Documents are prepared ahead of time; when a question arrives, the system searches that prepared collection and places selected passages into the prompt. The model then writes an answer using the supplied context. RAG can make knowledge easier to update than retraining a model, but it cannot rescue missing or misleading documents.
A USEFUL COMPARISON
Picture it this way
Think of answering a policy question with an open-book exam. First you find the pages likely to contain the answer, then you explain them in your own words. The analogy captures retrieval plus generation; unlike a person, a model may misuse a passage, so the system should check relevance and encourage citations.
FOLLOW THE JOURNEY
RAG has an indexing path and a question path
Distinguish document preparation from retrieval and grounded answer generation.
The first two nodes run during ingestion; retrieval and generation happen for each query. A production design also needs permissions, freshness, and evaluation.
Documents are chunked and embedded into an index; at query time the question retrieves relevant chunks, which are passed with the question to the language model for an answer.
How to read this visual
The first two nodes run during ingestion; retrieval and generation happen for each query. A production design also needs permissions, freshness, and evaluation.
UNDER THE HOOD
How it works, step by step
Prepare documents
Extract text, preserve useful metadata, and divide long files into chunks.
Why it matters: Search needs units small enough to retrieve but large enough to retain meaning.
Create a searchable index
Represent chunks for keyword, vector, or hybrid search and store them with source metadata.
Why it matters: The system needs a way to locate candidate evidence quickly.
Retrieve candidates
Use the question to find and optionally rerank relevant chunks.
Why it matters: Only relevant context should consume the model’s limited prompt space.
Generate with evidence
Supply the question and selected passages; ask for an answer grounded in them.
Why it matters: The response can reflect the supplied, updatable knowledge.
Evaluate and cite
Check relevance, answer support, permissions, and source references.
Why it matters: A plausible answer is not enough if evidence is weak or unauthorized.
OUTSIDE THE TEXTBOOK
A practical example
An employee asks, “How many days ahead should I request parental leave?” The service searches current policy documents, retrieves the relevant paragraph with its document date, and asks the model to summarize it with a link. If no suitable passage appears, the safer behavior is to say the policy could not be found rather than improvise.
GO A LITTLE DEEPER
Technical details that matter
- Chunk size and overlap affect both retrieval precision and context completeness. Very small chunks lose relationships; very large chunks dilute relevance.
- Embeddings map text into vectors so semantically similar passages may be found even when wording differs. Vector search is often combined with keyword search.
- Metadata filtering can restrict by document type, date, tenant, or access permissions before results reach the model.
- RAG adds retrieval latency and a new failure surface: ingestion, indexing, ranking, prompt construction, and generation all need evaluation.
MAKE A GOOD CHOICE
When is this useful?
- Knowledge assistants over private or changing documents.
- Question answering where evidence and citations are important.
- Applications that need to separate reusable model behavior from frequently updated knowledge.
Use structured queries for precise records and calculations. Consider fine-tuning for stable behavior or style, not as the default way to inject frequently changing facts.
WATCH OUT FOR
Common misunderstandings
- Assuming adding a vector database makes answers automatically correct.
- Ignoring document access controls when retrieving chunks.
- Evaluating only the final answer and not whether the right evidence was retrieved.
REAL-WORLD TRADE-OFFS
Limitations and failure cases
- Bad ingestion or stale sources yield bad evidence.
- Relevant passages may be missed or ranked too low.
- The model may contradict or overstate the retrieved text.
WHAT TO REMEMBER
Key takeaways
- 1RAG combines external retrieval with language generation.
- 2Ingestion and query-time retrieval are distinct workflows.
- 3Retrieval quality, source quality, and permissions determine the ceiling.
- 4Measure evidence retrieval as well as answer quality.
PRACTICE THE FOLLOW-UP
What might an interviewer ask next?
How do chunk size and overlap affect RAG?
Chunks must preserve enough context to be useful while staying specific to the query. Overlap can preserve boundaries but increases index size and duplicate matches.
When would you choose fine-tuning instead?
Fine-tuning can teach a stable task style or behavior. It is not a reliable substitute for retrieving frequently changing factual documents.
How do you evaluate a RAG system?
Measure retrieval relevance and coverage, then assess whether answers are supported by retrieved evidence, accurate, and appropriately cited.
SAY IT OUT LOUD
How to explain this in an interview
“RAG is a pattern where an application retrieves relevant external documents before asking a language model to answer. We prepare and index documents ahead of time, retrieve passages for each query, and pass those passages with the question to the model. This is useful for private or changing knowledge, but it adds retrieval and permission concerns. I would evaluate whether the right evidence was found and whether the answer is actually supported by it.”
Connect this question to the roadmap topic it supports.