What is RAG (retrieval-augmented generation)?
Ask a chatbot about your company's refund policy and it may confidently invent one. The model never saw your policy, so it fills the gap with something plausible. Retrieval-augmented generation, usually shortened to RAG, is the most common fix: look up the relevant text first, then hand it to the model along with the question.
What is RAG (retrieval-augmented generation)? It is a pattern where a system searches a collection of documents for passages related to your question, pastes the best matches into the prompt, and asks a language model to answer using that material. The model still just generates text; retrieval changes what it has in front of it.
Why RAG exists
A language model learns from its training data, and that knowledge is frozen at training time. It knows nothing about your private documents, and it can't reliably recall every detail of what it did see. Retraining a model every time a document changes is impractical. It is far easier to fetch the right text at question time and put it in the prompt, the way you'd let a student take an open-book exam.
Step 1: Prepare the documents
Before anyone asks a question, the documents are split into chunks of a manageable size, such as a few paragraphs each. Each chunk is run through an embedding model, which turns text into a list of numbers. Chunks with similar meaning end up with similar numbers. These vectors are stored in an index, often called a vector database, alongside the original text. (Our post on how a database index makes queries fast covers the general idea of indexing.)
Step 2: Retrieve what matches the question
When a question arrives, it is embedded the same way. The system then finds the stored chunks whose vectors are closest to the question's vector. This is a search by meaning rather than exact keywords, so "Can I get my money back?" can match a chunk titled "Refund policy". Many systems also combine this with ordinary keyword search, and some re-rank the top results with a second model.
Step 3: Augment the prompt
The retrieved chunks are placed into the prompt, along with instructions such as "answer using only the context below, and say so if the answer isn't there." The prompt now holds the question plus the evidence. This is the "augmented" part, and it must fit inside the model's context window, which is why only a handful of the best chunks are included.
Step 4: Generate the answer
The model reads that prompt and writes a response, as described in how large language models like ChatGPT work. Because the relevant text is right there, it can quote or summarise it. Many systems also show which chunks were used, so a reader can check the source.
What RAG does and doesn't fix
RAG helps with:
- Fresh or private knowledge. Update the documents and the answers change, with no retraining.
- Traceability. You can point to the passage behind an answer.
- Fewer invented details, when the right passage is retrieved.
It does not guarantee correctness:
- Bad retrieval means bad answers. If the wrong chunks come back, the model may answer confidently from irrelevant text.
- Chunking matters. Split a table or an argument in the wrong place and the key sentence may land in a different chunk from its context.
- The model can still misread the material it was given.
Most RAG quality problems are retrieval problems, so that is usually the first place to look.
RAG versus fine-tuning
Fine-tuning adjusts a model's parameters with extra training, which is good for teaching style, format, or a specialised task. RAG supplies facts at question time. They solve different problems and are often used together rather than as rivals.
Quick recap
- RAG = retrieve relevant text, add it to the prompt, then generate.
- Documents are chunked and stored as embeddings so search works by meaning.
- The question is embedded, and the closest chunks are fetched.
- It adds fresh or private knowledge and sources, without retraining.
- It is only as good as its retrieval.