RAG, or retrieval-augmented generation, means finding the useful passages in your documents for each question, then giving them to the model so that it writes an answer based on them, with sources.
Key points
- The model does not reread all your documents for every question: it only receives the passages closest to the question.
- Every answer can cite its sources, so it can be checked.
- Quality depends first on the documents: a scanned PDF with no text cannot be used as is.
How it works, in 4 steps
- Chunking: documents are split into short passages
- Indexing: each passage gets a numerical fingerprint of its meaning (a vector) and is stored in an index
- Retrieval: for each question, the index finds the passages whose meaning is closest
- Answer: the model writes the answer from these passages, and cites them
See RAG step by step
One question asked over three fictitious documents. Go step by step to see what the system actually does.
Try it
RAG step by step
Supplier framework contract.pdf
- Clause 11 · The contract is concluded for one year, renewed automatically.
- Clause 12 · Either party may terminate the contract with three months’ notice, by registered letter with acknowledgement of receipt.
- Clause 13 · Invoices are payable within thirty days.
Purchasing procedure.docx
- Any termination of a supplier contract is reported to the purchasing department before the letter is sent.
- Large orders are approved by management.
HR remote work memo.pdf
- Remote work is possible two days a week.
- Requests are sent to the team manager.
The three documents are split into short passages. Each passage keeps its origin: document and clause number.
Fictitious documents and scores
What makes a good answer
- Up-to-date documents, with usable text
- Chunking that keeps the context: headings, tables, clause numbers
- Clear instructions: answer only from the sources, and say when the information is missing
Limits
- A scanned PDF with no text first needs character recognition
- A question that is too vague retrieves passages that are not very relevant
- The model can summarise a passage badly: the sources shown let you check
Frequently asked questions
Further reading
- Private LLM for business: the guideWhat a private LLM is, its architecture, the GPU memory it needs (calculator), what stays on your side and its current limits.
- Fine-tuning: adapting a model to your businessWhat fine-tuning an open-source model changes, how it differs from RAG and instructions, how it is evaluated and what you own at the end.
- Private LLM or AI via API: how to choosePrivacy, hardware, models, time to start: the concrete criteria for choosing between an open-source private LLM and Claude or GPT through an API.
In 30 minutes, we look together at whether a pilot makes sense for you.