What is RAG? Retrieval-augmented generation explained simply

RAG (retrieval-augmented generation) is a process in which an AI, before answering, first looks up the matching passages from a stored document collection and bases its answer exclusively on them. In one sentence for management: the AI does not answer questions from memory but looks things up in your own documents first. This makes answers verifiable, up to date and considerably less often made up.

In brief

• RAG combines a search in your own documents with the language ability of an AI model.
• The process consists of three steps: search, enrich, answer.
• Because the answer is based on specific passages, every statement can be backed up with a source.
• RAG is not training: new documents can be queried immediately without changing the model.
• Limits exist where documents are missing, contradictory or where knowledge was never written down.

How does the librarian analogy explain RAG?

Imagine an experienced librarian who looks after a huge company library. If you ask them something, they first go to the shelves, pull out the three relevant files, read the decisive passages – and only then answer your question, with the open pages in front of them on the desk.

An ordinary chatbot without RAG works differently: it is like a well-read conversation partner who read a great deal years ago but currently has no access to your library. It answers from memory – often eloquently, sometimes correctly, occasionally made up, and it knows nothing at all about your company’s own files.

RAG turns the conversation partner into the librarian. The language model (LLM, large language model – an AI program that has learned from large amounts of text to understand and formulate language) still supplies the art of phrasing, but the facts come from the shelves of your own library.

Which three steps take place in RAG?

Behind every RAG answer is the same three-stage process, which happens in fractions of a second and invisibly for users. No mathematics is needed to understand it.

1. Search (retrieval): The system searches the stored document collection for passages that match the question. It searches by meaning, not just by exact words – a question about the “notice period” also finds the clause that talks about the “end of the contractual relationship”.
2. Enrich (augmentation): The passages found are passed to the language model together with the original question – comparable to the open files the librarian lays out in front of them.
3. Answer (generation): From exactly these passages, the language model formulates an understandable answer and outputs the passages as sources.

The order is decisive: first search, then answer. The model is instructed to stick to the passages supplied – and if there is nothing suitable there, it is supposed to say exactly that instead of improvising.

For users, all that remains visible of this is a chat window. Type in a question, receive an answer with a reference to the source, open the original document if needed – the three steps in the background require neither training nor technical understanding.

Why does RAG reduce hallucinations?

Hallucinations is the term for the phenomenon of language models producing convincing-sounding but false statements – invented facts, figures or sources. A freely answering model has no way of checking whether its memory is correct; it produces the linguistically most probable continuation, not the documented truth.

RAG defuses this problem at the root, because the answer is tied to specific sources. The model is no longer supposed to report from memory but to summarise the passages in front of it – a task at which language models work considerably more reliably than at free recall.

Hallucinations do not disappear completely as a result, and honesty requires saying so clearly. A RAG system, too, can summarise sources poorly or shorten nuances. The practical difference lies in verifiability: because every answer refers to a source document, checking costs one click instead of your own research.

What distinguishes RAG from training an AI?

“We train the AI with your data” sounds similar but is a fundamentally different – and, for most companies, unsuitable – approach. In training or fine-tuning, the inner parameters of the model itself are changed; the knowledge, so to speak, is built permanently into the model.

Criterion	RAG	Training / fine-tuning
New documents	Can be queried immediately	Only after retraining
Source	Source for every answer	Not possible – knowledge is “baked in”
Removing data	Deleting the document is enough	Hardly possible without retraining
Effort	Setup, then low on an ongoing basis	High and recurring
Control	Basis of the answer visible at any time	Behaviour hard to trace

For the use case “answers from your own company documents”, RAG is superior to training in almost every respect. The deletion aspect carries particular weight: removing a document from a RAG knowledge base takes one action – getting trained-in knowledge back out of a model is practically impossible.

Training has its place elsewhere, for example when a model is to adopt a particular writing style or technical jargon. Factual knowledge that changes continuously, on the other hand, belongs in the document base, not in the model parameters.

Where does RAG reach its limits?

RAG only finds what is in the documents. Practical experience that was never written down – the fitter’s knack, the history of a customer relationship – remains invisible to any system until it is captured at least as a note or voice recording.

Contradictory and outdated documents remain a weak point in terms of content. If three versions of a price list exist, the system can find and name all three but cannot reliably decide which one applies. Basic hygiene in file storage – marking or sorting out obsolete versions – remains the company’s task.

The way the question is asked also influences quality. Very vague questions (“What was the story with that project?”) deliver weaker results than specific ones (“Which payment terms were agreed in the Musterbau project?”). Experience shows that users learn how to use it quickly, but RAG is no magic wand for unclear requests either.

Finally, the search itself has its pitfalls. Questions whose answer only emerges from many documents – such as an evaluation across all projects of a year – overtax the approach more than a targeted individual question. RAG shines at looking things up, not at the statistical evaluation of entire collections.

What does RAG mean for companies in practice?

For small and medium-sized companies, RAG is the technical foundation that makes AI document search fit for business use in the first place: the entire collection instead of individual uploads, ongoing currency instead of frozen copies and a source instead of blind trust. Which alternatives exist and why they fail in long-term use is shown in the guide Using ChatGPT with your own data.

When comparing providers, it is worth looking at the quality features typical of RAG: does the system really answer only from your own documents? Does every answer lead to a verifiable source? Are scans captured with text recognition (OCR, the automatic conversion of images into searchable text)? And is the data stored on EU servers with contractually regulated processing?

Whether a solution meets these criteria in your own company is shown most reliably with your own documents and real everyday questions. Goma-it’s AI knowledge base relies on exactly this RAG principle and makes going through the answers to real questions with your own documents a fixed part of the introduction – so the benefit can be checked before a decision is made.