RAG stands for retrieval augmented generation, and spelling it out helps nobody. In practice it means something fairly simple: before answering, the system searches your documents and uses what it found to write the answer.
That is all. The hard part is the search, not the artificial intelligence.
It is worth understanding because it decides two very concrete things you care about: what happens when you correct a document, and how safe your material is.
What RAG is not: training a model on your documents
This is the most expensive confusion in the market, because it leads people to buy the wrong thing.
A language model is trained once, on an enormous amount of text, in a process that costs a fortune and takes weeks. What it learns there stays inside the model, blended in, impossible to point at. Training a model on your company’s documents, when it is genuinely offered, means taking an existing model and tuning it with your material. That has three problems for the use you have in mind.
It goes stale the day a document changes. The model learned the old version. For it to learn the new one, it has to be tuned again.
It cannot cite. It knows things, it does not know where from. And an answer with no source is useless inside a company for anything that matters.
It cannot be taken back out. If a document should never have gone in, there is no clean way of removing it.
RAG does the opposite. The model learns nothing about your company. Every time you ask, the system finds the relevant fragments in your documents, hands them to the model along with the question, and asks it to answer using only that.
The three consequences invert. Correct a document and the next answer already uses the correction. It can cite, because it knows which fragments it used. And removing a document from the index removes it from the answers, immediately.
How it works, in four steps
One. The documents get split. A forty-page contract does not fit inside a question, so it is cut into fragments of a few paragraphs. Where you cut matters quite a lot: cutting through the middle of a clause produces two fragments that say nothing.
Two. They get indexed. Each fragment is turned into a representation of its meaning and stored. This is the step that lets you find “commercial premises lease” when you searched for “office rental”.
Three. Retrieval. You ask. The system finds the most relevant fragments. The good ones run two searches at once, one by meaning and one by exact word, and combine both results, because the first is bad with invoice numbers and the second is bad with concepts.
Four. Generation. The model receives the question and the fragments, and writes. With the instruction to answer only from that and to say so when it cannot find the answer.
All the value is in step three. If retrieval brings back the wrong fragments, the best model in the world writes an immaculate, false answer.
Where it fails in practice
Four failures, in order of frequency according to what we have seen in real rollouts.
The right document was not there. The most common failure and the least technical. The answer is written nowhere, or it is sitting in someone’s inbox. The system answers with the closest thing it found, which is the worst thing it can do.
The wrong fragment was retrieved on surface similarity. You ask about the supplier advance payment policy and the system brings back the salary advance policy. They talk about the same thing in almost identical words and they are different things.
The split cut the information in half. The clause said “except as provided in the preceding paragraph” and that paragraph ended up in another fragment. The answer comes out with half the condition.
There were two versions and it brought the old one. This is the quietest and the most dangerous. A law firm we worked with had the same contract in three versions with no mark of which one governed. No retrieval system can know which is the good one if the file does not say.
Three of the four are documentation problems, not technology problems. That is the most useful practical conclusion in the whole article: RAG amplifies the quality of your archive in both directions.
What this means for the security of your documents
The consequence of the model learning nothing is that your documents do not travel anywhere whole: what travels are the fragments needed to answer that particular question, and only for as long as the answer takes. That is a real difference from the alternative of uploading the entire file to a generic tool every time somebody wants it summarized.
What does decide who sees what is the permission layer that sits before the search. If the system retrieves fragments from the whole archive without filtering by who is asking, search turns into an open corridor to documents that person should not open. The right question for a vendor is not whether they encrypt traffic, which everybody does, but whether retrieval respects the asker’s permissions before retrieving rather than after displaying.
What to ask a vendor who offers you RAG
Five questions. The answers separate whoever implemented it from whoever read about it.
Is the search semantic only, or hybrid? If it is semantic only, it will fail on invoice numbers, tax IDs and proper nouns. If they tell you semantic alone is enough, they have not tried it on a real billing archive.
Does it cite the exact fragment or only the document? Citing the file is fine. Citing the section is what makes verifying cost ten seconds instead of ten minutes.
What does it do when it finds nothing? The correct answer is that it says so. If the system always answers something, it is not measuring its own confidence and you will never know when to doubt it.
What does indexing and re-indexing cost? If you are charged per document, you will decide what to upload on price and the archive will end up incomplete, which is failure number one on the list above. In Ofivia indexing, re-indexing and searching run on our own server against a local model and consume no credit. Credit is only consumed when the agent works.
Are the documents used to train anything? The answer has to be no, in writing, in the data processing agreement. It is the question the buying committee asks and the one that ends the conversation fastest when the answer is ambiguous.
What you can do to make it work better
Three things, all of them documentation and none of them technology.
Mark which version is current. One line in the header with the date it takes effect and who answers for the document. It solves the most dangerous of the four failures.
Link the documents to each other. When a procedure links to the policy behind it and that policy to the minutes where it was approved, the system can follow the chain and you can walk it in both directions. In Ofivia those links are written wiki-style inside the text and the graph builds itself, so you can also ask which documents point at this one.
Archive what no longer applies. A procedure for a system the company abandoned in 2023 is not history, it is a trap. Take it out of the index and store it separately.
With those three, the same technology goes from being right half the time to being right nearly always. Without them, no vendor is going to fix it for you.
How the documents are stored, how the index is rebuilt from them, and why the disk outranks the database is explained on the vault page. The detail matters: if the index were deleted it regenerates, and if the folder were deleted it does not, and that asymmetry is what decides who really owns your knowledge.