RAG explained for business: grounding AI answers in your own documents

Retrieval-augmented generation (RAG) makes a large language model answer from your documents instead of from its memory. Before the model writes a word, the system searches your content for the passages most relevant to the question and hands them over as reading material, so the answer is grounded, current and checkable. For most organisations that want an assistant that actually knows their business, RAG beats fine-tuning: it is easier to update, easier to audit and far more forgiving when facts change.

What is retrieval-augmented generation?

A language model on its own answers from whatever it absorbed during training. That knowledge is broad but frozen, and it includes nothing about your price list, your policies or the contract your team signed last month. Ask about those things and the model will either admit it does not know or, worse, guess fluently. Confident guessing is what people mean by hallucination, and it is one of the main reasons businesses hesitate to put an assistant in front of staff or customers.

RAG closes that gap by splitting the job in two. A retrieval step finds the handful of passages in your document store that best match the question. A generation step then writes the answer using only those passages, citing them so a reader can check every claim. The model supplies language and reasoning; your documents supply the facts. When a policy changes, you update the document and re-index it, and the assistant is current the same day. No retraining, no release cycle.

Why does grounding beat fine-tuning for most cases?

Fine-tuning means continuing to train a model on your own examples until its internal weights shift. It is genuinely useful for teaching tone, format and specialised behaviour: a model that always drafts in your house style, or reliably produces a particular output structure. What it does poorly is store facts. Knowledge baked into weights cannot be traced, cannot be selectively removed and goes stale the moment reality moves. A fine-tuned model cannot tell you which document an answer came from, because there is no document any more, only a statistical residue of many.

Grounding turns each of those weaknesses into a strength. The knowledge lives in an ordinary document store you already control: change a file, and the next answer reflects it. Every answer can cite its sources, which matters for regulated work and for plain trust. Access control becomes enforceable, because retrieval can be filtered by the permissions of the person asking, something a fine-tuned model cannot do at all. And when a better base model ships, you swap it in and keep your whole knowledge base as it is. The two techniques also combine well: some teams fine-tune lightly for tone and rely on retrieval for facts. But if you only do one, do RAG first.

An assistant that shows its sources earns trust. One that answers from memory just asks for it.

How does a RAG pipeline actually work?

Four stages, each one ordinary plumbing and each one capable of sinking the whole system if rushed.

Chunking

Documents are split into passages, typically a few hundred words each, because retrieval works on pieces rather than whole files. Good chunking follows the document's own structure: headings, sections, list and table boundaries. Careless chunking cuts mid-sentence and separates a clause from the table that explains it. No later stage can repair that damage and no change of model will rescue it, which is why this plain, unglamorous step deserves real attention.

Embeddings

Each chunk is passed through an embedding model that converts it into a vector: a long list of numbers positioned so that texts with similar meaning sit near one another. The vectors go into a vector database or a search index that supports similarity queries; choosing and tuning that store is a data and database decision as much as an AI one. Embeddings are what let a question phrased as "can I carry over unused vacation?" find a policy paragraph that never uses the words "carry over".

Retrieval

At question time, the query is embedded the same way and the store returns the nearest chunks. Production systems usually run keyword search alongside vector search, called hybrid retrieval, because exact terms such as product codes and proper names are where pure semantic search is weakest. A reranking step often follows: a second model reads the candidates and reorders them by true relevance before the best few are passed on.

Generation with citations

The chosen passages are placed into the model's prompt with instructions to answer only from them and to cite the source of each claim. Done well, the assistant replies with references a reader can open, and when the passages do not contain the answer, it says so plainly instead of improvising. That refusal behaviour is something you design and test for, not something to hope for.

RAG, fine-tuning or long context: which fits your problem?

Models current at the time of writing accept very large prompts, so a third option exists: skip the index and paste entire documents into the context window. It works for small, stable collections, and falls over as the library grows. The honest comparison looks like this.

Where does RAG go wrong?

Grounding reduces hallucination; it does not abolish failure. The common modes are worth knowing before launch, because every one of them is preventable.

• Retrieval misses. The right document exists but the wrong chunks come back, so the model answers from something almost relevant. Hybrid search and reranking are the usual fixes.
• Broken chunks. A passage arrives without the heading or table that gave it meaning, and the model misreads it with full confidence.
• Stale indexes. The document was updated but never re-embedded, so the assistant confidently quotes last year's policy. Indexing must be wired to your content workflow, not run by hand.
• Conflicting sources. Two documents disagree, both are retrieved and the model blends them into one wrong answer. Deduplication and an authoritative-source policy matter more than model quality here.
• Ungrounded drift. The answer starts from the passages, then keeps going into territory they never covered. Strict prompting and groundedness checks catch this.
• Permission leaks. Retrieval that ignores who is asking will happily surface an executive document to a new hire. Filter at retrieval time, always.

How do you evaluate a RAG system?

Build a fixed evaluation set before you build much else: real questions your staff or customers ask, each paired with the passages that should answer it and, where possible, a reference answer. A short set of questions taken from real use beats a long set invented at a desk.

Then measure the two halves separately. Retrieval is scored by whether the known correct passages appear in the results, and how high they rank. Generation is scored on faithfulness, relevance and refusal: is every claim supported by the retrieved text, does the answer address the question asked, and does the system decline when the sources are silent? A useful working number is a groundedness rate: supported claims divided by total claims across the evaluation set. Automated judging with a second model handles scale, with regular human spot-checks keeping the judge honest. Re-run the whole set on every change, whether to chunking, prompts, models or the documents themselves, so a fix in one place cannot quietly break another.

How OlDevs builds grounded AI assistants

OlDevs is a full-stack technology studio in Vancouver, building software since 2014. Our AI development practice covers generative AI and LLM systems, natural language work and AI automation, and RAG assistants sit squarely in that work: pipeline design, document preparation, retrieval tuning, evaluation harnesses and the web or mobile product wrapped around them, all from one accountable team.

We work in the open. You see a working demo every week, evaluation results are shared rather than summarised, and you own all of the code, designs, accounts and IP from day one. If you have a document set your team keeps searching by hand, that is usually the sign a grounded assistant will pay its way. Request a quote and we will reply within one business day.