Prompting vs RAG vs fine-tuning: how to choose the right AI approach

If you are deciding how to make a large language model work for your business, start with prompting, add retrieval-augmented generation (RAG) when the model needs your data, and treat fine-tuning as a last resort for narrow, high-volume problems. Each approach changes a different part of the system: the instructions, the knowledge, or the model itself. Each carries a very different maintenance bill, and most organisations get everything they need from the first two.

What do prompting, RAG and fine-tuning actually change?

The three techniques get lumped together as "customising AI," but they operate on different layers of the system, and that is the single most useful thing to understand before spending money on any of them.

Prompting changes the instructions. A system prompt tells the model who it is, what it must and must not do, what format to answer in, and, through examples, what good output looks like. Modern models have large context windows, so a well-built prompt can carry detailed policy, worked examples and edge-case rules. Nothing about the model changes; you are steering a general-purpose engine with better directions.

RAG changes what the model can see. Before the model answers, your system searches a knowledge store, such as product documentation, contracts, support history or a database, and places the most relevant passages into the prompt. The model then answers from that material rather than from its training data alone. The model still has not changed; you have changed its reading list, one question at a time.

Fine-tuning changes the model itself. You take a base model and continue training it on hundreds or thousands of example input-output pairs until its default behaviour shifts. Done well, it makes a model follow a house format without being told, adopt a precise tone, or handle a narrow task with fewer instructions. Done casually, it produces a model that is confidently wrong in new ways and expensive to correct.

Prompting writes the job description, RAG hands over the filing cabinet, and fine-tuning retrains the employee. You rarely need to retrain someone who has clear instructions and the right documents.

What does each approach cost to build and maintain?

Build cost is the part everyone budgets for. Maintenance is the part that determines whether the project is still working a year later. The differences are stark.

A prompt is a text file. Changing it is an edit, a test run and a deployment, often the same afternoon a policy changes. A RAG system is real software: ingestion pipelines, chunking, embeddings, a vector or hybrid search index, and evaluation to confirm the right passages surface. Once built, though, updating knowledge usually means updating documents, not code. A fine-tuned model is the heaviest commitment: you must assemble and clean training data, run and evaluate training jobs, and repeat all of it whenever your requirements change or the base model you tuned is deprecated. Model providers retire versions regularly.

There is also an accuracy-of-fit cost. A prompt error is visible and fixable in minutes. A retrieval error can be traced to a document or an index setting. A fine-tuning error is baked into the weights, and diagnosing it means auditing training data and rerunning jobs. The heavier the technique, the slower the feedback loop when something goes wrong.

When is prompting enough?

More often than most teams expect. If the model already knows the domain, think general writing, summarising, classification, extraction, translation, code review or drafting replies, then the gap between a poor result and a production-ready one is usually instructions, not intelligence. Frontier models at the time of writing follow long, structured prompts well, and techniques such as few-shot examples, explicit output schemas and step-by-step task decomposition close most quality gaps.

Prompting is enough when the task depends on public knowledge or on text you can include directly, when requirements change often, and when a person reviews the output. It stops being enough when the model needs facts it cannot possibly know: your prices, your policies as of this morning, your customer's order history. No amount of clever wording fixes missing knowledge. That is a retrieval problem.

When does RAG earn its keep?

RAG is the workhorse of business AI, and for good reason: most valuable applications are really questions about your data. A support assistant that answers from your actual help centre. An internal tool that searches contracts, project files or a staff portal. A product advisor that reads the current catalogue rather than a memory of it.

Three properties make RAG the right choice. First, the knowledge changes: retrain a model weekly and you will regret it, but re-index documents nightly and nobody notices the effort. Second, answers need citations: RAG can point to the passage it used, which matters for trust, audits and regulated work. Third, access control: retrieval can respect user permissions, showing each person only what they are entitled to see, which a fine-tuned model cannot do because its knowledge is baked in for everyone.

The honest caveat is that RAG quality depends on retrieval quality. Poor chunking, a weak index or messy source documents produce confident answers built on the wrong passage. Budget for evaluation and for cleaning up the data layer underneath: that is where most RAG projects are won or lost.

When does fine-tuning actually make sense?

Fine-tuning earns its cost in a narrower set of cases than the marketing around it suggests. It is the right tool when you need a model to produce a strict output format at very high volume without lengthy instructions, when you want a smaller and cheaper model to match a larger one on a single repetitive task, when a precise brand voice must hold across a very large volume of output, or when a specialised domain has vocabulary and conventions the base model handles clumsily.

What fine-tuning does poorly is the thing people most often want it for: adding knowledge. Models absorb facts unreliably through fine-tuning, they cannot cite where a claim came from, and every update to your knowledge means another training run. If your goal is "the model should know our stuff," you want retrieval. If your goal is "the model should behave differently by default on a task that will not change for a long time," fine-tuning becomes worth discussing, usually layered on top of prompting and RAG, not instead of them.

How should you decide? A practical path

Work through these questions in order and stop at the first fit.

1. Is the output wrong in a way instructions could fix? Wrong tone, wrong format, missed steps, ignored rules: improve the prompt, add examples, and build a small evaluation set so you can measure the change. Stay here as long as quality keeps improving.
2. Is the output wrong because the model lacks your facts? Out-of-date answers, invented details, no knowledge of your documents: add retrieval. Start with one well-curated source, measure whether the right passages surface, then expand.
3. Is the task narrow, stable and high-volume, with prompting and RAG already in place? If a long prompt is driving up cost or latency at scale, or a strict format still slips, a fine-tune of a smaller model is now a reasonable optimisation with a clear payback.
4. None of the above? The problem is likely workflow design, data quality or evaluation, not model customisation. Fix those first; they are cheaper and they compound.

Notice where fine-tuning sits: at the end, as an optimisation of a system that already works. That is why most businesses never need it. Prompting plus retrieval covers knowledge, behaviour and freshness, and it leaves you free to adopt each new generation of models the day it ships instead of migrating a stack of custom weights.

How OlDevs helps you choose and build

OlDevs is a full-stack technology studio in Vancouver, building software since 2014, and our AI development practice covers exactly this ground: generative AI and LLM integration, retrieval pipelines, AI automation and agents, and the evaluation harnesses that tell you honestly whether any of it is working. We start with the cheapest technique that can meet the goal, show you a working demo every week, and hand over a system you own outright: code, prompts, indexes, accounts and IP. One accountable team, WCAG 2.2 AA accessibility, bilingual EN and FR capability, and video calls in your time zone wherever you are.

If you are weighing prompting, RAG or fine-tuning for a real use case, tell us what you are trying to build. Request a quote and we will reply within one business day with a clear-eyed recommendation, even if the recommendation is the simple option.