The three options change different components of a system, so the right one follows from what is actually failing.
What each approach actually changes
The three options are usually presented as a ladder of effort. The more useful distinction is that each one modifies a different component of the system.
Prompting changes the instructions and examples sent with each request. Retrieval changes the content that travels alongside those instructions, selected per request from a corpus you control. Fine-tuning changes the model weights by training on input and output pairs.
That distinction decides most of what follows. Prompting and retrieval take effect at request time, so a change is live when it is deployed and can be reverted the same way. Fine-tuning takes effect at build time: the behaviour is trained into a model artefact, and changing it means another training run.
Retrieval is the default when the answer has to come from your content
If a requirement is stated as answer from our documents, it is a retrieval requirement. The facts live in a corpus that changes, answers need attribution, and the correct answer depends on which version of a document is current and which documents the asker is allowed to see.
Fine-tuning on the same documents does not provide this. Training adjusts the distribution of text the model produces; it does not install a table you can look things up in. The model becomes more fluent in the vocabulary of your corpus, including when it is inventing.
Retrieval makes attribution, versioning and per-user permission achievable, but none of them comes for free. Citation plumbing has to carry the source through to the answer, a supersedence rule has to decide which version wins, and permission filtering has to run at retrieval time. Each is work to build; the point is that retrieval is the architecture in which they are buildable at all.
What fine-tuning genuinely helps with
Fine-tuning is the right tool when the problem is behaviour rather than knowledge. Four cases recur in the literature and in practice.
Output format. A system that consumes model output needs a fixed shape, and prompting produces that shape most of the time but not every time. Constrained decoding and structured output modes come first, because they enforce the shape rather than encouraging it, and they need no training run.
Tone and register. Output can be factually correct and still unusable because the voice is wrong for the channel. Style is hard to specify in instructions and straightforward to demonstrate in examples, which is what fine-tuning suits.
Narrow classification and extraction. For a fixed label set or a fixed schema with a reasonable volume of labelled examples, a fine-tuned smaller model is often a better fit than a long prompt carrying examples on every call.
Cost and latency at scale. A task a large model performs with a long prompt can sometimes be moved onto a smaller model trained on examples of that task. Check the licence first: most commercial providers restrict using their model outputs to train competing models, so a dataset distilled from a hosted model may not be usable for this at all. Note also that prompt caching already removes much of the per-request cost of a long stable instruction block, with no training run and no dataset, so it is worth measuring before a fine-tune is commissioned on cost grounds.
What these share is that the target behaviour can be written as input and output pairs, and that enough of them exist or can be produced. Where that dataset cannot be assembled, a fine-tune has nothing to learn from.
Decide from the failure cases
The choice becomes tractable once failures are collected as data: the input, what the system produced, and what it should have produced. Sorted that way, failures tend to point at one approach.
If the answer omits a fact that exists in your content, and you can name the passage that should have been quoted, check whether that passage was retrieved. If it was not, the work is in ingestion, chunking, hybrid search, reranking, freshness and permission filtering. If it was retrieved and the answer still went wrong, the problem is downstream of retrieval: the instructions, a task that needs splitting into steps, or a model that cannot do it.
Invented facts belong to both categories and have to be diagnosed rather than assumed. A model handed no supporting passage will fill the gap; a model handed the correct passage can still invent a figure in the middle of an otherwise grounded answer. The distinction is only visible if the evaluation records what was retrieved alongside what was answered.
This sorting depends on the labelled set existing. Without one the choice gets argued from preference, and no change can be shown to have helped. The set is also the regression suite for every later change.
The operational cost each one carries
Prompting is the cheapest to change and the easiest to break quietly. Prompts accumulate clauses, each added for a case someone hit once, and a late clause can cancel an earlier one. They need version control and an evaluation run like any other change.
Retrieval carries the cost of keeping an index faithful to a source that moves. Content changes require re-indexing, deletions have to propagate, and permission changes have to take effect at retrieval time rather than at the next rebuild. Changing the embedding model means re-embedding the whole corpus.
Fine-tuning carries two recurring costs. The dataset has to be maintained as the task drifts, and the fine-tune is bound to the base model it was trained from. With a hosted provider that base is replaced on the provider's schedule rather than yours, so a model upgrade means repeating the training run against the dataset, re-evaluating and redeploying. Serving an open-weight model yourself removes that forcing function, at the cost of running the infrastructure.
How hard a fine-tune is to reverse depends on its form. A served adapter sits alongside an unchanged base model and can be detached; a full-weight replacement is a larger undertaking to unwind. Both are slower to reverse than editing a prompt.
The three combine more often than they compete
These are not mutually exclusive, and a system that uses more than one is common. The prompt defines the task, the rules and the output contract. Retrieval supplies the facts and the citations. A fine-tuned model, where there is one, usually handles a sub-step rather than the whole task.
The ordering follows from the cost of changing each. Prompting comes first because it is the fastest to test. Retrieval comes in as soon as answers depend on content you own. Fine-tuning comes last, once the task has stopped changing shape and a dataset exists.
Which of the three a requirement needs is one of the questions a discovery sprint settles. AIVETech starts every engagement with a fixed-price discovery sprint at USD 2,500 over two weeks, which produces the architecture, initial screen designs, acceptance criteria, a running-cost model and a fixed price for the build. The sprint fee is deducted from the build.
Read about the AI Feature Implementation Sprint or discuss an AI project.