“Should we fine-tune or use RAG?” is usually the wrong question, because the two fix different problems. Retrieval changes what the model can see on each request. Fine-tuning changes how the model behaves on every request. Most agents that need either need retrieval first, and some need both.
This article lays out what each one is good at according to the research, a decision procedure for agents specifically, and back-of-the-envelope costs for both.
What each one actually does
Retrieval-augmented generation (RAG) searches an index for passages relevant to the request and puts them in the prompt. Lewis et al. introduced it as a way to give a model knowledge it can update and cite, without retraining.[1] Swap a document in the index and the next answer reflects it.
Fine-tuning continues training the model on your examples, so its weights shift toward producing outputs like them. With LoRA, that means training a small adapter rather than every weight. It changes defaults: format, tone, which tool to call, how terse to be, how to handle a category of request.
Fine-tuning is a poor way to add facts
Three findings explain why retrieval wins for knowledge:
- Ovadia et al. compared the two for knowledge injection and found RAG consistently outperformed unsupervised fine-tuning, both for facts the model had seen in pretraining and for entirely new ones. Models struggled to learn new facts by fine-tuning, though training on many paraphrases of each fact helped.[2]
- Gekhman et al. found models learn fine-tuning examples that contain new knowledge much more slowly than ones consistent with what they know, and that once learned, those examples linearly increased the model's tendency to hallucinate.[3]
- The LIMA paper argued that almost all of a model's knowledge comes from pretraining, and that alignment mostly teaches which format and style to use, which it showed with only 1,000 carefully chosen examples.[4]
Put together: fine-tuning shapes how a model uses what it knows. If the problem is that the model doesn't know your refund policy, give it the policy. Our earlier article, Can you just feed an LLM your documents?, goes deeper on why documents don't become knowledge.
Retrieval is a poor way to change behaviour
The reverse also holds. If your agent calls the right tool with malformed arguments, rambles when you need one word, or ignores a format no matter how the instructions are worded, stuffing more context into the prompt rarely fixes it. Long contexts have their own weakness: Liu et al. found models use information at the start and end of a long context much better than information in the middle.[5] Behaviour that has to hold on every request is what training is for.
A decision procedure for agents
- Write the failure down. Collect 20–50 failing cases and label each one: missing knowledge, wrong behaviour, or both.
- Missing knowledge that fits in the prompt? Put it in the prompt. Anthropic suggests that a knowledge base under about 200,000 tokens can go in the prompt whole, with prompt caching to cut the cost.[6]
- Missing knowledge that doesn't fit, or changes often? Build retrieval. Measure retrieval on its own first: if the right passage isn't in the top results, nothing downstream can fix it. Anthropic reported that adding context to chunks before embedding, combined with BM25 and reranking, cut retrieval failures by 67% in its tests.[6]
- Wrong behaviour? Tune the instructions first (it's cheap and fast), then try a different model, and fine-tune only if the behaviour still won't hold. See Prompt tuning or fine-tuning?
- Both? Combine them: retrieve, and fine-tune the model to use retrieved passages well.
When to combine them
RAFT showed the combination is more than the sum of its parts. It fine-tunes a model on questions paired with retrieved documents, including irrelevant “distractor” documents, and trains it to quote the passage that answers the question. It improved domain-specific RAG on PubMed, HotpotQA and Gorilla.[7] For an agent, the equivalent is training on transcripts where the tool results are realistic, including empty, wrong and adversarial ones, so the model learns to cope with what retrieval actually returns.
Side by side
| Retrieval (RAG) | Fine-tuning (LoRA) | |
|---|---|---|
| Fixes | Missing or changing knowledge | Behaviour, format, style, narrow skills |
| Update speed | Minutes: re-index a document | Hours: retrain and redeploy |
| Citations | Natural: you know which passage was used | None |
| Per-request cost | More input tokens and a search call | Often lower: shorter prompts |
| Upfront cost | Index, pipeline, retrieval evals | Labelled examples and GPU time |
| Main risk | Retrieval misses; prompt injection in documents | Hallucinating trained-in facts |
What each one costs, roughly
Retrieval's cost is per request: every call carries extra tokens. Here's an illustrative calculation; the price is an assumption, not a quote:
Illustrative assumptions (swap in your own prices): requests per month 100,000 retrieved context per call 4 chunks × 500 tokens = 2,000 extra input tokens input price $0.20 per million tokens (a small hosted model) extra tokens per month = 100,000 × 2,000 = 200 million extra cost per month = 200 × $0.20 = $40 plus: an embedding index, a search service, and someone to keep both current
A LoRA run's GPU cost is small and one-off. Using the method from what it costs to fine-tune an 8B model:
LoRA on an 8B model, 5,000 examples × 500 tokens × 2 epochs = 5M tokens compute ≈ 6 × 8.2B × 5M = 2.5 × 10¹⁷ FLOPs A100 at 35% of 312 TFLOPS ≈ 38 minutes of GPU time at $2.50/hour plus setup time ≈ $2 per training run plus: hosting the adapter, and retraining when behaviour needs to change
Neither number is the real cost. For retrieval, it's keeping the index fresh and evaluating retrieval quality. For fine-tuning, it's building a good dataset, serving the adapter, and retraining whenever the behaviour you want changes. Both need a test set to prove they helped.



