Skip to content

Training

Fine-tuning vs RAG: which one does your agent need?

Retrieval changes what a model can see; fine-tuning changes how it behaves. What the research says about each, a five-step decision procedure for agents, when to combine them, and rough costs.

By Will Schulz8 min read
A glass shelf of thin glass books with a red beam drawing one out, beside an obsidian sphere with a glowing red seam

“Should we fine-tune or use RAG?” is usually the wrong question, because the two fix different problems. Retrieval changes what the model can see on each request. Fine-tuning changes how the model behaves on every request. Most agents that need either need retrieval first, and some need both.

This article lays out what each one is good at according to the research, a decision procedure for agents specifically, and back-of-the-envelope costs for both.

What each one actually does

Retrieval-augmented generation (RAG) searches an index for passages relevant to the request and puts them in the prompt. Lewis et al. introduced it as a way to give a model knowledge it can update and cite, without retraining.[1] Swap a document in the index and the next answer reflects it.

Fine-tuning continues training the model on your examples, so its weights shift toward producing outputs like them. With LoRA, that means training a small adapter rather than every weight. It changes defaults: format, tone, which tool to call, how terse to be, how to handle a category of request.

Fine-tuning is a poor way to add facts

Three findings explain why retrieval wins for knowledge:

  • Ovadia et al. compared the two for knowledge injection and found RAG consistently outperformed unsupervised fine-tuning, both for facts the model had seen in pretraining and for entirely new ones. Models struggled to learn new facts by fine-tuning, though training on many paraphrases of each fact helped.[2]
  • Gekhman et al. found models learn fine-tuning examples that contain new knowledge much more slowly than ones consistent with what they know, and that once learned, those examples linearly increased the model's tendency to hallucinate.[3]
  • The LIMA paper argued that almost all of a model's knowledge comes from pretraining, and that alignment mostly teaches which format and style to use, which it showed with only 1,000 carefully chosen examples.[4]

Put together: fine-tuning shapes how a model uses what it knows. If the problem is that the model doesn't know your refund policy, give it the policy. Our earlier article, Can you just feed an LLM your documents?, goes deeper on why documents don't become knowledge.

Retrieval is a poor way to change behaviour

The reverse also holds. If your agent calls the right tool with malformed arguments, rambles when you need one word, or ignores a format no matter how the instructions are worded, stuffing more context into the prompt rarely fixes it. Long contexts have their own weakness: Liu et al. found models use information at the start and end of a long context much better than information in the middle.[5] Behaviour that has to hold on every request is what training is for.

A decision procedure for agents

  1. Write the failure down. Collect 20–50 failing cases and label each one: missing knowledge, wrong behaviour, or both.
  2. Missing knowledge that fits in the prompt? Put it in the prompt. Anthropic suggests that a knowledge base under about 200,000 tokens can go in the prompt whole, with prompt caching to cut the cost.[6]
  3. Missing knowledge that doesn't fit, or changes often? Build retrieval. Measure retrieval on its own first: if the right passage isn't in the top results, nothing downstream can fix it. Anthropic reported that adding context to chunks before embedding, combined with BM25 and reranking, cut retrieval failures by 67% in its tests.[6]
  4. Wrong behaviour? Tune the instructions first (it's cheap and fast), then try a different model, and fine-tune only if the behaviour still won't hold. See Prompt tuning or fine-tuning?
  5. Both? Combine them: retrieve, and fine-tune the model to use retrieved passages well.

When to combine them

RAFT showed the combination is more than the sum of its parts. It fine-tunes a model on questions paired with retrieved documents, including irrelevant “distractor” documents, and trains it to quote the passage that answers the question. It improved domain-specific RAG on PubMed, HotpotQA and Gorilla.[7] For an agent, the equivalent is training on transcripts where the tool results are realistic, including empty, wrong and adversarial ones, so the model learns to cope with what retrieval actually returns.

Side by side

Retrieval (RAG)Fine-tuning (LoRA)
FixesMissing or changing knowledgeBehaviour, format, style, narrow skills
Update speedMinutes: re-index a documentHours: retrain and redeploy
CitationsNatural: you know which passage was usedNone
Per-request costMore input tokens and a search callOften lower: shorter prompts
Upfront costIndex, pipeline, retrieval evalsLabelled examples and GPU time
Main riskRetrieval misses; prompt injection in documentsHallucinating trained-in facts

What each one costs, roughly

Retrieval's cost is per request: every call carries extra tokens. Here's an illustrative calculation; the price is an assumption, not a quote:

Illustrative assumptions (swap in your own prices):
  requests per month          100,000
  retrieved context per call  4 chunks × 500 tokens = 2,000 extra input tokens
  input price                 $0.20 per million tokens (a small hosted model)

extra tokens per month  = 100,000 × 2,000     = 200 million
extra cost per month    = 200 × $0.20         = $40
plus: an embedding index, a search service, and someone to keep both current

A LoRA run's GPU cost is small and one-off. Using the method from what it costs to fine-tune an 8B model:

LoRA on an 8B model, 5,000 examples × 500 tokens × 2 epochs = 5M tokens
compute  ≈ 6 × 8.2B × 5M          = 2.5 × 10¹⁷ FLOPs
A100 at 35% of 312 TFLOPS         ≈ 38 minutes of GPU time
at $2.50/hour plus setup time     ≈ $2 per training run
plus: hosting the adapter, and retraining when behaviour needs to change

Neither number is the real cost. For retrieval, it's keeping the index fresh and evaluating retrieval quality. For fine-tuning, it's building a good dataset, serving the adapter, and retraining whenever the behaviour you want changes. Both need a test set to prove they helped.

Frequently asked questions

What is the difference between fine-tuning and RAG?
RAG retrieves relevant passages and puts them in the prompt on each request, so it changes what the model can see and can be updated by re-indexing. Fine-tuning trains the model's weights on examples, so it changes how the model behaves on every request.
Is fine-tuning or RAG better for adding company knowledge?
RAG. Ovadia et al. found retrieval consistently beat unsupervised fine-tuning for knowledge injection, and Gekhman et al. found fine-tuning on new facts increases hallucination. Use fine-tuning for behaviour, format and style.
When should I fine-tune instead of using RAG?
When the failure is behaviour rather than knowledge: wrong output format, wrong tool or arguments, wrong tone or verbosity, after better instructions and a different model haven't fixed it.
Can you use fine-tuning and RAG together?
Yes. RAFT fine-tunes a model on questions with retrieved documents, including irrelevant distractors, and trains it to quote the relevant passage; it improved domain-specific RAG on PubMed, HotpotQA and Gorilla.
Do I need RAG if my documents fit in the context window?
Not necessarily. Anthropic suggests that a knowledge base under about 200,000 tokens can be included in the prompt whole, using prompt caching to reduce cost and latency.

Go deeper in the Academy

Free lessons with quizzes, citations and a hands-on exercise.

Sources

  1. [1]Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Lewis et al., arXiv (2020).
  2. [2]Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs. Ovadia et al., arXiv (2023).
  3. [3]Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?. Gekhman et al., arXiv (2024).
  4. [4]LIMA: Less Is More for Alignment. Zhou et al., arXiv (2023).
  5. [5]Lost in the Middle: How Language Models Use Long Contexts. Liu et al., arXiv (2023).
  6. [6]Introducing Contextual Retrieval. Anthropic (19 September 2024).
  7. [7]RAFT: Adapting Language Model to Domain Specific RAG. Zhang et al., arXiv (2024).

Put it into practice.

Kojen gives you the Forge, LLM-judged evaluations, red-team simulations and live endpoints in one place. Free to start, no card needed, then pay as you go.

Start training free

More reading

All articles →