RAG or Fine-Tuning: How to Choose for LLM Apps

Teams building on large language models keep asking a version of the question that has no answer: should we use retrieval or fine-tuning? The two techniques are not rival solutions to one problem. Retrieval changes what a model sees at request time; fine-tuning changes what the model is. The published research on which one fixes which failure is considerably clearer than most vendor blogs let on.

What each technique actually changes

Retrieval-augmented generation was named and formalised in a 2020 NeurIPS paper by Patrick Lewis and colleagues at what was then Facebook AI Research. Their RAG paper pairs a pre-trained sequence-to-sequence model with a dense vector index of Wikipedia reached through a neural retriever, and tests two variants: one that conditions on the same retrieved passages throughout generation, another that can swap passages per token. The authors reported state-of-the-art results on three open-domain question answering tasks and generation that was, in their words, “more specific, diverse and factual” than a parametric-only baseline.

Fine-tuning updates weights. The technique most startups actually use is low-rank adaptation, introduced in 2021 by Edward Hu and co-authors at Microsoft. LoRA freezes the pre-trained weights and injects small trainable rank-decomposition matrices into each transformer layer. Compared with fine-tuning GPT-3 175B using Adam, the paper claims a 10,000-fold reduction in trainable parameters and a threefold reduction in GPU memory, with no additional inference latency, unlike adapter approaches. That last point matters operationally.

That structural difference drives everything else. A retrieval system carries its knowledge outside the model, in rows you can add, edit and delete. A fine-tuned model carries whatever you taught it inside the weights, where you cannot see it, cite it or remove it.

The evidence says retrieval wins on facts

The cleanest head-to-head comparison is Oded Ovadia and colleagues’ “Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs”, published at EMNLP 2024. They tested unsupervised fine-tuning against RAG on knowledge-intensive tasks and found that RAG “consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge.” Their blunter conclusion: models struggle to learn new factual information through unsupervised fine-tuning at all.

That result lines up with the operational reality of shipping anything to paying customers. If your product answers questions about a document set that changes weekly, retrieval is the only approach where “update the knowledge” is a re-index rather than a training run. If you serve multiple tenants, retrieval is the only approach where tenant isolation is a filter on a query rather than a separate model per customer. And if a customer exercises a deletion right, deleting rows is a defensible answer where retraining is not.

Where fine-tuning earns its keep

Fine-tuning is good at behaviour, not knowledge. OpenAI’s own model optimization guide puts it in that order (build evaluations, write good prompts, then fine-tune) and lists what each method suits. Supervised fine-tuning is pitched at classification, nuanced translation, format-specific generation and correcting instruction-following failures. Direct preference optimization is pitched at summarisation and at getting tone and style right. Reinforcement fine-tuning is pitched at complex domain-specific reasoning, with medical diagnosis and legal case analysis as the worked examples.

There is a second, underrated payoff: prompt shrinkage. A 4,000-token instruction block that you send on every request can often be moved into the weights, which changes your unit economics rather than your accuracy.

The two are additive

A Microsoft Research team led by Angels Balaguer built both pipelines over agricultural data and measured them separately and together. Their 2024 case study reports “accuracy increase of over 6 p.p. when fine-tuning the model and this is cumulative with RAG, which increases accuracy by 5 p.p. further,” alongside an improvement in answer similarity from 47% to 72% across geographic contexts. That is the most useful single finding for a product team: the question is rarely which, and often what order.

Cost, latency and operational burden

Run the retrieval arithmetic before you architect around it. Eight retrieved passages of 500 tokens each add 4,000 input tokens to every request. At 40,000 requests a month and a $2 per million input token rate, that single design choice is $320 a month, which is trivial. At four million requests a month it is $32,000, and the case for fine-tuning a shorter prompt starts to write itself. Prompt caching changes the answer again: Anthropic’s caching documentation prices cache reads at 0.1x the base input rate for most models, so a stable retrieved prefix is an order of magnitude cheaper than a fresh one.

The hybrid approach worth copying

If you are going to do both, the interesting technique is retrieval-augmented fine-tuning. RAFT, from Tianjun Zhang and colleagues at Berkeley, trains the model on retrieved document sets that deliberately include distractor documents, and requires chain-of-thought answers that cite exact passages. The model learns to ignore irrelevant retrievals rather than dutifully summarising them, which is the single most common RAG failure in production. The authors evaluated on PubMed, HotpotQA and Gorilla.

The practical sequencing that follows: build retrieval first, log real traffic including the retrieved context, then fine-tune on those retrieval-conditioned traces. You end up teaching the model your task in the exact input distribution it will see at serving time.

Evaluate before you commit engineering time

1. Collect 100 to 300 real queries with human-graded answers before touching either technique. Without this you cannot tell an improvement from a mood.
2. Separate retrieval failures from generation failures. The open-source Ragas metric set splits neatly along that line: context precision and context recall grade the retriever, while faithfulness, response relevancy and noise sensitivity grade the generator.
3. Baseline with the best prompt you can write and no retrieval. A surprising share of “we need fine-tuning” turns out to be “we need a better prompt and a smaller output schema.”
4. Add retrieval and look at context recall first. If the answer is not in the retrieved passages, no amount of fine-tuning will conjure it.
5. Only then fine-tune, holding the evaluation set fixed. Grade format compliance and tool-call validity separately from factual accuracy, because that is where the gains should show up.
6. Re-run the whole suite every time you change base models. Fine-tunes do not travel across model versions; prompts and indexes mostly do.

A decision rule you can apply on Monday

Look at your last fifty bad outputs and sort them. Wrong facts, missing recent information, or answers that should have cited a document mean you have a retrieval problem. Wrong shape, wrong tone, malformed tool calls or ignored instructions mean you have a fine-tuning problem, or a prompt problem you have not exhausted yet. Wrong reasoning inside a narrow domain where correctness is machine-checkable is the one case where reinforcement fine-tuning is genuinely the right tool. And if the complaint is latency or the bill rather than quality, neither technique is your answer: caching, a smaller model and a shorter retrieved context are.

Sources

• arXiv — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., NeurIPS 2020)
• arXiv — LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
• arXiv — Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (Ovadia et al., EMNLP 2024)
• arXiv — RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture (Balaguer et al., Microsoft Research)
• arXiv — RAFT: Adapting Language Model to Domain Specific RAG (Zhang et al., 2024)
• OpenAI — Model optimization guide
• Anthropic — Prompt caching documentation
• Ragas — List of available metrics