RAG vs Fine-Tuning vs Prompting

Your company is adding an AI workflow to a real product. The model can already answer general questions, but the product needs more than general fluency: it needs the right facts, the right format, the right tone, and reliable behavior under messy user requests.

That's where the first architecture decision appears. Do you write better prompts, build retrieval, or fine-tune on your data? The clean rule: prompts steer the current request, retrieval supplies fresh evidence, and fine-tuning changes stable behavior.

The three paths to LLM customization

Three approaches can make an LLM better at your task:

These approaches aren't mutually exclusive. A fine-tuned model can still use retrieval, and a RAG pipeline still needs good prompts. The useful question is sequence: which measured failure should you fix next?

Sequence rule: Establish a prompt-only baseline first unless the task already requires private, changing, or citation-sensitive evidence.

How each approach works

Prompt engineering

You write a system prompt, add examples when needed^{[3]Reference 3Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165}, and make the output contract clear. For reasoning-heavy tasks, chain-of-thought style prompting can improve performance, though production systems usually hide or summarize the reasoning trace rather than exposing it directly.^{[4]Reference 4Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.https://arxiv.org/abs/2201.11903} Mechanically, the model's weights stay frozen. Adaptation lives in the input tokens.

In standard dense-attention Transformers, self-attention scales as O(N²) with respect to sequence length N^{[5]Reference 5Attention Is All You Need.https://arxiv.org/abs/1706.03762}, so giant prompts increase latency and token spend. Long prompts also have their own failure mode: research shows models can miss facts buried in the middle of context^{[6]Reference 6Lost in the Middle: How Language Models Use Long Contextshttps://arxiv.org/abs/2307.03172}. Some frontier APIs now offer 1M-token-class windows^{[7]Reference 71M context is now generally available for Opus 4.6 and Sonnet 4.6https://claude.com/blog/1m-context-ga}, but "paste the corpus" still doesn't give you indexing, access control, source attribution, or selective retrieval.

The useful pattern is simple: keep stable instructions small, include only examples that change behavior, and put the current input where the model can find it. Long static prefixes can sometimes be amortized with provider-side prompt caching^{[8]Reference 8Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching}, but caching doesn't add citations, permissions, or retrieval semantics.

Retrieval-Augmented Generation (RAG)

You build a pipeline that finds relevant documents and injects them into the prompt context. During indexing, you chunk documents, embed them, and write vectors plus metadata into a search index. At query time, you embed the user question, run approximate nearest-neighbor search, optionally rerank candidates, and inject the top chunks into the prompt.^{[9]Reference 9Retrieval-Augmented Generation for Large Language Models: A Survey.https://arxiv.org/abs/2312.10997[10]Reference 10Efficient and Robust Approximate Nearest Neighbor Using Hierarchical Navigable Small World Graphs.https://arxiv.org/abs/1603.09320[11]Reference 11Billion-scale similarity search with GPUs.https://arxiv.org/abs/1702.08734}

The architecture below shows both time scales in one vector space. Indexing turns chunks into searchable points ahead of time. Each request adds a query vector, selects a small neighborhood, and passes only those chunks into generation:

The model itself is unchanged. You're putting the right information in front of it at query time.^{[9]Reference 9Retrieval-Augmented Generation for Large Language Models: A Survey.https://arxiv.org/abs/2312.10997} That can lower hallucination risk on knowledge-heavy tasks, but only if retrieval recall, ranking, and grounding instructions are good. Bad retrieval still produces bad answers.

Fine-tuning

You train the model on your specific dataset so some learned parameters change. In full fine-tuning that means updating the base weights directly. In PEFT methods such as LoRA, it usually means learning attached adapter weights while the base model stays frozen. Either way, the behavior shift lives in the model stack itself rather than only in the request context.

A common production starting point is Parameter-Efficient Fine-Tuning (PEFT), rather than full fine-tuning. LoRA (Low-Rank Adaptation) freezes the pre-trained weight matrix $W_0$ and learns small adapter matrices:

LoRA arithmetic

W = W0 + ΔW = W0 + BA

For a 4096-by-4096 projection with rank 16, LoRA trains 16 * (4096 + 4096) = 131,072 adapter parameters while the 16.8M frozen base weights stay unchanged. In the original LoRA paper, this cut trainable parameters by orders of magnitude and reduced GPU memory requirements by about 3x relative to full fine-tuning in their experiments^{[2]Reference 2LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685}.

Facts still belong outside weights

Fine-tuning is usually a poor fit for fast-changing factual knowledge. It can help the model speak more fluently in a domain, but once those facts change you need another training cycle, and you still don't get explicit source attribution the way you do with retrieval.

Ovadia et al. tested this directly and found that for knowledge injection, RAG consistently outperformed unsupervised fine-tuning, including on entirely new facts.^{[12]Reference 12Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMshttps://aclanthology.org/2024.emnlp-main.15/} On new-fact suites, adding unsupervised fine-tuning on top of RAG often failed to beat RAG alone and sometimes hurt; on some MMLU-style knowledge subjects, FT+RAG was tied or slightly better than base+RAG. Treat hybrid FT+RAG as an evaluation, not a free upgrade, especially when the fine-tune is unsupervised continued pretraining rather than task SFT. That paper does not rule out supervised task fine-tuning (SFT/LoRA on question-answer or instruction pairs) for format, style, or stable behavior; it rules out unsupervised knowledge injection as a substitute for RAG. Fine-tuning is strongest when teaching behavior, style, or a stable input/output mapping, not facts you expect to update.

Production fine-tuning commonly uses Hugging Face peft and supervised fine-tuning trainers.^{[13]Reference 13PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora[14]Reference 14TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} A full run still needs clean examples, GPU memory, held-out evaluation, and model-specific chat-template handling.

The decision framework

Use this as a first filter, not as an accuracy scoreboard. Prompting is cheapest and fastest when the caller can provide all relevant facts. RAG adds ingestion, indexing, retrieval evaluation, and latency, but it's the strongest default for fresh or private knowledge. Fine-tuning costs more upfront and needs ML discipline, but it can be best for stable style, labels, format, or narrow behavior once the task is well measured.

Evaluate the remaining gap before choosing

The right layer depends on the failure you can measure:

• Prompting: task success rate, schema-valid output rate, and human rubric score.
• RAG: retrieval recall@k, MRR, nDCG, citation correctness, and answer faithfulness.
• Fine-tuning: held-out task accuracy, format adherence, and a regression suite on general skills.

That split tells you whether better instructions solve the task, retrieval missed the evidence, the generator ignored good evidence, or tuning introduced behavior drift.

MRR means Mean Reciprocal Rank and nDCG means normalized Discounted Cumulative Gain. Both ask whether the right chunk appears near the top of the retrieval list rather than somewhere deep in it.

Choosing by measured failure

Start with a prompt-only baseline unless the task needs retrieval or training. Keep improving the prompt when facts are present and the output mostly needs clearer structure. Add RAG when facts are missing, private, changing, or citation-sensitive. Test fine-tuning when evidence is present but format, labels, tone, or policy behavior fail repeatedly. If both facts and stable behavior are missing, use a hybrid: retrieval handles evidence, tuning handles habits.

Long context moved the threshold for when you need retrieval, but it didn't erase RAG. If a small, mostly static document set fits comfortably, long context can be the first version. Advertised capacity still isn't retrieval semantics: facts buried in the middle can be missed^{[6]Reference 6Lost in the Middle: How Language Models Use Long Contextshttps://arxiv.org/abs/2307.03172}, and longer inputs raise cost, latency, and degradation risk.^{[15]Reference 15Context Rot: How Increasing Input Tokens Impacts LLM Performancehttps://research.trychroma.com/context-rot}

Retrieval also became more agentic. Instead of one search-then-generate pass, an agent can decide whether to retrieve, split a question into sub-queries, read results, and search again^{[16]Reference 16Building Effective Agentshttps://www.anthropic.com/research/building-effective-agents}. Use that pattern only when the question needs iteration, because it adds model calls and evaluation complexity.

Frequent mistakes and fixes

Three mistakes cause most bad decisions:

• Fine-tuning to teach new facts gives stale answers after the next release. Use RAG for changing facts and keep fine-tuning data focused on format, tone, and stable reasoning patterns.
• Ignoring retrieval quality gives fluent answers grounded in irrelevant chunks. Measure retrieval recall and precision separately from answer quality, then add reranking, hybrid search, and cleaner chunks.
• Over-tuning can make a model good at the target task while losing general skills. Use PEFT, conservative learning rates, and regression tests on both target and general cases.

The hybrid playbook

Production systems often combine layers. The default for knowledge-heavy products is RAG plus good prompts: retrieval supplies evidence, and prompting controls output quality and format. Add fine-tuning when the model must follow a stable domain format while still citing fresh evidence. Add routing when volume is high and task difficulty varies.

Cost comparison: operating profile, not fixed pricing

Hard-coded dollar figures go stale fast, so think in terms of where effort shows up. Prompting is cheap until brittle prompts create review work; RAG adds ingestion, retrieval evaluation, and extra context tokens; and fine-tuning can look cheap at inference time while data curation, held-out evaluation, and refresh cycles dominate.

Decision map

Distill the trade-offs into two measured gaps after a prompt baseline. Move right as the model needs more external or changing knowledge. Move up as outputs need more stable learned behavior.

Re-plot after each change. If neither gap remains but cost is too high, optimize routing instead of adding capability.

Evaluation rule: Compare every added layer against the same held-out task set. Keep the simpler path when retrieval or tuning doesn't improve the target metric enough to justify its cost and failure modes.