RAG vs Fine-Tuning vs Prompting

Atlas is an internal policy assistant. A contractor asks, "Can I process a Q3 return after 30 days?" The model says yes, cites nothing, and pastes a paragraph from returns policy POL-441 that this caller isn't allowed to read.

Three repairs now compete for attention: a sharper prompt, retrieval-augmented generation (RAG), or a fine-tune. Before choosing one, ask what the trace actually shows. Was the rule missing, present but misused, or followed inconsistently across similar cases?

Atlas has an owner for each part of that miss. Retrieval can supply current policy text. A prompt can request a JSON decision and an abstention. Application code must authorize the caller before restricted text enters context, then validate any action after generation. Fine-tuning is only relevant if the same behavior still fails once those inputs and boundaries are sound. Prompt the active request, retrieve changing evidence, tune recurring behavior, and keep hard guarantees in code. None of those layers repairs a model that lacks the base capability or replaces a live system-of-record lookup when the answer depends on one.

Diagnose the failed case first

Start with one held-out Atlas question. Record the caller identity, retrieved context, model response, and application decision. Then ask which part of that trace was allowed to fail:

The table gives you a first hypothesis, not permission to add three systems. Score the same held-out cases after each change so the next result can answer one question: which layer moved the failure?

The order blocks expensive category errors. Fine-tuning can't make a stale POL-441 current. RAG can't make an unauthorized document safe. Prompt wording can't turn a probabilistic response into an access-control boundary.

Starting rule: Build the smallest system that can legally and factually satisfy the task. That's often a prompt baseline, but not when the first valid answer already needs external evidence, a live tool, or hard enforcement.

Locate the change before choosing the tool

Ask one concrete question before comparing features: where will this fix live after the request finishes? The answer predicts how you refresh it, test it, and roll it back.

The rows can stack. A RAG system still prompts its generator, a tuned model can consume retrieved passages, and a prompt can call a live API instead of searching documents. The useful choice is the first measured addition, not a permanent allegiance to one technique. Keep the cases and scoring fixed while you add it.

Prompting shapes the active request

Suppose the authorized POL-441 passage is already in context, but Atlas returns prose instead of the fields the application expects. Start with the request.

Prompt engineering changes instructions, examples, ordering, and response constraints while model weights stay fixed. Few-shot examples can teach a pattern inside the current request^{[2]Reference 2Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165}, but they don't update the model after the request ends.

Atlas's first prompt should name the task, mark policy text as evidence rather than instructions, require policy_id and source_version, and say when to abstain. A small JSON contract gives the application something concrete to parse.

Where a provider supports structured outputs, use that surface for schema shape and still validate business semantics in application code.^{[1]Reference 1Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs} Training a model to spell JSON is a poor first fix for a request contract.

That request may include more than instructions. Teams often use context engineering for assembling the prompt, evidence, tools, current state, and relevant history. Prompt design is one part of that working set, not an alternative to it.^{[3]Reference 3Effective context engineering for AI agentshttps://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents}

Now vary the amount of context. As of August 13, 2026, Anthropic's context-window docs list Claude Opus 5, Claude Sonnet 5, and several Claude 4.x routes with a one-million-token window by default.^{[4]Reference 4Context windowshttps://platform.claude.com/docs/en/build-with-claude/context-windows} That can make a small, mostly static corpus practical to send directly. It doesn't add source permissions, selective updates, or retrieval evaluation.

Capacity still isn't the same as reliable use of every token. Anthropic describes a performance gradient in which longer contexts can lose precision for retrieval and long-range reasoning, and labels this failure mode context rot.^{[4]Reference 4Context windowshttps://platform.claude.com/docs/en/build-with-claude/context-windows} Chroma's Context Rot report measured nonuniform degradation across 18 models as input length increased.^{[5]Reference 5Context Rot: How Increasing Input Tokens Impacts LLM Performancehttps://research.trychroma.com/context-rot} The 2024 Lost in the Middle study found that tested models often used information at the beginning or end of context more reliably than information in the middle.^{[6]Reference 6Lost in the Middle: How Language Models Use Long Contextshttps://aclanthology.org/2024.tacl-1.9/} Million-token context windows covers when to load a bounded corpus versus retrieve.

Repeated prefixes can still be cheaper and faster when provider caching applies. Current OpenAI documentation says prompt caching is enabled by default for supported models, while the GPT-5.6 model guide documents explicit reusable-prefix caching.^{[7]Reference 7Prompt cachinghttps://developers.openai.com/api/docs/guides/prompt-caching} Caching reuses prompt computation. It doesn't improve evidence quality or turn a long prompt into a search system.

Keep working on prompt and context when correct facts are already present and failures respond to clearer instructions, better examples, a smaller evidence packet, or stronger output constraints. Prompt optimization with DSPy becomes relevant once prompt variants, examples, and metrics are stable enough to compile and compare.

RAG builds an evidence path

If the authorized POL-441 passage isn't present, no wording can make the model quote it. Retrieval-Augmented Generation supplies that material at inference instead of expecting all task knowledge to live in model parameters.^{[8]Reference 8Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.https://arxiv.org/abs/2005.11401}

A typical document pipeline chunks and indexes sources ahead of time. At request time, it searches permitted candidates, reranks when useful, packs a small evidence set, and asks the model to answer from it.

Atlas's Q3 returns rule is a good retrieval target because POL-441 changes, the source ID matters, and a contractor shouldn't see the employee-only clause. RAG is strongest when answers depend on private documents, frequently revised policies, or citations a reviewer must inspect.

Retrieval also adds failure modes: stale indexes, weak recall, irrelevant ranking, permission leaks, contradictory chunks, prompt injection inside retrieved text, and extra request latency.

Retrieval doesn't grant access control automatically. Filter by trusted identity and policy before restricted text can enter model context, logs, or shared caches. Preserve source IDs and document versions, require an answer or abstention from supplied evidence, and evaluate retrieval separately from generation. Production RAG pipelines develops that full contract.

For a document QA task, measure three separate outcomes: did relevant evidence reach the top results, did the cited text support the answer, and did the system abstain when evidence was missing? Retrieval metrics such as recall@k, mean reciprocal rank (MRR), and normalized discounted cumulative gain (nDCG) cover ranking. Citation correctness and faithfulness cover answer support.

The original RAGAS paper proposed faithfulness, answer relevance, and context relevance as reference-free signals for retrieval-backed generation.^{[9]Reference 9RAGAS: Automated Evaluation of Retrieval Augmented Generation.https://arxiv.org/abs/2309.15217} No single final-answer score can tell you whether the retriever or generator failed. RAG evaluation separates those owners in a release gate.

RAG isn't a default for every external fact. An assistant asking for deployment status, feature flag state, or current incident severity should usually call an authoritative API instead of retrieving a possibly stale document snapshot. Use document retrieval for evidence collections and direct tools for live structured state.

There is a second retrieval boundary: when should the system fetch? Anthropic describes a just-in-time pattern in which the agent keeps identifiers such as paths or queries, then loads a passage at runtime instead of carrying a large pre-retrieved packet through every step.^{[3]Reference 3Effective context engineering for AI agentshttps://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents} That's useful when Atlas should open one cited section. It's extra evaluation work when a single permitted index lookup already has the passage.

Fine-tuning needs two decisions

Now take the opposite miss. The policy passage is present, the JSON keys are valid, and Atlas still emits the wrong escalation_reason shape across many held-out cases.

Training may be the next experiment, but "fine-tuning" still hides two separate choices:

1. Objective: What signal should the model learn from?
2. Update surface: Which parameters may change?

Supervised Fine-Tuning (SFT) learns from curated prompt-response examples. Continued pretraining keeps the next-token objective on raw domain text to adapt a model to the domain's language distribution.^{[10]Reference 10Don't Stop Pretraining: Adapt Language Models to Domains and Tasks.https://aclanthology.org/2020.acl-main.740/} Preference optimization learns from relative choices. Those signals can all change weights, but they don't teach the same behavior.

LoRA (Low-Rank Adaptation), quantized LoRA (QLoRA), and full-weight updates describe the update surface, not the objective. LoRA freezes base weights and learns low-rank adapter matrices^{[11]Reference 11LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685}; it can be used for SFT or another training objective. Current Hugging Face PEFT and TRL documentation expose these choices through LoRA configuration and SFT trainers.^{[12]Reference 12PEFT Documentation: LoRA Developer Guide.https://huggingface.co/docs/peft/main/developer_guides/lora[13]Reference 13TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer}

For a square projection $W_0 \in \mathbb{R}^{4096 \times 4096}$ and LoRA rank $r=16$, the update $\Delta W = BA$ trains:

$16 \times 4096 + 4096 \times 16 = 131{,}072$

adapter parameters for that projection, compared with $4096^2 = 16{,}777{,}216$ parameters in the full matrix. That arithmetic only counts one adapted matrix.

Whole-run memory still includes the frozen base, activations, adapter optimizer state, and every selected target module.

A rank-16 LoRA adapter replaces a 4096 x 4096 update. How many trainable parameters does this example add?

Answer

The two low-rank matrices contain 16 × 4096 + 4096 × 16 = 131,072 parameters. That is far below the 16,777,216-parameter full matrix, but whole-run memory still includes frozen weights, activations, optimizer state, and other target modules.

This script compares trainable parameter savings across rank choices and target projection modules for an 8B-scale architecture. Its State RAM column is a rough 16-byte-per-parameter planning estimate, not a promise about every optimizer or precision setting:

Fine-tuning can encode facts, so "facts never belong in weights" is too strong. The better question is whether weights are the right update and evidence interface. Ovadia and colleagues compared RAG with unsupervised fine-tuning on knowledge-intensive multiple-choice tasks. In their tested settings, RAG beat unsupervised fine-tuning; using a fine-tuned generator inside RAG helped some tasks but not consistently, and the current-events hybrid scored below RAG alone.

The paper focuses on unsupervised training and leaves supervised and reinforcement-based methods for further study.^{[14]Reference 14Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMshttps://aclanthology.org/2024.emnlp-main.15/} That supports RAG as a stronger default for fresh knowledge, not a universal claim that all supervised tuning or hybrids fail.

Test SFT when evidence is present but the same desired behavior keeps failing across a meaningful held-out set. For Atlas, that's the escalation-format miss: POL-441 is in context, the JSON keys are valid, and forty similar cases still emit the wrong escalation_reason shape. Test continued pretraining when raw domain text itself is poorly modeled and you have enough licensed, clean corpus data to measure domain gain against general regression. Neither training path supplies citations, enforces permissions, or removes the need for release evaluation.

⚠️ Common mistake: Fine-tuning Atlas on yesterday's policy PDF to "teach the rules." The next POL-441 revision makes the weights stale, and you still don't get source IDs or permission filters. Keep changing facts in retrieval or tools. Use SFT for a stable mapping you can hold out and re-test.

Run one controlled eval ladder

The three mechanisms describe ownership, but they don't tell you which change will pay off for Atlas. Create held-out cases before changing architecture: easy successes, known failures, missing-evidence cases, permission boundaries, adversarial retrieved text, and general capabilities you can't afford to regress. Keep the cases and scoring rules fixed across candidates.

Then move through layers:

1. Enforce hard boundaries in code. Validate identity, authorization, schema, tool arguments, and side effects outside the model.
2. Build a prompt and context baseline. Supply only the evidence needed for each case, state the answer contract, and log model, prompt, and context versions.
3. Label failures by owner. Separate missing retrieval, ignored evidence, bad instructions, base capability, and recurring behavior.
4. Add one candidate layer. Change retriever, prompt, model, or training artifact without moving every variable at once.
5. Re-run quality and operations checks. Compare task success, regressions, latency, token use, infrastructure cost, data work, and on-call burden.

The table separates evidence from operating cost. Prompt-heavy systems pay in input tokens, latency, and maintenance when instructions become brittle. RAG adds ingestion, storage, retrieval evaluation, and context tokens. Fine-tuning adds a dataset and release lifecycle even when adapter training is affordable. Compare total system cost per accepted task, not a training bill or token price in isolation.

Evaluation rule: Keep the simpler candidate unless the added layer improves the target metric enough to justify its latency, data lifecycle, failure modes, and rollback burden.

A hybrid is a set of owners, not a stack

Atlas may need all three techniques, but each should own a narrow failure:

• Application code authenticates the caller, checks policy scope, validates the final object, and authorizes any action.
• RAG supplies current permitted policy passages with source IDs.
• The prompt tells the generator how to use evidence, cite it, and abstain when it doesn't support the answer.
• An SFT adapter is optional when prompt and evidence are sound but escalation format or classification behavior still fails repeatedly.

The smallest design can change over time. A small static manual might start as long context, then move to RAG when updates, permissions, or corpus size demand indexing. Better embeddings or reranking may fix a domain retriever while the generator remains frozen. If the base model stays weak across prompt and evidence variants, replace it before collecting training data. Hybrid doesn't mean "turn on everything." It means assign each measured failure to one component.

Map remaining Atlas misses

After the prompt baseline and hard application boundaries, place the remaining Atlas misses on two axes. Move right when the answer needs more external or changing evidence. Move up only when a stable behavior gap remains across held-out cases.

Read the points as decisions, not scores. JSON keys stays with the prompt and application boundary. A missing POL-441 passage moves right toward RAG. A recurring escalation habit moves up toward SFT. Only the point with both gaps justifies a hybrid, and even that point needs component ablations before it earns another release.

Revisit the map after every improvement. If quality is acceptable but cost isn't, optimize context packing, caching, model routing, or retrieval stages instead of adding another adaptation method.

When retrieval is the next step, continue to Production RAG Pipelines. When recurring behavior remains, Supervised Fine-Tuning and LoRA separate the training objective from the parameter budget.