RAG vs Fine-Tuning vs Prompting

An engineer asks a release assistant, "Can I deploy this routine change on Friday at 17:00?" It answers yes. The old release policy, v11, allowed routine deployments until 18:00; the new policy, v12, requires an approved exception after 16:00.

A better prompt, a document search system, and a fine-tuned model could each change that answer. Only some would fix the reason it was wrong.

Call the assistant Atlas. Its policies and examples here are fictional. The engineering problem is familiar: separate a failure to obtain the right information from a failure to use it, without treating either as permission to automate a sensitive action.

Three ways to change the same answer

Prompt engineering changes the instructions and examples in a model's input. Retrieval-augmented generation (RAG) finds external evidence and supplies it with the question. Fine-tuning trains some or all of a model's parameters, its learned numerical weights, on additional data. The result may be a full model release or an adapter, a smaller learned update loaded with the base model.

They aren't mutually exclusive products. A retrieval system still needs to tell its generator what to do with the passages. A fine-tuned generator can read those same passages. The distinction is where the intervention enters the system:

The distinction also predicts what happens when policy changes:

Nothing in this table guarantees a correct answer. Even an excellent model can't cite a private policy it never received. Conversely, retrieving the right sentence doesn't guarantee that the model applies its exception correctly.

Try the correct passage before building retrieval

Open one failed request and inspect the text the generator actually saw. A search log saying "policy found" isn't enough: the result might contain an old revision, omit the exception, or have been dropped while assembling the request.

For the Friday question, replace the retrieved material with a human-checked excerpt. Keep the model, question, and instruction fixed. This is an oracle-evidence test: temporarily remove the search problem so you can test whether the generator can use the right input. It isn't a deployable solution or an expected production accuracy score.

The supplied excerpt could be:

[release-policy@v12, section 3] Routine deployments after Friday 16:00 require an approved exception. An exception is valid only when recorded in the release system. All times use the release calendar's configured time zone.

Assume the request identifies that time zone and the release system reports no approved exception. A supported answer is "This routine change requires an approved exception under v12," with the source attached. It isn't "never deploy on Friday," and it isn't "17:00 is before the old 18:00 cutoff."

If the answer becomes correct with that excerpt, examine ingestion, revision filtering, search, and context assembly. If it stays wrong, more search results won't necessarily help. The generator may need a clearer task, better examples, a more capable model, or a different workflow.

Repeat this comparison over representative cases, not just the one that inspired it. Include an allowed deployment, an exception-required deployment, an unknown approval state, and a question whose policy is unavailable. Supplying perfect evidence changes more than retrieval recall: it may remove distracting passages too. A large improvement identifies the evidence path as worth investigating; it doesn't identify the faulty component by itself.

Key insight: A retrieval-backed assistant can fail with the right source in its index. Check what reached the generator, then test what happens when you replace it with the exact evidence the answer needs.

Prompting can clarify a decision, but it can't authorize it

Suppose Atlas has v12 and still answers "allowed" whenever a policy mentions an exception. The instruction "Answer deployment questions" hasn't made the distinction between a possible exception and a verified one explicit.

A revised instruction can ask the model to separate the policy requirement, the known approval state, and any missing information. A few contrasting examples can show the intended pattern without changing model weights. Brown and colleagues' few-shot experiments used task demonstrations in the input without gradient updates at inference.^{[1]Reference 1Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165}

For our case, a useful output object might look like this. It's an example response, not an approval token:

{
  "recommendation": "exception_required",
  "policy_source": "release-policy@v12#section-3",
  "approval_state": "not_approved",
  "explanation": "The requested time is after the routine-deployment cutoff."
}

Where supported, schema-constrained generation can enforce the object's shape. It can't prove that exception_required is the correct value. OpenAI's structured-output documentation explicitly notes that schema-conforming responses can still contain mistakes.^{[2]Reference 2Structured outputshttps://developers.openai.com/api/docs/guides/structured-outputs} Check allowed fields and types in code, then separately evaluate the meaning of the response.

Actual deployment authorization belongs to the release service. It must check identity, current policy, and approval state independently of whatever Atlas says. If the policy is already represented as a complete set of structured rules, evaluate those rules in code and use the model to explain the result. A language model is more useful for interpreting a question or synthesizing prose than for replacing an existing deterministic policy engine.

For the narrow Friday rule, we can write the expected recommendations without a model. The function below assumes the caller has already converted Friday's requested time into the release calendar's time zone and validated the applicable policy. It doesn't implement identity checks, the full release policy, or deployment execution. Its purpose is to turn the four cases into a reference fixture for evaluating Atlas.

from typing import Literal

Approval = Literal["none", "approved", "unknown"]

def friday_recommendation(cutoff: int, requested: int, approval: Approval) -> str:
    if not 0 <= cutoff <= 24 * 60 or not 0 <= requested < 24 * 60:
        raise ValueError("times must be minutes within the release calendar day")
    if approval not in {"none", "approved", "unknown"}:
        raise ValueError("approval must be none, approved, or unknown")
    if requested <= cutoff:
        return "within_routine_window"
    if approval == "approved":
        return "exception_recorded"
    if approval == "unknown":
        return "check_approval_state"
    return "exception_required"

cases = [
    ("v11", 18 * 60, "none", "within_routine_window"),
    ("v12", 16 * 60, "none", "exception_required"),
    ("v12", 16 * 60, "approved", "exception_recorded"),
    ("v12", 16 * 60, "unknown", "check_approval_state"),
]
for revision, cutoff, approval, expected in cases:
    actual = friday_recommendation(cutoff, 17 * 60, approval)
    assert actual == expected
    print(f"{revision}, approval={approval}: {actual}")
assert friday_recommendation(16 * 60, 16 * 60, "none") == "within_routine_window"

v11, approval=none: within_routine_window
v12, approval=none: exception_required
v12, approval=approved: exception_recorded
v12, approval=unknown: check_approval_state

An unknown approval state produces a request to check, rather than a guessed denial or approval. None of these strings authorize a deployment. If policy authors intend the cutoff itself to be excluded, change that boundary and its assertion together; our fictional clause says after 16:00.

Prompting is also a reasonable first implementation for document questions when the application can supply the entire relevant document directly. A short, permitted release manual may not need an index. You still need to select the current revision and test whether the model uses its contents correctly. A large context window changes how much input fits; it doesn't decide which version applies.

If the prompt grows into dozens of examples, compare its quality, input cost, and maintenance effort with a training experiment. There's no requirement to exhaust every possible wording change before considering fine-tuning.

RAG makes evidence replaceable

Now suppose the oracle-evidence test works, but Atlas normally receives v11. RAG gives you an external place to update that knowledge without training new generator weights. Lewis and colleagues' original RAG work combined a language generator with retrieval from an external document index; their research system also trained model components, so "RAG always means a frozen model" would be too restrictive.^{[3]Reference 3Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.https://arxiv.org/abs/2005.11401}

In a common application implementation, document ingestion extracts passages and stores their text with source identifiers, revisions, effective dates, and access metadata. Search finds candidates for a question; a reranker can then compare candidates more closely before the generator receives a selected set. Exact-term search, meaning-based search, or a combination may fit the corpus. A vector database isn't the definition of RAG.

Atlas's update must do more than add another document. If v11 and v12 both match "Friday deployment," the system needs a rule for choosing the effective version. Keep historical versions when audit questions need them, but don't let a current-policy request silently retrieve an obsolete clause.

Permissions need their own enforcement. Restrict candidates to sources the authenticated caller may read, and recheck access when fetching passages if permissions can change after indexing. Retrieved text should also remain evidence, not a source of new tool permissions. A document that asks Atlas to upload a repository can't authorize that action.

An index update can still be stale, incomplete, or incorrectly filtered. Measure whether each test question receives its required passage and whether each answer's citation actually supports its conclusion. A source identifier is only useful if it resolves to the text and revision that the model used. Production RAG Pipelines develops these ingestion and request-time responsibilities in detail.

Not every missing fact belongs in document search. The policy explains what counts as an exception; the release system knows whether this particular change has one. Query that system directly. Copying approval records into a document index would introduce a freshness problem for a fact already available from an authoritative API.

Fine-tuning changes what examples teach

Suppose the checked passages, explicit instructions, and contrasting examples still leave a recurring problem: Atlas confuses "an exception can be approved" with "this exception is approved." You have enough reviewed cases to describe the error consistently. Fine-tuning becomes a testable candidate, alongside changing the base model or splitting interpretation from verification.

With supervised fine-tuning (SFT), the training data contains inputs and desired outputs. Optimization changes trainable parameters to increase the likelihood of the target responses.^{[4]Reference 4TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer} At serving time, those changes persist in a released model or adapter. This differs from copying examples into every request.

For Atlas, a training input should resemble a real request: the question, applicable policy excerpt, and known approval state. The target should demonstrate how to use those inputs. Curating only pairs of questions and answers such as "Friday at 17:00? No" teaches a shortcut that becomes wrong when the policy changes.

Contrast the examples you would want the dataset to distinguish:

These are hypothetical training cases, not measured model results. They make the policy and approval state vary while the question stays almost identical. Your evaluation set needs independently held-out examples, including different wording and policy revisions, to test whether the model learned the relationship rather than memorized rows.

Choosing the training objective is separate from choosing which parameters to update. SFT describes learning from target responses. LoRA, or Low-Rank Adaptation, freezes the base weights and learns smaller update matrices.^{[5]Reference 5LoRA: Low-Rank Adaptation of Large Language Models.https://arxiv.org/abs/2106.09685} It can be a way to implement the SFT experiment; it isn't a competing answer to whether the system needs retrieval. LoRA covers the parameter and memory arithmetic once training is justified.

The common advice "retrieve facts, tune behavior" is a useful starting hypothesis, but facts can be learned in weights and retrieval can carry behavioral examples. Ovadia and colleagues compared RAG with unsupervised fine-tuning for knowledge injection and found RAG stronger on their evaluated tasks.^{[6]Reference 6Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMshttps://aclanthology.org/2024.emnlp-main.15/} That result doesn't settle every supervised task or every hybrid. For Atlas, the stronger reason to keep the policy external is operational: reviewers need an inspectable revision, and the next cutoff change should not require a model-training release.

Common mistake: Training on an internal policy doesn't create document-level access controls. If users have different reading permissions, don't assume the resulting model will enforce those differences. Keep authorization outside the model and assess what the training data could expose.

Compare complete systems, then remove a component

A tuning run might improve Atlas, or it might conceal weak retrieval by memorizing old policies. A better retriever might fix the original questions while adding latency and irrelevant text to easy ones. Compare candidates on the same held-out questions, with the source revisions and caller permissions recorded for each case.

A useful experiment separates the evidence choice from the generator choice:

Keep the prompt and output schema fixed for this controlled comparison, and train with compatible inputs. You can optimize each candidate later, but then report those extra changes. Don't compare a tuned model with a carefully revised prompt against a base model with an old, weaker one and attribute the entire gain to training.

Score correct recommendations, appropriate requests for missing information, and citation support separately. Also inspect permission-boundary tests and whether any forbidden action executed. A model-based quality score can't replace an application authorization check.

If D improves on B, compare D with C to see what retrieval problems remain. If C improves but D doesn't, training didn't repair the deployed evidence path. If B already meets the quality target, adding tuning needs another measurable benefit, such as comparable quality from a cheaper serving configuration. Removing or swapping a component like this is an ablation: it tests what that component contributes.

The one-point gain could also hide many regressions. Here's one possible pairing behind those B and D totals. Each integer identifies the same held-out case in both runs. The sets record which cases received correct recommendations; they are fictional results, not outputs from a model benchmark.

base_correct = set(range(70))
tuned_correct = set(range(9, 80))
fixed = tuned_correct - base_correct
regressed = base_correct - tuned_correct

assert (len(base_correct), len(tuned_correct)) == (70, 71)
assert (len(fixed), len(regressed)) == (10, 9)
print(f"Correct recommendations: {len(base_correct)} -> {len(tuned_correct)}")
print(f"Fixed: {len(fixed)}; regressed: {len(regressed)}")

Correct recommendations: 70 -> 71
Fixed: 10; regressed: 9

Inspect the nine regressions before deciding whether to keep tuning. If they turn unknown approval states into confident recommendations, the extra correct answer doesn't compensate for that failure. Join real outputs by case identifier, retain missing or failed requests in the denominator, and compare the same source and approval snapshots. A changed policy or approval record would otherwise masquerade as a model regression. These paired records support an uncertainty analysis; the set arithmetic alone doesn't establish a reliable improvement.

Count total operating cost, not just model tokens or training GPU hours. Prompt examples consume request space; RAG adds ingestion, search, freshness monitoring, and access-control work; fine-tuning adds dataset review, training, evaluation, and artifact releases. Each candidate also needs a rollback path. For a hybrid, record the prompt, model, and index versions together so a regression can be reproduced.

Atlas doesn't need a permanent winner among three techniques. It needs current evidence, correct interpretation, and an independently enforced deployment boundary. The oracle-evidence test tells you where to investigate; the controlled comparison tells you whether the added machinery earned its place. When the evidence path is weak, continue with RAG Evaluation. When a training experiment is justified, Supervised Fine-Tuning covers the data and release process.