The LLM Lifecycle

The stale-key app for account acct_10234 (key id K10234 in policy and training fixtures) validated input, called a model boundary, stored evidence, and rendered a result under key-rotation-policy line P-7. One question was deliberately left open: where did the model behind that boundary come from?

Prereq debt from the first app: the local /rotation/decide lab still has no product-level caller principal. You are not done for public exposure until the gateway section below authenticates callers, scopes client keys, and isolates provider pools.

The answer starts with two connected loops in a large language model (LLM) system. A model-building loop changes weights through pre-training and post-training. A production loop serves frozen weights with prompts, retrieved evidence, traces, evals, and releases. Good debugging starts by knowing which loop owns the failure.

Deployment bundle: A deployed assistant isn't just a trained model. It's a checkpoint plus current context, serving code, evaluation, and feedback.

Two loops, one key decision

The requester sees one answer: "Eligible under P-7: stale key is at least 30 days old." Under that answer are several distinct engineering jobs.

Recipes differ. Some checkpoints receive several post-training passes; some products call a hosted model and never touch training. This map still matters because it separates changing the model from changing what the model sees or how the product runs it.

Stage 1: pre-training learns prediction

Before a model can answer a key-rotation question, it must learn patterns in language and code. Pre-training presents token sequences and asks the network to predict each next token. Its parameters are updated to assign more probability to continuations seen in useful data.

The tiny example below doesn't train a neural network. It exposes the signal a neural language model receives: after a context token, which continuation tends to occur?

Seeing frequent transitions isn't enough. Training needs a number that becomes smaller when the model assigns higher probability to the observed next token. Cross-entropy loss supplies that number: for the correct next token with probability p, loss is -log(p).

At real scale, researchers choose a model size, data mixture, token budget, and compute budget together. Kaplan et al. measured empirical power-law relationships between language-model loss, model size, dataset size, and compute.^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} Hoffmann et al. later showed that, under their compute budgets and experiments, training a smaller model on substantially more data could outperform a much larger undertrained model.^{[2]Reference 2Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} These papers don't prescribe every modern training run, but they explain why dataset and compute planning are first-class work.

What the base model still lacks

A base model can continue text impressively and still be a poor assistant. Given this prompt:

it may continue with another plausible request rather than produce a structured eligibility decision. It learned continuation, not your API contract, refusal policy, or evidence requirements.

Stage 2: SFT teaches the interface

Supervised fine-tuning (SFT) continues training on demonstrations: an instruction is paired with the answer the model should produce. The model still predicts tokens, but now the useful continuation is an assistant answer shaped for a task.

For the key-rotation app, one demonstration might require a JSON decision with cited evidence rather than a free-form requester-support answer.

SFT data should make desired behavior measurable. If the application requires eligible, policy_id, and evidence, a demonstration missing those fields teaches the wrong interface.

InstructGPT is a concrete published example of this progression: supervised demonstrations followed by preference data and reinforcement learning from human feedback (RLHF) improved instruction-following behavior relative to the underlying GPT-3 models in its evaluation setting.^{[3]Reference 3Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

SFT doesn't answer every product question

SFT can teach the model to cite policy. It can't guarantee that P-7 is today's policy. If the rotation threshold changes tomorrow, fresh retrieved policy text is usually a better first fix than training a new checkpoint.

Stage 3: post-training chooses better behavior

After SFT, multiple answers may be valid JSON and still differ in usefulness. One cites the exact policy clause; another guesses a rotation action. Post-training supplies a signal that favors the better behavior.

Two families matter for this map:

In InstructGPT's RLHF stage, a learned reward model represented human preferences inside a reinforcement-learning process.^{[3]Reference 3Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} DPO instead optimizes preference pairs with a classification loss, without fitting a separate reward model or running an RL optimization loop.^{[4]Reference 4Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} DeepSeek-R1-Zero is the paper's pure-reinforcement-learning experiment without preliminary SFT. The final DeepSeek-R1 used a multistage recipe: cold-start supervised data, reasoning-oriented RL, rejection-sampling and SFT, then another RL stage.^{[5]Reference 5DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} These are useful patterns to study, not default choices for every product.

The preference-data idea fits ordinary Python objects. This code builds a comparison record; an actual DPO or RLHF pipeline would convert many such records into parameter updates.

Verifiable rewards apply only when a checker represents the desired behavior. For this narrow decision, a checker can compare the output with trusted key facts, require policy evidence, and forbid a side effect that the model never executed.

That verifier is useful only for the part it checks. Here, owner_confirmed is a trusted fixture; free-form requester text couldn't set it safely. The checker says nothing about empathy, ambiguous ownership evidence, or whether policy P-7 is fair. A weak checker can train a model toward shortcuts, so every verifier needs adversarial tests and human-reviewed cases.

Stage 4: inference runs frozen weights

Once a checkpoint is deployed, inference is the work of generating responses for requests. Normal inference reads weights; it doesn't update them. Behavior can still change through retrieved evidence, prompt instructions, tool permissions, or decoding and serving settings.

The key-rotation assistant needs today's policy, so the application retrieves P-7 at request time and combines it with trusted key facts. The miniature function below stands in for the model boundary: it demonstrates that new evidence changes the served answer while the checkpoint identifier stays fixed.

Retrieval-Augmented Generation (RAG) was introduced as a model architecture combining generated text with retrieved external knowledge.^{[6]Reference 6Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.https://arxiv.org/abs/2005.11401} In products, the practical rule is simple: facts that change often should arrive as request-time evidence, not stale parametric memory.

Memory and latency become visible

Serving is constrained by GPU memory and latency. Weights alone require roughly weight bytes = parameter count * bytes per parameter.

An eight-billion-parameter checkpoint stored at two bytes per parameter needs about 14.9 GiB for weights alone. It still needs memory for the key-value (KV) cache, runtime buffers, and concurrent requests.

The four-bit result is an ideal packed-weight estimate, not a deployment promise. Real quantized formats can add metadata and higher-precision components. Measure the loaded model footprint before choosing hardware.

More context can also hurt quality as well as memory. Liu et al. observed in their long-context evaluations that models could perform worse when relevant information appeared in the middle of long inputs.^{[7]Reference 7Lost in the Middle: How Language Models Use Long Contextshttps://arxiv.org/abs/2307.03172} Dumping every policy revision into one huge prompt isn't a substitute for retrieval and evaluation.

Evaluation closes both loops

The model-building team evaluates checkpoints before promotion. The product team evaluates prompts, retrieval, serving, and releases after integration. Both need cases that look like real work.

For K10234, a golden case isn't "does the answer sound friendly?" It includes trusted key facts, a policy fact, a required decision, required evidence, and forbidden claims.

Open-ended response quality often needs a rubric rather than exact string matching. LLM judges can help scale rubric-based comparisons, and Zheng et al. studied their agreement and biases in MT-Bench and Chatbot Arena.^{[8]Reference 8Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} They aren't a substitute for calibrated human-reviewed cases, especially for policy and safety boundaries.

This small evaluator mixes deterministic policy checks with a human-review queue for answers that pass the rule but may still be confusing.

A failure should choose an owner

When an eval fails, "fine-tune the model" is rarely a complete diagnosis. First ask what must change.

Release gate can block a known regression before long-term fix ships.

Traces turn failures into concrete work items. A stored trace can already tell you whether to refresh evidence, revise a prompt, add preference examples, or optimize serving.

Keep the boundaries straight

Lifecycle gets simpler once you ask a precise question about each symptom.

What not to conclude

• Preference tuning isn't a reliable database update. It may shape how a model responds, but current policy facts belong in retrievable evidence.
• A verifier isn't automatically a truth oracle. It rewards only rules it can observe.
• Long context isn't guaranteed recall. Relevant evidence still needs selection and testing.
• A product incident isn't automatically a training incident. Most deployed fixes start with evidence, contracts, gates, or serving.

Secure the gateway before model work begins

The production loop starts before a prompt reaches the model. A gateway often authenticates callers, selects an upstream credential, and retries or fails over after a provider error. Those steps need their own security contract:

The next fixture exercises all four rules. It doesn't implement a complete identity service or provider retry library. Its raw token rows keep parameter binding visible; a production store should keep a one-way token verifier, and the server route should select required_scope. The fixture makes release invariants executable: a crafted token can't bypass authentication, expired or wrong-scope credentials stop before routing, and a credential-scoped production limit fails over without borrowing a staging credential. Production adapters must classify the provider's error scope. An account- or provider-wide 429 should not churn through sibling keys.

A flat shared provider key would erase the final isolation property: a staging loop could consume the same quota production needs. Scoped client keys and disjoint upstream pools solve different problems, so keep both in the deployment manifest. Also record circuit transitions by credential ID, never by secret value, and set a cooldown or probe policy before closing an open circuit.

Mastery check

Key concepts

• Pre-training predicts tokens and changes model weights.
• SFT teaches instruction behavior from demonstrations.
• Preference methods and verifiable rewards provide different post-training signals.
• Retrieval supplies current evidence without changing model weights.
• Inference exposes memory, latency, throughput, and reliability constraints.
• Evaluation connects production failures to the layer that owns the fix.

Evaluation rubric

• Foundational: Explains why a base next-token predictor may continue a key-rotation request instead of answering it.
• Foundational: Marks pre-training, SFT, and post-training as weight-updating stages, while separating retrieval and inference.
• Intermediate: Chooses SFT, preference data, a verifier, or retrieval for a concrete K10234 failure and states why.
• Intermediate: Calculates weight-only memory for a checkpoint and names the omitted serving memory.
• Intermediate: Designs a held-out policy case and a release gate that catches unsupported side-effect claims.

Follow-up questions

Common pitfalls

Treating post-training as fact storage

• Symptom: A team creates preference pairs to teach tomorrow's policy revision.

• Cause: It confuses preferred behavior with current evidence.

• Fix: Retrieve versioned policy text at request time and evaluate grounded decisions against it.

Rewarding a shortcut

• Symptom: Verifier scores rise while answers become unhelpful or game a formatting rule.

• Cause: Programmatic reward observes only a narrow proxy.

• Fix: Add adversarial verifier tests, held-out cases, and human-reviewed rubric samples.

Shipping without a product invariant

• Symptom: Assistant produces fluent output that claims a rotation job was executed.

• Cause: Eval set checks wording but not side effects.

• Fix: Gate release on deterministic constraints tied to recorded tool actions and policy evidence.

Blaming weights for serving failures

• Symptom: Latency spike triggers a request for another tuning run.

• Cause: Team hasn't separated inference capacity from model behavior.

• Fix: Inspect queue time, time to first token, KV-cache pressure, and trace errors before considering training.

Trusting a long prompt to retrieve itself

• Symptom: Correct policy is somewhere in a large context, but answer cites the wrong rule.

• Cause: Context availability isn't the same as reliable use of evidence.

• Fix: Retrieve focused policy passages, place evidence near the answer, and evaluate citation accuracy on held-out cases.