The LLM Lifecycle

In the previous chapter, you shipped a small app that decides whether damaged order A10234 qualifies under return-policy line P-7. The app validated input, called a model boundary, stored evidence, and rendered a result. One question was deliberately left open: where did the model behind that boundary come from?

This chapter answers that question. A large language model (LLM) product has two connected loops. A model-building loop changes weights through pre-training and post-training. A product loop serves frozen weights with prompts, retrieved evidence, traces, evals, and releases. Good debugging starts by knowing which loop owns the failure.

Key idea: A deployed assistant isn't just a trained model. It's a checkpoint plus current context, serving code, evaluation, and feedback.

Two loops, one return decision

The customer sees one answer: "Eligible under P-7: damaged item reported within 30 days." Under that answer are several distinct engineering jobs.

Recipes differ. Some checkpoints receive several post-training passes; some products call a hosted model and never touch training. This map still matters because it separates changing the model from changing what the model sees or how the product runs it.

Stage 1: pre-training learns prediction

Before a model can answer a return question, it must learn patterns in language and code. Pre-training presents token sequences and asks the network to predict each next token. Its parameters are updated to assign more probability to continuations seen in useful data.

The tiny example below doesn't train a neural network. It exposes the signal a neural language model receives: after a context token, which continuation tends to occur?

Seeing frequent transitions isn't enough. Training needs a number that becomes smaller when the model assigns higher probability to the observed next token. Cross-entropy loss supplies that number: for the correct next token with probability p, loss is -log(p).

At real scale, researchers choose a model size, data mixture, token budget, and compute budget together. Kaplan et al. measured empirical power-law relationships between language-model loss, model size, dataset size, and compute.^[1] Hoffmann et al. later showed that, under their compute budgets and experiments, training a smaller model on substantially more data could outperform a much larger undertrained model.^[2] These papers don't prescribe every modern training run, but they explain why dataset and compute planning are first-class work.

What the base model still lacks

A base model can continue text impressively and still be a poor assistant. Given this prompt:

it may continue with another plausible request rather than produce a structured eligibility decision. It learned continuation, not your API contract, refusal policy, or evidence requirements.

Stage 2: SFT teaches the interface

Supervised fine-tuning (SFT) continues training on demonstrations: an instruction is paired with the answer the model should produce. The model still predicts tokens, but now the useful continuation is an assistant answer shaped for a task.

For the return app, one demonstration might require a JSON decision with cited evidence rather than free-form customer-support prose.

SFT data should make desired behavior measurable. If the application requires eligible, policy_id, and evidence, a demonstration missing those fields teaches the wrong interface.

InstructGPT is a concrete published example of this progression: supervised demonstrations followed by preference data and reinforcement learning from human feedback (RLHF) improved instruction-following behavior relative to the underlying GPT-3 models in its evaluation setting.^[3]

SFT doesn't answer every product question

SFT can teach the model to cite policy. It can't guarantee that P-7 is today's policy. If the return window changes tomorrow, fresh retrieved policy text is usually a better first fix than training a new checkpoint.

Stage 3: post-training chooses better behavior

After SFT, multiple answers may be valid JSON and still differ in usefulness. One cites the exact policy clause; another guesses a refund action. Post-training supplies a signal that favors the better behavior.

Two families matter for this map:

In InstructGPT's RLHF stage, a learned reward model represented human preferences inside a reinforcement-learning process.^[3] DPO instead optimizes preference pairs with a classification loss, without fitting a separate reward model or running an RL optimization loop.^[4] DeepSeek-R1 is a published example where pure reinforcement learning improved measured reasoning on verifiable tasks.^[5] Useful pattern to study, not default choice for every product.

Here is the preference-data idea with ordinary Python objects. This code builds a comparison record; an actual DPO or RLHF pipeline would convert many such records into parameter updates.

Verifiable rewards apply only when a checker represents the desired behavior. For this narrow decision, a checker can compare the output with trusted order facts, require policy evidence, and forbid a side effect that the model never executed.

That verifier is useful only for the part it checks. Here, damage_confirmed is a trusted fixture; free-form customer text couldn't set it safely. The checker says nothing about empathy, ambiguous damage photos, or whether policy P-7 is fair. A weak checker can train a model toward shortcuts, so every verifier needs adversarial tests and human-reviewed cases.

Stage 4: inference runs frozen weights

Once a checkpoint is deployed, inference is the work of generating responses for requests. Normal inference reads weights; it doesn't update them. Behavior can still change through retrieved evidence, prompt instructions, tool permissions, or decoding and serving settings.

The return assistant needs today's policy, so the application retrieves P-7 at request time and combines it with trusted order facts. The miniature function below stands in for the model boundary: it demonstrates that new evidence changes the served answer while the checkpoint identifier stays fixed.

Retrieval-Augmented Generation (RAG) was introduced as a model architecture combining generated text with retrieved external knowledge.^[6] In products, practical rule is simple: facts that change often should arrive as request-time evidence, not stale parametric memory.

Memory and latency become visible

Serving is constrained by GPU memory and latency. Weights alone require roughly weight bytes = parameter count * bytes per parameter.

An eight-billion-parameter checkpoint stored at two bytes per parameter needs about 14.9 GiB for weights alone. It still needs memory for the key-value (KV) cache, runtime buffers, and concurrent requests.

The four-bit result is an ideal packed-weight estimate, not a deployment promise. Real quantized formats can add metadata and higher-precision components. Measure the loaded model footprint before choosing hardware.

More context can also be a quality issue, not only a memory issue. Liu et al. observed in their long-context evaluations that models could perform worse when relevant information appeared in the middle of long inputs.^[7] Dumping every policy revision into one huge prompt isn't a substitute for retrieval and evaluation.

Evaluation closes both loops

The model-building team evaluates checkpoints before promotion. The product team evaluates prompts, retrieval, serving, and releases after integration. Both need cases that look like real work.

For A10234, a golden case isn't "does the answer sound friendly?" It includes trusted order facts, a policy fact, a required decision, required evidence, and forbidden claims.

Open-ended response quality often needs a rubric rather than exact string matching. LLM judges can help scale rubric-based comparisons, and Zheng et al. studied their agreement and biases in MT-Bench and Chatbot Arena.^[8] They aren't a substitute for calibrated human-reviewed cases, especially for policy and safety boundaries.

This small evaluator mixes deterministic policy checks with a human-review queue for answers that pass the rule but may still be confusing.

A failure should choose an owner

When an eval fails, "fine-tune the model" is rarely a complete diagnosis. First ask what must change.

Release gate can block a known regression before long-term fix ships.

Traces turn failures into concrete work items. Stored trace can already tell you whether to refresh evidence, revise prompt, add preference examples, or optimize serving.

Keep the boundaries straight

Lifecycle gets simpler once you ask precise question about each symptom.

What not to conclude

• Preference tuning isn't a reliable database update. It may shape how a model responds, but current policy facts belong in retrievable evidence.
• A verifier isn't automatically a truth oracle. It rewards only rules it can observe.
• Long context isn't guaranteed recall. Relevant evidence still needs selection and testing.
• A product incident isn't automatically a training incident. Most deployed fixes start with evidence, contracts, gates, or serving.

A lab-sized lifecycle artifact

After completing this chapter, you can turn the snippets into a small repository artifact:

Small artifact, same habit: inspect failure, then choose smallest intervention backed by evidence instead of reflexively retraining.

Mastery check

Key concepts

• Pre-training predicts tokens and changes model weights.
• SFT teaches instruction behavior from demonstrations.
• Preference methods and verifiable rewards provide different post-training signals.
• Retrieval supplies current evidence without changing model weights.
• Inference exposes memory, latency, throughput, and reliability constraints.
• Evaluation connects production failures to the layer that owns the fix.

Evaluation rubric

• Foundational: Explains why a base next-token predictor may continue a return request instead of answering it.
• Foundational: Marks pre-training, SFT, and post-training as weight-updating stages, while separating retrieval and inference.
• Intermediate: Chooses SFT, preference data, a verifier, or retrieval for a concrete A10234 failure and states why.
• Intermediate: Calculates weight-only memory for a checkpoint and names the omitted serving memory.
• Intermediate: Designs a held-out policy case and a release gate that catches unsupported side-effect claims.

Follow-up questions

Common pitfalls

Treating post-training as fact storage

Symptom: A team creates preference pairs to teach tomorrow's policy revision.

Cause: It confuses preferred behavior with current evidence.

Fix: Retrieve versioned policy text at request time and evaluate grounded decisions against it.

Rewarding a shortcut

Symptom: Verifier scores rise while answers become unhelpful or game a formatting rule.

Cause: Programmatic reward observes only a narrow proxy.

Fix: Add adversarial verifier tests, held-out cases, and human-reviewed rubric samples.

Shipping without a product invariant

Symptom: Assistant produces fluent output that claims a refund or label was executed.

Cause: Eval set checks wording but not side effects.

Fix: Gate release on deterministic constraints tied to recorded tool actions and policy evidence.

Blaming weights for serving failures

Symptom: Latency spike triggers a request for another tuning run.

Cause: Team hasn't separated inference capacity from model behavior.

Fix: Inspect queue time, time to first token, KV-cache pressure, and trace errors before considering training.

Trusting a long prompt to retrieve itself

Symptom: Correct policy is somewhere in a large context, but answer cites the wrong rule.

Cause: Context availability isn't the same as reliable use of evidence.

Fix: Retrieve focused policy passages, place evidence clearly, and evaluate citation accuracy on held-out cases.