The LLM Lifecycle

Your assistant says that K10234, a stale 42-day-old service-account key, is eligible for rotation under fictional policy P-7. Tomorrow, the policy's minimum age changes from 30 to 60 days. The same key must now receive an ineligible verdict. Does that require retraining a 70-billion-parameter large language model (LLM), or updating the authoritative rule and evidence used by the application? These invented thresholds aren't security advice.

The preceding application lesson separated an interactive form, a server decision boundary, and a persistent trace. Here we follow a decoder language model's artifacts: training data, weights, evaluation reports, and the serving configuration. Other architectures and multimodal systems can use different objectives and additional training phases.

The age threshold itself belongs in deterministic code. An LLM can parse messy policy documents, explain the decision to engineers, or synthesize incident reports, but it shouldn't be the final unchecked authority executing privileged cryptographic rotations. The examples below make no external provider calls and execute no real key revocations.

A model-building loop changes learned parameters through pre-training and post-training; optional compression changes their representation or produces a smaller model. A production serving loop uses an approved, frozen checkpoint with application code, live evidence, access controls, and monitoring. Prompt changes affect request context while leaving weights fixed. Selected production traces can inform future evaluation after review.

View larger

Five lifecycle stages, one key decision

The user sees one sentence: "Eligible under P-7: stale key is 42 days old (minimum 30 days)." We can examine the system through five areas of work. This teaching map groups the stages; teams may move through them in different sequences:

Different organizations adopt different recipes. Some teams train foundation models from scratch on massive clusters; others fine-tune open checkpoints with Low-Rank Adaptation (LoRA); many consume hosted APIs directly and focus on retrieval and gateway routing. Regardless of where your team joins the pipeline, separating changing the weights from changing the context gives you your first reliable debugging intuition.

The application can supply K10234's live facts at request time, keeping them in the request path instead of training data. Review who receives them, including the hosted provider, and what is retained.

Stage 1: pre-training builds the base predictor

Pre-training gives later stages a base checkpoint to adapt. The model reads sequences of tokens and learns to predict the next token across a large corpus. Backpropagation adjusts its floating-point parameters, or weights, to improve those predictions. A saved snapshot of these weights is a base checkpoint. The application needs current policy storage or request-time evidence to identify the active version of P-7.

Builders first curate their data. Web crawls contain useful writing alongside duplicates, boilerplate, spam, and sensitive material. CCNet and C4 illustrate distinct curation pipelines:^{[1]Reference 1CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datahttps://arxiv.org/abs/1911.00359[2]Reference 2Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.https://arxiv.org/abs/1910.10683}

MinHash offers a separate near-duplicate approach; these papers describe other deduplication methods. Removing duplicates reduces repeated training text, while a model can still memorize the remaining corpus. Language and quality filters also change whose writing is represented, so inspect their exclusions as well as their throughput.

Byte-Pair Encoding (BPE) builds a subword vocabulary by repeatedly merging frequent adjacent symbols. The original NLP method starts with characters; byte-level variants start with bytes. Vocabulary size and preprocessing vary by tokenizer recipe. Other tokenizers, such as Unigram, make different choices.^{[3]Reference 3Neural Machine Translation of Rare Words with Subword Units.https://arxiv.org/abs/1508.07909[4]Reference 4Tokenization algorithmshttps://huggingface.co/docs/transformers/main/tokenizer_summary}

You can observe this merge dynamic with a minimal frequency counter:

from collections import Counter

# Frequencies of token tuples in a toy security corpus
corpus_words = {
    ("k", "e", "y"): 4,
    ("s", "t", "a", "l", "e"): 3,
    ("r", "o", "t", "a", "t", "i", "o", "n"): 2,
}

adjacent_pairs: Counter[tuple[str, str]] = Counter()
for word_tokens, frequency in corpus_words.items():
    for first, second in zip(word_tokens, word_tokens[1:]):
        adjacent_pairs[(first, second)] += frequency

most_frequent_pair, pair_count = adjacent_pairs.most_common(1)[0]
print(f"Top adjacent pair: {most_frequent_pair} ({pair_count} observations)")

def merge_once(tokens, pair):
    merged, index = [], 0
    while index < len(tokens):
        if tuple(tokens[index:index + 2]) == pair:
            merged.append("".join(pair))
            index += 2
        else:
            merged.append(tokens[index])
            index += 1
    return tuple(merged)

for word in [tuple("stale"), tuple("rotation")]:
    print("After one merge:", merge_once(word, most_frequent_pair))

Top adjacent pair: ('t', 'a') (5 observations)
After one merge: ('s', 'ta', 'l', 'e')
After one merge: ('r', 'o', 'ta', 't', 'i', 'o', 'n')

('t', 'a') wins with five observations: three in stale and two in rotation. ('k', 'e') has four. The new ta token replaces each matching pair, then the trainer recounts pairs for its next merge. This toy works inside words and omits real tokenizer preprocessing and boundary symbols.

For causal next-token training, cross-entropy assigns the observed token $x_t$ loss $-\log p(x_t\mid x_{<t})$. With natural logarithms, probabilities $0.80$ and $0.20$ give losses $0.223$ and $1.609$ nats. Lower probability receives a larger penalty; interpret loss magnitude in the context of the task, sequence length, and reduction. Predicting observed text measures fit to the training target; factual claims require evidence.

from math import log

def token_cross_entropy(probability_of_correct_token: float) -> float:
    return -log(probability_of_correct_token)

confident = token_cross_entropy(0.80)
uncertain = token_cross_entropy(0.20)

print(f"p=0.80 -> loss={confident:.3f}")
print(f"p=0.20 -> loss={uncertain:.3f}")
print("Lower loss rewards higher probability on the true observed continuation.")

p=0.80 -> loss=0.223
p=0.20 -> loss=1.609
Lower loss rewards higher probability on the true observed continuation.

At frontier scale, selecting model dimensions and data volume requires principled compute budgeting. Kaplan et al. initially proposed power-law relationships governing parameters, data, and compute.^{[5]Reference 5Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} Hoffmann et al. challenged those allocations with the Chinchilla scaling experiments: for dense transformers under a compute budget, parameters $N$ and tokens $D$ should scale in roughly equal proportion ($N \propto C^{0.5}, D \propto C^{0.5}$).^{[6]Reference 6Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556}

Their analysis revealed that earlier models like Gopher (280B parameters on 300B tokens) were severely undertrained. Under the same total training FLOP budget, a 70B parameter model trained on 1.4T tokens substantially outperformed the 280B giant. For dense transformers, training floating-point operations follow the approximation $\text{FLOPs} \approx 6ND$, where $N$ is parameter count and $D$ is training token count:

def estimate_dense_training_flops(parameter_count: float, token_count: float) -> float:
    """Hoffmann et al. standard approximation: FLOPs ≈ 6ND."""
    return 6 * parameter_count * token_count

chinchilla_flops = estimate_dense_training_flops(70e9, 1.4e12)
print(f"Chinchilla 70B x 1.4T tokens: {chinchilla_flops:.2e} FLOPs")
assert abs(chinchilla_flops - 5.88e23) / 5.88e23 < 1e-12

Chinchilla 70B x 1.4T tokens: 5.88e+23 FLOPs

The $6ND$ estimate counts approximate dense-model training arithmetic. Deployment, hardware utilization, and long-sequence attention costs require separate accounting.

Chinchilla asks how to allocate a training compute budget to minimize the fitted loss. A service owner may instead value a smaller model's recurring inference cost. Llama 3's builders trained the 8B and 70B models on about 15 trillion tokens and explicitly considered inference-efficient smaller models.^{[7]Reference 7The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} The familiar $D\approx20N$ training heuristic would put 8B near 160B tokens, making 15T about 94 times that rough reference. Treat the ratio as a heuristic; training allocation depends on achieved quality, request volume, architecture, hardware, and serving configuration.

A base model excels at text completion, yet it makes a clumsy software assistant. When prompted with stale key rotation request K10234:, a raw base model often continues with another hypothetical ticket or general prose instead of the structured decision the application needs. Teaching it how to behave requires post-training.

Stage 2: post-training aligns interface and behavior

Post-training adapts a base model's behavior using demonstrations, preferences, rewards, or combinations of them. It can improve instruction following and structured output; authorization and answer validation remain application responsibilities. Keep application checks in the serving path.

View larger

Three common data recipes help distinguish the objectives:

Data contract 1: raw text to turn messages

Our causal pre-training example predicts text tokens. Conversational SFT can instead use records with roles such as system, user, and assistant; tool-using recipes can include other roles. Match the checkpoint's actual chat template.

Chat templates serialize roles with model-specific delimiters. In an assistant-only loss recipe, user and system positions have no direct next-token loss; assistant positions supply the targets. Prompt tokens still condition the answer, and gradients can flow through computations that use them. SFT can also train on full sequences. TRL's documentation and configuration source, checked October 10, 2026, make assistant_only_loss an explicit setting, defaulting to false; assistant masking requires a compatible template.^{[8]Reference 8TRL Documentation: SFT Trainer.https://huggingface.co/docs/trl/sft_trainer}

Here's an SFT demonstration teaching the assistant to emit a structured JSON verdict:

import json

sft_record = {
    "messages": [
        {
            "role": "system",
            "content": "You are a key security assistant. Output JSON containing eligible, policy_id, and evidence.",
        },
        {
            "role": "user",
            "content": "Check stale key K10234 (age 42 days) against fictional P-7: Stale keys at least 30 days old are eligible for rotation.",
        },
        {
            "role": "assistant",
            "content": json.dumps({
                "eligible": True,
                "policy_id": "P-7",
                "evidence": "Stale keys at least 30 days old are eligible for rotation.",
            }, sort_keys=True),
        },
    ]
}

print("Role structure:", [m["role"] for m in sft_record["messages"]])
print("Target assistant payload:", sft_record["messages"][2]["content"])

Role structure: ['system', 'user', 'assistant']
Target assistant payload: {"eligible": true, "evidence": "Stale keys at least 30 days old are eligible for rotation.", "policy_id": "P-7"}

Bad training targets can encourage missing fields. This first audit checks field presence. A complete dataset audit also checks types, extra fields, evidence, and correctness:

required_fields = {"eligible", "policy_id", "evidence"}

training_targets = [
    {"eligible": True, "policy_id": "P-7", "evidence": "at least 30 days old"},
    {"eligible": True, "evidence": "age matches rule"},
]

for idx, target in enumerate(training_targets, start=1):
    missing = sorted(required_fields - target.keys())
    verdict = "FIELDS PRESENT" if not missing else f"REJECT (missing {missing})"
    print(f"Target {idx}: {verdict}")

Target 1: FIELDS PRESENT
Target 2: REJECT (missing ['policy_id'])

Data contract 2: turn messages to preference pairs

SFT makes the desired format likely, but it struggles when two candidates both produce valid JSON while differing in truthfulness or safety. For example, one candidate correctly states the key is eligible, while a second candidate states the key is eligible and fabricates an unperformed rotation event.

InstructGPT popularized Reinforcement Learning from Human Feedback (RLHF), training a separate reward model on ranked response pairs, then optimizing the generative model using Proximal Policy Optimization (PPO).^{[9]Reference 9Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

Direct Preference Optimization (DPO) uses a policy-based reward parameterization to derive a preference loss from the KL-regularized reward objective. It learns from chosen/rejected pairs without a separate reward model or PPO rollouts during that training step.^{[10]Reference 10Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} The derivation gives a closed-form optimal-policy relationship; gradient optimization trains the neural model. Llama 3 combined SFT and DPO in post-training.^{[7]Reference 7The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

A preference training pair compares two completions for the exact same prompt:

preference_pair = {
    "prompt": "Evaluate stale K10234 (age 42) under fictional P-7 (minimum age 30 days). No rotation ran.",
    "chosen": "Eligible under P-7. The stale key is at least 30 days old. No rotation job has been triggered.",
    "rejected": "Eligible under P-7. A rotation worker has successfully rotated K10234.",
    "rationale": "Chosen adheres to reality; rejected hallucinates an unauthorized mutation.",
}

print("Chosen response:  ", preference_pair["chosen"])
print("Rejected response:", preference_pair["rejected"])

Chosen response:   Eligible under P-7. The stale key is at least 30 days old. No rotation job has been triggered.
Rejected response: Eligible under P-7. A rotation worker has successfully rotated K10234.

Data contract 3: verifiable rewards and test-time reasoning compute

Human preference labeling is difficult for complex code, multi-step mathematical proofs, and strict policy rule engines. Annotators can make mistakes, and human evaluation is expensive to scale across millions of synthetic rollouts.

Reinforcement Learning with Verifiable Rewards (RLVR) uses checkable outcomes, such as an exact answer or executed tests, as reward signals. A checker can be incomplete or exploitable, and passing tests establishes performance only on the checked behavior. DeepSeek-R1-Zero used RL without preliminary SFT; DeepSeek-R1 used a multistage pipeline including supervised data and RL. The paper reports improvements in verifiable reasoning tasks and behaviors such as backtracking; assess broader reliability with application-specific evaluation.^{[11]Reference 11DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

Test-time compute is a different axis: the weights stay fixed while inference may spend more work on reasoning tokens, search, or multiple candidates scored by a verifier.^{[12]Reference 12Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314} Accuracy can improve on some tasks, but extra tokens can also waste time or reinforce mistakes. Compare quality, latency, and cost on the workload you actually serve.

The verifier below scores known-account candidates against a trusted fixture. Its compact Boolean format is a separate training exercise from the preceding app's three-way response contract. It demonstrates scoring logic; an RL training job is a separate step:

def verify_candidate_decision(output: dict[str, object], key_age: int, key_stale: bool) -> int:
    expected_eligible = key_stale and key_age >= 30
    passes_all_checks = (
        output.get("eligible") is expected_eligible
        and output.get("policy_id") == "P-7"
        and output.get("rotation_started") is False
    )
    return int(passes_all_checks)

candidate_honest = {"eligible": True, "policy_id": "P-7", "rotation_started": False}
candidate_invented = {"eligible": True, "policy_id": "P-7", "rotation_started": True}

candidate_contradictory = {**candidate_honest, "text": "Rotation completed successfully!"}
print("Honest candidate reward:  ", verify_candidate_decision(candidate_honest, 42, True))
print("Invented mutation reward: ", verify_candidate_decision(candidate_invented, 42, True))
print("Unchecked prose reward:   ", verify_candidate_decision(candidate_contradictory, 42, True))

Honest candidate reward:   1
Invented mutation reward:  0
Unchecked prose reward:    1

The verifier scores three fields in a typed fixture; its scope excludes the full response. It rejects rotation_started=True, yet rewards contradictory prose stored under text. That is a reward blind spot. Restrict what reaches the UI, or evaluate every claim channel against actual execution evidence. Authorization remains an application responsibility. A higher reward score can still coexist with incorrect behavior. What would you measure before calling the candidate ready?

Stage 3: evaluation tests capabilities and boundaries

Before promotion, measure the model together with the decision path around it. Public benchmarks cover selected capabilities; offline release gates should also cover safety and operational reliability in the target path.

Standardized benchmarks and contamination audits

Public benchmarks measure particular capabilities. Two historical examples are:

• Knowledge: Massive Multitask Language Understanding (MMLU) evaluates factual breadth across 57 academic subjects from elementary math to professional law.^{[13]Reference 13Measuring Massive Multitask Language Understanding (MMLU).https://arxiv.org/abs/2009.03300}
• Reasoning: GSM8K tests multi-step grade-school mathematical reasoning.^{[14]Reference 14Training Verifiers to Solve Math Word Problems (GSM8K).https://arxiv.org/abs/2110.14168}

Public questions and answers can enter training data through crawls, copied material, or later fine-tuning. This data contamination can inflate scores. A high score can reflect generalization, memorization, or both; further evidence is needed to distinguish them. These older benchmarks also provide limited evidence about today's frontier capabilities or this app's policy correctness.

Audits combine imperfect evidence:^{[15]Reference 15The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination.https://arxiv.org/abs/2503.16402}

1. Text overlap: Search accessible training data for matching benchmark text. Exact n-grams can miss paraphrases and flag ordinary shared language; a match needs inspection.
2. Controlled perturbations: Compare changed names or numbers while preserving the tested skill. A drop may reflect leakage, changed difficulty, or brittleness.
3. Fresh, versioned cases: LiveBench's maintained repository describes monthly new questions and objective ground-truth scoring without an LLM judge.^{[16]Reference 16LiveBench maintained benchmark repositoryhttps://github.com/LiveBench/LiveBench} Pin the release and model-training cutoff where known. Fresh cases reduce some leakage risk; audit conclusions still depend on available training-data evidence.

If a benchmark publishes a canary string, finding it in accessible training data is another clue. A missing canary leaves other routes for the questions to enter training data untested. When the model's training corpus is unavailable, report that audit limit.

LLM-as-a-judge and its systematic biases

Expert human review can assess conversational quality, and raters can disagree. Rubrics, blinded comparisons, and repeated ratings help. LLM judges offer cheaper repeatable scoring, with calibration against domain evidence and human judgments. The MT-Bench paper studies model judges; Chatbot Arena collects crowdsourced human preferences.^{[17]Reference 17Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

The paper documents biases whose strength depends on the judge and task:

1. Positional bias: Candidate ordering can change judgments. Evaluate both orders and report disagreements; the favored position and the effect of swapping depend on the judge.
2. Verbosity bias: Judges frequently rate wordy, verbose explanations as higher quality than concise, accurate answers.
3. Self-enhancement bias: Models tend to assign higher scores to responses generated by their own model family.

Golden test suites and deterministic release gating

For domain assistants like our key rotator, pair open benchmarks with private, version-pinned golden datasets that test explicit business constraints. The evaluator below grades candidate responses against golden fixtures:

test_cases = [
    # (key_id, age, stale, decision, policy, rotation_started, should_pass)
    ("K10234", 42, True, "eligible", "P-7", False, True),
    ("K10235", 15, True, "not_eligible", "P-7", False, True),
    ("K10236", 50, True, "eligible", "P-7", True, False),     # Invented rotation
    ("K10237", 10, True, "eligible", "P-7", False, False),    # Wrong verdict
    ("K10238", 45, True, "eligible", "P-9", False, False),    # Wrong policy ID
]

passes = 0
for key_id, age, stale, decision, policy, started, should_pass in test_cases:
    expected_decision = "eligible" if stale and age >= 30 else "not_eligible"
    candidate_pass = (
        decision == expected_decision
        and policy == "P-7"
        and started is False
    )
    agreed = (candidate_pass == should_pass)
    passes += int(agreed)
    print(f"Key {key_id}: evaluated_pass={candidate_pass} matches_expected={agreed}")

print(f"Harness audit score: {passes}/{len(test_cases)}")

Key K10234: evaluated_pass=True matches_expected=True
Key K10235: evaluated_pass=True matches_expected=True
Key K10236: evaluated_pass=False matches_expected=True
Key K10237: evaluated_pass=False matches_expected=True
Key K10238: evaluated_pass=False matches_expected=True
Harness audit score: 5/5

The preceding 5/5 audits the evaluator against five expected accept/reject labels. Only two of the five supplied candidates passed; this check made no model call and reports no model-accuracy score.

A release gate applies declared criteria to measured evidence. This next report is invented; its one-second full-decision budget is a lab setting. Full decision latency matches this UI, which waits for a checked decision before responding:

candidate_report = {
    "policy_cases_passed": 20,
    "policy_cases_total": 20,
    "unauthorized_side_effect_claims": 0,
    "p95_decision_ms": 780,
}

def evaluate_release_gate(metrics: dict[str, int]) -> tuple[str, list[str]]:
    required = {"policy_cases_passed", "policy_cases_total", "unauthorized_side_effect_claims", "p95_decision_ms"}
    if (not required <= metrics.keys()
        or any(type(metrics[name]) is not int or metrics[name] < 0 for name in required)
        or metrics["policy_cases_total"] == 0
        or metrics["policy_cases_passed"] > metrics["policy_cases_total"]):
        return "BLOCK", ["Missing or invalid evaluation evidence"]
    blockers = []
    if metrics["policy_cases_passed"] != metrics["policy_cases_total"]:
        blockers.append("Policy correctness regression")
    if metrics["unauthorized_side_effect_claims"] > 0:
        blockers.append("Unauthorized side-effect claim detected")
    if metrics["p95_decision_ms"] > 1000:
        blockers.append("P95 decision budget breached")

    verdict = "BLOCK" if blockers else "PROMOTE"
    return verdict, blockers

verdict, blockers = evaluate_release_gate(candidate_report)
print(f"Release gate verdict: {verdict}")
print("Blockers identified: ", blockers if blockers else "None (all checks pass)")
invalid_reports = [
    {**candidate_report, "policy_cases_passed": 0, "policy_cases_total": 0},
    {**candidate_report, "p95_decision_ms": -1},
    {**candidate_report, "policy_cases_passed": True},
    {**candidate_report, "policy_cases_passed": 21},
]
for report in invalid_reports:
    assert evaluate_release_gate(report)[0] == "BLOCK"
print("Invalid evidence cases blocked:", len(invalid_reports))

Release gate verdict: PROMOTE
Blockers identified:  None (all checks pass)
Invalid evidence cases blocked: 4

Passing metrics belong to the bundle that produced them. Save immutable revisions of the weights, tokenizer and chat template, app rule, runtime settings, and evaluation cases with the report, then compare each revision with what you intend to serve.

This local fixture records invented revisions, not real artifact hashes. All metrics pass, but changing only the chat template blocks reuse of the old report:

evaluated_bundle = {
    "weights": "weights-revision-17",
    "tokenizer": "tokenizer-revision-3",
    "template": "template-revision-4",
    "app_policy": "P-7@1",
    "runtime": "runtime-config-8",
    "cases": "private-cases-12",
}
report = {"bundle": evaluated_bundle.copy(), "metrics": {
    "policy_cases_passed": 20, "policy_cases_total": 20,
    "unauthorized_side_effect_claims": 0, "p95_decision_ms": 780,
}}

def report_matches_release(report, intended_bundle):
    return report["bundle"] == intended_bundle

unchanged = evaluated_bundle.copy()
changed_template = {**evaluated_bundle, "template": "template-revision-5"}
assert report_matches_release(report, unchanged)
assert not report_matches_release(report, changed_template)
print("Original bundle:", "metrics report applies")
print("Changed template:", "reevaluation required")

Original bundle: metrics report applies
Changed template: reevaluation required

Matching revisions makes the report applicable; quality still depends on passing the metric gate and measuring a representative workload. After compression, record the new weight artifact and evaluate that artifact again.

Stage 4: compression shrinks weights for serving

A trained 70-billion-parameter checkpoint stored in standard 16-bit brain floating point (bfloat16 or float16) consumes 2 bytes per parameter. Storing weights alone requires:

That's 140 GB in decimal units. Two 80 GB devices have enough aggregate capacity for those weights if the runtime partitions them appropriately; runtime buffers and the key-value (KV) cache require additional memory. Larger devices, offloading, and other layouts change the placement options.

At small decode batches, repeatedly reading active weights from High Bandwidth Memory (HBM) can dominate latency. Larger batches reuse weights across requests; long-context attention, communication, or other work may dominate instead. Lower precision can reduce traffic, but throughput gains depend on supported kernels and the workload.

Post-Training Quantization (PTQ) can reduce this weight traffic by converting 16-bit floating-point weights into 8-bit (INT8) or 4-bit (INT4) integers, without a full-model training run.

The calculation below estimates weight footprints across precisions:

def calculate_weight_gib(param_count_billions: float, bytes_per_param: float) -> float:
    return (param_count_billions * 1e9 * bytes_per_param) / (1024 ** 3)

bf16_gib = calculate_weight_gib(70, 2.0)
int8_gib = calculate_weight_gib(70, 1.0)
int4_gib = calculate_weight_gib(70, 0.5)

print(f"70B in BF16 (16-bit): {bf16_gib:.1f} GiB")
print(f"70B in INT8  (8-bit):  {int8_gib:.1f} GiB")
print(f"70B in INT4  (4-bit):  {int4_gib:.1f} GiB")

70B in BF16 (16-bit): 130.4 GiB
70B in INT8  (8-bit):  65.2 GiB
70B in INT4  (4-bit):  32.6 GiB

The ideal packed INT4 weights alone occupy 32.6 GiB. Real artifacts add scales, metadata, padding, and tensors retained at higher precision. A 48 GB card may still be insufficient at long contexts or high concurrency. Weight-only quantization leaves the KV cache at its configured precision.

For an illustrative 70B-like configuration with 80 layers, 8 KV heads, head dimension 128, and two-byte cache elements, count both keys and values:

layers, kv_heads, head_dim = 80, 8, 128
tokens_per_request, cache_bytes = 32768, 2
cache_gib = 2 * layers * kv_heads * head_dim * tokens_per_request * cache_bytes / 1024**3
ideal_weight_gib = 70e9 * 0.5 / 1024**3
print(f"KV cache per request: {cache_gib:.1f} GiB")
print(f"Ideal weights + two requests: {ideal_weight_gib + 2 * cache_gib:.1f} GiB")

KV cache per request: 10.0 GiB
Ideal weights + two requests: 52.6 GiB

Even that subtotal exceeds 48 GiB, before quantization overhead and runtime buffers. This calculation assumes ordinary full-attention K/V storage without sharing, compression, or offloading; inspect the actual architecture and runtime.

One symmetric INT8 recipe uses $[-127,127]$ and scale $s=\max(|W|)/127$. Other recipes can use an asymmetric range or a zero point. An all-zero tensor needs a nonzero arbitrary scale to avoid division by zero. The snippet uses Python floats; serving artifacts require the target tensor dtype and packed representation:

weights = [-0.84, -0.22, 0.05, 0.41, 0.96]

max_abs = max(abs(w) for w in weights)
scale = max_abs / 127.0 if max_abs else 1.0

quantized_int8 = [max(-127, min(127, round(w / scale))) for w in weights]
reconstructed = [q * scale for q in quantized_int8]
max_reconstruction_error = max(abs(w - r) for w, r in zip(weights, reconstructed))

print("Original float weights:", weights)
print("Quantized INT8 grid:   ", quantized_int8)
print("Reconstructed values:  ", [round(r, 4) for r in reconstructed])
print(f"Max reconstruction error: {max_reconstruction_error:.4f}")

Original float weights: [-0.84, -0.22, 0.05, 0.41, 0.96]
Quantized INT8 grid:    [-111, -29, 7, 54, 127]
Reconstructed values:   [-0.8391, -0.2192, 0.0529, 0.4082, 0.96]
Max reconstruction error: 0.0029

The outlier challenge: AWQ and GPTQ

At low bit widths, rounding error can significantly alter layer outputs. In weight-only quantization, a channel with large input activations can amplify its weight error. The quantizer sets the weight tensor's grid; activation magnitudes affect how its errors propagate. Outlier rates vary by model, so 0.1% is only a hypothetical fraction.

Two post-training methods respond to that amplification:

1. AWQ (Activation-aware Weight Quantization) uses calibration activations to search channel scales that protect important weights. Reciprocal activation scaling preserves the original unquantized operation; the scaling changes the subsequent quantization error. Its all-low-bit method keeps weights in low-bit form without a special 0.1% FP16 subset.^{[18]Reference 18AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.https://arxiv.org/abs/2306.00978}
2. GPTQ minimizes layer-output reconstruction error on calibration inputs. For a linear layer, that squared-error objective has Hessian $2XX^\top$. Damped inverse-Hessian information guides error compensation as remaining weights are quantized. GPTQ runs offline on calibration inputs; deployment uses the resulting quantized weights for inference.^{[19]Reference 19GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.https://arxiv.org/abs/2210.17323}

Both depend on calibration and implementation choices. Reevaluate the resulting artifact and the intended workload before deployment. Compression can also use distillation to train a smaller student, pruning to remove structure, or quantization-aware training to include quantization effects during training. Service readiness depends on whether the served artifact meets the available latency and memory budgets.

Stage 5: deployment optimizes latency and memory

Once the artifact is frozen, four measurements compete for the same hardware:

1. Time to First Token (TTFT): Time from a declared start boundary to the first output token. Client-observed time can include networking, retrieval, queueing, prefill, and the first decode step. First visible text is a different boundary.
2. Inter-Token Latency (ITL): Time between successive generated tokens. Client stream chunks can contain multiple tokens, so measure token timing separately from chunk spacing. Batching, attention, and scheduling affect it alongside memory bandwidth.
3. Throughput: Total output tokens generated per second across all concurrent streams.
4. VRAM footprint: Memory allocated across weights, runtime activations, and the KV cache.

Three important serving mechanisms illustrate the tradeoffs. Their availability and exact implementation depend on the engine, model, and backend:

PagedAttention manages KV-cache memory

Ordinary causal transformer serving caches previous key/value tensors so each decode step can reuse them. Total cache memory grows with layers, KV heads, head dimension, cache precision, sequence length, and concurrent requests. Cache memory can exceed weight memory depending on those settings. Sliding windows, shared prefixes, and other architectures change the calculation.

The original PagedAttention paper profiled systems where only 20.4% to 38.2% of allocated KV-cache memory held actual token states. Reservation and fragmentation account for the remaining allocated cache in those measurements.^{[20]Reference 20Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

PagedAttention maps logical cache blocks to fixed-size, noncontiguous physical blocks allocated as needed. It reduces external fragmentation and over-reservation; the last block can still have unused positions. Blocks can also support shared prefixes with reference counting and copy-on-write. The paper's throughput gains apply to its specified models and workloads; benchmark modern engines with the intended traffic. Block size depends on the backend. Cache payload still scales with actual token states.

Continuous batching prevents pipeline stalls

Static batching groups requests and waits for the batch to finish before admitting replacements. For example, if Request 1 finishes after 15 decode iterations while Request 2 needs 600, its slot remains idle for the remaining 585 iterations. Some implementations also compute padding or finished rows; that extra arithmetic isn't required by every static-batch implementation. Iteration-level scheduling can reuse the available capacity sooner.

Orca introduced iteration-level scheduling, which lets batch membership change between generation iterations.^{[21]Reference 21Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu} A completed request can yield capacity to another request. Admission latency depends on prefill work, available cache, and scheduling policy; shared prefix blocks may remain cached.

Speculative decoding: propose tokens, then verify

Small-batch decoding may have spare compute while weight traffic dominates. Speculative decoding uses a draft model or another proposal mechanism to suggest several tokens, then lets the target verify them in a parallel pass.^{[22]Reference 22Fast Inference from Transformers via Speculative Decoding.https://arxiv.org/abs/2211.17192} Accepted proposals form a contiguous prefix.

In the exact stochastic algorithm, acceptance probabilities and correction sampling preserve the target's specified sampling distribution under the paper's assumptions. If the first four of five proposals pass and the fifth fails, emit those four plus a correction token. If all five pass, the pass can also emit one bonus token. Drafting and verification add work, so speedup depends on acceptance rate, model sizes, and runtime costs. Approximate acceptance methods and different sampling settings need separate quality checks.

The runtime gateway: authentication, authorization, and rate limits

An application can enforce identity and account permissions before contacting a provider. This standalone routing fixture illustrates parameterized token lookup, expiry, and environment scope. Its strings are public invented tokens, stored in plaintext only for the exercise. The preceding FastAPI app would need its own authentication middleware, credential storage, account authorization, revocation, request limits, and concurrency controls.

import sqlite3

NOW = 100
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE client_keys (token TEXT PRIMARY KEY, scope TEXT, expires_at INTEGER)")
db.executemany(
    "INSERT INTO client_keys VALUES (?, ?, ?)",
    [
        ("client-prod", "rotation:read", 130),
        ("client-staging", "staging:rotation", 130),
        ("client-expired", "rotation:read", 90),
    ],
)

provider_pools = {
    "rotation:read": ["prod-primary", "prod-backup"],
    "staging:rotation": ["staging-primary"],
}
provider_status = {
    "prod-primary": 503,
    "prod-backup": 200,
    "staging-primary": 200,
}
open_until: dict[str, int] = {}

def authenticate(token: str, required_scope: str) -> str | None:
    # Parameterized query keeps untrusted bearer tokens out of SQL syntax
    row = db.execute("SELECT scope, expires_at FROM client_keys WHERE token = ?", (token,)).fetchone()
    if row is None:
        return None
    scope, expires_at = row
    return scope if scope == required_scope and expires_at > NOW else None

def route(token: str, required_scope: str) -> str:
    scope = authenticate(token, required_scope)
    if scope is None:
        return "unauthorized"

    for pool_id in provider_pools[scope]:
        if open_until.get(pool_id, 0) > NOW:
            continue
        status = provider_status[pool_id]
        if status == 429:
            return "provider_rate_limited"
        if status == 503:
            open_until[pool_id] = NOW + 10
            continue
        if 200 <= status < 300:
            return f"ok:{pool_id}"
        return f"provider_error:{status}"
    return "provider_pool_exhausted"

crafted_token = "missing' OR 1=1 --"
print("Crafted SQL token:   ", route(crafted_token, "rotation:read"))
print("Expired token:       ", route("client-expired", "rotation:read"))
print("Wrong scope token:   ", route("client-staging", "rotation:read"))
print("Production routing:  ", route("client-prod", "rotation:read"))
print("Primary open until:  ", open_until.get("prod-primary"))
open_until.clear()
provider_status["prod-primary"] = 429
print("Shared quota routing:", route("client-prod", "rotation:read"))
db.close()

Crafted SQL token:    unauthorized
Expired token:        unauthorized
Wrong scope token:    unauthorized
Production routing:   ok:prod-backup
Primary open until:   110
Shared quota routing: provider_rate_limited

The fixture temporarily skips a provider pool reporting 503 and selects a backup within the authorized environment. It stops on 429: another key may share the same project, organization, or model quota. The ten-unit cooldown is invented and the fixture has no real request, deadline, half-open probe, or concurrent breaker. Production failover also requires approved data handling, spare capacity, and evaluation of the backup's behavior.^{[23]Reference 23Rate Limitshttps://developers.openai.com/api/docs/guides/rate-limits}

Five symptoms, five investigations

One symptom can have several causes. Start with the versioned input, delivered context, completion status, application checks, and serving measurements before choosing a fix. A trace can narrow an investigation; establish root cause from versioned inputs and execution evidence.

View larger

The threshold is stale. Compare the active policy with the exact version delivered to the model and the rule used by the deterministic checker. Check response-cache keys too. Stale chunks are one possibility; a hardcoded old constant or cached answer is another. Update the artifact that owns the old rule, whether it is retrieved context or application code.

The JSON is malformed. Inspect the provider's terminal status, refusal path, output limit, and raw-to-object adapter first. A cut-off response or mismatched parser can look like a format-learning problem. Then verify schema support and chat-template configuration; audit training demonstrations only if evidence points there.

The answer claims an action that never ran. Check execution receipts and the code that assembles the display. A status message derived from a durable action outcome can establish what ran. Preference or reward training may improve future model behavior.

The model loops after quantization. Compare original and compressed artifacts on the same versioned prompts, template, and decoding settings. If only the compressed artifact fails, investigate calibration, quantizer, packing, and kernel compatibility. If both fail, context or decoding may be responsible. A smaller group size changes overhead; test its effect on quality for this model.

First output is slow, or the GPU runs out of memory. Split queueing, prefill, decode, network time, and memory categories under the same load. Reduce or schedule real cache demand if capacity is exhausted. A block-pool setting changes allocation policy while physical VRAM stays fixed. Speculative decoding targets decode work; investigate prefill and queueing separately.

This toy chooses a first evidence check from known symptom phrases. Diagnosis, model changes, and remedies require the corresponding evidence and operational steps. Real incidents can require several of these checks:

def first_check_from_symptom(trace_log: dict[str, str]) -> str:
    symptom = trace_log["symptom"].lower()
    if "stale policy" in symptom:
        return "Compare active, delivered, and server-rule policy versions"
    if "missing field" in symptom or "malformed json" in symptom:
        return "Inspect completion status, schema support, and parsing"
    if "invented rotation" in symptom or "hallucinated action" in symptom:
        return "Compare action receipts with model and UI claims"
    if "repetitive loop" in symptom or "numeric degradation" in symptom:
        return "Compare original and quantized artifacts under matched settings"
    if "slow first token" in symptom or "kv oom" in symptom:
        return "Measure queue, prefill, decode, and memory under load"
    return "Triage manually with full trace telemetry"

traces = [
    {"symptom": "K10234 used stale policy threshold 30 days"},
    {"symptom": "Response missing field policy_id in JSON payload"},
    {"symptom": "Response claimed invented rotation execution without tool trigger"},
    {"symptom": "Generated text exhibited repetitive loop after 4-bit conversion"},
    {"symptom": "Slow first token latency spike during traffic surge"},
]

for t in traces:
    check = first_check_from_symptom(t)
    print(f"[{t['symptom'][:32]}...] -> {check}")

[K10234 used stale policy thresho...] -> Compare active, delivered, and server-rule policy versions
[Response missing field policy_id...] -> Inspect completion status, schema support, and parsing
[Response claimed invented rotati...] -> Compare action receipts with model and UI claims
[Generated text exhibited repetit...] -> Compare original and quantized artifacts under matched settings
[Slow first token latency spike d...] -> Measure queue, prefill, decode, and memory under load

A known version mismatch supports a narrower plan. The following function proposes an investigation and preserves the checkpoint. Deployment, update-time measurement, and incident resolution require separate operational checks:

incident_trace = {
    "key_id": "K10234",
    "checkpoint": "security-assistant-70b-v2-awq",
    "retrieved_policy_version": "P-7-v1",
    "active_policy_version": "P-7-v2",
    "failure": "stale policy threshold",
}

def plan_incident_checks(trace: dict[str, str]) -> tuple[str, str]:
    target_checkpoint = trace["checkpoint"]
    if trace["retrieved_policy_version"] != trace["active_policy_version"]:
        action = f"Verify source and server rule {trace['active_policy_version']}; inspect stale context caches"
    else:
        action = "Inspect delivered context, completion, checker, and display before selecting a fix"
    return target_checkpoint, action

checkpoint, investigation = plan_incident_checks(incident_trace)
print("Model checkpoint status: UNCHANGED ->", checkpoint)
print("Proposed investigation:", investigation)

Model checkpoint status: UNCHANGED -> security-assistant-70b-v2-awq
Proposed investigation: Verify source and server rule P-7-v2; inspect stale context caches

A stale threshold belongs to the artifact that owns it: training data, weights, retrieved context, or the checker in application code. Fitting a small model yourself, and reading held-out loss, is the next way to see what a learned weight actually captured.

Review questions

Test your operational reasoning against these five boundary scenarios:

1. Does changing the policy from 30 days to 60 days require retraining?

Keep the model weights fixed for this rule change. Update the authoritative policy and deterministic threshold, deliver matching evidence, invalidate stale answers where needed, and test boundary cases. Verify that retrieval supplies the new rule and the checker uses the new threshold.

2. Can a rule-based verifier catch a text contradiction if structured fields pass?

Only if the verifier evaluates both channels. If a candidate returns {"rotation_started": false} in JSON while appending "Rotation completed successfully!" in user text, a verifier that checks only JSON keys will reward the deception. Check text claims against system execution logs, or assemble user-facing status only from verified structured fields outside the model.

3. How does AWQ protect 4-bit models from perplexity collapse?

AWQ uses calibration activations to find weight-channel scaling that reduces important output errors. Reciprocal scaling preserves the original operation before rounding. Quality changes depend on the model and workload, so evaluate the quantized artifact against both. Its rationale follows measured activation effects; the outlier fraction varies with the model and workload.

4. Why does PagedAttention reduce VRAM requirements under concurrent traffic?

It reduces KV-cache over-reservation and fragmentation by mapping logical blocks to noncontiguous physical storage allocated as needed. The last block can have unused positions. More efficient cache allocation can enable higher concurrency; the token-state payload still scales with actual cached tokens, and throughput gains depend on the workload.

5. Why do modern teams overtrain small models past Chinchilla optimality?

Chinchilla optimizes the parameter/data allocation for a training compute budget. A serving system has recurring inference costs too. Training a smaller model longer may reach useful quality at a lower serving cost; Llama 3 illustrates that choice. The economic result depends on quality, lifetime traffic, and serving configuration.