From GPT to Modern LLMs

Suppose CI run RUN-842 stops at auth_fixture_test.py:41 with an unhandled exception: expired auth fixture. Your team wants an assistant to label review comments, pull the exact run ID from a noisy 2,000-line log, and explain the failure. Those sound like jobs for different tools. Could one text model handle them through the same interface, and where would a simpler tool do better?

GPT stands for Generative Pre-trained Transformer, the model family introduced by OpenAI; large language model (LLM) is the broader category. GPT-style causal decoders generate text by predicting the next token from a prefix. That continuation interface can express classification, extraction, explanation, and bug-fix drafting. Accuracy and task fit still require evaluation.

The original Transformer debuted in 2017 as an encoder-decoder machine translation model.^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762} Later work explored causal decoders, bidirectional encoders, larger training runs, instruction following, and more efficient serving. These developments overlap, and each changes a different part of the system. We'll follow RUN-842 through those design choices, asking what each one buys and what still needs to be checked.

One generation loop, many tasks

Traditional systems solve the first two jobs directly: a classifier predicts logits over a closed set of three labels, {"Bug", "Docs", "Security}, and an extraction model returns character spans [start_idx, end_idx] into the log. A generative model can attempt both through the same text-continuation interface.

A causal decoder predicts continuations using left-to-right attention. Instructions and demonstrations describe the intended output: Docs after Label:, or RUN-842 after Extracted Run ID:. Those strings may require several tokenizer pieces. A base model may not recognize a system role; a chat model expects its trained conversation format.

A software assistant encounters three standard task shapes:

1. Classification: A review note states, "This docs page is stale." You want the category label Docs.
2. Extraction: A CI trace contains RUN-842 inside a stack frame. You want the exact identifier.
3. Generation: An on-call engineer asks, "Why did tests fail?" You want a concise diagnosis grounded in the trace.

An encoder like BERT can support classification and extraction heads. Its standard masked-token objective trains bidirectional representations; a causal decoder provides the usual left-to-right explanation interface. A suitably trained causal decoder can attempt all three tasks through prompting. Here is a few-shot classification prompt:

Classify the review note into Bug, Docs, or Security.

Note: The auth test fails on main.
Label: Bug

Note: This docs page is stale.
Label: Docs

Note: The API key is printed in logs.
Label: Security

---

Note: The README still mentions the old CLI flag.
Label:

The model ingests the prompt prefix, attends across the task rules and demonstrations, and predicts the continuation at the empty Label: slot. The prompt pattern sets up an inductive bias for Docs.

A fixed classifier head returns scores for its configured labels. An unconstrained generator can produce Documentation or Sure, the label is Docs. Parse and validate the result, or use supported constrained decoding with an enumerated schema. Correct syntax still needs an accuracy check. For exact extraction, compare the identifier against the source log; a regular expression may handle this job more reliably.

View larger

This code builds the review-note prompt. A model call and an accuracy evaluation are separate steps. Docs is the intended answer for this note:

def build_review_label_prompt(note: str) -> str:
    examples = [
        ("The auth test fails on main.", "Bug"),
        ("This docs page is stale.", "Docs"),
        ("The API key is printed in logs.", "Security"),
    ]
    header = "Classify the review note into Bug, Docs, or Security."
    shots = "\n\n".join(
        f"Note: {text}\nLabel: {label}" for text, label in examples
    )
    return f"{header}\n\n{shots}\n\n---\n\nNote: {note}\nLabel:"

prompt = build_review_label_prompt("The README still mentions the old CLI flag.")
print(prompt)
print("pattern_target=Docs")
print("why=the prompt pattern ends with a label slot")

Classify the review note into Bug, Docs, or Security.

Note: The auth test fails on main.
Label: Bug

Note: This docs page is stale.
Label: Docs

Note: The API key is printed in logs.
Label: Security

---

Note: The README still mentions the old CLI flag.
Label:
pattern_target=Docs
why=the prompt pattern ends with a label slot

Why encoders read both ways and decoders write left to right

Why did GPT-style decoders become common in assistants? Autoregressive generation gives the application one flexible output interface. The attention mask describes visibility; adoption and task performance require separate evidence.

The 2017 Transformer consisted of two stacks: an encoder that processed the full source sentence bidirectionally, and an autoregressive decoder that generated target tokens one by one while cross-attending to the encoder representations.^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762} In 2018, researchers demonstrated useful designs with a single stack for some tasks:

1. BERT (2018). Its bidirectional encoder lets a non-padding position attend to both earlier and later non-padding positions.^{[2]Reference 2BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.https://arxiv.org/abs/1810.04805} Original BERT trained with masked language modeling and next-sentence prediction. Of the 15% of positions selected for masked-token supervision, 80% were replaced by [MASK], 10% by a random token, and 10% left unchanged. Encoder representations support classification and span heads. Sentence-BERT adds training specifically for useful sentence-similarity embeddings; arbitrary pooled BERT states need task-specific evaluation for retrieval.^{[3]Reference 3Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.https://arxiv.org/abs/1908.10084} Mask filling and adapted generation procedures are possible. GPT-style models use left-to-right training for assistant-style generation.
2. GPT (2018). Its causal Transformer uses self-attention and feed-forward blocks without encoder cross-attention.^{[4]Reference 4Improving Language Understanding by Generative Pre-Training.https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf} This is more than keeping an unchanged translation decoder. A lower-triangular causal mask, including the diagonal, lets the state at position $i$ predict $x_{i+1}$ without seeing future tokens. Original pretraining used causal language modeling. Supervised adaptation added task objectives.

The useful distinction is interface flexibility. Text generation lets you express many tasks through prompts; task performance still needs evaluation. Encoder-decoder models also generate text, as T5 demonstrates.^{[5]Reference 5Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.https://arxiv.org/abs/1910.10683} Keep architecture, training objective, and application quality separate.

A decoder writes left to right

In causal attention, token $t$ attends only to positions $\le t$. When predicting the token after failed, the model attends to <bos>, tests, and failed; future-token positions are masked.

During teacher-forced training, the known input sequence supports parallel position computations within each layer. Mask scores strictly above the diagonal with $-\infty$, then normalize the allowed scores. Compare each position's logits with the following token, excluding unsupervised targets. Layer dependencies and total work remain. Basic inference selects successive tokens, while speculative methods can verify multiple candidate positions together.

tokens = ["<bos>", "tests", "failed", "<eos>"]
n = len(tokens)
causal = [[1 if col <= row else 0 for col in range(n)] for row in range(n)]
bidirectional = [[1] * n for _ in range(n)]

print("tokens:", tokens)
print("causal row for 'failed':      ", causal[2])
print("bidirectional row for 'failed':", bidirectional[2])
print("future visible in causal row:", causal[2][3])

tokens: ['<bos>', 'tests', 'failed', '<eos>']
causal row for 'failed':       [1, 1, 1, 0]
bidirectional row for 'failed': [1, 1, 1, 1]
future visible in causal row: 0

Here, 1 means allowed and 0 means blocked; the table shows visibility independent of any library's mask argument. A real bidirectional encoder also masks padding. The encoder's visibility pattern supports full-context representations; causal decoders use the shifted training contract illustrated here.

The life of a token: prefill, KV cache, and decode

Trace the engineer's prompt through the decoder: "Why did tests fail?".

Text enters the tokenizer and breaks into a sequence of discrete token IDs. An embedding lookup table maps each ID to a vector of dimension $d_{\text{model}}$. Position information is injected into each token's representation.

For a conventional pre-normalization decoder, the tokens flow through $N$ layers. A common arrangement is:

1. Normalization rescales each position's features before its attention branch.
2. Self-attention projects hidden states into Queries ($Q$), Keys ($K$), and Values ($V$).
3. Softmax-weighted attention scores combine values across all allowed causal positions.
4. An output projection maps the attended values back to $d_{\text{model}}$.
5. A normalized feed-forward branch transforms each position through nonlinear activations. Residual additions combine both branches with the running state.

After the final normalization, the last position's row vector projects through $W_{\text{vocab}} \in \mathbb{R}^{d_{\text{model}} \times |V|}$ to produce vocabulary logits.^{[6]Reference 6GPT-2 Source Implementation.https://github.com/openai/gpt-2/blob/master/src/model.py} An implementation may instead store its transpose as $[|V|,d_{\text{model}}]$. Applying softmax yields normalized probabilities. In our hand-chosen four-token fixture, greedy argmax selects Auth; stochastic decoding could select another token.

Inference splits into two distinct execution phases:

1. Prefill: Process the known prompt positions and populate the KV cache. Projections can combine $BT$ rows into matrix multiplications, often giving better weight reuse than a one-row operation. Long prompts may be processed in chunks. The last prompt position supplies logits for the first answer token.
2. Cached decode: Feed the newly selected token as the next input. Each layer computes its new K/V entry, then the token's query scores visible keys in the previous cache plus that current entry. The cache reuses earlier tokens' K/V projections, while attention work for the new query grows with visible context. Projection and feed-forward work continue for the new token.

Long-prefill projections often have high arithmetic intensity, while small-batch decode can be limited by weight and cache bandwidth.^{[7]Reference 7Efficiently Scaling Transformer Inferencehttps://proceedings.mlsys.org/paper_files/paper/2023/hash/c4be71ab8d24cdfb45e3d06dbfca2780-Abstract-mlsys2023.html} These are workload tendencies. At batch size $B$, a decode projection processes $B$ rows, so it needn't be GEMV; batching, quantization, attention variants, hardware, and communication can change the bottleneck. Tensor-core utilization depends on the full workload and implementation.

View larger

Inspect a hand-chosen output projection and the logical cache dimensions. The stacked K/V shape below is an accounting convention; runtime layouts vary:

import math

prompt_tokens, d_model, vocab_size = 5, 4, 4
last_hidden = [0.6, -0.2, 0.4, 0.1]
output_rows = [
    [0.8, 0.1, -0.2, 0.0],
    [0.2, 0.5, 0.1, -0.1],
    [-0.1, 0.2, 0.4, 0.3],
    [0.0, -0.3, 0.2, 0.4],
]
labels = ["Auth", "The", "I", "Fix"]
logits = [sum(h * w for h, w in zip(last_hidden, row)) for row in output_rows]
peak = max(logits)
exps = [math.exp(x - peak) for x in logits]
z = sum(exps)
probs = [e / z for e in exps]

print("prompt tokens:", prompt_tokens, "d_model:", d_model, "vocab:", vocab_size)
print("last-position logits:", [round(x, 2) for x in logits])
print("argmax:", labels[logits.index(max(logits))])
print("prob sum:", round(sum(probs), 3))
layers, kv_heads, head_dim = 4, 2, 2
print("cached K/V shape per layer:", (2, 1, kv_heads, prompt_tokens, head_dim))
assert math.isclose(sum(probs), 1.0)
assert logits.index(max(logits)) == 0

prompt tokens: 5 d_model: 4 vocab: 4
last-position logits: [0.38, 0.05, 0.09, 0.18]
argmax: Auth
prob sum: 1.0
cached K/V shape per layer: (2, 1, 2, 5, 2)

Notice the sequence length discrepancy right after sampling. Selecting Auth adds it to the text output, bringing the sequence to 6 tokens. The KV cache still contains only 5 positions. The next decode step takes Auth as its single input token, computes its Key and Value vectors, and expands the cache to length 6.

How GPT grew from text completion into assistants

To answer the engineer, a base model that continues a log line needs additional training and application controls. Transfer learning, prompt-based evaluation, instruction following, training allocation, and architecture each changed a different part of that gap, and their timelines overlap. Instruction tuning complements few-shot prompting, and fine-tuning remains useful.

View larger

1. GPT-1: Generative pretraining + task fine-tuning (2018)

Generative pretraining followed by supervised adaptation predates GPT. Radford et al. cite earlier language-model transfer work and demonstrate an influential Transformer version using BooksCorpus, a collection of over 7,000 books.^{[4]Reference 4Improving Language Understanding by Generative Pre-Training.https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf}

GPT-1 consisted of a 12-layer decoder with 117 million parameters. Its core workflow was a two-stage process:

• Unsupervised pretraining: Train the decoder on raw text using standard autoregressive cross-entropy loss.
• Supervised fine-tuning: Add a task-specific linear output layer and update the model on labeled tasks. The paper also retains an auxiliary language-model objective.

The reported supervised adaptations use task-specific checkpoints. The paper also examines zero-shot behavior, so prompt-based evaluation didn't first appear in GPT-2.

2. GPT-2: Zero-shot transfer without task-specific weights (2019)

GPT-2 expanded the largest model to about 1.5 billion parameters and trained on WebText, roughly 40 GB of text collected from links shared on Reddit.^{[8]Reference 8Language Models are Unsupervised Multitask Learners.https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf} Its report evaluated tasks without task-specific parameter updates, with results that varied substantially by task.

An illustrative zero-shot task framing is Input: The service was awful. Sentiment:. The intended continuation is negative; this prompt demonstrates the format and reports no GPT-2 measurement. Zero-shot evaluation omits worked task demonstrations from the prompt; pretraining can still contain related examples or task information.

3. GPT-3: In-context few-shot emergence (2020)

Brown et al. trained models up to 175 billion parameters, about 117 times GPT-2's largest model, on 300 billion tokens.^{[9]Reference 9Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165} Their evaluations showed stronger few-shot performance with scale on many tasks, alongside weak results on others. The 175B value is one study scale point; capability depends on data, task, and training.

Few-shot evaluation supplies input-output demonstrations in the prefix without gradient updates. The GPT-3 study often used many more than two or three examples, up to what fit its context. Demonstrations can influence predictions; that evaluation protocol leaves the internal learning mechanism unresolved. Example selection, formatting, order, and task difficulty all matter.

4. InstructGPT: Preference training (2022)

Base-model training may focus on text continuation before instruction following. Explain this test failure can elicit an answer, another question, or a continuation of a transcript. Prompting can help; a production assistant also needs evaluated instruction-following behavior.

Ouyang et al.'s InstructGPT applied an existing line of human-feedback research to instruction following.^{[10]Reference 10Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} Its particular RLHF pipeline was:

1. Supervised Fine-Tuning (SFT): Human annotators write thousands of high-quality demonstrations of instructions and ideal responses. The base model fine-tunes on this instruction dataset, learning the conversational assistant format.
2. Reward Modeling (RM): The SFT model generates multiple candidate responses for various prompts. Human labelers rank these candidates from best to worst. A reward model trains on these pairwise comparisons to output a scalar score predicting human preference.
3. Reinforcement Learning (PPO): Initialize the policy from SFT and optimize a learned reward with a KL penalty relative to the SFT reference. The paper's PPO-ptx variant also mixes in a pretraining objective to reduce task regressions.

On the study's held-out API prompt distribution, labelers preferred the 1.3B InstructGPT outputs to the 175B GPT-3 outputs.^{[10]Reference 10Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} That result measures preference on this evaluation; knowledge coverage and performance on other tasks require separate tests. Preference training can still reward plausible mistakes.

The following fixture stores preferred and rejected responses and checks whether the preferred text repeats a required phrase. A substring check measures phrase presence; truth evaluation and preference-model training require different methods. Try adding There was no expired auth fixture: it passes the phrase check while contradicting the log.

comparisons = [
    {
        "prompt": "CI run RUN-842: auth_fixture_test.py:41 reports 'expired auth fixture'. Explain the failure.",
        "chosen": "RUN-842 failed at auth_fixture_test.py:41: the log reports an expired auth fixture.",
        "rejected": "The build failed for an unknown reason.",
        "required_fact": "expired auth fixture",
    },
    {
        "prompt": "Review note says the API key is printed in logs. Classify the issue.",
        "chosen": "Security: redact the API key and rotate the exposed credential.",
        "rejected": "Docs: update the README wording.",
        "required_fact": "API key",
    },
]

for row in comparisons:
    keeps_required_fact = (
        row["required_fact"] in row["prompt"]
        and row["required_fact"] in row["chosen"]
    )
    print(f"prompt={row['prompt'][:22]}... chosen_keeps_fact={keeps_required_fact}")
    print("  preferred:", row["chosen"])

prompt=CI run RUN-842: auth_f... chosen_keeps_fact=True
  preferred: RUN-842 failed at auth_fixture_test.py:41: the log reports an expired auth fixture.
prompt=Review note says the A... chosen_keeps_fact=True
  preferred: Security: redact the API key and rotate the exposed credential.

These checkpoint labels describe intended use and can overlap across training recipes. An instruct model may support multi-turn chat, and a chat model can use objectives other than PPO:

A chat template serializes messages using the checkpoint's expected roles and control tokens.^{[11]Reference 11Transformers Documentation: Chat templates.https://huggingface.co/docs/transformers/main/en/chat_templating} Formats differ even between models derived from the same base. Use the checkpoint's supplied template or official encoding implementation, including its generation prefix. Some releases, such as DeepSeek-V4.1-Flash, provide an encoding implementation in place of a Jinja chat template.^{[12]Reference 12DeepSeek-V4.1-Flash Model Cardhttps://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash} Avoid adding duplicate special tokens if you tokenize an already formatted string. Training shapes behavior; formatting supplies the expected structure. The application must supply retained history because separate API requests carry no automatic memory.

What scaling laws say about a fixed training budget

As teams trained larger models, an engineering question arose: given a fixed computational budget (measured in floating-point operations, or FLOPs), how should you divide your budget between adding parameters and feeding more training tokens?

Kaplan et al. (2020) fitted empirical relationships between language-model loss, parameter count $N$, data, and training compute $C$.^{[13]Reference 13Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} Their compute-allocation estimates scaled model size faster than training tokens $D$:

Several prominent runs increased parameters while retaining roughly 300B training tokens. Gopher used 280B parameters; GPT-3 used 175B.^{[14]Reference 14Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} GPT-3's ratio was about 1.7 tokens per parameter. That ratio describes training allocation; evaluate task quality separately.

Hoffmann et al. varied training horizons and matched learning-rate schedules to those horizons, using over 400 models from 70M to over 16B parameters.^{[14]Reference 14Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} They explain that intermediate losses from a fixed long schedule can overestimate the loss achievable by a shorter run with its own schedule. Their larger-model coverage and fitted methods also differ from Kaplan's analysis. The Chinchilla study found approximately equal parameter/token scaling:

Under these fitted training conditions, doubling parameters calls for roughly doubling training tokens. About 20 tokens per parameter is a useful landmark from this study; choose an allocation for the specific data mixture, architecture, and compute range. Its three fitting approaches gave slightly different exponents.

Chinchilla used the same reported training-compute budget as Gopher and improved aggregate MMLU and BIG-bench performance.^{[14]Reference 14Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} Some individual tasks still favored Gopher. The LLaMA and Llama 3 rows come from their own papers; the Chinchilla paper reports the Chinchilla results.^{[15]Reference 15LLaMA: Open and Efficient Foundation Language Models.https://arxiv.org/abs/2302.13971[16]Reference 16The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}

runs = {
    "Gopher": {"params_b": 280, "tokens_b": 300},
    "GPT-3": {"params_b": 175, "tokens_b": 300},
    "Chinchilla": {"params_b": 70, "tokens_b": 1400},
    "Llama-3-8B": {"params_b": 8, "tokens_b": 15000},  # rounded 15T landmark
}

for name, run in runs.items():
    ratio = run["tokens_b"] / run["params_b"]
    print(f"{name:12s} tokens/param={ratio:7.1f}")

Gopher       tokens/param=    1.1
GPT-3        tokens/param=    1.7
Chinchilla   tokens/param=   20.0
Llama-3-8B   tokens/param= 1875.0

To compare allocations, a common approximation for dense Transformer training is $C \approx 6ND$ FLOPs.^{[14]Reference 14Training Compute-Optimal Large Language Models.https://arxiv.org/abs/2203.15556} The factor roughly combines two FLOPs per weight multiply-add with forward and backward work. Attention, embeddings, and other operations can change the estimate; use profiling for wall-clock time and a separate formula for MoE compute.

Predict the budget change when both $N$ and $D$ double. This arithmetic example keeps the model architecture and accounting approximation fixed:

baseline_flops = 6 * 1e9 * 20e9
allocations = {
    "baseline": (1e9, 20e9),
    "double both": (2e9, 40e9),
    "parameters only": (4e9, 20e9),
    "tokens only": (1e9, 80e9),
}

for name, (parameters, tokens) in allocations.items():
    budget_ratio = 6 * parameters * tokens / baseline_flops
    print(f"{name:15s} compute={budget_ratio:.1f}x tokens/param={tokens / parameters:.1f}")

baseline        compute=1.0x tokens/param=20.0
double both     compute=4.0x tokens/param=20.0
parameters only compute=4.0x tokens/param=5.0
tokens only     compute=4.0x tokens/param=80.0

Doubling both quantities costs roughly four times the baseline, just like the other two larger allocations. Their token/parameter ratios differ. Equal estimated compute can produce different loss values, so choose an allocation using measured scaling behavior for the data and training recipe. The 20-token ratio illustrates one allocation; use observed scaling behavior to select a compute-optimal ratio. $D$ counts training token exposures, so repeated data can increase $D$ without adding unique examples.

Training compute versus lifetime serving costs

The Chinchilla ratio represents training-compute optimality: minimizing the loss achieved for a given training budget. In real systems, total cost includes inference serving.

The original LLaMA paper explicitly considers inference efficiency.^{[15]Reference 15LLaMA: Open and Efficient Foundation Language Models.https://arxiv.org/abs/2302.13971} Later Llama 3 work also trains smaller models beyond training-compute optimality.^{[16]Reference 16The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} A smaller model can reduce per-request weight storage and arithmetic, making additional pretraining worthwhile for a large serving workload. Break-even depends on request volume, model quality, hardware, quantization, and required context. An 8B model's ability to fit one GPU depends on that GPU and the complete runtime memory budget.

Three newer building blocks: RMSNorm, SwiGLU, and RoPE

Many later models modify the 2017 architecture. RMSNorm, gated feed-forward networks, and rotary positions are common choices, including in original LLaMA.^{[15]Reference 15LLaMA: Open and Efficient Foundation Language Models.https://arxiv.org/abs/2302.13971} Architectures still vary: original Gemma uses GeGLU instead of SwiGLU, for example.^{[17]Reference 17Gemma: Open Models Based on Gemini Research and Technologyhttps://arxiv.org/abs/2403.08295} Public reports provide limited detail about undisclosed hosted models.

1. Root Mean Square Normalization (RMSNorm)

Standard LayerNorm computes both the mean $\mu$ and variance $\sigma^2$ across the feature dimension $d$ of hidden vector $x$:^{[18]Reference 18LayerNormhttps://docs.pytorch.org/docs/2.14/generated/torch.nn.LayerNorm.html}

Zhang and Sennrich tested a simpler normalization that omits centering.^{[19]Reference 19Root Mean Square Layer Normalization.https://arxiv.org/abs/1910.07467} Its empirical success covers their tested architectures; gradient stability still requires evaluation in each full network.

RMSNorm discards the mean calculation and the learned additive bias $\beta$, scaling strictly by root mean square energy:^{[19]Reference 19Root Mean Square Layer Normalization.https://arxiv.org/abs/1910.07467[20]Reference 20RMSNormhttps://docs.pytorch.org/docs/2.14/generated/torch.nn.RMSNorm.html}

RMSNorm uses a mean-square statistic; LayerNorm uses centered variance. Both normalization types can use optimized single-pass or fused kernels, so the speed difference depends on the implementation and workload. The original paper reports runtime savings in its tested models; evaluate accelerator speed and stability on the target model and workload.

2. SwiGLU activations

Early transformers used standard ReLU or GELU activations inside a two-layer feed-forward network: $\text{FFN}(x) = \text{activation}(x W_1) W_2$.

Gated Linear Units (GLUs) multiply a transformed gate projection by another projection. Shazeer's comparison found lower perplexity for selected gated variants in its tested Transformer setup.^{[21]Reference 21GLU Variants Improve Transformerhttps://arxiv.org/abs/2002.05202} Evaluate the effect under each training recipe.

SwiGLU uses the Swish (or SiLU) activation function $\text{Swish}(z) = z \cdot \sigma(z)$ as the gating mechanism:

Ignoring biases, a two-matrix FFN has $2dd_{\text{ff}}$ weights and a three-matrix SwiGLU FFN has $3dd_{\text{ff}}$. Matching a conventional width $4d$ therefore gives a SwiGLU width $8d/3$.^{[21]Reference 21GLU Variants Improve Transformerhttps://arxiv.org/abs/2002.05202} Rounding and architecture choices can alter this width. The matrix multiply-add budgets are comparable under these assumptions; nonlinearities, elementwise products, kernels, and wall time needn't be identical. SwiGLU's gate is signed and unbounded, unlike a sigmoid gate constrained to $[0,1]$.

3. Rotary Position Embeddings (RoPE)

The original Transformer used added sinusoidal positions and also evaluated learned positions.^{[1]Reference 1Attention Is All You Need.https://arxiv.org/abs/1706.03762} Sinusoids support algebraic relationships between offsets; useful extrapolation still requires evaluation. A learned position table has a finite number of rows unless extended.

Su et al. (2021) introduced Rotary Position Embeddings (RoPE).^{[22]Reference 22RoFormer: Enhanced Transformer with Rotary Position Embedding.https://arxiv.org/abs/2104.09864} Instead of adding positional vectors at the input layer, RoPE rotates 2D coordinate pairs in the Query and Key vectors inside each attention head by a position-dependent angle $m\theta_i$:

For fixed unrotated content vectors $\mathbf q$ and $\mathbf k$, the positional rotation depends on the relative offset because $R_{\Theta,m}^T R_{\Theta,n}=R_{\Theta,n-m}$:

Positions $(14,12)$ and $(4,2)$ have the same relative rotation. Their complete attention scores can differ because the unrotated vectors depend on content and prior layer context. Individual RoPE scores also needn't decrease monotonically with distance. Context extension can modify rotary frequencies; evaluate each method with its target model because the identity alone says nothing about extrapolation quality.

Run the mechanics of RMSNorm and SwiGLU in Python:

import math

def rms_norm(x: list[float], gamma: list[float], eps: float = 1e-6) -> list[float]:
    if not x or len(x) != len(gamma):
        raise ValueError("x and gamma must have the same nonzero length")
    if not math.isfinite(eps) or eps <= 0:
        raise ValueError("eps must be positive and finite")
    d = len(x)
    rms = math.sqrt(sum(xi ** 2 for xi in x) / d + eps)
    return [(xi / rms) * gi for xi, gi in zip(x, gamma, strict=True)]

def silu(z: float) -> float:
    if z >= 0:
        return z / (1.0 + math.exp(-z))
    exp_z = math.exp(z)
    return z * exp_z / (1.0 + exp_z)

x = [1.2, -0.8, 2.5, -1.5]
gamma = [1.0, 1.0, 1.0, 1.0]
normed = rms_norm(x, gamma)

w_gate = [0.5, -0.3]
w_up = [1.2, 0.8]
gate_val = sum(xi * w for xi, w in zip(x[:2], w_gate))
up_val = sum(xi * w for xi, w in zip(x[:2], w_up))
swiglu_val = silu(gate_val) * up_val

print("rms_norm output:", [round(v, 4) for v in normed])
print("rms verification:", round(math.sqrt(sum(v**2 for v in normed) / len(normed)), 4))
print("gate:", round(gate_val, 4), "up:", round(up_val, 4), "swiglu:", round(swiglu_val, 4))

rms_norm output: [0.7379, -0.4919, 1.5372, -0.9223]
rms verification: 1.0
gate: 0.84 up: 0.8 swiglu: 0.4694

The printed RMS rounds to 1.0. Before rounding, it is slightly below one because the denominator includes positive eps. This check also uses unit gamma; learned per-feature scales can change the output's RMS.

How attention design changes KV cache costs

Conventional full-attention decode reads visible cached keys and values. Sliding-window, sparse, compressed, or shared-cache architectures have different access and storage patterns. Cache size depends on these choices and batch/context lengths as well as the model's parameters.

To estimate that cost in a conventional layout, count keys and values for every cached position across the batch. With separate K/V arrays, equal head dimensions and precision, and $B$ requests each caching $L$ positions, the raw KV payload is:

Here $P$ is bytes per stored scalar (two for FP16/BF16), and the factor of two counts keys and values. For unequal request lengths, replace $BL$ with the sum of cached lengths; sum layer-specific payloads when their layouts differ. Quantization metadata, page rounding, reserved capacity, weights, and runtime buffers add memory beyond this formula.

These three attention designs differ in how many K/V heads they keep for the queries:

1. Multi-Head Attention (MHA): Each query head has an independent key head and value head ($n_{\text{kv\_heads}} = n_{\text{q\_heads}}$). For 32 query heads, you cache 32 key heads and 32 value heads.
2. Multi-Query Attention (MQA): Share one key head and one value head across query heads.^{[23]Reference 23Fast Transformer Decoding: One Write-Head is All You Need.https://arxiv.org/abs/1911.02150} With 32 query heads and otherwise equal cache dimensions, the raw KV payload is $1/32$ of MHA. Quality and speed need evaluation.
3. Grouped-Query Attention (GQA): Share a KV head within each query-head group.^{[24]Reference 24GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.https://arxiv.org/abs/2305.13245} Eight KV heads for 32 query heads use one quarter of MHA's raw KV payload. The original study obtained quality close to MHA in its tested, uptrained models. Evaluate each checkpoint, task, and kernel. A trained MHA checkpoint requires compatible weights and implementation before using a GQA layout.

Compare raw payloads for a hypothetical uniform batch. These byte calculations isolate the tensor payload; GPU allocation and throughput require runtime measurements:

batch, layers, tokens, head_dim, bytes_per_value = 8, 32, 8192, 128, 2
query_heads = 32

for label, kv_heads in [("MHA", 32), ("GQA", 8), ("MQA", 1)]:
    cached_values = batch * layers * tokens * kv_heads * head_dim * 2
    cache_gib = cached_values * bytes_per_value / (1024 ** 3)
    relative = kv_heads / query_heads
    print(f"{label}: kv_heads={kv_heads:2d} cache={cache_gib:5.2f} GiB relative={relative:.3f}")

MHA: kv_heads=32 cache=32.00 GiB relative=1.000
GQA: kv_heads= 8 cache= 8.00 GiB relative=0.250
MQA: kv_heads= 1 cache= 1.00 GiB relative=0.031

This RoPE lab holds the content vectors fixed, isolates one coordinate pair, and checks equal relative offsets. Change key for the late case to see why equal gaps don't imply equal scores for different content:

import math

def rotate(vector: tuple[float, float], position: int, theta: float = 0.4) -> tuple[float, float]:
    angle = position * theta
    cosine, sine = math.cos(angle), math.sin(angle)
    x, y = vector
    return (cosine * x - sine * y, sine * x + cosine * y)

def dot(left: tuple[float, float], right: tuple[float, float]) -> float:
    return left[0] * right[0] + left[1] * right[1]

query = (1.0, 0.2)
key = (0.3, 0.9)
same_gap_early = dot(rotate(query, 4), rotate(key, 2))
same_gap_late = dot(rotate(query, 14), rotate(key, 12))
different_gap = dot(rotate(query, 14), rotate(key, 2))
print("same gap, early:", round(same_gap_early, 6))
print("same gap, late: ", round(same_gap_late, 6))
print("different gap:  ", round(different_gap, 6))
print("same gap equal:", math.isclose(same_gap_early, same_gap_late))

same gap, early: 0.936998
same gap, late:  0.936998
different gap:   -0.794779
same gap equal: True

Head sharing reduces KV payload. Other serving techniques reduce attention's memory traffic or cache-allocation waste:

• FlashAttention: Compute dense attention in tiles without materializing the full $T\times T$ score/probability arrays in HBM.^{[25]Reference 25FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.https://arxiv.org/abs/2205.14135} This reduces memory traffic and uses linear auxiliary storage in sequence length for fixed head width. Dense attention arithmetic remains quadratic. Its original HBM-access bound depends on SRAM size $M$, $\Theta(T^2d^2/M)$; it isn't universally linear in $T$.
• PagedAttention: Map logical KV blocks to non-contiguous physical blocks.^{[26]Reference 26Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} Paging reduces allocation waste and supports cache sharing. Partial final pages and other runtime allocations still consume capacity. Reported throughput gains apply to the paper's evaluated workloads; benchmark the target runtime and traffic pattern.

Sparse architectures: Mixture of Experts (MoE)

Dense transformer layers apply their attention and feed-forward projections to each position. Parameter count roughly tracks this work; embeddings use lookups, attention work depends on context, and a multiply-add convention often counts two FLOPs. Sparse experts increase available feed-forward capacity while activating only a subset per token.

Shazeer et al. (2017) introduced the sparsely gated Mixture of Experts layer, later developed in Mixtral 8x7B (Jiang et al., 2024) and DeepSeek models.^{[27]Reference 27Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.https://arxiv.org/abs/1701.06538[28]Reference 28Mixtral of Experts.https://arxiv.org/abs/2401.04088}

In a Mixtral-style layer, $E$ expert feed-forward networks replace the dense FFN. The router selects $k$ experts and combines their output vectors. The following convention retains selected logits, sets all others to $-\infty$, then normalizes the selected weights; other MoE designs use different gates, shared experts, and values of $k$:

The fundamental MoE tradeoff: VRAM vs FLOPs

Mixtral 8x7B contains 47 billion total parameters across its 8 experts per layer. But for any given token, only 2 experts activate, touching approximately 13 billion parameters.

The two counts answer different questions:

• Active parameters (~13B): Help estimate per-token projection work. Latency also depends on routing, expert utilization, batch size, kernels, and device communication.
• Total parameters (~47B): Determine the uncompressed checkpoint weight pool. At two bytes per weight, the rounded count gives 94 GB / 87.54 GiB before runtime overhead. A fully GPU-resident deployment must hold this pool across its GPUs. CPU/disk offloading or quantization can change GPU residency, with transfer, speed, and quality tradeoffs.^{[29]Reference 29Accelerate Documentation: Big Model Inference.https://huggingface.co/docs/accelerate/usage_guides/big_modeling}

Poor load balance can overload a few experts and leave others with little training signal. Routers can use an auxiliary load-balancing loss as one design option. DeepSeek-V3 primarily adjusts routing biases using observed load, while retaining a small complementary sequence-level balance loss.^{[30]Reference 30DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437} Routing balance manages load distribution; each token can use selected experts, and an expert need not have one human-interpretable specialty.

Inspect top-2 expert routing in code:

import math

experts = ["E0", "E1", "E2", "E3"]
tokens = ["import", "assertion"]
router_logits = [
    [2.4, 0.3, 0.5, -0.2],
    [0.1, 2.2, 0.7, 0.0],
]

for token, logits in zip(tokens, router_logits):
    ranked = sorted(range(len(logits)), key=lambda i: logits[i], reverse=True)[:2]
    selected = [logits[i] for i in ranked]
    peak = max(selected)
    weights = [math.exp(value - peak) for value in selected]
    total = sum(weights)
    weights = [weight / total for weight in weights]
    assert len(ranked) == 2 and math.isclose(sum(weights), 1.0)
    routes = list(zip((experts[i] for i in ranked), (round(weight, 3) for weight in weights)))
    print(f"{token:10s} -> {routes}")

import     -> [('E0', 0.87), ('E2', 0.13)]
assertion  -> [('E1', 0.818), ('E2', 0.182)]

Reasoning models and the cost of extra inference compute

An autoregressive model can generate an explanation or scratchpad before its answer. Test-time compute also includes repeated sampling, search, verification, and tool calls. Reasoning-trained models emphasize some of these behaviors; reasoning and chat describe overlapping capabilities, and a model can support both.

OpenAI o1 (2024) and DeepSeek-R1 (2025) are prominent milestones in training models to use additional inference computation for reasoning.^{[31]Reference 31Learning to reason with LLMshttps://openai.com/index/learning-to-reason-with-llms/[32]Reference 32DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.https://arxiv.org/abs/2501.12948} Test-time compute predates both systems, and its effect varies by task.

A reasoning model may generate a reasoning trace, with visibility depending on the product. That trace can include:

1. Formulate a hypothesis and test it against the evidence.
2. Backtrack when a reasoning path leads to a contradiction.
3. Compare supplied stack frames and configurations. Retrieve commit history from the repository or include it in the prompt; additional reasoning needs evidence access.

As checked on 30 October 2026, OpenAI's Responses API exposes reasoning.effort on supported models.^{[33]Reference 33Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} Supported effort levels vary by model. Reasoning can occur between tool calls and visible output segments, as well as before final output. For a factual audit of RUN-842, verify the final explanation against line 41 and distinguish the reported expired fixture from the still-unconfirmed reason it expired.

OpenAI's reasoning tokens occupy context and are billed as output tokens even when the trace isn't visible.^{[33]Reference 33Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} Reserve budget for both reasoning and the final answer; an exhausted output budget can leave the answer incomplete. Evaluate effort levels on the same held-out cases, recording verified success, time to first useful output, complete-response latency, token costs, and tool execution costs. Include routine cases as well as hard ones.

Open weights, licensing, and local serving

Meta's original LLaMA release in 2023 was influential, but downloadable language-model weights predate it.^{[15]Reference 15LLaMA: Open and Efficient Foundation Language Models.https://arxiv.org/abs/2302.13971} Original LLaMA access was for approved non-commercial research; its terms excluded unrestricted commercial deployment.^{[34]Reference 34Introducing LLaMA: A Foundational, 65-Billion-Parameter Large Language Modelhttps://ai.meta.com/blog/large-language-model-llama-meta-ai/} Later releases have their own terms. Downloadable weights permit local execution and supported adaptation techniques, subject to licensing and runtime compatibility. Check what your team can actually do with the specific checkpoint before building around it.

When evaluating downloadable models, software teams must distinguish between three terms:

OSI's AI definition requires detailed data information; sufficient documentation can meet that requirement without releasing every raw training example. Copyleft licenses can also qualify.^{[35]Reference 35The Open Source AI Definition - 1.0https://opensource.org/ai/open-source-ai-definition}

Two releases checked on 30 October 2026 illustrate why release identity, licensing, and runtime support need separate checks. These are vendor-published specifications; use workload benchmarks to assess performance:

The local checkpoint and a hosted API alias are different identities. DeepSeek's changelog says older V4 Flash API names route to V4.1 Flash; its 17 October update maps deepseek-flash to V4.1-Flash-1017. A downloaded July checkpoint remains its own revision.^{[40]Reference 40DeepSeek API Change Loghttps://api-docs.deepseek.com/updates/} Pin model and runtime revisions for reproducible evaluations.

Self-hosting can keep inference inside your chosen boundary. Privacy depends on telemetry, remote tools, logging, storage, and network configuration. The team also owns capacity, uptime, updates, and serving performance.

Evaluating a released model on your workload

Use public leaderboard scores to shortlist candidates, then evaluate them on your proprietary logs, company idioms, and latency constraints.

Start with a pilot set, for example 50 cases of each task: failing CI logs, review-note labels, and multi-turn debugging questions. Treat those counts as an initial planning value; choose final evaluation size based on the decision and desired uncertainty. Include missing evidence, contradictory logs, unfamiliar projects, and relevant log text that contains misleading instructions. Keep development and final evaluation cases separate, grouping related runs or repositories to avoid leakage.

Give each case an output contract and a verifiable reference. Expired auth fixture is a reported failure; a claimed root cause such as clock skew needs further evidence. Score exact IDs and label validity separately from factual correctness, citation support, and useful abstention. Test candidates on the same cases and inspect uncertainty, paired disagreements, and failures by task alongside the pooled average.

Measure latency under representative concurrency and prompt/output lengths, distinguishing first-token from complete-response latency. Account for retries, tools, and local operating costs. Apply mandatory license, privacy, quality, and latency constraints first. Then compare eligible candidates using an explicit utility or cost objective.

This scorecard uses invented measurements to illustrate a weighted tradeoff. quality is a hypothetical verified-success fraction, p95_ms is complete-response p95 for a fixed short task, and cost is USD per 1,000 requests. Live code help has a mandatory 750 ms p95 deadline; nightly audit has no hard p95 cap in this example. Both scenarios enforce license eligibility before arbitrary latency and cost weights rank the remaining candidates:

candidates = [
    {"name": "hosted-fast", "quality": 0.86, "p95_ms": 250, "cost": 0.50, "license_ok": True},
    {"name": "hosted-deep", "quality": 0.94, "p95_ms": 1200, "cost": 1.50, "license_ok": True},
    {"name": "local-restricted", "quality": 0.90, "p95_ms": 500, "cost": 0.20, "license_ok": False},
]

def score(model: dict, latency_penalty: float, cost_penalty: float) -> float:
    if not model["license_ok"]:
        return float("-inf")
    return 100 * model["quality"] - latency_penalty * model["p95_ms"] - cost_penalty * model["cost"]

def is_eligible(model: dict, p95_deadline_ms: int | None) -> bool:
    meets_deadline = p95_deadline_ms is None or model["p95_ms"] <= p95_deadline_ms
    return model["license_ok"] and meets_deadline

scenarios = {
    "live code help": {"weights": (0.03, 8.0), "p95_deadline_ms": 750},
    "nightly audit": {"weights": (0.003, 4.0), "p95_deadline_ms": None},
}

for scenario, policy in scenarios.items():
    eligible = [model for model in candidates if is_eligible(model, policy["p95_deadline_ms"])]
    if not eligible:
        raise ValueError("no candidate satisfies the hard constraints")
    ranked = sorted(eligible, key=lambda model: score(model, *policy["weights"]), reverse=True)
    winner = ranked[0]
    deadline = policy["p95_deadline_ms"]
    deadline_label = f"{deadline}ms" if deadline is not None else "none"
    eligible_names = ",".join(model["name"] for model in eligible)
    print(f"{scenario}: p95_deadline={deadline_label}; eligible={eligible_names}; winner={winner['name']} score={score(winner, *policy['weights']):.2f}")

live_policy = scenarios["live code help"]
nightly_policy = scenarios["nightly audit"]
hosted_deep = next(model for model in candidates if model["name"] == "hosted-deep")
assert not is_eligible(hosted_deep, live_policy["p95_deadline_ms"])
assert is_eligible(hosted_deep, nightly_policy["p95_deadline_ms"])

live code help: p95_deadline=750ms; eligible=hosted-fast; winner=hosted-fast score=74.50
nightly audit: p95_deadline=none; eligible=hosted-fast,hosted-deep; winner=hosted-deep score=84.40

The 1,200 ms hosted-deep candidate fails live code help's 750 ms deadline and is removed before scoring. Nightly audit has no hard p95 cap in this fixture, so both licensed hosted candidates reach the weighted comparison, where hosted-deep wins. These invented values illustrate how hard constraints and preference weights play different roles.

Longer context and the evidence that gets missed

Large advertised context windows can tempt us to assume a model will find anything we include. Capacity measures how much text can fit; retrieval quality depends on position and task.

Liu et al. (2023) varied evidence position in multi-document question answering and key-value retrieval.^{[41]Reference 41Lost in the Middle: How Language Models Use Long Contextshttps://aclanthology.org/2024.tacl-1.9/} Many tested models did worse with evidence in the middle than at the edges, producing a U-shaped curve. The effect varied by model and task; some synthetic retrieval settings were solved perfectly. Treat its findings as model- and task-specific; evaluate the position curve for each current model.

View larger

The observed answer-accuracy curve leaves its cause unresolved. Attention mass on initial tokens measures attention weights; evaluate separately whether the model uses that evidence faithfully. Full attention can directly connect an answer position to a distant visible key. Attribute a retrieval error to attention diffusion only after a targeted evaluation isolates that mechanism.

In the study, the query-before-and-after intervention greatly improved synthetic key-value retrieval and had little effect on the multi-document QA trend.^{[41]Reference 41Lost in the Middle: How Language Models Use Long Contextshttps://aclanthology.org/2024.tacl-1.9/} Test prompt changes on the workload you intend to improve; gains on one benchmark may not transfer to debugging.

Calculate the figure's hypothetical scores:

cases_per_position = 20
correct_by_slot = {1: 15, 5: 10, 10: 7, 15: 10, 20: 14}

for slot, correct in correct_by_slot.items():
    assert 0 <= correct <= cases_per_position
    print(f"slot={slot:2d} correct={correct:2d}/20 accuracy={correct / cases_per_position:.0%}")
print("fixture_only: no model was evaluated")

slot= 1 correct=15/20 accuracy=75%
slot= 5 correct=10/20 accuracy=50%
slot=10 correct= 7/20 accuracy=35%
slot=15 correct=10/20 accuracy=50%
slot=20 correct=14/20 accuracy=70%
fixture_only: no model was evaluated

Practical prompt engineering mitigations

When feeding large logs or multiple retrieved documents to your developer assistant:

1. Put governing instructions in the model's supported system/developer role. Mark log text as untrusted evidence and delimit it clearly.
2. Compare task-before-context, task-after-context, and repeated-task layouts on the chosen model. Preserve chronology where event order matters.
3. Test evidence near either boundary and in the middle. Shorten irrelevant material when possible, while keeping the context needed to support the diagnosis.

policy = "Rule: explain CI failures using cited log lines."
older_notes = [
    "History: dependency cache restored.",
    "History: formatter completed.",
    "History: unit tests started.",
]
case = "Evidence: RUN-842 failed at auth_fixture_test.py:41 with expired auth fixture."
question = "Question: explain the failure and cite the evidence."

prompt_lines = [policy, *older_notes, case, question]
for position, line in enumerate(prompt_lines, start=1):
    print(f"{position}: {line}")
print("evidence_adjacent_to_question:", prompt_lines[-2] == case)

1: Rule: explain CI failures using cited log lines.
2: History: dependency cache restored.
3: History: formatter completed.
4: History: unit tests started.
5: Evidence: RUN-842 failed at auth_fixture_test.py:41 with expired auth fixture.
6: Question: explain the failure and cite the evidence.
evidence_adjacent_to_question: True

This code verifies adjacency in the constructed prompt. Measure model quality by comparing the layout with a position-shuffled baseline and inspecting supported answers, misses, and false diagnoses.

Match the architecture to the job

Different jobs around the same CI failure may call for different models. Match the architecture to the measurement you actually need:

Common misconceptions

Practice: diagnose the failing assistant

Your first prototype of the RUN-842 assistant exhibits three operational failures:

1. Behavioral failure: When an engineer sends "Explain why RUN-842 failed at line 41", the model responds by outputting: "Explain why RUN-843 failed at line 12\nExplain why RUN-844 failed at line 99".
2. Retrieval failure: When you provide a 20-line snippet, the model correctly identifies the expired auth fixture. When you provide the complete 50,000-line build log containing the exact same snippet in the middle, the model claims: "The build succeeded without errors".
3. Memory failure: A newly proposed 47B MoE model advertises 13B active parameters per token. Your ops team attempts to deploy it on a single GPU with 24 GB of VRAM, and the container crashes immediately with an OutOfMemory CUDA error.

Validate the continuation against evidence. Size deployment from total serving memory, including weights, KV cache, scales, and runtime overhead. The RUN-842 assistant also needs the right evidence and a testable output format.