Perplexity & Model Evaluation

The embedding lesson followed token IDs into vectors. Now put the model's predictions to a test: when the real next token appears, how much probability did the model assign to it?

Suppose a held-out incident log begins The request returned. A model that assigns high probability to 500 and almost none to volcano fits this local pattern. A model that treats both as equally plausible doesn't. Perplexity turns that held-out behavior into one number you can compare across checkpoints, as long as the measurement contract stays fixed.

That number can still point in the wrong product direction. One checkpoint may predict incident prose better yet recommend an unsafe rollback, while another scores worse on tokens and passes the source and policy checks. Perplexity tells you which model fits this text; it doesn't choose the release.

Prediction-fit metric: Perplexity is a next-token fit metric for causal language models. It's useful when you keep the evaluation contract fixed. It isn't a score for factuality, helpfulness, or safe product behavior.

From surprise to a metric

A causal language model assigns a probability to every possible next token. During evaluation, you don't sample a continuation. You feed the actual preceding text and score the probability assigned to the next token that occurred. This is the same teacher-forced prediction used in next-token training, now measured on text withheld from training.

If the observed token has probability $p$, its negative log-likelihood (NLL), or surprise, is:

A certain correct prediction has probability 1 and surprise 0. A probability close to 0 produces a large penalty. Confident misses hurt language-model loss the most.

The following probabilities are illustrative entries from one distribution; omitted tokens carry the remaining 0.149 probability. Treat each displayed string as one token. Which observed token would contribute the largest surprise?

import math

probabilities = {
    "500": 0.60,
    "200": 0.25,
    "volcano": 0.001,
}

for token in ["500", "200", "volcano"]:
    surprise = -math.log(probabilities[token])
    print(f"{token:9s} probability={probabilities[token]:.3f} surprise={surprise:.3f} nats")

500       probability=0.600 surprise=0.511 nats
200       probability=0.250 surprise=1.386 nats
volcano   probability=0.001 surprise=6.908 nats

volcano contributes the largest loss because its assigned probability is closest to zero. The held-out token matters: if the log says 500, only the first score counts. It doesn't matter that 200 also sounded reasonable.

Average surprise becomes perplexity

For $N$ scored held-out tokens, the average NLL is:

Here $\theta$ identifies the model weights, and $x_{<i}$ is the actual text before target $x_i$, restricted to the context the evaluator supplies. $N$ counts scored targets, not padding or input-only context. Whether the first text token has a beginning-of-sequence token to condition on is a policy we'll make explicit below.

For one-hot next-token targets, that average NLL is also the causal cross-entropy loss: the model's average penalty on the observed next tokens.

Perplexity exponentiates that average:

This is the standard definition used for autoregressive, or causal, language models. It isn't the standard metric for masked models such as BERT, because they predict masked positions rather than the next token in sequence.^{[1]Reference 1Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

A masked model can instead be evaluated with pseudo-perplexity: mask each text token separately, score its original value using the other tokens, and exponentiate the average penalty. That conditioning includes right context, so the resulting number isn't a causal next-token PPL. Label it separately rather than ranking BERT against a causal model by those two raw scores.^{[2]Reference 2Masked Language Model Scoringhttps://aclanthology.org/2020.acl-main.240/}

Start with three observed token probabilities. The 0.10 token should dominate the average NLL, but the final PPL should be computed only after that average:

import math

observed_probabilities = [0.50, 0.10, 0.80]
token_nll = [-math.log(probability) for probability in observed_probabilities]
average_nll = sum(token_nll) / len(token_nll)
perplexity = math.exp(average_nll)

print(f"token NLL: {[round(value, 3) for value in token_nll]}")
print(f"average NLL: {average_nll:.3f} nats")
print(f"perplexity: {perplexity:.2f}")

token NLL: [0.693, 2.303, 0.223]
average NLL: 1.073 nats
perplexity: 2.92

The output means the model behaved, on average, as though it faced about 2.92 equally likely choices at each prediction step. That effective choice count is an interpretation, not a claim that exactly 2.92 vocabulary tokens were available.

Precisely, PPL is the geometric mean of the reciprocals of the observed-token probabilities. It measures surprise on those targets, rather than how uncertain the model feels about its own preferred predictions.

Nor is PPL bounded by vocabulary size. A two-token model that assigns only 0.01 probability to every observed target gets PPL 100. It can be confidently wrong. This is cross-entropy on observed data, not the entropy of the model's own probability distribution.

A uniform guess among $k$ options has probability $1/k$ and perplexity $k$. That's why people read PPL as a branching factor:

import math

for equally_likely_options in [1, 4, 20, 100]:
    probability = 1 / equally_likely_options
    loss = -math.log(probability)
    perplexity = math.exp(loss)
    print(f"{equally_likely_options:3d} options -> loss={loss:.3f}, PPL={perplexity:.1f}")

1 options -> loss=-0.000, PPL=1.0
  4 options -> loss=1.386, PPL=4.0
 20 options -> loss=2.996, PPL=20.0
100 options -> loss=4.605, PPL=100.0

Training logs usually show the same average in nats (natural log). Information theory often uses bits. They're the same surprise in different units: divide nats by $\ln 2$ to get bits per token, and $\mathrm{PPL} = \exp(L) = 2^{L / \ln 2}$.

import math

average_nll_nats = 1.073
bits_per_token = average_nll_nats / math.log(2)
from_nats = math.exp(average_nll_nats)
from_bits = 2 ** bits_per_token

print(f"{bits_per_token:.3f} bits/token")
print(f"exp(nats)={from_nats:.2f}")
print(f"2**(bits)={from_bits:.2f}")

1.548 bits/token
exp(nats)=2.92
2**(bits)=2.92

Because the map is exponential, a constant loss step multiplies PPL by a constant factor. Adding $\ln 2 \approx 0.693$ nats doubles the score:

Those formulas assume you already have probabilities. A real evaluator starts from logits, and a naive conversion can fail before the average is ever computed.

Compute from logits without numerical failure

Models produce logits, not probabilities. A naive implementation calls exp(logit) directly. Large logits can overflow even though the eventual softmax probabilities are ordinary values. Stable log-softmax subtracts the largest logit before exponentiating.

Before running the example, predict which path fails: raw exp(1000), not the shifted log-softmax calculation.

import math

def stable_log_softmax(logits: list[float]) -> list[float]:
    if not logits or not all(math.isfinite(value) for value in logits):
        raise ValueError("expected nonempty finite logits")
    maximum = max(logits)
    shifted = [value - maximum for value in logits]
    log_normalizer = math.log(sum(math.exp(value) for value in shifted))
    return [value - log_normalizer for value in shifted]

logits = [1000.0, 998.0, 997.0]
observed_token_id = 0

try:
    math.exp(logits[observed_token_id])
except OverflowError:
    print("naive exp(logit) overflowed")

log_probabilities = stable_log_softmax(logits)
nll = -log_probabilities[observed_token_id]
print(f"stable NLL={nll:.3f}, PPL={math.exp(nll):.3f}")

naive exp(logit) overflowed
stable NLL=0.170, PPL=1.185

In a framework evaluator, cross-entropy normally applies this stable computation for you.^{[3]Reference 3CrossEntropyLosshttps://docs.pytorch.org/docs/main/generated/torch.nn.CrossEntropyLoss.html} Keeping the subtraction in shifted coordinates also avoids adding a tiny normalizer to a huge logit and losing it to rounding. This helper requires finite logits; reject NaN rather than publishing a score from a broken run.

Stable log-softmax doesn't guarantee that the final exponential fits in a floating-point number. A finite average NLL of 1,000 nats overflows math.exp; keep and compare the loss itself rather than clipping it to make a presentable PPL. An exactly zero probability for an observed target gives infinite NLL and PPL by definition. Distinguish that case from a NaN caused by a broken computation.

A finite PPL is still only a number. The next problem is whether two of those numbers are even comparable.

The comparison contract

A perplexity score is never complete without its units and conditioning rules. At minimum, log:

Record the checkpoint revision as well. For ordinary base-model PPL, use evaluation mode with dropout disabled, no sampling, and unmodified logits. Temperature scaling or top-k filtering changes the distribution being scored. A generation-only API that doesn't expose likelihoods for the supplied target text isn't enough to compute this metric.

Matching those fields makes a checkpoint comparison interpretable, but it doesn't make one corpus represent every domain. Paloma (Perplexity Analysis for Language Model Assessment) was built because perplexity on a single held-out distribution can hide large gaps elsewhere: a model that looks healthy on one source can look much worse on another, even under a controlled protocol.^{[4]Reference 4PALOMA: A Benchmark for Evaluating Language Model Fit.https://arxiv.org/abs/2312.10523}

Log each domain separately. Don't treat an incident-log score as a legal-contract score.

Represent that contract in code before you compare checkpoints:

from dataclasses import dataclass

@dataclass(frozen=True)
class EvaluationContract:
    dataset: str
    tokenizer: str
    context_tokens: int
    stride_tokens: int
    text_policy: str = "raw-text-no-external-normalization"
    objective: str = "causal-next-token"
    special_token_policy: str = "skip-padding"
    first_token_policy: str = "skip-unconditioned-first-token"
    masking_policy: str = "score-each-target-once"
    document_policy: str = "reset-context-per-document"
    scored_span: str = "all-text-targets"

def comparable(left: EvaluationContract, right: EvaluationContract) -> bool:
    return left == right

baseline = EvaluationContract("incident-holdout-v3", "bpe-v7", 2048, 512)
new_checkpoint = EvaluationContract("incident-holdout-v3", "bpe-v7", 2048, 512)
short_context_run = EvaluationContract("incident-holdout-v3", "bpe-v7", 512, 512)
different_masking_run = EvaluationContract(
    "incident-holdout-v3",
    "bpe-v7",
    2048,
    512,
    masking_policy="score-all-window-targets",
)

print("baseline vs new checkpoint:", comparable(baseline, new_checkpoint))
print("baseline vs short context:", comparable(baseline, short_context_run))
print("baseline vs different masking:", comparable(baseline, different_masking_run))

baseline vs new checkpoint: True
baseline vs short context: False
baseline vs different masking: False

Now compare two hypothetical checkpoints on the same five token outcomes. Their probabilities are hand-assigned fixtures; this example doesn't query or load either model.

import math

def perplexity(observed_probabilities: list[float]) -> float:
    if not observed_probabilities or any(
        not math.isfinite(p) or not 0 < p <= 1 for p in observed_probabilities
    ):
        raise ValueError("expected nonempty probabilities in (0, 1]")
    average_nll = math.fsum(-math.log(p) for p in observed_probabilities) / len(
        observed_probabilities
    )
    return math.exp(average_nll)

held_out_probabilities = {
    "checkpoint-0400": [0.31, 0.44, 0.18, 0.52, 0.24],
    "checkpoint-0800": [0.42, 0.59, 0.29, 0.61, 0.35],
}

for name, probabilities in held_out_probabilities.items():
    print(f"{name}: PPL={perplexity(probabilities):.2f}")

checkpoint-0400: PPL=3.18
checkpoint-0800: PPL=2.31

The result supports a narrow statement: checkpoint-0800 predicts tokens in this held-out set better under this protocol. It doesn't yet prove better incident answers. It also doesn't prove the same ranking on a different domain. This helper deliberately requires strictly positive probabilities; don't silently replace a zero-probability target with a small positive value to conceal infinite loss.

Aggregate loss once

Evaluation windows are rarely the same size. Averaging window PPLs gives a short window as much influence as a long one. There's a second problem even when counts match: exponentiation is nonlinear, so averaging exponentiated losses isn't the same as exponentiating their average. Sum NLL weighted by scored-token count, divide once, and exponentiate once.

import math

windows = [
    {"average_nll": 0.50, "scored_tokens": 2},
    {"average_nll": 2.00, "scored_tokens": 8},
]

wrong = sum(math.exp(window["average_nll"]) for window in windows) / len(windows)
total_nll = sum(
    window["average_nll"] * window["scored_tokens"] for window in windows
)
total_tokens = sum(window["scored_tokens"] for window in windows)
right = math.exp(total_nll / total_tokens)

print(f"average of window PPLs: {wrong:.2f}")
print(f"token-weighted corpus PPL: {right:.2f}")

average of window PPLs: 4.52
token-weighted corpus PPL: 5.47

A matched contract still assumes both models emit the same kind of token. When vocabularies differ, the denominator itself has changed.

Different tokenizers need common units

Token-level perplexity depends on tokenization. The string Compile CI log may be four subword tokens for one model and fourteen character tokens for another. A probability event per large subword isn't the same unit as a probability event per character. Hugging Face's perplexity documentation explicitly warns that tokenization affects PPL comparisons.^{[1]Reference 1Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

For the same raw UTF-8 evaluation text, bits per byte (BPB) gives both models one shared denominator. The Pile preferred BPB as a practical comparison unit when tokenizers differ.^{[5]Reference 5The Pile: An 800GB Dataset of Diverse Text for Language Modeling.https://arxiv.org/abs/2101.00027}

Paloma adopts that unit when vocabulary can't be held fixed, and still prefers one shared vocabulary whenever the experiment allows it. BPB scores the canonical token sequence each tokenizer chose; it doesn't marginalize over every valid segmentation of the same bytes.^{[4]Reference 4PALOMA: A Benchmark for Evaluating Language Model Fit.https://arxiv.org/abs/2312.10523}

where $B$ is the byte count of the original scored text. A related metric, bits per character, is useful when a benchmark defines character units instead of bytes. UTF-8 bytes and characters coincide for this ASCII example, but not for text such as é (two UTF-8 bytes).

For this constructed comparison, both models receive a beginning-of-sequence (BOS) token and score every text token, including the first. The BOS itself isn't scored or included in the byte count; neither is an end-of-sequence token. The segmentations and losses below are invented to isolate the denominator effect, not measurements of named tokenizers. If a protocol skips an initial token, its skipped byte span can differ across tokenizers, so record that policy rather than silently comparing different scored text.

Predict the ranking before running the example: raw token PPL should favor the character model because its loss is spread across more tokens. BPB should favor the subword model because its total NLL is lower on the same bytes.

import math

text = "Compile CI log"
byte_count = len(text.encode("utf-8"))
evaluations = [
    {"name": "subword model", "tokens": 4, "total_nll": 8.4},
    {"name": "character model", "tokens": 14, "total_nll": 9.0},
]

for run in evaluations:
    ppl = math.exp(run["total_nll"] / run["tokens"])
    bpb = run["total_nll"] / (byte_count * math.log(2))
    print(f"{run['name']:15s} token PPL={ppl:.2f}, BPB={bpb:.3f}")

print("Lower BPB identifies less surprise on identical bytes.")

subword model   token PPL=8.17, BPB=0.866
character model token PPL=1.90, BPB=0.927
Lower BPB identifies less surprise on identical bytes.

In this fixture the character model looks dramatically better under raw token PPL: its slightly larger total loss is spread over more prediction events. BPB reverses that ranking here: the subword model assigned less surprise to the same bytes. Smaller token units alone don't guarantee a lower PPL for every pair of trained models.

Raw token PPL mixes event size with model fit. A larger vocabulary can emit fewer, coarser tokens, while a larger choice set can change per-token surprise. Those effects can pull in opposite directions, so don't infer a tokenizer or model win from vocabulary size alone. BPB fixes the raw-byte denominator, but it doesn't erase every segmentation effect. Keep logging the tokenizer and use a fixed vocabulary when you can.

Fixing vocabulary is a control for models trained with that vocabulary. Swapping an existing checkpoint's tokenizer to create a shared denominator can make its input IDs incompatible with its learned rows, as the tokenization lesson showed.

Check what survives normalization

A common byte denominator doesn't restore information discarded before prediction. SentencePiece, for example, can apply a custom NFKC variant and whitespace cleanup before segmentation.^{[6]Reference 6Text Normalization in SentencePiecehttps://github.com/google/sentencepiece/blob/master/doc/normalization.md} If one run erases case or repeated spaces, its loss may describe an easier, different target than a run that preserves them.

Try the following deliberately lossy normalizer. The first input contains a full-width capital A and two spaces. Can its output tell you which original input was used?

import math
import unicodedata

def normalize_for_fixture(text: str) -> str:
    return " ".join(unicodedata.normalize("NFKC", text).casefold().split())

inputs = ["Ａ  a", "A a", "a a"]
for raw in inputs:
    normalized = normalize_for_fixture(raw)
    print(f"{raw!r} -> {normalized!r}")

raw = inputs[0]
normalized = normalize_for_fixture(raw)
total_nll = 1.2  # Invented loss for the normalized text, not a model measurement.
raw_bytes = len(raw.encode("utf-8"))
normalized_bytes = len(normalized.encode("utf-8"))
print(f"raw bytes={raw_bytes}; normalized bytes={normalized_bytes}")
print(f"loss / raw bytes: {total_nll / (raw_bytes * math.log(2)):.3f}")
print(f"loss / normalized bytes: {total_nll / (normalized_bytes * math.log(2)):.3f}")

'Ａ  a' -> 'a a'
'A a' -> 'a a'
'a a' -> 'a a'
raw bytes=6; normalized bytes=3
loss / raw bytes: 0.289
loss / normalized bytes: 0.577

The 0.289 value doesn't demonstrate lossless prediction or compression of the six original bytes: the scored text has forgotten which A and spacing occurred. For a normalized-text benchmark, define that target and denominator consistently for both runs. For a raw-text benchmark, audit what each tokenizer preserves, including unknown-token behavior. This fixture uses Python's NFKC, case folding, and whitespace splitting; it isn't an implementation of SentencePiece.

Also report how much raw-text history fits into each token context limit. BPB aligns the output denominator, while 1,024 input tokens can cover different amounts of text under different tokenizers.

BPB fixes the unit. It doesn't tell you how to score a document that's longer than the model's context.

Long documents need a scoring policy

A real evaluation file may contain thousands of tokens, while a model accepts only a fixed number of context tokens. Cutting text into disjoint blocks is fast, but tokens at each block boundary lose usable left context. A strided sliding window reuses context and scores only newly exposed target tokens.

Disjoint blocks can also lose their first target when no BOS or preceding token is supplied. Track which positions were scored, not just how many. A stride smaller than the window leaves overlap; a smaller stride usually gives targets more history but requires more forward passes.

Hugging Face's documentation reports this comparison for GPT-2 Large on WikiText-2: no-overlap stride = 1024 gives PPL 19.44, while stride = 512 gives 16.44 for the same model and corpus. These are published example results, not runs performed by the small lab below. More context improved that score; the model weights didn't change. Extra context can still hurt individual predictions, so a lower score isn't a mathematical guarantee.^{[1]Reference 1Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

The no-overlap number sits near the 19.93 WikiText-2 result reported for the 762M GPT-2 model. That paper used its own preprocessing and invertible de-tokenizers, so treat 19.44 versus 19.93 as the same ballpark rather than a reproduction.^{[7]Reference 7Language Models are Unsupervised Multitask Learners.https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf} As checked on October 10, 2026, the guide's sample code also subtracts batch size from the valid-label count before checking which labels the causal shift removes. That undercounts overlapping windows whose first label is already ignored. Its published numbers illustrate the context trade-off; use the shifted-target count below for your own evaluator.^{[1]Reference 1Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

Before reading the output, predict which positions contribute loss: A supplies initial context, and B through J should each be scored exactly once.

Now simulate which positions a sliding-window loop scores:

tokens = list("ABCDEFGHIJ")
windows = [
    {"context": (0, 5), "score": (1, 5)},
    {"context": (3, 8), "score": (5, 8)},
    {"context": (5, 10), "score": (8, 10)},
]

scored_tokens: list[str] = []
for index, window in enumerate(windows, start=1):
    begin, end = window["context"]
    score_begin, score_end = window["score"]
    context = "".join(tokens[begin:end])
    scored = "".join(tokens[score_begin:score_end])
    scored_tokens.extend(tokens[score_begin:score_end])
    print(f"window {index}: context={context}, newly scored={scored}")

print("scored exactly once:", scored_tokens == tokens[1:])

window 1: context=ABCDE, newly scored=BCDE
window 2: context=DEFGH, newly scored=FGH
window 3: context=FGHIJ, newly scored=IJ
scored exactly once: True

This example has no BOS token, so A is input context rather than a scored target. A model that accepts a BOS prefix can predict the first text token from that prefix instead. In a framework implementation, context-only labels are commonly masked with -100 so cross-entropy ignores them.^{[1]Reference 1Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

This dependency-free evaluation loop uses precomputed token NLL values. A real model supplies the losses; aggregation logic stays the same.

import math

new_target_losses = [
    [0.30, 0.72, 0.51, 0.43],
    [0.27, 0.61, 0.38],
    [0.56, 0.48],
]

total_nll = sum(sum(window) for window in new_target_losses)
scored_tokens = sum(len(window) for window in new_target_losses)
perplexity = math.exp(total_nll / scored_tokens)

print(f"scored tokens={scored_tokens}")
print(f"average NLL={total_nll / scored_tokens:.3f}")
print(f"PPL={perplexity:.2f}")

scored tokens=9
average NLL=0.473
PPL=1.61

For every reported PPL, store max_context_tokens, stride_tokens, the first-token policy, and the masking policy beside the score. Those details are measurement settings, not implementation trivia.

Count targets after shifting labels

The preceding loop assumed somebody had already extracted the right losses. Here's the boundary where evaluators often go wrong. A logit vector at input position 0 predicts the label at position 1. We therefore pair logits[:, :-1] with labels[:, 1:], then count the labels that remain unmasked.

Run this with PyTorch installed. It uses fixed CPU tensors, so there's no model download or paid API call. The fixture gives each alphabetical next token logit 3 and all other tokens logit 0. It tests alignment and aggregation, not the quality of a trained model.

import math
import torch
import torch.nn.functional as F

token_ids = torch.arange(10)  # A=0 through J=9
windows = [(0, 5, 1), (3, 8, 5), (5, 10, 8)]
total_nll = 0.0
total_targets = 0
wrong_targets = 0

for begin, end, score_begin in windows:
    input_ids = token_ids[begin:end].unsqueeze(0)
    labels = input_ids.clone()
    labels[:, :score_begin - begin] = -100

    logits = torch.zeros(1, end - begin, 10, dtype=torch.float64)
    next_ids = (input_ids + 1) % 10
    logits.scatter_(2, next_ids.unsqueeze(-1), 3.0)

    shifted_logits = logits[:, :-1, :].reshape(-1, 10)
    shifted_labels = labels[:, 1:].reshape(-1)
    count = int((shifted_labels != -100).sum())
    loss_sum = F.cross_entropy(
        shifted_logits, shifted_labels, ignore_index=-100, reduction="sum"
    )
    # Wrong: position 0 was already masked, so subtracting one drops a target.
    wrong_count = int((labels != -100).sum()) - labels.shape[0]
    print(f"window {begin}:{end}: correct={count}, wrong={wrong_count}")
    total_nll += loss_sum.item()
    total_targets += count
    wrong_targets += wrong_count

assert total_targets == 9
expected_nll = math.log1p(9 * math.exp(-3))
assert math.isclose(total_nll, 9 * expected_nll, rel_tol=1e-12)
print(f"correct targets={total_targets}; wrong targets={wrong_targets}")
print(f"fixture PPL={math.exp(total_nll / total_targets):.3f}")

# Two padded rows contribute different numbers of targets and different losses.
batch_labels = torch.tensor([[-100, 1, 2], [-100, -100, 3]])
batch_logits = torch.zeros(2, 3, 10, dtype=torch.float64)
batch_logits[0, 0, 1] = batch_logits[0, 1, 2] = 3.0
shifted = batch_labels[:, 1:]
losses = F.cross_entropy(
    batch_logits[:, :-1].reshape(-1, 10), shifted.reshape(-1),
    ignore_index=-100, reduction="none",
).reshape(2, 2)
counts = (shifted != -100).sum(dim=1)
assert counts.tolist() == [2, 1]
correct_mean = losses.sum() / counts.sum()
wrong_mean = (losses.sum(dim=1) / counts).mean()
print(f"uneven batch targets={counts.tolist()}")
print(f"token-weighted NLL={correct_mean.item():.3f}; row-averaged NLL={wrong_mean.item():.3f}")

empty_labels = torch.full((2,), -100)
empty_loss = F.cross_entropy(torch.zeros(2, 10), empty_labels, reduction="mean")
assert torch.isnan(empty_loss)
print("all-ignored mean loss is NaN:", bool(torch.isnan(empty_loss)))

window 0:5: correct=4, wrong=3
window 3:8: correct=3, wrong=2
window 5:10: correct=2, wrong=1
correct targets=9; wrong targets=6
fixture PPL=1.448
uneven batch targets=[2, 1]
token-weighted NLL=1.014; row-averaged NLL=1.336
all-ignored mean loss is NaN: True

Counting after the shift works whether position 0 was masked or not. Subtracting the batch size from the number of valid unshifted labels doesn't: it removes a valid target even when the shifted-away label was already ignored. PyTorch's ignore_index removes ignored labels from both the loss and the mean denominator.^{[3]Reference 3CrossEntropyLosshttps://docs.pytorch.org/docs/main/generated/torch.nn.CrossEntropyLoss.html}

The padded batch contributes two easier targets from its first row and one harder target from its second. Averaging the two row means gives the single harder target half the weight, instead of one third. Accumulate the summed losses and valid-target counts across batches, or across workers in a distributed evaluation, before dividing. An all-ignored batch has no defined mean loss; skip its contribution after checking its target count. If the entire evaluation has no targets, reject the run.

With a real model, call model.eval() and use torch.inference_mode() around the forward pass, then use its logits here. Don't shift twice if you use a model wrapper's already-shifted loss. Padding needs both an attention mask for the model and ignored target labels for the loss. Reject an evaluation with zero valid targets rather than dividing by zero.

Try removing the masking line. The three windows will then score 4 + 4 + 4 targets, repeating E, G, and H. The assertion should fail. Restore masking, then change the preferred logit from 3 to 0: all ten vocabulary items become equally likely, so PPL should become 10 (update the analytic check too).

A carefully measured PPL still answers only one question: how surprised was the model by held-out tokens?

PPL answers one question, not every question

Suppose your incident assistant predicts common status-log language fluently but recommends the wrong rollback step. Perplexity can reward fluent next-token prediction without detecting that operational failure. Likewise, changing decoding strategy can change generated text quality even when the underlying model is unchanged, as Holtzman et al. demonstrated when studying repetitive neural generation.^{[8]Reference 8The Curious Case of Neural Text Degeneration.https://arxiv.org/abs/1904.09751}

Use PPL for the question it answers:

candidates = [
    {"name": "fluent-wrong", "ppl": 8.9, "policy_checks_passed": 1},
    {"name": "grounded-answer", "ppl": 9.8, "policy_checks_passed": 3},
]

best_language_fit = min(candidates, key=lambda row: row["ppl"])
best_product_answer = max(candidates, key=lambda row: row["policy_checks_passed"])

print("best held-out language fit:", best_language_fit["name"])
print("best incident answer result:", best_product_answer["name"])

best held-out language fit: fluent-wrong
best incident answer result: grounded-answer

For this fixture, the supplied incident says the status is blocked, cites incident_policy_483, and forbids an automatic rollback. Check these structured fields before reaching for a model judge. A matching source ID checks citation identity, not whether a free-text claim is supported by that source.

EXPECTED_STATUS = "blocked"
REQUIRED_SOURCE = "incident_policy_483"

def score_answer(answer: dict[str, object]) -> tuple[int, list[str]]:
    failures: list[str] = []
    if answer.get("status") != EXPECTED_STATUS:
        failures.append("wrong status")
    if answer.get("source") != REQUIRED_SOURCE:
        failures.append("wrong source ID")
    if answer.get("automatic_rollback") is not False:
        failures.append("unsafe or missing rollback decision")
    return 3 - len(failures), failures

answers = [
    {"name": "grounded-answer", "status": "blocked",
     "source": "incident_policy_483", "automatic_rollback": False},
    {"name": "fluent-wrong", "status": "blocked",
     "source": "unrelated_runbook", "automatic_rollback": True},
]

for answer in answers:
    score, failures = score_answer(answer)
    print(answer["name"], score, failures or ["pass"])

grounded-answer 3 ['pass']
fluent-wrong 1 ['wrong source ID', 'unsafe or missing rollback decision']

Open-ended tone and partial credit may need a rubric. LLM judges can scale that review, but the 2023 MT-Bench study found position bias and verbosity bias in its tested judges. It also investigated possible preference for a judge's own responses, without establishing that effect conclusively. Those experiments don't determine the biases of every current judge; measure the model, prompt, and rubric you actually use.^{[9]Reference 9Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

Treat a judge as a calibrated instrument, not as ground truth. LLM Benchmarks & Limitations builds those controls after you have source documents you can cite. Keep PPL in its lane here: it doesn't substitute for product evidence.

Even a clean split between PPL and product checks fails if the evaluation records were in the training set.

Keep evaluation data clean

PPL needs held-out text. If training data includes your evaluation records, lower loss may reflect memorization rather than generalization. Product task suites have the same failure: if prompt examples or fine-tuning rows include hidden test tickets, release metrics lose meaning.

training_record_ids = {"ticket-101", "ticket-102", "ticket-103"}
validation_record_ids = {"ticket-201", "ticket-202", "ticket-103"}

overlap = training_record_ids & validation_record_ids
if overlap:
    print("FAIL leaked record ids:", sorted(overlap))
else:
    print("PASS validation set is disjoint")

FAIL leaked record ids: ['ticket-103']

Disjoint IDs catch exact record reuse, not copied text under new IDs or near-duplicates. Audit normalized text and provenance too. Removing a record from today's training manifest doesn't remove it from an already trained checkpoint; replace the contaminated evaluation material or retrain without it. Keep checkpoint-selection data separate from the final test set, since repeatedly tuning against test results also weakens generalization evidence.

For public LLM benchmarks, test content can also enter later training corpora. LiveBench addresses that risk with frequently updated questions from recent sources and objective ground-truth scoring; it limits contamination risk rather than making every future score immune to leakage.^{[10]Reference 10LiveBench: A Challenging, Contamination-Limited LLM Benchmarkhttps://arxiv.org/abs/2406.19314}

Build an evaluation report

Bring the pieces together in a report. These illustrative values describe a hypothetical run, not the three-token or PyTorch fixture. Store the scored-target count as well as PPL so a tiny sample can't look like a full evaluation. The eligibility check below enforces a few necessary conditions; passing three fixture checks isn't evidence that a real deployment is safe.

from dataclasses import dataclass
import math

@dataclass(frozen=True)
class Report:
    checkpoint: str
    perplexity: float
    dataset: str
    tokenizer: str
    text_policy: str
    context_tokens: int
    stride_tokens: int
    objective: str
    special_token_policy: str
    first_token_policy: str
    masking_policy: str
    document_policy: str
    scored_span: str
    scored_tokens: int
    leaked_records: int
    policy_checks_passed: int
    policy_checks_required: int

def release_gate(report: Report) -> str:
    if (
        type(report.perplexity) not in (int, float)
        or not math.isfinite(report.perplexity)
        or report.perplexity < 1
    ):
        return "BLOCK: invalid perplexity"
    counts = (
        report.context_tokens, report.stride_tokens, report.scored_tokens,
        report.leaked_records, report.policy_checks_passed,
        report.policy_checks_required,
    )
    if any(type(value) is not int for value in counts):
        return "BLOCK: counts must be integers"
    if any(value < 0 for value in counts):
        return "BLOCK: negative counts"
    if report.scored_tokens <= 0:
        return "BLOCK: no scored targets"
    if not 0 < report.stride_tokens < report.context_tokens:
        return "BLOCK: invalid overlapping-window protocol"
    if not 0 <= report.policy_checks_passed <= report.policy_checks_required:
        return "BLOCK: invalid product-check counts"
    if report.leaked_records:
        return "BLOCK: contaminated evaluation set"
    if report.policy_checks_required <= 0 or report.policy_checks_passed < report.policy_checks_required:
        return "BLOCK: product regressions"
    return "ELIGIBLE: recorded checks passed; inspect the full report"

report = Report(
    checkpoint="incident-lm-0800",
    perplexity=9.81,
    dataset="incident-holdout-v3",
    tokenizer="bpe-v7",
    text_policy="raw-text-no-external-normalization",
    context_tokens=2048,
    stride_tokens=512,
    objective="causal-next-token",
    special_token_policy="skip-padding",
    first_token_policy="skip-unconditioned-first-token",
    masking_policy="score-each-target-once",
    document_policy="reset-context-per-document",
    scored_span="all-text-targets",
    scored_tokens=12000,
    leaked_records=0,
    policy_checks_passed=3,
    policy_checks_required=3,
)

print(f"{report.checkpoint}: PPL={report.perplexity} @ {report.context_tokens}/{report.stride_tokens}")
print(
    "protocol:",
    report.objective,
    report.special_token_policy,
    report.first_token_policy,
    report.masking_policy,
)
print(release_gate(report))

incident-lm-0800: PPL=9.81 @ 2048/512
protocol: causal-next-token skip-padding skip-unconditioned-first-token score-each-target-once
ELIGIBLE: recorded checks passed; inspect the full report

The gate requires a finite score for this fixture's eligibility check; infinite PPL isn't accepted, even though it has a mathematical meaning. The dataclass annotations don't validate values at runtime, so the gate rejects Boolean and fractional counts explicitly. It can't verify a self-reported leaked_records=0: link the report to the audit and corpus revision.

Likewise, keep total NLL, per-document losses, domain breakdowns, and product failures alongside this summary. For small checkpoint differences, compare paired document losses. An uncertainty estimate should respect the sampling unit: resample independent documents, or larger groups when documents share a source, rather than treating adjacent tokens as independent. A single corpus PPL hides both variation and the amount of evidence behind a ranking.

Use dataclasses.replace to test the gate: set perplexity=float("nan"), scored_tokens=0, leaked_records=1, or policy_checks_passed=2. Each must block for its corresponding reason. These failures are more informative than another happy-path score.

Perplexity rules worth keeping

• Perplexity is exp(average NLL): an interpretable view of held-out next-token surprise.
• Raw PPL comparison requires the same dataset, tokenizer, objective, context, stride, and masking policy. One domain still doesn't stand in for another.
• Bits per byte puts models with different tokenizers onto one raw-text denominator. A shared vocabulary is even better when you can hold it fixed.
• Long-document evaluation must score new target tokens once while reusing context and aggregating loss before exponentiating.
• Low PPL doesn't establish factual, useful, or safe outputs; application checks and calibrated review answer those questions.
• Leakage invalidates confident evaluation claims, whether the set measures PPL or product behavior.

Mastery check

Key concepts

• Held-out next-token likelihood
• Cross-entropy to perplexity
• Stable log-probability scoring
• Evaluation protocol contracts
• Bits-per-byte normalization
• Strided context windows
• Domain-specific held-out fit
• Intrinsic versus product quality
• Leakage-resistant evaluation sets

Evaluation rubric

• Foundational: Computes token surprise, average NLL, and PPL from observed probabilities.
• Intermediate: Explains effective choice count without treating it as a vocabulary-size claim.
• Intermediate: Rejects invalid raw PPL comparisons by checking protocol fields.
• Intermediate: Refuses to treat one domain's PPL as a score for every other domain.
• Advanced: Uses BPB when tokenizers differ and aggregates strided loss correctly.
• Advanced: Designs a report that separates language-fit metrics from product and leakage gates.

Follow-up questions

Common pitfalls

Comparing scores without a protocol

• Symptom: A team declares victory from PPL 10 versus PPL 12 but can't name the tokenizer, data split, or stride.

• Cause: The score was treated as a universal model rating instead of a metric with units and conditioning rules.

• Fix: Store the evaluation contract with every result and compare raw PPL only when contracts match.

Averaging window perplexities

• Symptom: Long-document PPL changes when window boundaries move, even though scored token losses are unchanged.

• Cause: Per-window perplexities were averaged directly.

• Fix: Sum token NLL across all windows, divide by scored-token count once, then exponentiate.

Selecting chat behavior using language fit alone

• Symptom: A fluent model ships a wrong policy answer because it had the lowest PPL.

• Cause: Intrinsic next-token evaluation was confused with application correctness.

• Fix: Gate releases on deterministic task checks and calibrated review in addition to base-model fit metrics.

Treating one domain as the whole story

• Symptom: Incident-log PPL improved, so the team skips measuring legal-contract or code holdouts.

• Cause: A single held-out mix was treated as a universal language-fit score.

• Fix: Keep the protocol fixed, then measure each domain you care about. Paloma exists because those gaps can be large.

Testing on leaked records

• Symptom: Evaluation looks unusually strong, then fails on genuinely new tickets.

• Cause: Training or prompt examples overlap with hidden evaluation data.

• Fix: Enforce disjoint identifiers, keep private held-out records, and rotate realistic challenge cases.