LearnCore LLM FoundationsPerplexity & Model Evaluation

📊MediumEvaluation & Benchmarks

Perplexity & Model Evaluation

Compute perplexity from held-out token probabilities, compare models under a fixed protocol, normalize across tokenizers, and decide what PPL can't tell you.

20 min read

Learning path

Step 56 of 196 in the full curriculum

Static to Contextual Embeddings File Ingestion for AI

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

After tokenization, a model looks up an embedding for each token ID. Those vectors are the starting geometry for prediction. The remaining question is narrower: when the real next token appears, how much probability did the model assign to it?

Suppose a held-out incident log begins The request returned. A model that assigns high probability to 500 and almost none to volcano fits this local pattern. A model that treats both as equally plausible doesn't. Perplexity turns that held-out behavior into one number you can compare across checkpoints, as long as the measurement contract stays fixed.

That number can still point in the wrong product direction. One checkpoint may predict incident prose better yet recommend an unsafe rollback, while another scores worse on tokens and passes the source and policy checks. Perplexity tells you which model fits this text; it doesn't choose the release.

Prediction-fit metric: Perplexity is a next-token fit metric for causal language models. It's useful when you keep the evaluation contract fixed. It isn't a score for factuality, helpfulness, or safe product behavior.

From surprise to a metric

A causal language model assigns a probability to every possible next token. During evaluation, you don't reward the model for a token it could have emitted. You score the probability it assigned to the token that occurred in held-out text.

If the observed token has probability $p$ , its negative log-likelihood (NLL), or surprise, is:

\text{surprise}(p) = -\ln(p)

A certain correct prediction has probability 1 and surprise 0. A probability close to 0 produces a large penalty. Confident misses hurt language-model loss the most.

Before running the small calculation, predict which token contributes the largest surprise. It should be volcano: its probability is closest to zero.

compare-token-surprise.py

import math

probabilities = {
    "500": 0.60,
    "200": 0.25,
    "volcano": 0.001,
}

for token in ["500", "200", "volcano"]:
    surprise = -math.log(probabilities[token])
    print(f"{token:9s} probability={probabilities[token]:.3f} surprise={surprise:.3f} nats")

Output

500       probability=0.600 surprise=0.511 nats
200       probability=0.250 surprise=1.386 nats
volcano   probability=0.001 surprise=6.908 nats

The held-out token matters. If the log says 500, the first score counts. It doesn't matter that 200 also sounded reasonable: likelihood evaluates the text the model was asked to predict.

Average surprise becomes perplexity

For held-out tokens $x_1, x_2, \ldots, x_N$ , the average NLL is:

L = -\frac{1}{N}\sum_{i=1}^{N}\ln p_\theta(x_i \mid x_{<i})

For one-hot next-token targets, that average NLL is also the causal cross-entropy loss: the model's average penalty on the observed next tokens.

Perplexity exponentiates that average:

\text{PPL} = \exp(L)

This is the standard definition used for autoregressive, or causal, language models. Speech-recognition work already treated perplexity as an effective branching factor; causal LMs report the same exponentiated-loss form today.^{[1]Reference 1Perplexity — a Measure of the Difficulty of Speech Recognition Tasks.https://doi.org/10.1121/1.2016299} It isn't the standard metric for masked models such as BERT, because they predict masked positions rather than the next token in sequence.^{[2]Reference 2Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

Start with three observed token probabilities. The 0.10 token should dominate the average NLL, but the final PPL should be computed only after that average:

compute-perplexity-from-probabilities.py

import math

observed_probabilities = [0.50, 0.10, 0.80]
token_nll = [-math.log(probability) for probability in observed_probabilities]
average_nll = sum(token_nll) / len(token_nll)
perplexity = math.exp(average_nll)

print(f"token NLL: {[round(value, 3) for value in token_nll]}")
print(f"average NLL: {average_nll:.3f} nats")
print(f"perplexity: {perplexity:.2f}")

Output

token NLL: [0.693, 2.303, 0.223]
average NLL: 1.073 nats
perplexity: 2.92

The output means the model behaved, on average, as though it faced about 2.92 equally likely choices at each prediction step. That effective choice count is an interpretation, not a claim that exactly 2.92 vocabulary tokens were available.

Worked perplexity pipeline converting observed probabilities 0.50, 0.10, and 0.80 into token NLL values, average NLL 1.073, and perplexity 2.92. — The worked example keeps every aggregation step visible: transform each observed probability, average the three token losses, and exponentiate 1.073 to obtain PPL 2.92.

A uniform guess among $k$ options has probability $1/k$ and perplexity $k$ . That's why people read PPL as a branching factor:

recover-effective-choice-count.py

import math

for equally_likely_options in [1, 4, 20, 100]:
    probability = 1 / equally_likely_options
    loss = -math.log(probability)
    perplexity = math.exp(loss)
    print(f"{equally_likely_options:3d} options -> loss={loss:.3f}, PPL={perplexity:.1f}")

Output

options -> loss=-0.000, PPL=1.0
options -> loss=1.386, PPL=4.0
options -> loss=2.996, PPL=20.0
options -> loss=4.605, PPL=100.0

Training logs usually show the same average in nats (natural log). Information theory often uses bits. They're the same surprise in different units: divide nats by $\ln 2$ to get bits per token, and $\mathrm{PPL} = \exp(L) = 2^{L / \ln 2}$ .

convert-nats-to-bits.py

import math

average_nll_nats = 1.073
bits_per_token = average_nll_nats / math.log(2)
from_nats = math.exp(average_nll_nats)
from_bits = 2 ** bits_per_token

print(f"{bits_per_token:.3f} bits/token")
print(f"exp(nats)={from_nats:.2f}")
print(f"2**(bits)={from_bits:.2f}")

Output

1.548 bits/token
exp(nats)=2.92
2**(bits)=2.92

Because the map is exponential, a constant loss step multiplies PPL by a constant factor. Adding $\ln 2 \approx 0.693$ nats doubles the score:

Continuous exponential curve mapping average negative log-likelihood to perplexity, with equal 0.693-nat steps doubling perplexity from 5 to 10 to 20. — The x-axis uses a continuous loss scale: each additional ln(2), about 0.693 nats, doubles perplexity, so 5 becomes 10 and then 20.

Those formulas assume you already have probabilities. A real evaluator starts from logits, and a naive conversion can fail before the average is ever computed.

Compute from logits without numerical failure

Models produce logits, not probabilities. A naive implementation calls exp(logit) directly. Large logits can overflow even though the eventual softmax probabilities are ordinary values. Stable log-softmax subtracts the largest logit before exponentiating.

Before running the example, predict which path fails: raw exp(1000), not the shifted log-softmax calculation.

stable-log-softmax-for-perplexity.py

import math

def stable_log_softmax(logits: list[float]) -> list[float]:
    maximum = max(logits)
    log_normalizer = maximum + math.log(
        sum(math.exp(value - maximum) for value in logits)
    )
    return [value - log_normalizer for value in logits]

logits = [1000.0, 998.0, 997.0]
observed_token_id = 0

try:
    math.exp(logits[observed_token_id])
except OverflowError:
    print("naive exp(logit) overflowed")

log_probabilities = stable_log_softmax(logits)
nll = -log_probabilities[observed_token_id]
print(f"stable NLL={nll:.3f}, PPL={math.exp(nll):.3f}")

Output

naive exp(logit) overflowed
stable NLL=0.170, PPL=1.185

In a framework evaluator, cross-entropy normally applies this stable computation for you. You still need to know the principle when debugging inf losses, implementing metrics, or reviewing a custom evaluation loop.

A finite PPL is still only a number. The next problem is whether two of those numbers are even comparable.

The comparison contract

A perplexity score is never complete without its units and conditioning rules. At minimum, log:

Contract field	Why it changes the score
Dataset and split	An incident-log corpus isn't a legal-contract corpus; train data isn't held-out data.
Tokenizer revision	Tokens set the denominator and the events being predicted.
Context length and stride	More usable left context generally makes token prediction easier.
Special-token and masking policy	Scoring or skipping initial and padding tokens changes the aggregate.
Model objective	A causal next-token model isn't directly comparable to a masked-language objective.

Matching those fields makes a checkpoint comparison interpretable, but it doesn't make one corpus represent every domain. Paloma (Perplexity Analysis for Language Model Assessment) was built because perplexity on a single held-out distribution can hide large gaps elsewhere: a model that looks healthy on one source can look much worse on another, even under a controlled protocol.^{[3]Reference 3PALOMA: A Benchmark for Evaluating Language Model Fit.https://arxiv.org/abs/2312.10523}

Log each domain separately. Don't treat an incident-log score as a legal-contract score.

Two evaluation-contract matrices showing a valid checkpoint comparison with every field matched and an invalid comparison blocked by a 2048-versus-512 context mismatch. — The left matrix supports the exact 3.18-to-2.31 checkpoint comparison. On the right, changing context from 2048 to 512 changes the measurement, so the verdict is withheld.

Represent that contract in code before you compare checkpoints:

enforce-a-perplexity-contract.py

from dataclasses import dataclass

@dataclass(frozen=True)
class EvaluationContract:
    dataset: str
    tokenizer: str
    context_tokens: int
    stride_tokens: int
    objective: str = "causal-next-token"
    special_token_policy: str = "skip-padding"
    first_token_policy: str = "skip-unconditioned-first-token"
    masking_policy: str = "score-each-target-once"

def comparable(left: EvaluationContract, right: EvaluationContract) -> bool:
    return left == right

baseline = EvaluationContract("incident-holdout-v3", "bpe-v7", 2048, 512)
new_checkpoint = EvaluationContract("incident-holdout-v3", "bpe-v7", 2048, 512)
short_context_run = EvaluationContract("incident-holdout-v3", "bpe-v7", 512, 512)
different_masking_run = EvaluationContract(
    "incident-holdout-v3",
    "bpe-v7",
    2048,
    512,
    masking_policy="score-all-window-targets",
)

print("baseline vs new checkpoint:", comparable(baseline, new_checkpoint))
print("baseline vs short context:", comparable(baseline, short_context_run))
print("baseline vs different masking:", comparable(baseline, different_masking_run))

Output

baseline vs new checkpoint: True
baseline vs short context: False
baseline vs different masking: False

Now compare two model checkpoints on the same token outcomes:

compare-checkpoints-on-one-held-out-set.py

import math

def perplexity(observed_probabilities: list[float]) -> float:
    average_nll = sum(-math.log(p) for p in observed_probabilities) / len(
        observed_probabilities
    )
    return math.exp(average_nll)

held_out_probabilities = {
    "checkpoint-0400": [0.31, 0.44, 0.18, 0.52, 0.24],
    "checkpoint-0800": [0.42, 0.59, 0.29, 0.61, 0.35],
}

for name, probabilities in held_out_probabilities.items():
    print(f"{name}: PPL={perplexity(probabilities):.2f}")

Output

checkpoint-0400: PPL=3.18
checkpoint-0800: PPL=2.31

The result supports a narrow statement: checkpoint-0800 predicts tokens in this held-out set better under this protocol. It doesn't yet prove better incident answers. It also doesn't prove the same ranking on a different domain.

Aggregate loss once

Evaluation windows are rarely the same size. Averaging each window's already exponentiated PPL gives every window equal influence, so a short window can offset a long one too much. Sum NLL weighted by scored-token count, divide once, and exponentiate once.

aggregate-loss-before-exponentiating.py

import math

windows = [
    {"average_nll": 0.50, "scored_tokens": 2},
    {"average_nll": 2.00, "scored_tokens": 8},
]

wrong = sum(math.exp(window["average_nll"]) for window in windows) / len(windows)
total_nll = sum(
    window["average_nll"] * window["scored_tokens"] for window in windows
)
total_tokens = sum(window["scored_tokens"] for window in windows)
right = math.exp(total_nll / total_tokens)

print(f"average of window PPLs: {wrong:.2f}")
print(f"token-weighted corpus PPL: {right:.2f}")

Output

average of window PPLs: 4.52
token-weighted corpus PPL: 5.47

A matched contract still assumes both models emit the same kind of token. When vocabularies differ, the denominator itself has changed.

Different tokenizers need common units

Token-level perplexity depends on tokenization. The string Compile CI log may be four subword tokens for one model and fourteen character tokens for another. A probability event per large subword isn't the same unit as a probability event per character. Hugging Face's perplexity documentation explicitly warns that tokenization affects PPL comparisons.^{[2]Reference 2Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

For the same raw UTF-8 evaluation text, bits per byte (BPB) gives both models one shared denominator. The Pile preferred BPB as a practical comparison unit when tokenizers differ.^{[4]Reference 4The Pile: An 800GB Dataset of Diverse Text for Language Modeling.https://arxiv.org/abs/2101.00027}

Paloma adopts that unit when vocabulary can't be held fixed, and still prefers one shared vocabulary whenever the experiment allows it. BPB scores the canonical token sequence each tokenizer chose; it doesn't marginalize over every valid segmentation of the same bytes.^{[3]Reference 3PALOMA: A Benchmark for Evaluating Language Model Fit.https://arxiv.org/abs/2312.10523}

\text{BPB} = \frac{-\sum_i \ln p_\theta(x_i \mid x_{<i})} {B \ln 2}

where $B$ is the byte count of the original text. A related metric, bits per character, is useful when a benchmark defines character units instead of bytes.

Byte-aligned comparison of four subword tokens and fourteen character tokens for Compile CI log, showing raw token PPL ranks the character model lower while bits per byte ranks the subword model lower. — Centered dots mark spaces. The same fourteen bytes produce four subword events or fourteen character events, so raw token PPL favors the smaller units while BPB compares total surprise over the shared byte count.

Predict the ranking before running the example: raw token PPL should favor the character model because its loss is spread across more tokens. BPB should favor the subword model because its total NLL is lower on the same bytes.

compare-tokenizers-with-bits-per-byte.py

import math

text = "Compile CI log"
byte_count = len(text.encode("utf-8"))
evaluations = [
    {"name": "subword model", "tokens": 4, "total_nll": 8.4},
    {"name": "character model", "tokens": 14, "total_nll": 9.0},
]

for run in evaluations:
    ppl = math.exp(run["total_nll"] / run["tokens"])
    bpb = run["total_nll"] / (byte_count * math.log(2))
    print(f"{run['name']:15s} token PPL={ppl:.2f}, BPB={bpb:.3f}")

print("Lower BPB identifies less surprise on identical bytes.")

Output

subword model   token PPL=8.17, BPB=0.866
character model token PPL=1.90, BPB=0.927
Lower BPB identifies less surprise on identical bytes.

The character model looks dramatically better under raw token PPL because it predicts smaller units. BPB reverses that ranking here: the subword model assigned less surprise to the same bytes.

Raw token PPL mixes event size with model fit. A larger vocabulary can emit fewer, coarser tokens, while a larger choice set can change per-token surprise. Those effects can pull in opposite directions, so don't infer a tokenizer or model win from vocabulary size alone. BPB fixes the raw-byte denominator, but it doesn't erase every segmentation effect. Keep logging the tokenizer and use a fixed vocabulary when you can.

BPB fixes the unit. It doesn't tell you how to score a document that's longer than the model's context.

Long documents need a scoring policy

A real evaluation file may contain thousands of tokens, while a model accepts only a fixed number of context tokens. Cutting text into disjoint blocks is fast, but tokens at each block boundary lose usable left context. A strided sliding window reuses context and scores only newly exposed target tokens.

Hugging Face demonstrates this protocol for GPT-2 Large on WikiText-2: a no-overlap stride = 1024 run reports PPL 19.44, while stride = 512 reports 16.44 for the same model and corpus. More context improved the score; the model weights didn't change.^{[2]Reference 2Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

The no-overlap number sits near the 19.93 WikiText-2 result reported for the 762M GPT-2 model. That paper used its own preprocessing and invertible de-tokenizers, so treat 19.44 versus 19.93 as the same ballpark rather than a reproduction.^{[5]Reference 5Language Models are Unsupervised Multitask Learners.https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf}

Exact three-window causal language model evaluation over tokens A through J, with reused context in blue, newly scored tokens in green, and every target B through J scored exactly once. — The matrix matches the worked loop exactly: BCDE, FGH, and IJ contribute nine losses total, while overlapping blue tokens provide context without entering the denominator again.

Before reading the output, predict which positions contribute loss: A supplies initial context, and B through J should each be scored exactly once.

Now simulate which positions a sliding-window loop scores:

score-each-target-token-once.py

tokens = list("ABCDEFGHIJ")
windows = [
    {"context": (0, 5), "score": (1, 5)},
    {"context": (3, 8), "score": (5, 8)},
    {"context": (5, 10), "score": (8, 10)},
]

scored_tokens: list[str] = []
for index, window in enumerate(windows, start=1):
    begin, end = window["context"]
    score_begin, score_end = window["score"]
    context = "".join(tokens[begin:end])
    scored = "".join(tokens[score_begin:score_end])
    scored_tokens.extend(tokens[score_begin:score_end])
    print(f"window {index}: context={context}, newly scored={scored}")

print("scored exactly once:", scored_tokens == tokens[1:])

Output

window 1: context=ABCDE, newly scored=BCDE
window 2: context=DEFGH, newly scored=FGH
window 3: context=FGHIJ, newly scored=IJ
scored exactly once: True

The first token is input context because a causal model needs a previous position before it can score a next-token target. In a framework implementation, context-only labels are commonly masked with -100 so cross-entropy ignores them.^{[2]Reference 2Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

This dependency-free evaluation loop uses precomputed token NLL values. A real model supplies the losses; aggregation logic stays the same.

aggregate-a-strided-evaluation-run.py

import math

new_target_losses = [
    [0.30, 0.72, 0.51, 0.43],
    [0.27, 0.61, 0.38],
    [0.56, 0.48],
]

total_nll = sum(sum(window) for window in new_target_losses)
scored_tokens = sum(len(window) for window in new_target_losses)
perplexity = math.exp(total_nll / scored_tokens)

print(f"scored tokens={scored_tokens}")
print(f"average NLL={total_nll / scored_tokens:.3f}")
print(f"PPL={perplexity:.2f}")

Output

scored tokens=9
average NLL=0.473
PPL=1.61

For every reported PPL, store max_context_tokens, stride_tokens, the first-token policy, and the masking policy beside the score. Those details are measurement settings, not implementation trivia.

A carefully measured PPL still answers only one question: how surprised was the model by held-out tokens?

PPL answers one question, not every question

Suppose your incident assistant predicts common status-log language fluently but recommends the wrong rollback step. Perplexity can reward fluent next-token prediction without detecting that operational failure. Likewise, changing decoding strategy can change generated text quality even when the underlying model is unchanged, as Holtzman et al. demonstrated when studying repetitive neural generation.^{[6]Reference 6The Curious Case of Neural Text Degeneration.https://arxiv.org/abs/1904.09751}

Use PPL for the question it answers:

Decision	Useful measurement
Did a base-model checkpoint get better at held-out next-token prediction?	PPL under fixed protocol, or BPB across tokenizers
Did the assistant provide the correct incident status and cite supplied evidence?	Task-specific deterministic checks
Did an open-ended reply follow a rubric for clarity and groundedness?	Calibrated judge or human review
Is a release safe for a high-impact workflow?	Task regressions plus human-reviewed edge cases

separate-language-fit-from-product-correctness.py

candidates = [
    {"name": "fluent-wrong", "ppl": 8.9, "policy_checks_passed": 1},
    {"name": "grounded-answer", "ppl": 9.8, "policy_checks_passed": 3},
]

best_language_fit = min(candidates, key=lambda row: row["ppl"])
best_product_answer = max(candidates, key=lambda row: row["policy_checks_passed"])

print("best held-out language fit:", best_language_fit["name"])
print("best incident answer result:", best_product_answer["name"])

Output

best held-out language fit: fluent-wrong
best incident answer result: grounded-answer

Two incident-assistant candidates on the same axes: fluent-wrong has PPL 8.9 and passes 1 of 3 product checks, while grounded-answer has PPL 9.8 and passes all 3, so the release ranking reverses the language-fit ranking. — Lower PPL picks fluent-wrong. The release gate picks grounded-answer, because status, evidence, and rollback checks are the product question and they don't move with next-token fit.

Status, citation ID, and a forbidden runbook claim are deterministic. Score those first, before anyone reaches for a model judge.

build-a-deterministic-incident-eval.py

EXPECTED_STATUS = "blocked"
REQUIRED_SOURCE = "incident_policy_483"

def score_answer(answer: dict[str, str]) -> tuple[int, list[str]]:
    failures: list[str] = []
    if answer["status"] != EXPECTED_STATUS:
        failures.append("wrong status")
    if answer["source"] != REQUIRED_SOURCE:
        failures.append("missing evidence")
    return 2 - len(failures), failures

answers = [
    {"name": "A", "status": "blocked", "source": "incident_policy_483"},
    {"name": "B", "status": "approved", "source": "incident_policy_483"},
]

for answer in answers:
    score, failures = score_answer(answer)
    print(answer["name"], score, failures or ["pass"])

Output

A 2 ['pass']
B 1 ['wrong status']

Open-ended tone and partial credit may need a rubric. LLM judges can scale that review, but they bring their own measurement problems: position bias, verbosity bias, and a preference for model-like answers.^{[7]Reference 7Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

Treat a judge as a calibrated instrument, not as ground truth. LLM Benchmarks & Limitations builds those controls after you have source documents you can cite. Keep PPL in its lane here: it doesn't substitute for product evidence.

Even a clean split between PPL and product checks fails if the evaluation records were in the training set.

Keep evaluation data clean

PPL needs held-out text. If training data includes your evaluation records, lower loss may reflect memorization rather than generalization. Product task suites have the same failure: if prompt examples or fine-tuning rows include hidden test tickets, release metrics lose meaning.

fail-on-evaluation-data-leakage.py

training_record_ids = {"ticket-101", "ticket-102", "ticket-103"}
validation_record_ids = {"ticket-201", "ticket-202", "ticket-103"}

overlap = training_record_ids & validation_record_ids
if overlap:
    print("FAIL leaked record ids:", sorted(overlap))
else:
    print("PASS validation set is disjoint")

Output

FAIL leaked record ids: ['ticket-103']

For public LLM benchmarks, test content can also enter later training corpora. LiveBench addresses that risk with frequently updated questions from recent sources and objective ground-truth scoring; it limits contamination risk rather than making every future score immune to leakage.^{[8]Reference 8LiveBench: A Challenging, Contamination-Limited LLM Benchmarkhttps://arxiv.org/abs/2406.19314}

Set-overlap diagram showing ticket-103 shared between training and validation records, followed by a clean evaluation design with a stable private holdout and fresh rotating challenge cases. — The left side reproduces the code failure: ticket-103 appears in both sets. After removal, stable private records support regression tracking while rotating cases test behavior beyond one frozen benchmark.

Build an evaluation report

An engineering metric becomes useful when it ships with enough context to reproduce a decision. A compact report should include metric value, protocol fields, leakage checks, and product task gates.

emit-a-release-evaluation-report.py

from dataclasses import dataclass

@dataclass(frozen=True)
class Report:
    checkpoint: str
    perplexity: float
    dataset: str
    tokenizer: str
    context_tokens: int
    stride_tokens: int
    objective: str
    special_token_policy: str
    first_token_policy: str
    masking_policy: str
    leaked_records: int
    policy_pass_rate: float

def release_gate(report: Report) -> str:
    if report.leaked_records:
        return "BLOCK: contaminated evaluation set"
    if report.policy_pass_rate < 1.0:
        return "BLOCK: product regressions"
    return "PASS: protocol recorded and product checks passed"

report = Report(
    checkpoint="incident-lm-0800",
    perplexity=9.81,
    dataset="incident-holdout-v3",
    tokenizer="bpe-v7",
    context_tokens=2048,
    stride_tokens=512,
    objective="causal-next-token",
    special_token_policy="skip-padding",
    first_token_policy="skip-unconditioned-first-token",
    masking_policy="score-each-target-once",
    leaked_records=0,
    policy_pass_rate=1.0,
)

print(f"{report.checkpoint}: PPL={report.perplexity} @ {report.context_tokens}/{report.stride_tokens}")
print(
    "protocol:",
    report.objective,
    report.special_token_policy,
    report.first_token_policy,
    report.masking_policy,
)
print(release_gate(report))

Output

incident-lm-0800: PPL=9.81 @ 2048/512
protocol: causal-next-token skip-padding skip-unconditioned-first-token score-each-target-once
PASS: protocol recorded and product checks passed

The report refuses two common shortcuts: treating an untrusted held-out set as evidence, and treating language fit as a substitute for application correctness.

Perplexity rules worth keeping

Perplexity is exp(average NLL): an interpretable view of held-out next-token surprise.
Raw PPL comparison requires the same dataset, tokenizer, objective, context, stride, and masking policy. One domain still doesn't stand in for another.
Bits per byte puts models with different tokenizers onto one raw-text denominator. A shared vocabulary is even better when you can hold it fixed.
Long-document evaluation must score new target tokens once while reusing context and aggregating loss before exponentiating.
Low PPL doesn't establish factual, useful, or safe outputs; application checks and calibrated review answer those questions.
Leakage invalidates confident evaluation claims, whether the set measures PPL or product behavior.

Mastery check

Key concepts

Held-out next-token likelihood
Cross-entropy to perplexity
Stable log-probability scoring
Evaluation protocol contracts
Bits-per-byte normalization
Strided context windows
Domain-specific held-out fit
Intrinsic versus product quality
Leakage-resistant evaluation sets

Evaluation rubric

Foundational: Computes token surprise, average NLL, and PPL from observed probabilities.
Intermediate: Explains effective choice count without treating it as a vocabulary-size claim.
Intermediate: Rejects invalid raw PPL comparisons by checking protocol fields.
Intermediate: Refuses to treat one domain's PPL as a score for every other domain.
Advanced: Uses BPB when tokenizers differ and aggregates strided loss correctly.
Advanced: Designs a report that separates language-fit metrics from product and leakage gates.

Follow-up questions

Common pitfalls

Comparing scores without a protocol

Symptom: A team declares victory from PPL 10 versus PPL 12 but can't name the tokenizer, data split, or stride.
Cause: The score was treated as a universal model rating instead of a metric with units and conditioning rules.
Fix: Store the evaluation contract with every result and compare raw PPL only when contracts match.

Averaging window perplexities

Symptom: Long-document PPL changes when window boundaries move, even though scored token losses are unchanged.
Cause: Per-window perplexities were averaged directly.
Fix: Sum token NLL across all windows, divide by scored-token count once, then exponentiate.

Selecting chat behavior using language fit alone

Symptom: A fluent model ships a wrong policy answer because it had the lowest PPL.
Cause: Intrinsic next-token evaluation was confused with application correctness.
Fix: Gate releases on deterministic task checks and calibrated review in addition to base-model fit metrics.

Treating one domain as the whole story

Symptom: Incident-log PPL improved, so the team skips measuring legal-contract or code holdouts.
Cause: A single held-out mix was treated as a universal language-fit score.
Fix: Keep the protocol fixed, then measure each domain you care about. Paloma exists because those gaps can be large.

Testing on leaked records

Symptom: Evaluation looks unusually strong, then fails on genuinely new tickets.
Cause: Training or prompt examples overlap with hidden evaluation data.
Fix: Enforce disjoint identifiers, keep private held-out records, and rotate realistic challenge cases.

Next Step

Continue to File Ingestion for AI

You can now score held-out next-token fit and refuse a contaminated evaluation set. Next you'll turn PDFs, HTML, scans, and Markdown into traceable records, because a PPL run or a status check is only as honest as the source text that reached the corpus.

PreviousStatic to Contextual Embeddings

Share this article