LearnCore LLM FoundationsPerplexity & Model Evaluation

📊MediumEvaluation & Benchmarks

Perplexity & Model Evaluation

Compute perplexity from held-out token probabilities, compare models under a fixed protocol, normalize across tokenizers, and decide what PPL can't tell you.

16 min read

Learning path

Step 51 of 177 in the full curriculum

Static to Contextual Embeddings File Ingestion for AI

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Tokens gave the model its vocabulary. Embeddings gave those tokens useful geometry. Now an engineer needs a measurement: when the real next token appears, how much probability did the model assign to it?

Suppose an incident log begins The request returned. A model that assigns high probability to 500 and low probability to volcano understands this local pattern better than a model that treats both as equally plausible. Perplexity converts that held-out prediction behavior into one number.

You'll build that number from probabilities, implement its failure-resistant details, and write an evaluation report that answers a more important question than "is the score low?": is this score comparable to the previous run, and is it enough for the product decision?

Prediction-fit metric: Perplexity is a next-token fit metric for causal language models. It's useful when you keep the evaluation contract fixed. It isn't a score for factuality, helpfulness, or safe product behavior.

From surprise to a metric

A causal language model assigns a probability to every possible next token. During evaluation, you don't reward the model for a token it could have emitted. You score the probability it assigned to the token that occurred in held-out text.

If the observed token has probability $p$ , its negative log-likelihood (NLL), or surprise, is:

\text{surprise}(p) = -\ln(p)

A certain correct prediction has probability 1 and surprise 0. A probability close to 0 produces a large penalty. Confident misses hurt language-model loss the most.

compare-token-surprise.py

import math

probabilities = {
    "500": 0.60,
    "200": 0.25,
    "volcano": 0.001,
}

for token in ["500", "200", "volcano"]:
    surprise = -math.log(probabilities[token])
    print(f"{token:9s} probability={probabilities[token]:.3f} surprise={surprise:.3f} nats")

Output

500       probability=0.600 surprise=0.511 nats
200       probability=0.250 surprise=1.386 nats
volcano   probability=0.001 surprise=6.908 nats

The held-out token matters. If the log says 500, the first score counts. It doesn't matter that 200 also sounded reasonable: likelihood evaluates the text the model was asked to predict.

Average surprise becomes perplexity

For held-out tokens $x_1, x_2, \ldots, x_N$ , the average NLL is:

L = -\frac{1}{N}\sum_{i=1}^{N}\ln p_\theta(x_i \mid x_{<i})

Perplexity exponentiates that average:

\text{PPL} = \exp(L)

This is the standard definition used for autoregressive, or causal, language models. It isn't the standard metric for masked models such as BERT, because they are trained to predict masked positions rather than the next token in sequence.^{[1]Reference 1Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

Start with three observed token probabilities:

compute-perplexity-from-probabilities.py

import math

observed_probabilities = [0.50, 0.10, 0.80]
token_nll = [-math.log(probability) for probability in observed_probabilities]
average_nll = sum(token_nll) / len(token_nll)
perplexity = math.exp(average_nll)

print(f"token NLL: {[round(value, 3) for value in token_nll]}")
print(f"average NLL: {average_nll:.3f} nats")
print(f"perplexity: {perplexity:.2f}")

Output

token NLL: [0.693, 2.303, 0.223]
average NLL: 1.073 nats
perplexity: 2.92

The output means the model behaved, on average, as though it faced about 2.92 equally likely choices at each prediction step. That effective choice count is an interpretation, not a claim that exactly 2.92 vocabulary tokens were available.

recover-effective-choice-count.py

import math

for equally_likely_options in [1, 4, 20, 100]:
    probability = 1 / equally_likely_options
    loss = -math.log(probability)
    perplexity = math.exp(loss)
    print(f"{equally_likely_options:3d} options -> loss={loss:.3f}, PPL={perplexity:.1f}")

Output

options -> loss=-0.000, PPL=1.0
options -> loss=1.386, PPL=4.0
options -> loss=2.996, PPL=20.0
options -> loss=4.605, PPL=100.0

Continuous exponential curve mapping average negative log-likelihood to perplexity, with equal 0.693-nat steps doubling perplexity from 5 to 10 to 20. — The x-axis uses a continuous loss scale: each additional ln(2), about 0.693 nats, doubles perplexity, so 5 becomes 10 and then 20.

Worked perplexity pipeline converting observed probabilities 0.50, 0.10, and 0.80 into token NLL values, average NLL 1.073, and perplexity 2.92. — The worked example keeps every aggregation step visible: transform each observed probability, average the three token losses, and exponentiate 1.073 to obtain PPL 2.92.

Compute from logits without numerical failure

Models produce logits, not probabilities. A naive implementation calls exp(logit) directly. Large logits can overflow even though the eventual softmax probabilities are ordinary values. Stable log-softmax subtracts the largest logit before exponentiating.

stable-log-softmax-for-perplexity.py

import math

def stable_log_softmax(logits: list[float]) -> list[float]:
    maximum = max(logits)
    log_normalizer = maximum + math.log(
        sum(math.exp(value - maximum) for value in logits)
    )
    return [value - log_normalizer for value in logits]

logits = [1000.0, 998.0, 997.0]
observed_token_id = 0

try:
    math.exp(logits[observed_token_id])
except OverflowError:
    print("naive exp(logit) overflowed")

log_probabilities = stable_log_softmax(logits)
nll = -log_probabilities[observed_token_id]
print(f"stable NLL={nll:.3f}, PPL={math.exp(nll):.3f}")

Output

naive exp(logit) overflowed
stable NLL=0.170, PPL=1.185

In a framework evaluator, cross-entropy normally applies this stable computation for you. You still need to know the principle when debugging inf losses, implementing metrics, or reviewing a custom evaluation loop.

The comparison contract

A perplexity score is never complete without its units and conditioning rules. At minimum, log:

Contract field	Why it changes the score
Dataset and split	An incident-log corpus isn't a legal-contract corpus; train data isn't held-out data.
Tokenizer revision	Tokens set the denominator and the events being predicted.
Context length and stride	More usable left context generally makes token prediction easier.
Special-token and masking policy	Scoring or skipping initial and padding tokens changes the aggregate.
Model objective	A causal next-token model isn't directly comparable to a masked-language objective.

Two evaluation-contract matrices showing a valid checkpoint comparison with every field matched and an invalid comparison blocked by a 2048-versus-512 context mismatch. — The left matrix supports the exact 3.18-to-2.31 checkpoint comparison. On the right, changing context from 2048 to 512 changes the measurement, so the verdict is withheld.

Represent that contract in code before you compare checkpoints:

enforce-a-perplexity-contract.py

from dataclasses import dataclass

@dataclass(frozen=True)
class EvaluationContract:
    dataset: str
    tokenizer: str
    context_tokens: int
    stride_tokens: int
    objective: str = "causal-next-token"
    special_token_policy: str = "skip-padding"
    first_token_policy: str = "skip-unconditioned-first-token"
    masking_policy: str = "score-each-target-once"

def comparable(left: EvaluationContract, right: EvaluationContract) -> bool:
    return left == right

baseline = EvaluationContract("incident-holdout-v3", "bpe-v7", 2048, 512)
new_checkpoint = EvaluationContract("incident-holdout-v3", "bpe-v7", 2048, 512)
short_context_run = EvaluationContract("incident-holdout-v3", "bpe-v7", 512, 512)
different_masking_run = EvaluationContract(
    "incident-holdout-v3",
    "bpe-v7",
    2048,
    512,
    masking_policy="score-all-window-targets",
)

print("baseline vs new checkpoint:", comparable(baseline, new_checkpoint))
print("baseline vs short context:", comparable(baseline, short_context_run))
print("baseline vs different masking:", comparable(baseline, different_masking_run))

Output

baseline vs new checkpoint: True
baseline vs short context: False
baseline vs different masking: False

Now compare two model checkpoints on the same token outcomes:

compare-checkpoints-on-one-held-out-set.py

import math

def perplexity(observed_probabilities: list[float]) -> float:
    average_nll = sum(-math.log(p) for p in observed_probabilities) / len(
        observed_probabilities
    )
    return math.exp(average_nll)

held_out_probabilities = {
    "checkpoint-0400": [0.31, 0.44, 0.18, 0.52, 0.24],
    "checkpoint-0800": [0.42, 0.59, 0.29, 0.61, 0.35],
}

for name, probabilities in held_out_probabilities.items():
    print(f"{name}: PPL={perplexity(probabilities):.2f}")

Output

checkpoint-0400: PPL=3.18
checkpoint-0800: PPL=2.31

The result supports a narrow statement: checkpoint-0800 predicts tokens in this held-out set better under this protocol. It doesn't yet prove better incident answers.

Aggregate loss once

Evaluation windows are rarely the same size. Averaging each window's already exponentiated PPL gives a short hard window too much influence. Sum NLL weighted by scored-token count, divide once, and exponentiate once.

aggregate-loss-before-exponentiating.py

import math

windows = [
    {"average_nll": 0.50, "scored_tokens": 2},
    {"average_nll": 2.00, "scored_tokens": 8},
]

wrong = sum(math.exp(window["average_nll"]) for window in windows) / len(windows)
total_nll = sum(
    window["average_nll"] * window["scored_tokens"] for window in windows
)
total_tokens = sum(window["scored_tokens"] for window in windows)
right = math.exp(total_nll / total_tokens)

print(f"average of window PPLs: {wrong:.2f}")
print(f"token-weighted corpus PPL: {right:.2f}")

Output

average of window PPLs: 4.52
token-weighted corpus PPL: 5.47

Different tokenizers need common units

Token-level perplexity depends on tokenization. The string Compile CI log may be four subword tokens for one model and fourteen character tokens for another. A probability event per large subword isn't the same unit as a probability event per character. Hugging Face's perplexity documentation explicitly warns that tokenization affects PPL comparisons.^{[1]Reference 1Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

For the same raw UTF-8 evaluation text, bits per byte (BPB) gives both models one shared denominator. PALOMA uses BPB as a practical compromise when tokenizers differ, while noting that it still scores the canonical token sequence chosen by each tokenizer rather than marginalizing over every valid segmentation.^{[2]Reference 2PALOMA: A Benchmark for Evaluating Language Model Fit.https://arxiv.org/abs/2312.10523}

\text{BPB} = \frac{-\sum_i \ln p_\theta(x_i \mid x_{<i})} {B \ln 2}

where $B$ is the byte count of the original text. A related metric, bits per character, is useful when a benchmark defines character units instead of bytes.

Byte-aligned comparison of four subword tokens and fourteen character tokens for Compile CI log, showing raw token PPL ranks the character model lower while bits per byte ranks the subword model lower. — Centered dots mark spaces. The same fourteen bytes produce four subword events or fourteen character events, so raw token PPL favors the smaller units while BPB compares total surprise over the shared byte count.

compare-tokenizers-with-bits-per-byte.py

import math

text = "Compile CI log"
byte_count = len(text.encode("utf-8"))
evaluations = [
    {"name": "subword model", "tokens": 4, "total_nll": 8.4},
    {"name": "character model", "tokens": 14, "total_nll": 9.0},
]

for run in evaluations:
    ppl = math.exp(run["total_nll"] / run["tokens"])
    bpb = run["total_nll"] / (byte_count * math.log(2))
    print(f"{run['name']:15s} token PPL={ppl:.2f}, BPB={bpb:.3f}")

print("Lower BPB identifies less surprise on identical bytes.")

Output

subword model   token PPL=8.17, BPB=0.866
character model token PPL=1.90, BPB=0.927
Lower BPB identifies less surprise on identical bytes.

The character model looks dramatically better under raw token PPL because it predicts smaller units. BPB gives the fairer comparison: the subword model assigned more total probability to the same bytes. It doesn't erase every tokenization effect, so keep logging the tokenizer and use a fixed vocabulary when possible.

Long documents need a scoring policy

A real evaluation file may contain thousands of tokens, while a model accepts only a fixed number of context tokens. Cutting text into disjoint blocks is fast, but tokens at each block boundary lose usable left context. A strided sliding window reuses context and scores only newly exposed target tokens.

Hugging Face demonstrates this protocol for GPT-2 Large on WikiText-2: a no-overlap stride = 1024 run reports PPL 19.44, while stride = 512 reports 16.44 for the same model and corpus. More context improved the score; the model weights didn't change.^{[1]Reference 1Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

Exact three-window causal language model evaluation over tokens A through J, with reused context in blue, newly scored tokens in green, and every target B through J scored exactly once. — The matrix matches the worked loop exactly: BCDE, FGH, and IJ contribute nine losses total, while overlapping blue tokens provide context without entering the denominator again.

First simulate which positions a sliding-window loop scores:

score-each-target-token-once.py

tokens = list("ABCDEFGHIJ")
windows = [
    {"context": (0, 5), "score": (1, 5)},
    {"context": (3, 8), "score": (5, 8)},
    {"context": (5, 10), "score": (8, 10)},
]

scored_tokens: list[str] = []
for index, window in enumerate(windows, start=1):
    begin, end = window["context"]
    score_begin, score_end = window["score"]
    context = "".join(tokens[begin:end])
    scored = "".join(tokens[score_begin:score_end])
    scored_tokens.extend(tokens[score_begin:score_end])
    print(f"window {index}: context={context}, newly scored={scored}")

print("scored exactly once:", scored_tokens == tokens[1:])

Output

window 1: context=ABCDE, newly scored=BCDE
window 2: context=DEFGH, newly scored=FGH
window 3: context=FGHIJ, newly scored=IJ
scored exactly once: True

The first token is input context because a causal model needs a previous position before it can score a next-token target. In a framework implementation, context-only labels are commonly masked with -100 so cross-entropy ignores them.^{[1]Reference 1Perplexity of fixed-length modelshttps://huggingface.co/docs/transformers/perplexity}

This dependency-free evaluation loop uses precomputed token NLL values. A real model supplies the losses; aggregation logic stays the same.

aggregate-a-strided-evaluation-run.py

import math

new_target_losses = [
    [0.30, 0.72, 0.51, 0.43],
    [0.27, 0.61, 0.38],
    [0.56, 0.48],
]

total_nll = sum(sum(window) for window in new_target_losses)
scored_tokens = sum(len(window) for window in new_target_losses)
perplexity = math.exp(total_nll / scored_tokens)

print(f"scored tokens={scored_tokens}")
print(f"average NLL={total_nll / scored_tokens:.3f}")
print(f"PPL={perplexity:.2f}")

Output

scored tokens=9
average NLL=0.473
PPL=1.61

For every reported PPL, store max_context_tokens, stride_tokens, the first-token policy, and the masking policy beside the score. Those details are measurement settings, not implementation trivia.

PPL answers one question, not every question

Suppose your incident assistant predicts common status-log language fluently but recommends the wrong rollback step. Perplexity can reward fluent next-token prediction without detecting that operational failure. Likewise, changing decoding strategy can change generated text quality even when the underlying model is unchanged, as Holtzman et al. demonstrated when studying repetitive neural generation.^{[3]Reference 3The Curious Case of Neural Text Degeneration.https://arxiv.org/abs/1904.09751}

Use PPL for the question it answers:

Decision	Useful measurement
Did a base-model checkpoint get better at held-out next-token prediction?	PPL under fixed protocol, or BPB across tokenizers
Did the assistant provide the correct incident status and cite supplied evidence?	Task-specific deterministic checks
Did an open-ended reply follow a rubric for clarity and groundedness?	Calibrated judge or human review
Is a release safe for a high-impact workflow?	Task regressions plus human-reviewed edge cases

separate-language-fit-from-product-correctness.py

candidates = [
    {"name": "fluent-wrong", "ppl": 8.9, "policy_checks_passed": 1},
    {"name": "grounded-answer", "ppl": 9.8, "policy_checks_passed": 3},
]

best_language_fit = min(candidates, key=lambda row: row["ppl"])
best_product_answer = max(candidates, key=lambda row: row["policy_checks_passed"])

print("best held-out language fit:", best_language_fit["name"])
print("best incident answer result:", best_product_answer["name"])

Output

best held-out language fit: fluent-wrong
best incident answer result: grounded-answer

Qualitative map of evaluation methods by application fidelity and cost, placing perplexity in cheap language-fit space and task checks, rubric judges, and human audits in product-evidence space. — This is a qualitative workflow map, not benchmark data. PPL stays on the language-fit side; release decisions require checks or review that directly represent product behavior.

When deterministic checks end

Many incident cases can be checked without a model judge: expected status, required citation identifier, and forbidden runbook claim are deterministic. Use those checks first.

build-a-deterministic-incident-eval.py

EXPECTED_STATUS = "blocked"
REQUIRED_SOURCE = "incident_policy_483"

def score_answer(answer: dict[str, str]) -> tuple[int, list[str]]:
    failures: list[str] = []
    if answer["status"] != EXPECTED_STATUS:
        failures.append("wrong status")
    if answer["source"] != REQUIRED_SOURCE:
        failures.append("missing evidence")
    return 2 - len(failures), failures

answers = [
    {"name": "A", "status": "blocked", "source": "incident_policy_483"},
    {"name": "B", "status": "approved", "source": "incident_policy_483"},
]

for answer in answers:
    score, failures = score_answer(answer)
    print(answer["name"], score, failures or ["pass"])

Output

A 2 ['pass']
B 1 ['wrong status']

Open-ended tone, clarity, and partial correctness may require rubric review. LLM judges can scale that review, but the MT-Bench and Chatbot Arena study documents position bias, verbosity bias, and preference for model-like answers. Treat a judge as a calibrated measurement instrument, not truth.^{[4]Reference 4Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685}

Three-part LLM judge control diagram showing a frozen rubric, A/B order swaps that preserve candidate identity, and judge scores calibrated against held-out human labels. — The diagram is schematic. A usable judge metric keeps its rubric fixed, counterbalances candidate order, logs rationale, and tracks agreement on held-out human-labeled cases.

That boundary matters. Later evaluation lessons build judge calibration, benchmark selection, and online experimentation in depth.

Keep evaluation data clean

PPL needs held-out text. If training data includes your evaluation records, lower loss may reflect memorization rather than generalization. Product task suites have the same failure: if prompt examples or fine-tuning rows include hidden test tickets, release metrics lose meaning.

fail-on-evaluation-data-leakage.py

training_record_ids = {"ticket-101", "ticket-102", "ticket-103"}
validation_record_ids = {"ticket-201", "ticket-202", "ticket-103"}

overlap = training_record_ids & validation_record_ids
if overlap:
    print("FAIL leaked record ids:", sorted(overlap))
else:
    print("PASS validation set is disjoint")

Output

FAIL leaked record ids: ['ticket-103']

For public LLM benchmarks, test content can also enter later training corpora. LiveBench addresses that risk with frequently updated questions from recent sources and objective ground-truth scoring; it limits contamination risk rather than making every future score immune to leakage.^{[5]Reference 5LiveBench: A Challenging, Contamination-Limited LLM Benchmarkhttps://arxiv.org/abs/2406.19314}

Set-overlap diagram showing ticket-103 shared between training and validation records, followed by a clean evaluation design with a stable private holdout and fresh rotating challenge cases. — The left side reproduces the code failure: ticket-103 appears in both sets. After removal, stable private records support regression tracking while rotating cases test behavior beyond one frozen benchmark.

Build an evaluation report

An engineering metric becomes useful when it ships with enough context to reproduce a decision. A compact report should include metric value, protocol fields, leakage checks, and product task gates.

emit-a-release-evaluation-report.py

from dataclasses import dataclass

@dataclass(frozen=True)
class Report:
    checkpoint: str
    perplexity: float
    dataset: str
    tokenizer: str
    context_tokens: int
    stride_tokens: int
    objective: str
    special_token_policy: str
    first_token_policy: str
    masking_policy: str
    leaked_records: int
    policy_pass_rate: float

def release_gate(report: Report) -> str:
    if report.leaked_records:
        return "BLOCK: contaminated evaluation set"
    if report.policy_pass_rate < 1.0:
        return "BLOCK: product regressions"
    return "PASS: protocol recorded and product checks passed"

report = Report(
    checkpoint="incident-lm-0800",
    perplexity=9.81,
    dataset="incident-holdout-v3",
    tokenizer="bpe-v7",
    context_tokens=2048,
    stride_tokens=512,
    objective="causal-next-token",
    special_token_policy="skip-padding",
    first_token_policy="skip-unconditioned-first-token",
    masking_policy="score-each-target-once",
    leaked_records=0,
    policy_pass_rate=1.0,
)

print(f"{report.checkpoint}: PPL={report.perplexity} @ {report.context_tokens}/{report.stride_tokens}")
print(
    "protocol:",
    report.objective,
    report.special_token_policy,
    report.first_token_policy,
    report.masking_policy,
)
print(release_gate(report))

Output

incident-lm-0800: PPL=9.81 @ 2048/512
protocol: causal-next-token skip-padding skip-unconditioned-first-token score-each-target-once
PASS: protocol recorded and product checks passed

The report refuses two common shortcuts: treating an untrusted held-out set as evidence, and treating language fit as a substitute for application correctness.

Perplexity rules worth keeping

Perplexity is exp(average NLL): an interpretable view of held-out next-token surprise.
Raw PPL comparison requires the same dataset, tokenizer, objective, context, stride, and masking policy.
Bits per byte puts models with different tokenizers onto one raw-text denominator.
Long-document evaluation must score new target tokens once while reusing context and aggregating loss before exponentiating.
Low PPL doesn't establish factual, useful, or safe outputs; application checks and calibrated review answer those questions.
Leakage invalidates confident evaluation claims, whether the set measures PPL or product behavior.

Mastery check

Key concepts

Held-out next-token likelihood
Cross-entropy to perplexity
Stable log-probability scoring
Evaluation protocol contracts
Bits-per-byte normalization
Strided context windows
Intrinsic versus product quality
Leakage-resistant evaluation sets

Evaluation rubric

Foundational: Computes token surprise, average NLL, and PPL from observed probabilities.
Intermediate: Explains effective choice count without treating it as a vocabulary-size claim.
Intermediate: Rejects invalid raw PPL comparisons by checking protocol fields.
Advanced: Uses BPB when tokenizers differ and aggregates strided loss correctly.
Advanced: Designs a report that separates language-fit metrics from product and leakage gates.

Follow-up questions

Common pitfalls

Comparing scores without a protocol

Symptom: A team declares victory from PPL 10 versus PPL 12 but can't name the tokenizer, data split, or stride.
Cause: The score was treated as a universal model rating instead of a metric with units and conditioning rules.
Fix: Store the evaluation contract with every result and compare raw PPL only when contracts match.

Averaging window perplexities

Symptom: Long-document PPL changes when window boundaries move, even though scored token losses are unchanged.
Cause: Per-window perplexities were averaged directly.
Fix: Sum token NLL across all windows, divide by scored-token count once, then exponentiate.

Selecting chat behavior using language fit alone

Symptom: A fluent model ships a wrong policy answer because it had the lowest PPL.
Cause: Intrinsic next-token evaluation was confused with application correctness.
Fix: Gate releases on deterministic task checks and calibrated review in addition to base-model fit metrics.

Testing on leaked records

Symptom: Evaluation looks unusually strong, then fails on genuinely new tickets.
Cause: Training or prompt examples overlap with hidden evaluation data.
Fix: Enforce disjoint identifiers, keep private held-out records, and rotate realistic challenge cases.

Next Step

Continue to File Ingestion for AI

You can now measure prediction fit and protect evaluation sets; next you'll turn source documents into clean, traceable records that a system can evaluate and cite.

PreviousStatic to Contextual Embeddings

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Perplexity of fixed-length models

Hugging Face · 2026

https://huggingface.co/docs/transformers/perplexity

PALOMA: A Benchmark for Evaluating Language Model Fit.

Magnusson, I., et al. · 2024 · NeurIPS 2024 Datasets and Benchmarks Track

https://arxiv.org/abs/2312.10523

The Curious Case of Neural Text Degeneration.

Holtzman, A., et al. · 2020 · ICLR 2020

https://arxiv.org/abs/1904.09751

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.

Zheng, L., et al. · 2023 · NeurIPS 2023

https://arxiv.org/abs/2306.05685

LiveBench: A Challenging, Contamination-Limited LLM Benchmark

White, C., et al. · 2024

https://arxiv.org/abs/2406.19314

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.

Perplexity & Model Evaluation

From surprise to a metric

A model assigns probability 0.8 to the observed token on one step and 0.02 on another. Which step dominates its loss, and why?

Average surprise becomes perplexity

Compute from logits without numerical failure

Why should an evaluator accumulate negative log-likelihood instead of multiplying token probabilities together?

The comparison contract

Aggregate loss once

Run A reports PPL 12 and run B reports PPL 10. What must be checked before declaring B better?

Different tokenizers need common units

Long documents need a scoring policy

PPL answers one question, not every question

When deterministic checks end

Keep evaluation data clean

Build an evaluation report

Perplexity rules worth keeping

Mastery check

Key concepts

Evaluation rubric

Follow-up questions

Why can two models not be compared by raw PPL when their tokenizers differ?

Why does decreasing stride often lower PPL for the same fixed-context model?

A runbook model has lower PPL but fails rollback-safety checks. Which model should ship?

Common pitfalls

Comparing scores without a protocol

Averaging window perplexities

Selecting chat behavior using language fit alone

Testing on leaked records

Mastery Check

Discussion