LLM Benchmarks & Limitations

The chunking chapter made deploy-policy evidence retrievable. Now CodeAssist, a developer assistant built around a large language model (LLM), is about to answer a release question. Its retriever returned this exact rule:

Retrieved evidence: Production deploys during a freeze require incident commander approval.

One candidate preserves that rule: "During a freeze, get incident commander approval before deploying." A second candidate says: "Any on-call engineer can deploy during a freeze." The second answer sounds helpful, but it invents permission that the evidence never granted.

That is the release problem: a fluent answer can still be unsafe. Before replacing a deployed model, you need evidence that its answers are more reliable without breaking latency, cost, or output contracts. You'll build that evidence from a private evaluation set, deterministic checks, calibrated model judges, and explicit release thresholds. Along the way, you'll separate public benchmark claims from product evidence and decide when code-generation pass@k belongs in the report.

Start with questions you must answer correctly

A benchmark is a collection of tasks plus a scoring procedure. For CodeAssist, that means a private set whose rows mirror the policy questions the service must answer. For freeze-deploy, the test asks whether an answer preserves the retrieved commander-approval clause, not whether the model can answer unrelated public multiple-choice questions.

Start with a small private evaluation set. The policy statements are fictional platform rules for this lab, not general deployment advice.

Before reading the rows, ask two questions: which fact must a release engineer trust, and which extra claim would create risk? Each row records both answers.

These rows are more useful than a generic "answer quality" prompt because each one names the behavior that passes and the risky extra claim that fails.

Before scoring a model, validate the evaluation set itself. A duplicated ID can silently overwrite a result, and a required phrase absent from its cited evidence creates an impossible test. If you duplicated freeze-deploy or required a phrase missing from its evidence, the validator should stop before any model answer is scored.

A score is only meaningful with its contract

The earlier perplexity lesson made this point for next-token fit: a number without a protocol isn't a comparison. Product evaluation needs that same contract, then adds retrieval, output schema, and operational gates.

When a model card says "86%," the number is incomplete without the task and harness that produced it. Record five parts so another engineer can reproduce the result:

Chunking solved the evidence handoff. Evaluation starts after retrieval, so follow the request from left to right:

When step 4 reports a miss, trace it upstream before blaming the model. If the required fact never reached the context, retrieval failed. If the context contains it but the answer contradicts or omits it, inspect generation or instruction following.

First gate: preserve policy evidence

Begin with deterministic checks whenever the answer has explicit requirements. This isn't a complete semantic evaluator. It's a fast test that catches answers missing required policy terms or adding known unsafe claims.

This lab uses three fixture answers. It also asserts that the evaluation rows themselves are valid: every required phrase must be present in the cited evidence.

Predict the output before running the check. freeze-deploy and secret-rotation should pass; rollback-runbook should fail because its answer drops both required policy terms and adds an unsafe instruction.

The failed rollback-runbook answer is fluent, but it omits both required terms and invents a dangerous continuation. A helpful-sounding response isn't enough when an engineer may act on unsupported policy.

This split between retrieved context and generated answer also appears in RAG evaluation research. Ragas names faithfulness for support by retrieved context and answer relevance for addressing the question, rather than collapsing both failures into one vague score.^{[1]Reference 1RAGAS: Automated Evaluation of Retrieval Augmented Generation.https://arxiv.org/abs/2309.15217}

Diagnose retrieval and generation separately

Suppose an answer omits incident commander approval. That observation alone doesn't identify the failing component. Compare the gold requirement with both the retrieved context and the generated answer:

Predict the label first: a missing phrase in retrieved context points to bad_retrieval; a missing phrase after successful retrieval points to bad_generation.

Why check that required phrases occur in the evidence before evaluating model answers?

Answer

An incorrect golden row can make a correct answer look wrong or reward an unsupported claim. Evaluation data is part of the product and needs its own validation.

Public benchmarks answer narrower questions

Private tests tell you whether your application meets its contract. Public benchmarks help compare capabilities under standardized tasks, but each family measures a different property.

MMLU (Massive Multitask Language Understanding) evaluates multiple-choice accuracy across 57 academic and professional subjects.^{[2]Reference 2Measuring Massive Multitask Language Understanding (MMLU).https://arxiv.org/abs/2009.03300} That breadth can reveal knowledge differences under a stated prompt protocol. It can't prove that a policy assistant quotes the right deploy-freeze clause.

The HLE paper motivates a harder closed-ended exam by pointing to models above 90% on popular benchmarks such as MMLU.^{[3]Reference 3Humanity's Last Examhttps://arxiv.org/abs/2501.14249} MMLU-Pro keeps the exam format but expands each question from four answer options to ten and removes trivial or noisy items, making chance guessing less useful when MMLU scores plateau.^{[4]Reference 4MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkhttps://arxiv.org/abs/2406.01574}

Stanford's HELM, a multidimensional foundation-model evaluation suite, scores models across standard taxonomies covering accuracy, calibration, robustness, fairness, and bias under fixed prompt formats.^{[5]Reference 5Holistic Evaluation of Language Models (HELM).https://arxiv.org/abs/2211.09110} That standardization helps compare foundation models, but HELM still can't evaluate proprietary policy retrieval or system-level service commitments.

GPQA (Graduate-Level Google-Proof Question Answering) narrows the task to difficult expert questions. Its 448 multiple-choice questions were written by domain experts in biology, physics, and chemistry.^{[6]Reference 6GPQA: A Graduate-Level Google-Proof Q&A Benchmark.https://arxiv.org/abs/2311.12022} Skilled non-expert validators scored 34% with unrestricted web access, which explains the name. A strong GPQA result can signal expert science question-answering ability under that harness, but it still doesn't prove that a developer workflow follows policy.

Math benchmarks narrow the question further. GSM8K (Grade School Math 8K) uses grade-school word problems, while MATH uses harder competition-style problems. Both help when a workflow must produce checkable mathematical answers, but neither measures open-ended support quality.^{[7]Reference 7Training Verifiers to Solve Math Word Problems (GSM8K).https://arxiv.org/abs/2110.14168[8]Reference 8Measuring Mathematical Problem Solving with the MATH Dataset.https://arxiv.org/abs/2103.03874}

At the harder frontier, HLE's initial preprint described 3,000 questions, while the project later finalized a public set of 2,500; FrontierMath targets research-level mathematics.^{[3]Reference 3Humanity's Last Examhttps://arxiv.org/abs/2501.14249[9]Reference 9Humanity's Last Examhttps://lastexam.ai/[10]Reference 10FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIhttps://arxiv.org/abs/2411.04872} ARC-AGI-2, the second Abstraction and Reasoning Corpus for artificial general intelligence, asks systems to infer novel transformations from small visual-grid demonstrations rather than recall subject matter.^{[11]Reference 11ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systemshttps://arxiv.org/abs/2505.11831} These benchmarks answer different research questions. A model isn't "better at reasoning" in every product merely because it improves on one of them.

HumanEval provides 164 handwritten Python functions with unit tests and introduced functional-correctness reporting with repeated samples and pass@k.^{[12]Reference 12Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374} SWE-bench moves from isolated functions to real GitHub issues whose patches are tested in repository environments.^{[13]Reference 13SWE-bench: Can Language Models Resolve Real-World GitHub Issues?.https://arxiv.org/abs/2310.06770}

The commonly reported SWE-bench Verified subset contains 500 human-filtered tasks. Official leaderboards also split results by agent scaffold, so a resolved rate describes a whole system, not a raw-model score.^{[14]Reference 14SWE-bench Official Leaderboardshttps://www.swebench.com/} These benchmarks matter when your system edits code, but neither evaluates developer-facing policy truthfulness.

For open-ended responses, MT-Bench uses a model judge, while Chatbot Arena collects blind pairwise human preferences and estimates rankings with Bradley-Terry-style models.^{[15]Reference 15Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685[16]Reference 16Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference.https://arxiv.org/abs/2403.04132} These methods can compare tone, fluency, or perceived helpfulness, but they still measure preference rather than policy truth.

Using familiar Elo-scaled ratings, the probability that model $A$ beats model $B$ with latent ratings $R_A$ and $R_B$ is:

$$P(A \succ B) = \frac{1}{1 + 10^{(R_B - R_A) / 400}}$$

The equation turns a pairwise vote into a relative ranking. User preference still doesn't measure factual ground truth: an answer can win because it sounds polite or confident while citing an unsupported policy.

Compare results only when the dataset, input path, output contract, prompt, temperature and other sampling settings, tools, and scorer match. 84% on a multiple-choice set and 42% on repository fixes aren't competing measurements of the same property.

Encode that rule as a field-by-field equality check. Before reading the output, predict the result: A and B share one harness, while C sees five chunks instead of three and should be incomparable.

A model is stronger on MMLU than another model is on SWE-bench. Which model should answer deploy-policy questions?

Answer

You can't decide from those two numbers. They score different tasks and neither measures your policy-answer contract. Run both candidates on the private policy set with the same retrieval and generation harness.

pass@k: measure a workflow that can test several drafts

For code generation, one answer isn't always the real workflow. A developer may generate several candidates and run tests until one passes. HumanEval's pass@k estimator measures the probability that at least one of $k$ selected samples is correct, given $n$ generated programs and $c$ correct programs.^{[12]Reference 12Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Suppose a config-parser task generated 100 candidates and tests accepted 20. One random candidate succeeds with probability 0.20. Before looking at the formula, predict what happens when the workflow can test 10 distinct candidates: the all-miss risk should shrink, but the metric should still describe this test-and-select workflow, not the model's one-shot accuracy.

$$\operatorname{pass}@k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

Here, $n$ is the number of generated programs, $c$ is the number that pass tests, and $k$ is the number you're allowed to try. Chen et al. warn against estimating the same quantity as $1-(1-\hat{p})^k$ from the empirical pass@1. That form treats the $k$ draws as independent with replacement; the combinatorial estimator is the unbiased one used on HumanEval.

At k=10, the shortcut is 0.8926 and the unbiased estimator is 0.9049. The gap is modest because 100 generated samples leave a large pool outside the 10 tested draws. It grows when n is closer to k.

This distinction matters at the product boundary. Multiple tested drafts make sense for a generated parser because unit tests select a passing implementation. For a developer-facing policy answer, sampling more drafts and choosing the most confident-looking one isn't a correctness test unless a policy-grounded scorer selects the winner.

Open-ended answers need controlled judgment

Deterministic checks catch explicit clauses, but they don't settle every question. Two answers can both cite the correct deadline while differing in clarity or empathy. For that remainder, teams often use humans or an LLM judge.

The choice of rubric changes what the judge can see. Point-wise scoring rates one answer on a 1-to-5 scale and often compresses scores near the top. Pairwise comparison ranks candidate A against candidate B and can discriminate more sharply, but presentation order can sway the result. Ground either approach with explicit reference criteria: pass the retrieved policy text into the judge prompt, name the facts the response must contain, and flag forbidden or unsupported claims.

The MT-Bench study found that capable LLM judges can approximate human preferences in its setting, but also documents position bias, verbosity bias, and cases where a judge favored its own model family.^{[15]Reference 15Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} A judge result is measurement output, not ground truth.

Position bias is the failure this lab measures: most judges in that study preferred whichever answer appeared first. Verbosity bias lets a longer rewrite win even when it adds no facts. Self-enhancement was harder to isolate from style, so treat a same-family judge as a risk to control, not as proof that its generator wins.

The small simulation below gives a toy judge an explicit position bias: when two answers contain the same required facts, it chooses whichever appears first. Predict the result before running it. The equivalent pair should become unstable after swapping; the factual miss should leave the grounded answer ahead in both orders.

An unstable comparison isn't a failure of either candidate. It means the measurement can't distinguish them reliably under its current rubric. Send such cases to a human reviewer or improve the rubric before aggregating a winner.

Public test sets can lose their meaning

A public benchmark is reproducible because everyone can access its tasks. That visibility creates a second property to manage: questions or solutions may appear in training corpora. Once a model has encountered a test item during training, its score no longer describes performance on a genuinely unseen example.

When a static public benchmark becomes the primary release target, Goodhart's Law takes over: when a measure becomes a target, it ceases to be a good measure. Teams can optimize prompt formats, synthetic fine-tuning mixes, and post-training data for leaderboard points. Over time, a high score may reflect benchmark-specific overfitting rather than task generalization.

Don't jump from risk to accusation. A public score alone doesn't prove contamination in a particular model. It does mean that a report should state its decontamination checks, while high-stakes selection should add fresh or private tasks.

LiveCodeBench was designed around this problem. It continually collects coding tasks released over time and evaluates models on problems that appear after their stated training cutoff.^{[17]Reference 17LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codehttps://arxiv.org/abs/2403.07974} LiveBench applies a similar idea more broadly: it adds and updates questions monthly from recent sources and scores them against objective answers, so a static public dump is less useful as training leakage.^{[18]Reference 18LiveBench: A Challenging, Contamination-Limited LLM Benchmarkhttps://arxiv.org/abs/2406.19314} Time splits reduce exposure risk, although they still depend on accurate cutoff information and a stable harness.

For public benchmarks, document the decontamination check instead of assuming the training corpus is clean. GPT-3's evaluation analysis, for example, used n-gram overlap checks between benchmark examples and training data and reported limitations when overlap remained.^{[19]Reference 19Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165} Treat that overlap as a lower bound on exposure: paraphrase, translation, and reordering can leak task content while looking "clean."

A time split reduces exposure risk from older public copies. A private holdout covers the workload you'll deploy, so high-stakes selection still needs fresh or private tasks even when public decontamination reports look good.

For the CodeAssist policy assistant, keep private policy cases separate from prompt examples, demo transcripts, few-shot packs, synthetic seed rows, SFT data, retrieval fixtures, human-rater instructions, and internal runbooks that may later be used for tuning.

If an evaluation case becomes a training example, appears in a few-shot pack, seeds synthetic data, enters a judge rubric that quotes gold answers, or is indexed for retrieval, retire it from the holdout set. Keep it as regression evidence, but stop calling it unseen generalization.

Private golden questions and answers are also a retrieval contamination surface. Never ingest eval fixtures into the production policy index: a retriever that can look up the test is no longer measuring generalization. Keep an eval/ corpus checksum-blocked from the production index, separate fixture IDs from indexable document IDs, and treat index membership the same way you treat holdout membership. Regression suites may intentionally include known cases; holdout claims may not.

A time-split suite also helps when new policy versions arrive. A model tuned using cases created through March shouldn't be evaluated as "unseen" on those same cases. Hold out cases authored after the tuning snapshot. With the snapshot below, predict the split: the February case belongs in regression, while the April and May cases remain fresh holdout evidence.

Turn results into a release gate

Your evaluation is useful when it changes an engineering decision. One model may give grounded answers but miss your p95 latency target for live chat. Another may be fast and cheap, but fail critical policy cases. Define the bar before comparing models, then make each failure route explicit.

The fixture values below represent measurements from a fictional private evaluation run. Predict the decisions before reading the code: fast-small should be rejected for policy quality, balanced should ship, and slow-specialist should route because it misses an online gate. The code makes the release gate explicit: a candidate that misses policy or schema quality is rejected, while a correct but expensive model may be routed only to difficult cases.

The public benchmark shortlist never appears inside decision. That's intentional. Public scores help decide which candidates deserve testing; the release gate uses measurements from the workload you're about to serve.

Inspect failure slices, not the average alone

An aggregate score can hide the category with the largest operational impact. The run below clears four of six cases overall, but freeze-deploy answers fail most often. Predict which number should govern a release when that slice is critical: the slice, not the average.

That report should block a release if deploy-freeze policy mistakes are critical, even when the average seems acceptable. Slices turn a vague quality miss into a routing or remediation decision.

Small samples need an uncertainty margin

Three passing examples make a good tutorial, not a production release argument. Earlier statistics lessons introduced confidence intervals for estimated rates. Before running the next check, predict which rows clear a 0.95 floor: only the large sample should have enough evidence. For a binary pass metric, a Wilson lower bound gives a conservative view of the pass rate supported by a finite sample:

Even perfect performance on three examples has a weak lower bound. Build a sufficiently large, risk-stratified holdout before claiming a system clears a production-quality bar.

Write down enough to reproduce the claim

Evaluation reports should let another engineer rerun the comparison and inspect its failures. Without the input path, scorer, and holdout history, a ranking can't be reproduced or diagnosed. Record these fields before sharing it:

That prompt-and-chat-format row points to the next lesson. If a model regularly omits required structure or speaks in the wrong role, evaluation has exposed a behavior gap. You then need to understand how instruction tuning and chat templates teach the interaction contract.

Failure patterns to diagnose

Mastery check

Key concepts

• Evaluation contracts and private golden sets
• Grounded answer checks for RAG
• Benchmark families and harness comparability
• HumanEval and pass@k
• Judge bias controls
• Contamination and time-split holdouts
• Quality, latency, and cost release gates

Evaluation rubric

• Foundational: Explains why public benchmark scores can't replace a private policy-answer evaluation contract.
• Intermediate: Builds a grounded-answer check that attributes retrieval versus generation failures.
• Intermediate: Explains when pass@k is appropriate and why it doesn't equal policy-answer accuracy.
• Advanced: Designs judge-order, failure-slice, uncertainty, latency, and cost controls for a release decision.

Common pitfalls

• Treating a public benchmark gain as evidence that policy answers are grounded.
• Comparing model scores produced by different prompts, tools, or scoring rules.
• Accepting an LLM judge result without checking position bias or factual anchors.
• Releasing on a mean quality score while ignoring critical slices, latency, or cost.

Follow-up questions

Why does the evaluation set include retrieved evidence as well as expected answers?

Answer

A RAG system can fail before or after generation. Keeping the evidence lets you tell a retrieval miss from an answer that ignored or contradicted an available clause.

Why can't pass@10 be treated as the same claim as policy-answer accuracy?

Answer

pass@10 assumes you can generate several code candidates and select one with executable tests. A developer-facing policy response needs its own grounding and safety checks; sampling more unscored answers doesn't establish correctness.

When should an LLM judge comparison be sent to human review?

Answer

When the winner changes after answer-order swaps, factual anchors disagree, or the case carries a risk that the rubric doesn't safely score. Instability is evidence about the evaluator, not permission to pick a convenient winner.

Practice extension

Add a fourth policy case whose retrieved context is intentionally wrong. Extend score-grounded-policy-answers.py to label failures as bad_retrieval or bad_generation. Then add a source_page field to the candidate response and reject answers whose citation doesn't match the evidence row. You have now built the first slice of an evaluation dashboard for a RAG assistant.