LLM Benchmarks & Limitations

Start a large language model (LLM) evaluation from a deploy-policy document split into chunks that preserve answerable evidence. One retrieved chunk contained this exact rule:

Retrieved evidence: Production deploys during a freeze require incident commander approval.

Now a developer assistant answers: "During a freeze, get incident commander approval before deploying." That looks good. A second model answers: "Any on-call engineer can deploy during a freeze." That sounds helpful, but the retrieved clause doesn't support it.

Before replacing a deployed model, you need evidence that the new system answers these policy questions more reliably without breaking latency, cost, or output contracts. This chapter builds that evidence from a private evaluation set, deterministic checks, calibrated model judges, and explicit release thresholds. It also separates public benchmark claims from product evidence and explains when code-generation pass@k belongs in the report.

Start with questions you must answer correctly

A benchmark is a collection of tasks plus a scoring procedure. A production evaluation is the same idea applied to your own workload. Instead of asking only whether a model knows academic facts, you ask whether the entire system retrieves the right clause, answers from that clause, follows the output contract, and meets latency and cost limits.

Start with a tiny private evaluation set for CodeAssist. The policy statements are fictional platform rules for this lab, not general deployment advice.

These rows are more useful than a generic "answer quality" prompt because each one names the behavior that passes and the risky extra claim that fails.

Before scoring a model, validate the evaluation set itself. A duplicated ID can silently overwrite a result, and a required phrase absent from its cited evidence creates an impossible test.

A score is only meaningful with its contract

When a model card says "86%," the number is incomplete without the task and harness that produced it. Five parts make an evaluation result reproducible:

The preceding chunking lesson addressed the second column of this pipeline: getting complete evidence into the context. Now measure what happens after retrieval. Work from left to right:

If a response fails at step 4, don't immediately blame the model. Inspect the retrieved chunk first. A missing fact may be a retrieval failure, while a contradicted fact may be a generation or instruction-following failure.

First gate: preserve policy evidence

Begin with deterministic checks whenever the answer has explicit requirements. This isn't a complete semantic evaluator. It's a fast test that catches answers missing required policy terms or adding known unsafe claims.

This lab uses three fixture answers. It also asserts that the evaluation rows themselves are valid: every required phrase must be present in the cited evidence.

The failed rollback-runbook answer is fluent, but it never mentions the required threshold or runbook and invents a dangerous continuation. A helpful-sounding response isn't enough when an engineer may act on an unsupported policy.

This split between retrieved context and generated answer also appears in RAG evaluation research. Ragas evaluates whether retrieved context supports the answer and whether the answer addresses the question, rather than collapsing all failures into one vague score.^{[1]Reference 1RAGAS: Automated Evaluation of Retrieval Augmented Generation.https://arxiv.org/abs/2309.15217}

Diagnose retrieval and generation separately

Suppose an answer omits incident commander approval. That doesn't tell you which component failed. Compare the gold requirement with both the retrieved context and the generated answer:

Why check that required phrases occur in the evidence before evaluating model answers?

Answer

An incorrect golden row can make a correct answer look wrong or reward an unsupported claim. Evaluation data is part of the product and needs its own validation.

Public benchmarks answer narrower questions

Private tests tell you whether your application meets its contract. Public benchmarks help compare capabilities under standardized tasks, but each family measures a different property.

MMLU evaluates multiple-choice accuracy across 57 academic and professional subjects.^{[2]Reference 2Measuring Massive Multitask Language Understanding (MMLU).https://arxiv.org/abs/2009.03300} It can reveal broad knowledge differences under a stated prompt protocol. It can't prove that a policy assistant quotes the right deploy-freeze clause.

GPQA narrows the question to difficult expert QA. Its 448 multiple-choice questions were written by domain experts in biology, physics, and chemistry.^{[3]Reference 3GPQA: A Graduate-Level Google-Proof Q&A Benchmark.https://arxiv.org/abs/2311.12022} A strong GPQA result can signal expert STEM question-answering ability under that harness, but it still doesn't prove that a developer workflow follows policy.

Math benchmarks narrow the question. GSM8K uses grade-school word problems, while MATH uses harder competition-style problems; both are useful when a workflow must produce checkable mathematical answers, but they don't measure open-ended support quality.^{[4]Reference 4Training Verifiers to Solve Math Word Problems (GSM8K).https://arxiv.org/abs/2110.14168[5]Reference 5Measuring Mathematical Problem Solving with the MATH Dataset.https://arxiv.org/abs/2103.03874} At the harder frontier, Humanity's Last Exam (HLE) collects expert-level questions across domains, and FrontierMath targets advanced mathematical problems. ARC-AGI-2 asks systems to infer novel transformations from small visual-grid demonstrations rather than recall subject matter.^{[6]Reference 6Humanity's Last Examhttps://arxiv.org/abs/2501.14249[7]Reference 7FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIhttps://arxiv.org/abs/2411.04872[8]Reference 8ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systemshttps://arxiv.org/abs/2505.11831} These benchmarks answer different research questions; a model isn't "better at reasoning" in every product merely because it improves on one of them.

HumanEval provides handwritten Python functions with tests and introduced functional correctness reporting with repeated samples and pass@k.^{[9]Reference 9Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374} SWE-bench moves from isolated functions to real GitHub issues whose patches are tested in repository environments.^{[10]Reference 10SWE-bench: Can Language Models Resolve Real-World GitHub Issues?.https://arxiv.org/abs/2310.06770} These are relevant if your system edits code, but neither evaluates developer-facing policy truthfulness.

MT-Bench uses a model judge for open-ended, multi-turn responses, while Chatbot Arena collects blind pairwise human preferences.^{[11]Reference 11Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685[12]Reference 12Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference.https://arxiv.org/abs/2403.04132} Preference is useful for tone and helpfulness, but an answer that users prefer can still cite the wrong policy.

Compare results only when the dataset, input path, output contract, prompt, temperature and other sampling settings, tools, and scorer match. 84% on a multiple-choice set and 42% on repository fixes aren't competing measurements of the same property.

That rule is easy to encode. This helper allows comparisons only when the benchmark and the harness fields are identical.

A model is stronger on MMLU than another model is on SWE-bench. Which model should answer deploy-policy questions?

Answer

You can't decide from those two numbers. They score different tasks and neither measures your policy-answer contract. Run both candidates on the private policy set with the same retrieval and generation harness.

pass@k: measure a workflow that can test several drafts

For code generation, a single answer isn't always the real workflow. A developer may generate several candidates and run tests until one passes. HumanEval's pass@k estimator measures the probability that at least one of $k$ selected samples is correct, given $n$ generated programs and $c$ correct programs.^{[9]Reference 9Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Suppose a config-parser code task generated 100 candidates and tests accepted 20. Checking one random candidate succeeds with probability 0.20. Checking 10 distinct candidates succeeds much more often:

$$\operatorname{pass}@k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

Here, $n$ is the number of generated programs, $c$ is the number that pass tests, and $k$ is the number you're allowed to try.

This distinction matters. Multiple tested drafts make sense for a generated parser because unit tests select a passing implementation. You shouldn't sample several developer-facing policy answers and silently choose the most confident-looking one without a policy-grounded scorer.

Open-ended answers need controlled judgment

Deterministic checks catch explicit clauses, but they don't settle every question. Two answers can both cite the correct deadline while differing in clarity or empathy. For open-ended comparisons, teams often use humans or an LLM judge.

The MT-Bench study found that capable LLM judges can approximate human preferences in its setting, but also documents position bias, verbosity bias, and preference for model-like answers.^{[11]Reference 11Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.https://arxiv.org/abs/2306.05685} A judge result is measurement output, not ground truth.

The small simulation below intentionally gives a toy judge a position bias: if two answers contain the same required facts, it chooses whichever answer appears first. Running each comparison in both orders exposes that instability.

An unstable comparison isn't a failure of either candidate. It means the measurement can't distinguish them reliably under its current rubric. Send such cases to a human reviewer or improve the rubric before aggregating a winner.

Public test sets can lose their meaning

A public benchmark is reproducible because everyone can access the tasks. That same visibility creates a risk: benchmark questions or solutions may appear in training corpora. A model that encountered a test item during training is no longer being measured on a genuinely unseen example.

Don't jump from risk to accusation. A public score alone doesn't prove contamination in a particular model. It means the report should state what decontamination checks were used, and high-stakes selection should include fresh or private tasks.

LiveCodeBench was designed around this problem: it continually collects coding tasks released over time and evaluates models on problems that appear after their stated training cutoff.^{[13]Reference 13LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codehttps://arxiv.org/abs/2403.07974} Time splits reduce exposure risk, although they still depend on accurate cutoff information and a stable harness.

For public benchmarks, document the decontamination check instead of assuming the training corpus is clean. GPT-3's evaluation analysis, for example, used n-gram overlap checks between benchmark examples and training data and reported limitations when overlap remained.^{[14]Reference 14Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165} Treat n-gram overlap as a lower bound on exposure: paraphrase, translation, and reordering can still leak task content while looking "clean." A time split reduces exposure risk from older public copies. A private holdout covers the workload you'll deploy, and high-stakes selection still needs fresh or private tasks even when public decontamination reports look good.

For the CodeAssist policy assistant, keep private policy cases separate from prompt examples, demo transcripts, few-shot packs, synthetic seed rows, SFT data, retrieval fixtures, human-rater instructions, and internal runbooks that may later be used for tuning. If an evaluation case becomes a training example, appears in a few-shot pack, seeds synthetic data, enters a judge rubric that quotes gold answers, or is indexed for retrieval, retire it from the holdout set or record that it now measures regression behavior rather than generalization.

Private golden questions and answers are also a retrieval contamination surface. Never ingest eval fixtures into the production policy index: a retriever that can look up the test is no longer measuring generalization. Keep an eval/ corpus checksum-blocked from the production index, separate fixture IDs from indexable document IDs, and treat index membership the same way you treat holdout membership. Regression suites may intentionally include known cases; holdout claims may not.

A time-split suite also helps when new policy versions arrive. A model tuned using cases created through March shouldn't be evaluated as "unseen" on those same cases. Hold out cases authored after the tuning snapshot:

Turn results into a release gate

Your evaluation is useful when it changes an engineering decision. A model may give grounded answers but miss your p95 latency target for live chat. Another may be fast and cheap, but fail critical policy cases. Define the bar before comparing models.

The fixture values below represent measurements from a fictional private evaluation run. The code makes the release gate explicit: a candidate that misses policy or schema quality is rejected, while a correct but expensive model may be routed only to difficult cases.

The public benchmark shortlist never appears inside decision. That's intentional. Public scores help decide which candidates deserve testing. A release gate uses measurements taken on the workload you're about to serve.

Inspect failure slices, not the average alone

An aggregate score can hide the one category with the largest operational impact. The run below clears four of six cases overall, but freeze-deploy answers fail most often.

That report should block a release if deploy-freeze policy mistakes are critical, even when the average seems acceptable.

Small samples need an uncertainty margin

Three passing examples make a good tutorial, not a production release argument. Earlier statistics lessons introduced confidence intervals for estimated rates. For a binary pass metric, a Wilson lower bound gives a conservative view of the pass rate supported by a finite sample:

Even perfect performance on three examples has a weak lower bound. Build a sufficiently large, risk-stratified holdout before claiming a system clears a production-quality bar.

Write down enough to reproduce the claim

Evaluation reports should let another engineer rerun the comparison and inspect its failures. Record these fields before sharing any model ranking:

That prompt-and-chat-format row points to the next lesson. If a model regularly omits required structure or speaks in the wrong role, evaluation has exposed a behavior gap. You then need to understand how instruction tuning and chat templates teach the interaction contract.

Failure patterns to diagnose

Mastery check

Key concepts

• Evaluation contracts and private golden sets
• Grounded answer checks for RAG
• Benchmark families and harness comparability
• HumanEval and pass@k
• Judge bias controls
• Contamination and time-split holdouts
• Quality, latency, and cost release gates

Evaluation rubric

• Foundational: Explains why public benchmark scores can't replace a private policy-answer evaluation contract.
• Intermediate: Builds a grounded-answer check that attributes retrieval versus generation failures.
• Intermediate: Explains when pass@k is appropriate and why it doesn't equal policy-answer accuracy.
• Advanced: Designs judge-order, failure-slice, uncertainty, latency, and cost controls for a release decision.

Common pitfalls

• Treating a public benchmark gain as evidence that policy answers are grounded.
• Comparing model scores produced by different prompts, tools, or scoring rules.
• Accepting an LLM judge result without checking position bias or factual anchors.
• Releasing on a mean quality score while ignoring critical slices, latency, or cost.

Follow-up questions

Why does the evaluation set include retrieved evidence as well as expected answers?

Answer

A RAG system can fail before or after generation. Keeping the evidence lets you tell a retrieval miss from an answer that ignored or contradicted an available clause.

Why can't pass@10 be treated as the same claim as policy-answer accuracy?

Answer

pass@10 assumes you can generate several code candidates and select one with executable tests. A developer-facing policy response needs its own grounding and safety checks; sampling more unscored answers doesn't establish correctness.

When should an LLM judge comparison be sent to human review?

Answer

When the winner changes after answer-order swaps, factual anchors disagree, or the case carries a risk that the rubric doesn't safely score. Instability is evidence about the evaluator, not permission to pick a convenient winner.

Practice extension

Add a fourth policy case whose retrieved context is intentionally wrong. Extend score-grounded-policy-answers.py to label failures as bad_retrieval or bad_generation. Then add a source_page field to the candidate response and reject answers whose citation doesn't match the evidence row. You have now built the first slice of an evaluation dashboard for a RAG assistant.