LLM Benchmarks & Limitations

In the previous lesson, you turned a returns-policy document into chunks that preserve answerable evidence. One retrieved chunk contained this exact rule:

Retrieved evidence: Damaged electronics: report within 48 hours with photos.

Now a support assistant answers: "Send photos within 48 hours so we can review the damage." That looks good. A second model answers: "You can return any electronics within 30 days." That sounds helpful, but the retrieved clause doesn't support it.

This chapter answers the question that comes next: how do you measure whether a large language model (LLM) system is safe to improve or deploy? You'll build a small private evaluation set, run deterministic checks, learn what public benchmark scores do and don't prove, measure code generation with pass@k, control judge bias, and turn results into a release decision.

Start With Questions You Must Answer Correctly

A benchmark is a collection of tasks plus a scoring procedure. A production evaluation is the same idea applied to your own workload. Instead of asking only whether a model knows academic facts, you ask whether the entire system retrieves the right clause, answers from that clause, follows the output contract, and meets latency and cost limits.

Here is a tiny private evaluation set for ShopFlow. The policy statements are fictional product rules for this lab, not general retail advice.

These rows are more useful than a generic "answer quality" prompt because each one names the behavior that passes and the risky extra claim that fails.

Before scoring a model, validate the evaluation set itself. A duplicated ID can silently overwrite a result, and a required phrase absent from its cited evidence creates an impossible test.

A Score Is Only Meaningful With Its Contract

When a model card says "86%," the number is incomplete without the task and harness that produced it. Five parts make an evaluation result reproducible:

The preceding chunking lesson addressed the second column of this pipeline: getting complete evidence into the context. This lesson measures what happens after retrieval. Work from left to right:

If a response fails at step 4, don't immediately blame the model. Inspect the retrieved chunk first. A missing fact may be a retrieval failure, while a contradicted fact may be a generation or instruction-following failure.

First Gate: Preserve Policy Evidence

Begin with deterministic checks whenever the answer has explicit requirements. This isn't a complete semantic evaluator. It is a fast test that catches answers missing required policy terms or adding known unsafe claims.

The following lab uses three fixture answers. It also asserts that the evaluation rows themselves are valid: every required phrase must be present in the cited evidence.

The failed late-shipment answer is fluent, but it never mentions the required investigation and invents an automatic refund. A helpful-sounding response isn't enough when a customer may act on an unsupported policy.

This split between retrieved context and generated answer also appears in RAG evaluation research. Ragas evaluates whether retrieved context supports the answer and whether the answer addresses the question, rather than collapsing all failures into one vague score.^[1]

Diagnose Retrieval and Generation Separately

Suppose an answer omits the 48 hours deadline. That doesn't tell you which component failed. Compare the gold requirement with both the retrieved context and the generated answer:

Public Benchmarks Answer Narrower Questions

Private tests tell you whether your application meets its contract. Public benchmarks help compare capabilities under standardized tasks, but each family asks a different question.

MMLU evaluates multiple-choice accuracy across 57 academic and professional subjects.^[2] It can reveal broad knowledge differences under a stated prompt protocol. It can't prove that a policy assistant quotes the right return clause.

GPQA narrows the question to difficult expert QA. Its 448 multiple-choice questions were written by domain experts in biology, physics, and chemistry.^[3] A strong GPQA result can signal expert STEM question-answering ability under that harness, but it still doesn't prove that a support workflow follows policy.

Math benchmarks narrow the question. GSM8K uses grade-school word problems, while MATH uses harder competition-style problems; both are useful when a workflow must produce checkable mathematical answers, but they don't measure open-ended support quality.^[4][5] At the harder frontier, Humanity's Last Exam (HLE) collects expert-level questions across domains, and FrontierMath targets advanced mathematical problems. ARC-AGI-2 asks systems to infer novel transformations from small visual-grid demonstrations rather than recall subject matter.^[6][7][8] These benchmarks answer different research questions; a model isn't "better at reasoning" in every product merely because it improves on one of them.

HumanEval provides handwritten Python functions with tests and introduced functional correctness reporting with repeated samples and pass@k.^[9] SWE-bench moves from isolated functions to real GitHub issues whose patches are tested in repository environments.^[10] These are relevant if your system edits code, but neither evaluates customer-facing policy truthfulness.

MT-Bench uses a model judge for open-ended, multi-turn responses, while Chatbot Arena collects blind pairwise human preferences.^[11][12] Preference is useful for tone and helpfulness, but an answer that users prefer can still cite the wrong policy.

The practical rule is simple: compare results only when the dataset, input path, output contract, prompt, sampling settings, tools, and scorer match. 84% on a multiple-choice set and 42% on repository fixes aren't competing measurements of the same property.

That rule is easy to encode. This helper allows comparisons only when the benchmark and the harness fields are identical.

pass@k: Measure a Workflow That Can Test Several Drafts

For code generation, a single answer isn't always the real workflow. A developer may generate several candidates and run tests until one passes. HumanEval's pass@k estimator measures the probability that at least one of $k$ selected samples is correct, given $n$ generated programs and $c$ correct programs.^[9]

Suppose an inventory-code task generated 100 candidates and tests accepted 20. Checking one random candidate succeeds with probability 0.20. Checking 10 distinct candidates succeeds much more often:

$$\operatorname{pass}@k = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

Here, $n$ is the number of generated programs, $c$ is the number that pass tests, and $k$ is the number you are allowed to try.

This distinction matters. Multiple tested drafts make sense for a generated inventory function because unit tests select a passing implementation. You shouldn't sample several customer-facing refund answers and silently choose the most confident-looking one without a policy-grounded scorer.

Open-Ended Answers Need Controlled Judgment

Deterministic checks catch explicit clauses, but they don't settle every question. Two answers can both cite the correct deadline while differing in clarity or empathy. For open-ended comparisons, teams often use humans or an LLM judge.

The MT-Bench study found that capable LLM judges can approximate human preferences in its setting, but also documents position, verbosity, and self-enhancement biases.^[11] A judge result is measurement output, not ground truth.

The small simulation below intentionally gives a toy judge a position bias: if two answers contain the same required facts, it chooses whichever answer appears first. Running each comparison in both orders exposes that instability.

An unstable comparison isn't a failure of either candidate. It means the measurement can't distinguish them reliably under its current rubric. Send such cases to a human reviewer or improve the rubric before aggregating a winner.

Public Test Sets Can Lose Their Meaning

A public benchmark is reproducible because everyone can access the tasks. That same visibility creates a risk: benchmark questions or solutions may appear in training corpora. A model that encountered a test item during training is no longer being measured on a genuinely unseen example.

Do not jump from risk to accusation. A public score alone doesn't prove contamination in a particular model. It means the report should state what decontamination checks were used, and high-stakes selection should include fresh or private tasks.

LiveCodeBench was designed around this problem: it continually collects coding tasks released over time and evaluates models on problems that appear after their stated training cutoff.^[13] Time splits reduce exposure risk, although they still depend on accurate cutoff information and a stable harness.

For the ShopFlow policy assistant, keep private policy cases separate from prompt examples, demo transcripts, and support documentation that may later be used for tuning. If an evaluation case becomes a training example, retire it from the holdout set or record that it now measures regression behavior rather than generalization.

A time-split suite also helps when new policy versions arrive. A model tuned using cases created through March shouldn't be evaluated as "unseen" on those same cases. Hold out cases authored after the tuning snapshot:

Turn Results Into a Release Gate

Your evaluation is useful when it changes an engineering decision. A model may give grounded answers but be too slow for live chat. Another may be fast and cheap, but fail critical policy cases. Define the bar before comparing models.

The fixture values below represent measurements from a fictional private evaluation run. The code makes the release rule explicit: a candidate that misses policy or schema quality is rejected, while a correct but expensive model may be routed only to difficult cases.

The public benchmark shortlist never appears inside decision. That is intentional. Public scores help decide which candidates deserve testing. A release gate uses measurements taken on the workload you are about to serve.

Inspect Failure Slices, Not Only the Average

An aggregate score can hide the one category with the largest customer impact. The run below clears four of six cases overall, but damaged-electronics answers fail most often.

That report should block a release if damaged-item policy mistakes are critical, even when the average seems acceptable.

Small Samples Need an Uncertainty Margin

Three passing examples make a good tutorial, not a production release argument. Earlier statistics lessons introduced uncertainty in estimated rates. For a binary pass metric, a Wilson lower bound gives a conservative view of the pass rate supported by a finite sample:

Even perfect performance on three examples has a weak lower bound. Build a sufficiently large, risk-stratified holdout before claiming a system clears a production-quality bar.

Write Down Enough to Reproduce the Claim

Evaluation reports should let another engineer rerun the comparison and inspect its failures. Record these fields before sharing any model ranking:

That prompt-and-chat-format row points to the next lesson. If a model regularly omits required structure or speaks in the wrong role, evaluation has exposed a behavior gap. You then need to understand how instruction tuning and chat templates teach the interaction contract.

Failure Patterns to Diagnose

Mastery check

Key concepts

• Evaluation contracts and private golden sets
• Grounded answer checks for RAG
• Benchmark families and harness comparability
• HumanEval and pass@k
• Judge bias controls
• Contamination and time-split holdouts
• Quality, latency, and cost release gates

Evaluation rubric

• Foundational: Explains why public benchmark scores can't replace a private policy-answer evaluation contract.
• Intermediate: Builds a grounded-answer check that attributes retrieval versus generation failures.
• Intermediate: Explains when pass@k is appropriate and why it doesn't equal policy-answer accuracy.
• Advanced: Designs judge-order, failure-slice, uncertainty, latency, and cost controls for a release decision.

Common pitfalls

• Treating a public benchmark gain as evidence that policy answers are grounded.
• Comparing model scores produced by different prompts, tools, or scoring rules.
• Accepting an LLM judge result without checking position bias or factual anchors.
• Shipping on a mean quality score while ignoring critical slices, latency, or cost.

Follow-up questions

Practice extension

Add a fourth policy case whose retrieved context is intentionally wrong. Extend score-grounded-policy-answers.py to label failures as bad_retrieval or bad_generation. Then add a source_page field to the candidate response and reject answers whose citation doesn't match the evidence row. You have now built the first slice of an evaluation dashboard for a RAG assistant.