Hypothesis Tests, Intervals, and pass@k

Sampling rates in Distributions and Sampling moved from sample to sample. Hypothesis tests build on that wobble: a one-task win isn't yet a model replacement. It matters when a platform engineering team evaluates a coding assistant that writes small functions for timestamp parsing, retry limits, permission checks, and cache invalidation. Hidden tests mark each generated function as pass or fail.

Model B passes one more task than Model A on a six-task evaluation. Is that enough evidence to replace the old model? Learn how to answer without confusing a promising result with a proven improvement.

The numbers below are deliberately small so you can calculate them by hand. They teach the method, not a production launch threshold.

Start with paired pass or fail outcomes

Each prompt asks for one Python helper used in an internal developer workflow. Both models receive the same prompt and are checked by the same hidden tests.

Convert pass to 1 and fail to 0. Then Model A has a pass rate of 3 / 6 = 0.500, Model B has 4 / 6 = 0.667, and the observed lift is 1 / 6 = 0.167, or 16.7 percentage points.

Run the calculation rather than trusting a headline.

A paired evaluation preserves more information than two totals. Subtract A from B for each task:

Only disagreement tasks tell you which model won. Five ties make the benchmark look larger without giving any directional evidence.

A hypothesis test asks how surprising the win is

The null hypothesis says that, on disagreement tasks, Model A and Model B are equally likely to win. The directional alternative says Model B wins more often.

Under that null hypothesis, each disagreement is like a fair coin: B wins or A wins. Our six-task benchmark has one disagreement, and it went to B. A one-sided p-value for the planned claim "B is better" asks:

If both models were equally likely to win a disagreement, how often would B win at least this many of the disagreements?

With one disagreement, B wins it with probability 0.5. That result isn't rare. The sample moved upward, but the evidence is weak.

This exact coin calculation is easy to implement with the binomial coefficients you already know.

The second line shows why more disagreement evidence matters. A result with 13 B wins and 3 A wins under a predeclared directional question is much harder to explain with an equal-win coin.

If you had planned to detect a difference in either direction, you would use a two-sided version instead. Pick the question before seeing the winning direction.

Put an interval around the quantity you care about

A confidence interval should target the decision. If your question is "How much better is B than A on the same tasks?", calculate an interval for the paired lift B - A. Comparing two separate pass-rate intervals can hide the pairing and isn't the right decision rule.

A paired bootstrap interval repeatedly resamples complete task rows with replacement. Each resample keeps Model A's outcome beside Model B's outcome, then recomputes the mean difference. The bootstrap is a general resampling method for estimating sampling uncertainty from observed data.^{[1]Reference 1Bootstrap Methods: Another Look at the Jackknife.https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods--Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full}

Six tasks are useful for intuition but painfully sparse. Use a slightly larger illustrative evaluation with 40 paired tasks:

The observed lift is (8 - 5) / 40 = 0.075, or +7.5 percentage points. Bootstrap the paired differences to see how unstable that lift remains.

Bootstrap intervals are approximate, especially with tiny or highly discrete samples. They help you see uncertainty; they don't turn weak evidence into certainty. Here the correct statement is: "B gained 7.5 points in this paired sample, and the interval still includes zero."

Write that conclusion as a rule your evaluation report can enforce.

pass@k measures attempts

So far, each task used one candidate completion from each model. A coding assistant can also generate several candidate functions and let an evaluator check whether at least one passes hidden tests. That's the question measured by pass@k in functional code-generation evaluations such as HumanEval.^{[2]Reference 2Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Consider this illustrative ordered sample of three platform-helper tasks with three completions apiece. It shows one realized batch, not the HumanEval estimator:

Here first_attempt_hit records whether Attempt 1 passed, and any_of_3_hit asks whether any of three completions passed. These columns illustrate how a larger attempt budget can solve more tasks. They are not what a HumanEval-style harness reports.

For reporting HumanEval-style numbers, treat the n candidates as an unordered bag and use the combinatorial estimator on the count of correct outcomes. For the same three tasks with n=3 per task, correct counts are c = [1, 1, 0], so unbiased pass@1 is the average of c/n:

$$\frac{1}{3}\left(\frac{1}{3} + \frac{1}{3} + \frac{0}{3}\right) \approx 0.222$$

That is not "column 0 of an ordered table." When k=1, the combinatorial formula always reduces to c/n, regardless of which sample happened to appear first. The model isn't being awarded the same budget at k=1 and k=3.

For pass@k, a report must publish k, the number of generated samples, the decoding policy, and the tests used to judge correctness. A higher score under a larger attempt budget isn't evidence of stronger single-candidate behavior.

Derive the HumanEval estimator by hand

Suppose the assistant generates n = 10 candidate implementations for one function and hidden tests accept c = 2 of them. You want the expected pass@5 result if you select five candidates from that pool.

Counting success cases is tedious. Count the failure case instead:

1. There are 10 - 2 = 8 failing candidates.
2. There are C(10, 5) = 252 ways to choose five candidates.
3. There are C(8, 5) = 56 all-failing choices.
4. The chance of at least one pass is 1 - 56 / 252 = 0.778.

In notation:

$$\text{pass@k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

The quantity we care about is still generative: the probability that at least one of k i.i.d. model samples passes, often written $1 - (1-p)^k$ for unknown per-sample success rate $p$. The combinatorial formula is an unbiased U-statistic for that generative quantity; combinatorially it equals the fraction of $k$-subsets of the $n$ draws that contain a success. The paper therefore samples $n \ge k$ and reports the U-statistic, not a with-replacement plug-in. Chen and colleagues use this unbiased estimator for HumanEval evaluation after generating more samples per task than the reported k.^{[2]Reference 2Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Naive estimator bias: with replacement vs. without replacement

A tempting plug-in estimate for pass@k is:

$$\text{pass@k}_{\text{naive}} = 1 - (1 - \hat{p})^k$$

where $\hat{p} = \frac{c}{n}$ is the observed single-sample success rate. The plug-in $1-(1-\hat{p})^k$ is a biased estimator of the generative pass@k probability $1-(1-p)^k$. The combinatorial HumanEval form is unbiased for that same generative quantity; combinatorially it also equals the chance that at least one of $k$ distinct selections from the frozen pool of $n$ candidates succeeds.^{[2]Reference 2Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

After drawing a failing candidate without replacement, correct candidates make up a larger share of the remaining pool. For $0 < c < n$ and $k > 1$, the plug-in calculation is strictly smaller than the unbiased combinatorial estimate on the same $(n,c,k)$.

For $n=10$, $c=2$, and $k=5$, compare both calculations:

Two boundary checks should feel right:

• When c = 0, no selected group can pass.
• When fewer than k failures exist, every k-sized group contains at least one passing candidate.

For larger values of n, avoid assembling enormous combinations. The HumanEval paper gives an equivalent product implementation that stays numerically well behaved.^{[2]Reference 2Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Average per-task estimates, not raw completions

A benchmark score averages task-level results. One difficult permission-check function counts as one task; it shouldn't disappear beneath hundreds of samples from an easier string formatter.

Assume four tasks each produce n = 10 candidates, with the following correct-count vector:

Compute each task's estimate first, then average those four estimates.

Notice that pass@5 can rise dramatically while pass@1 remains modest. That's useful information if your product can test several generated candidates, but it isn't a substitute for measuring single-candidate behavior under the same protocol.

Failure case: changed protocols create fake wins

pass@k only supports a comparison when the protocol matches. Keep the same task set, hidden tests, number of generated samples n, retained attempt budget k, and decoding rule.

A deterministic decoder illustrates the trap. If every generated candidate is identical for a task, extra attempt slots can't discover a different correct solution. In a controlled deterministic fixture, pass@5 collapses to pass@1.

Real serving stacks can introduce nondeterminism from outside the sampling policy, so record the actual decoder and run settings. Operationally, multiple attempts only buy search when they produce meaningfully different candidates.

A second failure is subtler: generated code that passes weak hidden tests can still be wrong on missing cases or unsafe to execute. Unit-test passing measures functional correctness under that test suite. It doesn't authorize running untrusted code in a production environment. HumanEval's authors evaluated generated code in a sandbox for that reason.^{[2]Reference 2Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Build an evaluation report

Put the pieces together into one report for a model-review meeting. The first comparison measures a paired pass@1 lift. The second metric measures multi-attempt capability under an explicitly recorded stochastic sampling setup.

pass@k is a nonlinear U-statistic averaged over tasks. A bare point estimate reintroduces the overconfidence habit this chapter fights. Resample tasks, recompute macro pass@k on each draw, and report the percentile interval beside the point. With four toy tasks the interval is enormous; that is honest, not a bug.

Before anyone calls a winner, your report still needs:

• an interval for the paired lift, not two unrelated headline intervals
• the macro pass@k point estimate plus an interval or another uncertainty summary
• the directional or two-sided hypothesis question chosen before inspecting results
• task and hidden-test quality checks
• latency, token cost, and sandboxing constraints for any deployment decision

Statistical significance and product value answer different questions. Even convincing statistical evidence can't tell you whether a gain justifies additional candidate generation, test execution, latency, or risk.

Practice: review a benchmark claim

A teammate writes: "Model B is better because its pass@5 is 64%, while Model A's pass@1 is 58%."

Write a review comment with three corrections:

1. Request the same k, n, task set, tests, and decoding policy for both models.
2. Ask for the paired lift and its uncertainty interval under that matched protocol.
3. Ask whether added generation and test-execution budget is acceptable for the product.

A good rewritten claim would sound like this:

Under the same 200 paired platform-helper tasks and fixed pass@1 protocol, Model B improved hidden-test pass rate by 2.5 percentage points; the paired interval and cost guardrails are reported below. pass@5 is listed separately because it measures a larger candidate-search budget.