Hypothesis Tests, Intervals, and pass@k

Model B passed one extra coding task. Is that a durable upgrade, or just a lucky roll of the dice?

A batch solve rate wobbles from run to run, and an interval around one model's score still doesn't settle whether it beats its predecessor. Take a coding-assistant upgrade on six platform-helper tasks: parsing timestamp windows, permission fallbacks, retry budgets, event deduplication, quota rounding, and cache invalidation. Both models face identical prompts and hidden tests that mark each candidate pass or fail. The numbers stay small enough to verify every calculation by hand.

The estimation lesson bounded uncertainty around a single rate; sampling and simulation showed how repeated draws scatter an estimate. Here the target is the difference between two systems. All outcomes below are illustrative fixtures rather than benchmarks of commercial checkpoints. The Python examples rely purely on the standard library.

Before inspecting the task rows, consider a simple question: if five tasks tie and one favors Model B, which rows tell you who actually won?

View larger

Start with paired pass or fail outcomes

Both models receive the exact same prompt and face the exact same hidden unit tests. That pairing anchors the benchmark design. If prompts, test harnesses, or decoding parameters also change, the result compares those complete configurations; it doesn't isolate the checkpoint change. Treat each task as a single paired observation rather than two disconnected scores.

The evaluation flow keeps that paired structure visible: one prompt fans out to two completions, which face identical unit tests before computing the task difference.

View larger

Now inspect the six task outcomes. Look for rows where the two result columns disagree.

Convert pass to 1 and fail to 0. Model A's pass rate is 3 / 6 = 0.500, Model B's is 4 / 6 = 0.667, and the observed lift is 1 / 6 = 0.167, or 16.7 percentage points.

The next snippet prints those three quantities so you can check the arithmetic against the table.

model_a = [1, 0, 1, 0, 1, 0]
model_b = [1, 1, 1, 0, 1, 0]

rate_a = sum(model_a) / len(model_a)
rate_b = sum(model_b) / len(model_b)
lift = rate_b - rate_a

print(f"Model A pass@1: {rate_a:.3f}")
print(f"Model B pass@1: {rate_b:.3f}")
print(f"observed lift:   {lift:+.3f} ({lift * 100:+.1f} percentage points)")

Model A pass@1: 0.500
Model B pass@1: 0.667
observed lift:   +0.167 (+16.7 percentage points)

Looking only at aggregate totals conceals the pairing. Subtract Model A's score from Model B's score on each row:

Only disagreement tasks reveal which model performed better. Ties still shape both overall pass rates and the magnitude of the lift, but not the sign test's conditional win count. That distinction drives the first hypothesis test.

model_a = [1, 0, 1, 0, 1, 0]
model_b = [1, 1, 1, 0, 1, 0]

differences = [b - a for a, b in zip(model_a, model_b, strict=True)]
b_wins = differences.count(1)
a_wins = differences.count(-1)
ties = differences.count(0)

print("paired differences:", differences)
print(f"B wins={b_wins}, A wins={a_wins}, ties={ties}")
print("directional evidence comes from disagreements:", b_wins + a_wins)

paired differences: [0, 1, 0, 0, 0, 0]
B wins=1, A wins=0, ties=5
directional evidence comes from disagreements: 1

zip(..., strict=True) rejects unequal list lengths instead of silently discarding an unmatched task. Equal lengths alone don't establish pairing: real evaluation records must be joined by task ID, not their completion order. A missing model result must be resolved under the declared timeout or failure policy before comparing the same tasks.

A one-task lead exists in this sample. If both models are equally capable, count how often a fair comparison produces at least this many wins for Model B.

How surprising is one B win?

Only one row disagreed, and Model B won it. If the two models were equally strong, that row could have fallen either way. The null hypothesis ($H_0$) sets that baseline: on disagreement tasks, Model A and Model B are equally likely to win ($p = 0.5$). The directional alternative hypothesis ($H_1$) states that Model B wins more frequently ($p > 0.5$).

Drop the ties and treat each remaining pair as a fair coin toss under the null: Model B wins or Model A wins. That is the exact sign test on disagreements. Dixon and Mood presented the sign test in 1946.^{[1]Reference 1The Statistical Sign Testhttps://doi.org/10.1080/01621459.1946.10501898} For paired binary outcomes, this conditional binomial calculation also underlies the exact McNemar test of equal marginal pass probabilities; its usual two-sided form asks about either direction.^{[2]Reference 2statsmodels.stats.contingency_tables.mcnemarhttps://www.statsmodels.org/stable/generated/statsmodels.stats.contingency_tables.mcnemar.html}

This calculation assumes independent task pairs drawn from the task distribution you want to evaluate. Twenty near-duplicate prompts aren't twenty independent pieces of evidence. For related tasks derived from the same repository or template, use a cluster-level analysis instead of treating each variant as an independent observation.

Our six-task benchmark has one disagreement, and Model B won it. A one-sided p-value for the planned claim "Model B is better" asks:

If both models were equally likely to win a disagreement, how often would random chance hand Model B at least this many wins?

Before running the code, predict what happens with $d = 1$ and $b = 1$: can a single win ever be rare when a fair coin has only two outcomes?

If $d$ represents the count of disagreements and $b$ represents the count of Model B wins, the one-sided right-tail probability is:

The sum counts every outcome at least as favorable to Model B as the observed result. A p-value is this tail probability under the null hypothesis. It isn't the probability that the null hypothesis is true, and it isn't the probability that Model B is better.^{[3]Reference 3ASA Statement on Statistical Significance and P-Valueshttps://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf}

With one disagreement, Model B wins it with probability $0.5$. That result isn't surprising in the slightest.

View larger

The next function evaluates that exact tail for the one-disagreement fixture and for a larger 13-to-3 outcome:

from math import comb

def b_wins_one_sided_p_value(b_wins: int, a_wins: int) -> float:
    if b_wins < 0 or a_wins < 0:
        raise ValueError("win counts must be nonnegative")
    disagreements = b_wins + a_wins
    if disagreements == 0:
        return 1.0
    tail_count = sum(comb(disagreements, wins) for wins in range(b_wins, disagreements + 1))
    return tail_count / (2 ** disagreements)

print(f"one B win, zero A wins: p={b_wins_one_sided_p_value(1, 0):.3f}")
print(f"thirteen B wins, three A wins: p={b_wins_one_sided_p_value(13, 3):.4f}")

one B win, zero A wins: p=0.500
thirteen B wins, three A wins: p=0.0106

Thirteen wins to three is a completely different story. If you had declared "Model B is better" before seeing the outcomes, that one-sided tail probability is $0.0106$. If you had planned to detect a difference in either direction, the symmetric two-sided binomial test yields about $0.0213$ here. Never pick the favorable direction after inspecting who won.^{[4]Reference 4scipy.stats.binomtesthttps://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binomtest.html}

A p-value answers whether a win is surprising under the null. It doesn't tell you how large the lift is, or whether the test had enough power to detect a real gain.

Type I errors, Type II errors, and statistical power

Every model-review decision is a choice made under uncertainty. In statistical evaluation, two distinct mistakes can occur:

A Type I error is a false positive: rejecting a true null hypothesis. The fair-coin calculation uses equality as the boundary of the directional null that B wins no more often than A. A valid, preplanned level-$\alpha$ test controls its false-rejection probability at or below $\alpha$ under its assumptions. For our discrete exact test, the achievable probability can be smaller than the chosen threshold. Rejecting this statistical null is evidence about pass rates, not an automatic instruction to deploy.

A Type II error ($\beta$) is a false negative: failing to reject the null hypothesis when Model B is genuinely superior. An engineer trains a refined checkpoint that produces a real 3 percentage point lift, but the evaluation suite contains too few tasks to confirm it, so the model gets shelved.

Statistical power ($1 - \beta$) measures the probability of detecting a real improvement of a specific effect size $\delta$. Power depends directly on four interacting variables:

1. Effect size ($\delta$): At the same noise level, larger improvements are easier to detect; a one- or two-percentage-point gain may need many more tasks.
2. Sample size ($N$): Larger task suites provide more chances to observe disagreements and narrow the sampling error.
3. Variance ($\sigma^2$): Noisy prompts obscure the underlying difference.
4. Significance threshold ($\alpha$): Stricter false-positive gates (like $\alpha = 0.01$) demand stronger evidence, which lowers power unless sample size increases.

Conditional on observing just one disagreement, neither possible outcome can reject at the one-sided 5% threshold: even a B win has $p = 0.50$. More generally, the smallest possible one-sided p-value with $d$ disagreements is $2^{-d}$, so at least five disagreements are needed even if B wins every one. That's a limitation of the conditional test, not a power calculation for every possible six-task experiment. Power must average over future outcomes under a specified alternative, including how many disagreements arise.

A result with $p > 0.05$ doesn't establish that the models are identical. It may reflect a small effect, limited power, or no improvement. To establish that differences are smaller than a useful tolerance, plan an equivalence analysis around that tolerance rather than interpreting non-rejection as equality.

Looking for a winner changes the test

Suppose the 13-to-3 result was the best of twenty prompt variants tried against Model A. Reporting only its $p = 0.0106$ leaves out nineteen opportunities to find a small p-value by pure chance.

A significance level, such as $\alpha = 0.05$, limits the false-rejection probability for a single, preplanned test under its null. Searching a catalog changes the family-wise error rate: if twenty independent tests each falsely reject with probability $0.05$, the probability of at least one false rejection is:

One standard safeguard is the Bonferroni correction: for twenty planned comparisons, test each individual comparison at $\alpha / M = 0.05 / 20 = 0.0025$. This adjustment bounds the family-wise error rate even when tests are dependent, provided each p-value is valid. Our $0.0106$ result doesn't clear that bar. Bonferroni can be conservative, but the core discipline is to account for all comparisons searched, not just the winner.^{[5]Reference 5NIST/SEMATECH e-Handbook: Bonferroni's Methodhttps://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm}

Repeatedly inspecting the same benchmark and halting execution the moment $p < 0.05$ is another form of hidden search. Fix the sample size and analysis protocol in advance, or use sequential testing methods designed for continuous monitoring. If an evaluation suite guided prompt or hyperparameter selection, validate the chosen checkpoint on an untouched holdout benchmark. A low p-value isn't a replacement for clean data splits.^{[3]Reference 3ASA Statement on Statistical Significance and P-Valueshttps://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf}

When pairing reduces variance

Pairing removes a change in prompt mix from the comparison. Its effect on precision depends on the covariance between the two models' outcomes.

Let $X_i \in \{0, 1\}$ represent Model A's pass/fail outcome on task $i$, and let $Y_i \in \{0, 1\}$ represent Model B's outcome.

If you test Model A on $N$ independent prompts and Model B on a separate, unpaired batch of $N$ prompts, the variance of the estimated difference between the two sample means is the sum of their individual variances:

For example, both models might solve the timestamp helper and both might miss a cache-invalidation edge case. Shared difficulty can produce positive covariance. Specialized models can instead succeed on different tasks, so positive covariance and a particular correlation aren't guaranteed. The paired analysis must preserve the relationship actually present in the outcomes.^{[6]Reference 6STAT 504: Matched Pairs and McNemar's Testhttps://online.stat.psu.edu/stat504/Lesson11}

In a paired design, the estimator is the sample mean of the row-by-row differences $D_i = Y_i - X_i$:

Positive covariance reduces this variance; zero covariance leaves it unchanged, and negative covariance increases it relative to independent batches with the same marginal outcome distributions.

For a separate hypothetical task population, let 45% of tasks produce two passes, 45% two failures, 5% a B-only pass, and 5% an A-only pass. Both marginal pass rates are 50%, so $\sigma_X^2 = \sigma_Y^2 = 0.25$. The covariance is $E[XY] - E[X]E[Y] = 0.45 - 0.25 = 0.20$, giving correlation $\rho = 0.80$. These chosen assumptions define a population for comparing designs; the later 40-row fixture is separate. Compare designs with $N = 40$ observations per model under this population.

In an unpaired test, the variance is:

The standard error is $\sqrt{0.0125} \approx 0.1118$ (11.2 percentage points). A nominal normal-approximation margin ($\pm 1.96 \times \text{SE}$) is $\pm 21.9$ percentage points. The figure centers both reference bands at this population's true lift of zero to compare their widths; they aren't computed confidence intervals from observed task rows.

In a paired test, the covariance term activates:

The variance drops by a factor of five. The standard error shrinks to $\sqrt{0.0025} = 0.050$ (5.0 percentage points), giving a nominal normal margin of $\pm 9.8$ points. With only four expected disagreements in this hypothetical population, normal coverage can be poor; the exact variance calculation doesn't make the normal interval exact.

View larger

Under these chosen variances, 200 independent observations per model would match the variance of 40 paired observations. This comparison concerns variance under the stated assumptions; statistical power also depends on the test, effect size, and decision threshold.

An interval for B minus A

A confidence interval should target the quantity behind the decision. If the question is "How much better is Model B than Model A on this task suite?", build an interval around the paired lift $B - A$.

Comparing two separate pass-rate intervals can hide the pairing. The evaluation report needs a single interval for the row-by-row difference.

The 95% label describes the intended long-run coverage of the interval procedure over repeated experiments. It doesn't mean this specific realized interval has a 95% probability of containing the true parameter.^{[7]Reference 7NIST/SEMATECH e-Handbook: What Are Confidence Intervals?https://www.itl.nist.gov/div898/handbook/prc/section1/prc14.htm} A common shortcut, the Wald interval $\hat{p} \pm 1.96\sqrt{\hat{p}(1-\hat{p})/N}$, uses a symmetric normal approximation. On small binary samples it can produce impossible bounds outside $[0, 1]$ or collapse to zero width when every task passes or fails. The estimation chapter used Wilson intervals for one proportion; here we need an interval for a paired difference.^{[8]Reference 8NIST/SEMATECH e-Handbook: Confidence Intervals for Proportionshttps://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm}

In the estimation chapter, bootstrap resampling placed an interval around a single metric by drawing observations with replacement.^{[9]Reference 9Bootstrap Methods: Another Look at the Jackknife.https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods--Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full} Here the unit of resampling is the paired task row: keep Model A's outcome beside Model B's outcome, then recompute the mean difference. The percentile bootstrap interval takes the 2.5th and 97.5th percentiles of those resampled lifts as its bounds. This task bootstrap reflects sampling variation across a broader population of comparable tasks.^{[10]Reference 10scipy.stats.bootstraphttps://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html}

Prediction check: Six tasks help develop intuition but are too few for a stable interval. Consider a slightly larger fixture with 40 paired tasks. Before running the resampler, predict whether a +7.5-point lift from only 13 disagreements will remain strictly positive:

The observed lift is $(8 - 5) / 40 = 0.075$, or $+7.5$ percentage points. Bootstrap the paired differences to inspect how unstable that lift remains. The next cell resamples complete paired differences and reports the percentile interval.

from random import Random

differences = [0] * 27 + [1] * 8 + [-1] * 5
rng = Random(7)
n = len(differences)
bootstrap_lifts = [
    sum(rng.choice(differences) for _ in range(n)) / n
    for _ in range(20_000)
]

def quantile(values: list[float], q: float) -> float:
    ordered = sorted(values)
    pos = (len(ordered) - 1) * q
    lo = int(pos)
    hi = min(lo + 1, len(ordered) - 1)
    frac = pos - lo
    return ordered[lo] * (1 - frac) + ordered[hi] * frac

low = quantile(bootstrap_lifts, 0.025)
high = quantile(bootstrap_lifts, 0.975)
observed = sum(differences) / n
at_or_below_zero = sum(lift <= 0 for lift in bootstrap_lifts) / len(bootstrap_lifts)

print(f"observed paired lift: {observed * 100:+.1f} percentage points")
print(f"approximate 95% bootstrap interval: {low * 100:+.1f} to {high * 100:+.1f} points")
print("interval includes zero:", low <= 0 <= high)
print(f"share of resamples with lift <= 0: {at_or_below_zero:.0%}")

observed paired lift: +7.5 percentage points
approximate 95% bootstrap interval: -10.0 to +25.0 points
interval includes zero: True
share of resamples with lift <= 0: 24%

Resampling the differences array is equivalent to resampling complete rows for this binary example because each row contributes one difference. With richer multi-metric evals, keep the full row together so the analysis preserves the pairing. The task row, not an isolated model score, remains the fundamental unit of resampling.

The printed 24% is a fraction of bootstrap resamples, not a p-value and not a Bayesian posterior probability that Model B is worse. These resamples center around the observed sample lift; they weren't generated under the zero-lift null hypothesis.

View larger

Bootstrap intervals are approximate, especially with small or discrete samples. For instance, resampling six observed ties produces only zero lifts and a [0, 0] interval, even though six ties don't prove the models are identical on future tasks. Running more bootstrap iterations can't conjure missing evidence out of thin air. For our 40-task fixture, the actionable conclusion is: "Model B gained 7.5 points in this paired sample, but the approximate interval spans losses as well as gains."

Set a minimum useful lift threshold before examining results. If deploying Model B requires at least two percentage points of improvement to justify higher hosting costs, an interval of [+1, +14] points excludes zero but doesn't establish the required threshold. The next function formalizes those decisions:

def comparison_claim(
    observed_lift: float,
    interval: tuple[float, float],
    minimum_useful_lift: float = 0.02,
) -> str:
    low, high = interval
    if low > high or minimum_useful_lift < 0:
        raise ValueError("require ordered endpoints and a nonnegative useful lift")
    if low > minimum_useful_lift:
        return f"clears useful-lift threshold: estimated lift {observed_lift:+.3f}"
    if low > 0:
        return f"positive lift, useful size not established: estimate {observed_lift:+.3f}"
    if high < 0:
        return f"evidence of regression: estimated lift {observed_lift:+.3f}"
    return f"inconclusive: estimated lift {observed_lift:+.3f}, interval includes zero"

print(comparison_claim(0.075, (-0.100, 0.250)))
print(comparison_claim(0.075, (0.010, 0.140)))
print(comparison_claim(0.075, (0.030, 0.120)))

inconclusive: estimated lift +0.075, interval includes zero
positive lift, useful size not established: estimate +0.075
clears useful-lift threshold: estimated lift +0.075

So far each task evaluated a single completion from each model. Coding evaluations often permit several candidate attempts. That measures a different capability, and it demands its own evaluation protocol.

pass@k measures attempts

A coding assistant can generate multiple candidate functions, allowing an automated test harness to verify whether at least one candidate passes hidden tests. That is the capability measured by pass@k in functional code-generation benchmarks like HumanEval.^{[11]Reference 11Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374} The metric fixes a budget of $k$ candidate attempts per task. An alternative completion may solve a task that the model's first attempt missed.

HumanEval comprises 164 handwritten Python coding problems with an average of 7.7 unit tests each. In that paper, the authors generated $n = 200$ samples per task ($n \ge k$) and reported $k$ up to $100$. Those parameters reflect their specific experimental design, not a rigid requirement. The estimator below works for any predeclared $n \ge k$ that you run.

Consider this illustrative sample of three platform-helper tasks with three completions each. Follow the retry-state row: only the third candidate passes.

first_attempt_hit records whether Attempt 1 passed. any_of_3_hit records whether any of the three completions passed. Extra attempts discover additional solutions. These realized trials illustrate the underlying concept; a HumanEval-style harness evaluates all sampled candidates to compute an unbiased estimate.

Count each task once in both perspectives:

attempts = [
    [1, 0, 0],
    [0, 0, 1],
    [0, 0, 0],
]

first_attempt_hit = sum(row[0] for row in attempts) / len(attempts)
any_of_3_hit = sum(any(row[:3]) for row in attempts) / len(attempts)

print(f"first_attempt_hit: {first_attempt_hit:.3f}")
print(f"any_of_3_hit: {any_of_3_hit:.3f}")
print("extra solved tasks from extra attempts:", int((any_of_3_hit - first_attempt_hit) * len(attempts)))

first_attempt_hit: 0.333
any_of_3_hit: 0.667
extra solved tasks from extra attempts: 1

The ordered table answered a product question about a known first completion. For reporting standard HumanEval metrics, treat the $n$ candidates as an unordered pool and evaluate the combinatorial estimator on the count of correct completions. For the same three tasks with $n = 3$ samples each, the correct counts are $c = [1, 1, 0]$. The unbiased pass@1 score is the mean of $c/n$:

When $k = 1$, the combinatorial formula simplifies to $c/n$, regardless of which completion was generated first. When $n = k = 3$, the combinatorial estimator equals the any_of_3_hit column. These aren't conflicting definitions: one is a single realized draw, while the estimator computes the expected value over all possible subsets drawn from the generated pool.

Keep two budgets distinct: $k$ specifies the operational attempt budget being evaluated; $n$ is the number of samples drawn to estimate that metric. You can estimate pass@1 using 200 samples without giving the end product 200 attempts.

Publish $k$, the sample pool size $n$, the decoding parameters, and the test suites used to verify functional correctness. A higher score under a larger candidate budget doesn't imply stronger single-completion performance. Chen and colleagues also observed that higher sampling temperature often benefits larger $k$, because more diverse candidates increase the chance that at least one completion solves the problem.^{[11]Reference 11Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Derive the HumanEval estimator by hand

Suppose the assistant generates $n = 10$ candidate implementations for a function, and hidden tests accept $c = 2$ of them. You want to compute the expected pass@5 score if you pick five candidates from that pool. Before calculating, predict which event is easier to count: at least one candidate passing, or all five candidates failing?

Counting successful subsets directly is tedious. Count the failure case instead. The event "at least one candidate passes" is the complement of "all five candidates fail":

1. There are $10 - 2 = 8$ failing candidates.
2. There are $\binom{10}{5} = 252$ total ways to pick five candidates from the pool.
3. There are $\binom{8}{5} = 56$ ways to pick five candidates that all fail.
4. The probability that at least one candidate passes is $1 - 56 / 252 = 0.778$.

In general notation:

The target parameter is generative: the probability that at least one of $k$ independent model completions succeeds, written $1 - (1-p)^k$ for this task's unknown per-completion pass probability $p$. The combinatorial formula is an unbiased minimum-variance estimator (a U-statistic) under those independent, identically distributed draws. It calculates the exact fraction of $k$-sized subsets of the $n$ completions that contain at least one passing solution. The HumanEval protocol samples $n \ge k$ and reports this combinatorial estimator rather than a naive plug-in.^{[11]Reference 11Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Why is averaging these subsets unbiased? Any fixed subset of k distinct positions contains k i.i.d. completions, so its pass indicator has expected value $1-(1-p)^k$. Averaging all those indicators keeps that same expected value. The subsets overlap and aren't independent of one another, so their count $\binom{n}{k}$ isn't a new sample size.^{[11]Reference 11Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Here i.i.d. signifies independent draws from the same decoding distribution for this task. Fix $n$ in advance and don't discard duplicate completions: identical outputs still carry probability mass. Halting early after the first passing completion, deduplicating outputs, or feeding test errors back into subsequent attempts alters the experimental distribution. A multi-turn agent that inspects error traces requires an evaluation of that complete iterative harness, not this single-turn formula.

Hidden tests serve as an evaluation oracle: pass@k credits a candidate pool whenever it contains a passing solution. It doesn't assess whether an unguided production system, without access to hidden unit tests, can pick out that passing candidate. Evaluate your reranker or selection model separately if the product must return a single answer.

The next snippet verifies the hand calculation with comb:

from math import comb

n = 10
c = 2
k = 5
all_groups = comb(n, k)
all_failing_groups = comb(n - c, k)
score = 1 - all_failing_groups / all_groups

print("all groups:", all_groups)
print("all-failing groups:", all_failing_groups)
print(f"pass@5: {score:.3f}")

all groups: 252
all-failing groups: 56
pass@5: 0.778

How the naive plug-in underestimates pass@k

A tempting shortcut calculates the empirical success rate and substitutes it into the generative formula:

where $\hat{p} = c / n$ is the observed single-sample pass rate. Chen and colleagues demonstrated that $1 - (1 - \hat{p})^k$ is a biased underestimate of the generative pass@k probability $1 - (1 - p)^k$.^{[11]Reference 11Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Two distinct mathematical principles explain this shortfall:

1. Jensen's inequality on concave functions: Consider the function $g(p) = 1 - (1 - p)^k$. For $k > 1$ and $p \in (0, 1)$, the second derivative with respect to $p$ is strictly negative: $$g''(p) = -k(k - 1)(1 - p)^{k - 2} < 0$$ Because $g(p)$ is strictly concave, Jensen's inequality guarantees that the expected value of the plug-in estimator is strictly smaller than the true function evaluated at the mean: $$\mathbb{E}\left[1 - (1 - \hat{p})^k\right] < 1 - \left(1 - \mathbb{E}[\hat{p}]\right)^k = 1 - (1 - p)^k$$ Plugging a noisy empirical proportion into a concave curve inevitably pushes the estimate downward.
2. Sampling with replacement versus without replacement: The plug-in expression $(1 - c/n)^k$ represents drawing $k$ samples with replacement from the finite pool of $n$ completions. Each draw assumes an unchanged failure rate of $(n - c) / n$. But in an actual without-replacement draw, pulling a failing completion removes that failure from the denominator. The proportion of successes among the remaining candidates rises from $c/n$ to $c / (n - 1)$. Drawing without replacement depletes the failure pool, making a success more probable on subsequent picks.

For $0 < c < n$ and $k > 1$, the naive plug-in is strictly smaller than the unbiased combinatorial estimate on the same $(n, c, k)$.

View larger

For $n = 10$, $c = 2$, and $k = 5$, compare both calculations directly:

p_hat = c / n
naive_score = 1.0 - (1.0 - p_hat) ** k
unbiased_score = score  # From previous cell: 1 - comb(8, 5) / comb(10, 5)

shortfall = unbiased_score - naive_score
print(f"Plug-in pass@5 1-(1-c/n)^k:            {naive_score:.4f}")
print(f"Unbiased combinatorial pass@5:         {unbiased_score:.4f}")
print(f"Plug-in shortfall vs unbiased estimate: {shortfall:.4f}")

assert abs(naive_score - 0.67232) < 1e-5

Plug-in pass@5 1-(1-c/n)^k:            0.6723
Unbiased combinatorial pass@5:         0.7778
Plug-in shortfall vs unbiased estimate: 0.1055

Two boundary checks confirm the logic:

• When $c = 0$, no selected group can pass.
• When fewer than $k$ failures exist ($n - c < k$), every $k$-sized group contains at least one passing candidate.

Floating-point implementations of large combinations or factorials can overflow. Python's math.comb returns exact integers, but constructing those integers can still be costly. The HumanEval paper provides an equivalent product formulation that avoids the large combinations:^{[11]Reference 11Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

It evaluates the failure probability factor by factor, avoiding factorials of large numbers.

from math import prod

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    failure_probability = prod(1.0 - k / i for i in range(n - c + 1, n + 1))
    return 1.0 - failure_probability

print(f"n=10, c=2, k=1: {pass_at_k(10, 2, 1):.3f}")
print(f"n=10, c=2, k=5: {pass_at_k(10, 2, 5):.3f}")
print(f"no passing candidates: {pass_at_k(10, 0, 5):.3f}")
print(f"not enough failures: {pass_at_k(10, 8, 5):.3f}")

try:
    pass_at_k(10, 11, 5)
except ValueError as error:
    print(error)

n=10, c=2, k=1: 0.200
n=10, c=2, k=5: 0.778
no passing candidates: 0.000
not enough failures: 1.000
require n > 0, 0 <= c <= n, and 1 <= k <= n

Average pass@k across tasks

A benchmark score macro-averages task-level results. A complex permission-verification function counts as one task; it shouldn't be drowned out by hundreds of completions from an easy string helper. The estimator runs per task before the benchmark computes its overall mean. Tasks can have different success probabilities: if half always pass and half always fail, both macro pass@1 and macro pass@5 equal 0.5. Applying $1-(1-0.5)^5$ to that aggregate rate instead gives 0.969. The nonlinear transformation must happen within each task before averaging; an aggregate pass@5 generally can't be inverted to recover aggregate pass@1.

Suppose four tasks each produce $n = 10$ candidates, with the following counts of correct completions:

[0, 1, 2, 4]

Compute each task's estimate first, then average those four estimates. A task with zero passing completions contributes zero across every value of $k$.

correct_counts = [0, 1, 2, 4]
for k in (1, 3, 5):
    task_scores = [pass_at_k(10, correct, k) for correct in correct_counts]
    mean_score = sum(task_scores) / len(task_scores)
    print(f"pass@{k}: {mean_score:.3f}  per-task={[round(score, 3) for score in task_scores]}")

pass@1: 0.175  per-task=[0.0, 0.1, 0.2, 0.4]
pass@3: 0.417  per-task=[0.0, 0.3, 0.533, 0.833]
pass@5: 0.563  per-task=[0.0, 0.5, 0.778, 0.976]

Notice that pass@5 climbs to $0.563$ while pass@1 sits at $0.175$. That extra search capacity helps if your production product can verify multiple candidates. It doesn't indicate that single-completion quality improved. The metric grew because the attempt budget expanded, not because the underlying tasks became easier.

Changed protocols create fake wins

To compare models fairly under a fixed sampling policy, keep the task suite, unit tests, candidate budget $k$, and decoding rules matched. Record token cutoffs, execution timeouts, and candidate selection rules. A timeout shouldn't quietly vanish from the denominator.

Record $n$ as well, but don't confuse it with $k$. Two independently sampled pools with different predeclared $n \ge k$ estimate the same underlying pass@k; their precision differs. Matching $n$ is a convenient balanced setup, not a mathematical necessity for a valid fixed-$k$ benchmark. Don't omit challenging tasks simply because they generated fewer completions than requested: finish the protocol or report that the score can't be computed.^{[12]Reference 12HumanEval: Functional Correctness Evaluation and pass@k Estimatorhttps://github.com/openai/human-eval/blob/master/human_eval/evaluation.py}

A deterministic decoder illustrates the risk. If every generated candidate is identical, extra sampling slots can't uncover an alternative correct solution. For a fixed deterministic decoding pipeline, pass@5 equals pass@1. A temperature-zero setting alone doesn't establish reproducibility of the whole serving pipeline.

identical_failed_candidates = [0] * 10
identical_passing_candidates = [1] * 10

for name, outcomes in [
    ("same failed candidate", identical_failed_candidates),
    ("same passing candidate", identical_passing_candidates),
]:
    correct = sum(outcomes)
    print(name, f"pass@1={pass_at_k(10, correct, 1):.1f}", f"pass@5={pass_at_k(10, correct, 5):.1f}")

same failed candidate pass@1=0.0 pass@5=0.0
same passing candidate pass@1=1.0 pass@5=1.0

Serving engines can introduce nondeterminism from external system factors, so record the exact runtime flags. A degenerate deterministic output distribution remains compatible with independent draws: every draw returns the identical completion. Duplicate completions alone don't prove a protocol violation. They remain in $n$ and $c$. A larger $k$ represents a larger compute expenditure, not an unearned model upgrade.

Another risk is subtler: code that passes basic unit tests can still harbor edge-case bugs or malicious side effects. Unit-test evaluation establishes functional correctness under that specific suite. It doesn't authorize executing untrusted code directly on host infrastructure.

HumanEval's authors described an isolated sandbox using gVisor and network restrictions.^{[11]Reference 11Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374} The released harness isn't that complete isolation environment: its README requires users to supply a robust security sandbox. Treat functional correctness testing and execution isolation as separate layers.^{[13]Reference 13HumanEval: Usage and Execution Isolation Requirementshttps://github.com/openai/human-eval}

Build an evaluation report

A model review needs one scorecard, not two disconnected headlines. The 40-task paired fixture evaluates single-attempt lift. A separate four-task pool evaluates Model B's multi-candidate search behavior. Both require reported uncertainty and a documented protocol. The second fixture isn't a direct comparison with Model A.

This self-contained script runs the calculations and reports each fixture alongside its protocol metadata:

from itertools import product
from math import prod
from random import Random

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    return 1.0 - prod(1.0 - k / i for i in range(n - c + 1, n + 1))

def quantile(values: list[float], q: float) -> float:
    ordered = sorted(values)
    pos = (len(ordered) - 1) * q
    lo = int(pos)
    hi = min(lo + 1, len(ordered) - 1)
    frac = pos - lo
    return ordered[lo] * (1 - frac) + ordered[hi] * frac

model_a = [1] * 17 + [0] * 10 + [0] * 8 + [1] * 5
model_b = [1] * 17 + [0] * 10 + [1] * 8 + [0] * 5
paired_differences = [b - a for a, b in zip(model_a, model_b, strict=True)]
paired_lift = sum(paired_differences) / len(paired_differences)
rng = Random(7)
lift_resamples = [
    sum(rng.choice(paired_differences) for _ in paired_differences)
    / len(paired_differences)
    for _ in range(20_000)
]
lift_lo, lift_hi = quantile(lift_resamples, 0.025), quantile(lift_resamples, 0.975)

correct_counts_b = [0, 1, 2, 4]
task_scores = [pass_at_k(10, correct, 5) for correct in correct_counts_b]
pass5 = sum(task_scores) / len(task_scores)
# Task-level bootstrap of macro pass@k: every with-replacement resample of the
# four task scores (4^4 = 256), then percentile interval of the resample means.
# With only four toy tasks the interval is huge; the point is the reporting contract.
boot_means = [
    sum(task_scores[i] for i in idxs) / len(task_scores)
    for idxs in product(range(len(task_scores)), repeat=len(task_scores))
]
pass5_lo, pass5_hi = quantile(boot_means, 0.025), quantile(boot_means, 0.975)
protocol = {
    "paired_fixture_tasks": len(model_a),
    "paired_candidates_per_model_per_task": 1,
    "candidate_pool_fixture_tasks": len(correct_counts_b),
    "metric": "hidden-test functional correctness",
    "samples_per_task": 10,
    "reported_k": 5,
    "outcomes": "illustrative fixtures, not live generations",
}

print(f"paired pass@1 lift: {paired_lift * 100:+.1f} percentage points")
print(f"approximate 95% lift interval: [{lift_lo * 100:+.1f}, {lift_hi * 100:+.1f}] points")
print(f"Model B pass@5 on candidate pool: {pass5:.3f}")
print(f"pass@5 95% task-bootstrap interval: [{pass5_lo:.3f}, {pass5_hi:.3f}]")
for key, value in protocol.items():
    print(f"{key}: {value}")

paired pass@1 lift: +7.5 percentage points
approximate 95% lift interval: [-10.0, +25.0] points
Model B pass@5 on candidate pool: 0.563
pass@5 95% task-bootstrap interval: [0.194, 0.877]
paired_fixture_tasks: 40
paired_candidates_per_model_per_task: 1
candidate_pool_fixture_tasks: 4
metric: hidden-test functional correctness
samples_per_task: 10
reported_k: 5
outcomes: illustrative fixtures, not live generations

For $k > 1$, pass@k is a nonlinear function of task-level completion counts, which is subsequently averaged across tasks. Resampling tasks and recalculating macro pass@k models task-sampling uncertainty. Comparing Model A and Model B at the same $k$ requires per-task estimates from both models: resample the paired differences between them, not two disconnected sets of tasks.

With four toy tasks the bootstrap interval is wide. Its nominal coverage depends on the sampling design and assumptions, so this calculation alone does not support a launch decision. This task bootstrap holds the generated completions fixed when resampling rows; it does not model how an entirely new batch of completions on the same tasks would vary. For grouped benchmark sets, resample independent clusters rather than treating every prompt as independent.

Before approving a production rollout, verify that your evaluation report contains:

• An interval for the paired lift rather than two disconnected headline numbers
• The macro pass@k point estimate alongside an empirical bootstrap interval
• The planned hypothesis direction chosen before inspecting test results
• The total count of comparisons evaluated and any family-wise error adjustments
• Explicit checkpoint hashes, prompt templates, decoding seeds, and timeout configs
• Sandboxing guarantees and test suite coverage audits
• Latency and cost ceilings for candidate generation in production

Statistical significance and operational value address different questions. Evidence for a pass-rate gain doesn't establish that its size justifies additional generation, verification latency, serving cost, or security exposure.

Practice: review a benchmark claim

A teammate submits this update: "Model B is better because its pass@5 reached 64%, whereas Model A's pass@1 was 58%."

Write a code review comment addressing three specific issues:

1. Request matched attempt budgets $k$, prompt suites, test suites, and decoding settings, while noting the sample size $n \ge k$.
2. Ask for the paired lift and its confidence interval under that matched protocol.
3. Inquire whether the extra generation budget, verification latency, and compute cost are viable for production serving.

A rigorous rewritten claim should read:

Under the same 200 paired platform-helper tasks and a matched pass@1 protocol, Model B's observed pass rate was 2.5 percentage points higher. The paired 95% interval and serving cost guardrails are reported below; the point estimate alone doesn't justify a production swap. pass@5 is documented separately because it reflects a larger candidate search budget.