Hypothesis Tests, Intervals, and pass@k

Model B passed one extra coding task. Is that a durable upgrade, or just a lucky roll of the dice?

A batch solve rate wobbles from run to run, and putting an interval around one model's score still doesn't settle whether it beats its predecessor. In this lesson, we test a coding-assistant upgrade using a suite of six platform-helper tasks: parsing timestamp windows, permission fallbacks, retry budgets, event deduplication, quota rounding, and cache invalidation. Both models face identical prompts and hidden tests that mark each candidate pass or fail. The numbers stay small enough to verify every calculation by hand.

The estimation lesson bounded uncertainty around a single rate; sampling and simulation showed how repeated draws scatter an estimate. Here the target is the difference between two systems. All outcomes below are illustrative fixtures rather than benchmarks of commercial checkpoints. The Python examples rely purely on the standard library.

Before inspecting the task rows, consider a simple question: if five tasks tie and one favors Model B, which rows tell you who actually won?

Start with paired pass or fail outcomes

Both models receive the exact same prompt and face the exact same hidden unit tests. That pairing anchors the entire benchmark design. If prompts, test harnesses, or decoding parameters drift between runs, you're no longer measuring a model upgrade; you're measuring environmental noise. Treat each task as a single paired observation rather than two disconnected scores.

The evaluation flow keeps that paired structure visible: one prompt fans out to two completions, which face identical unit tests before computing the task difference.

Now inspect the six task outcomes. Look for rows where the two result columns disagree.

Convert pass to 1 and fail to 0. Model A's pass rate is 3 / 6 = 0.500, Model B's is 4 / 6 = 0.667, and the observed lift is 1 / 6 = 0.167, or 16.7 percentage points.

The next snippet prints those three quantities so you can check the arithmetic against the table.

Looking only at aggregate totals conceals the pairing. Subtract Model A's score from Model B's score on each row:

Only disagreement tasks reveal which model performed better. Ties still shape both overall pass rates and the magnitude of the lift, but not the sign test's conditional win count. That distinction drives the first hypothesis test.

A one-task lead exists in this sample. The central question is whether that lead is surprising if both models are equally capable. In other words, how often would a fair comparison produce at least this many wins for Model B?

How surprising is one B win?

Only one row disagreed, and Model B won it. If the two models were equally strong, that row could have fallen either way. The null hypothesis ($H_0$) sets that baseline: on disagreement tasks, Model A and Model B are equally likely to win ($p = 0.5$). The directional alternative hypothesis ($H_1$) states that Model B wins more frequently ($p > 0.5$).

Drop the ties and treat each remaining pair as a fair coin toss: Model B wins or Model A wins. That is the exact sign test on disagreements. Dixon and Mood presented this paired sign-test formulation in 1946; the test conditions strictly on the number of disagreements and ignores tied outcomes.^{[1]Reference 1The Statistical Sign Testhttps://doi.org/10.1080/01621459.1946.10501898}

This calculation assumes independent task pairs drawn from the task distribution you want to evaluate. Twenty near-duplicate prompts aren't twenty independent pieces of evidence. For related tasks derived from the same repository or template, use a cluster-level analysis instead of treating each variant as an independent observation.

Our six-task benchmark has one disagreement, and Model B won it. A one-sided p-value for the planned claim "Model B is better" asks:

If both models were equally likely to win a disagreement, how often would random chance hand Model B at least this many wins?

Before running the code, predict what happens with $d = 1$ and $b = 1$: can a single win ever be rare when a fair coin has only two outcomes?

If $d$ represents the count of disagreements and $b$ represents the count of Model B wins, the one-sided right-tail probability is:

$$p = \sum_{w=b}^{d} \binom{d}{w} \left(\frac{1}{2}\right)^d$$

The sum counts every outcome at least as favorable to Model B as the observed result. A p-value is this tail probability under the null hypothesis. It isn't the probability that the null hypothesis is true, and it isn't the probability that Model B is better.^{[2]Reference 2ASA Statement on Statistical Significance and P-Valueshttps://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf}

With one disagreement, Model B wins it with probability $0.5$. That result isn't surprising in the slightest.

The next function evaluates that exact tail for the one-disagreement fixture and for a larger 13-to-3 outcome:

Thirteen wins to three is a completely different story. If you had declared "Model B is better" before seeing the outcomes, that one-sided tail probability is $0.0106$. If you had planned to detect a difference in either direction, the symmetric two-sided binomial test yields about $0.0213$ here. Never pick the favorable direction after inspecting who won.^{[3]Reference 3scipy.stats.binomtesthttps://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.binomtest.html}

Model B passes four tasks and Model A passes three, but there is only one disagreement. What is the safe conclusion?

Answer

Model B scored higher in this small sample. The single disagreement gives weak evidence for a durable improvement, so you should collect more paired tasks before claiming the model is better.

A p-value answers whether a win is surprising under the null. It doesn't tell you how large the lift is, or whether the test had enough power to detect a real gain.

Type I errors, Type II errors, and statistical power

Every model-review decision is a choice made under uncertainty. In statistical evaluation, two distinct mistakes can occur:

A Type I error ($\alpha$) is a false positive: rejecting the null hypothesis when Model B is actually no better (or worse) than Model A. Shipping a false win burns GPU memory, invalidates inference caches, complicates rollbacks, and risks silent user regressions. Setting a significance threshold such as $\alpha = 0.05$ caps the probability of making a Type I error at 5% under the baseline assumption that the null hypothesis is true.

A Type II error ($\beta$) is a false negative: failing to reject the null hypothesis when Model B is genuinely superior. An engineer trains a refined checkpoint that produces a real 3 percentage point lift, but the evaluation suite contains too few tasks to confirm it, so the model gets shelved.

Statistical power ($1 - \beta$) measures the probability of detecting a real improvement of a specific effect size $\delta$. Power depends directly on four interacting variables:

1. Effect size ($\delta$): Large leaps in model capability are easy to detect; subtle 1% to 2% gains require extensive data.
2. Sample size ($N$): Larger task suites provide more chances to observe disagreements and narrow the sampling error.
3. Variance ($\sigma^2$): Noisy prompts obscure the underlying difference.
4. Significance threshold ($\alpha$): Stricter false-positive gates (like $\alpha = 0.01$) demand stronger evidence, which lowers power unless sample size increases.

In our six-task benchmark with a single disagreement, statistical power against even a massive 20-point lift is practically zero. Because a single fair coin toss has probability $0.50$, it's mathematically impossible for any one-disagreement result to produce $p \le 0.05$. You could never reject the null on one disagreement, no matter how much better Model B truly is.

Even with 16 disagreements, power against subtle improvements remains modest. When an evaluation yields $p > 0.05$, that doesn't mean the two models are identical; it often just means the evaluation lacked the statistical power to resolve the difference. Never mistake absence of evidence for evidence of absence.

Looking for a winner changes the test

Suppose the 13-to-3 result was the best of twenty prompt variants tried against Model A. Reporting only its $p = 0.0106$ leaves out nineteen opportunities to find a small p-value by pure chance.

A significance level, such as $\alpha = 0.05$, limits the false-rejection probability for a single, preplanned test under its null. It doesn't guarantee a five-percent chance of error when searching through a catalog of experiments. If twenty independent tests each falsely reject with probability $0.05$, the probability of observing at least one false rejection across the family is:

$$1 - (1 - 0.05)^{20} \approx 64\%$$

One standard safeguard is the Bonferroni correction: for twenty planned comparisons, test each individual comparison at $\alpha / M = 0.05 / 20 = 0.0025$. This adjustment bounds the family-wise error rate even when tests are dependent, provided each p-value is valid. Our $0.0106$ result doesn't clear that bar. Bonferroni can be conservative, but the core discipline is to account for all comparisons searched, not just the winner.^{[4]Reference 4NIST/SEMATECH e-Handbook: Bonferroni's Methodhttps://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm}

Repeatedly inspecting the same benchmark and halting execution the moment $p < 0.05$ is another form of hidden search. Fix the sample size and analysis protocol in advance, or use sequential testing methods designed for continuous monitoring. If an evaluation suite guided prompt or hyperparameter selection, validate the chosen checkpoint on an untouched holdout benchmark. A low p-value isn't a replacement for clean data splits.^{[2]Reference 2ASA Statement on Statistical Significance and P-Valueshttps://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf}

Why paired evaluations crush variance

Why do rigorous LLM benchmarks insist on paired evaluations rather than testing Model A on one set of prompts and Model B on another? The mathematical explanation rests on prompt difficulty covariance.

Let $X_i \in \{0, 1\}$ represent Model A's pass/fail outcome on task $i$, and let $Y_i \in \{0, 1\}$ represent Model B's outcome.

If you test Model A on $N$ independent prompts and Model B on a separate, unpaired batch of $N$ prompts, the variance of the estimated difference between the two sample means is the sum of their individual variances:

$$\text{Var}(\bar{Y} - \bar{X}) = \frac{\sigma_X^2 + \sigma_Y^2}{N}$$

In code generation, prompt difficulty varies across an enormous range. Writing an inclusive timestamp filter is straightforward for modern models (roughly a 95% solve rate), while resolving concurrency race conditions or subtle cache invalidations is brutally difficult (often below a 10% solve rate).

When both models attempt the exact same prompt, they share that prompt's difficulty. If task $i$ is trivial, both models pass; if task $i$ is exceptionally hard, both models fail. Because prompt difficulty affects both models simultaneously, $X_i$ and $Y_i$ exhibit strong positive correlation ($\text{Cov}(X, Y) > 0$, typically with correlation $\rho \in [0.70, 0.90]$).

In a paired design, the estimator is the sample mean of the row-by-row differences $D_i = Y_i - X_i$:

$$\text{Var}(\bar{D}) = \frac{\text{Var}(Y_i - X_i)}{N} = \frac{\sigma_X^2 + \sigma_Y^2 - 2\text{Cov}(X, Y)}{N}$$

The covariance term $-2\text{Cov}(X, Y)$ directly subtracts out the between-prompt variance.

Consider concrete numbers for a benchmark of $N = 40$ tasks. Suppose each model has individual pass variance $\sigma_X^2 = \sigma_Y^2 = 0.25$ (corresponding to a 50% baseline pass rate), and the prompt difficulty correlation is $\rho = 0.80$, giving $\text{Cov}(X, Y) = \rho \sigma_X \sigma_Y = 0.80 \times 0.5 \times 0.5 = 0.20$.

In an unpaired test, the variance is:

$$\text{Var}(\bar{Y} - \bar{X}) = \frac{0.25 + 0.25}{40} = \frac{0.50}{40} = 0.0125$$

The standard error is $\sqrt{0.0125} \approx 0.1118$ (11.2 percentage points). An approximate 95% margin of error ($\pm 1.96 \times \text{SE}$) spans $\pm 21.9$ percentage points. A genuine +7.5 point lift gets lost inside a wide $[-14.4, +29.4]$ interval.

In a paired test, the covariance term activates:

$$\text{Var}(\bar{D}) = \frac{0.25 + 0.25 - 2(0.20)}{40} = \frac{0.50 - 0.40}{40} = \frac{0.10}{40} = 0.0025$$

The variance drops by a factor of 5. The standard error shrinks to $\sqrt{0.0025} = 0.050$ (5.0 percentage points), narrowing the 95% error margin to $\pm 9.8$ percentage points ($[-2.3, +17.3]$).

Pairing filters out prompt-to-prompt noise. It gives 40 paired tasks the statistical resolving power of 200 unpaired tasks.

An interval for B minus A

A confidence interval should target the quantity behind the decision. If the question is "How much better is Model B than Model A on this task suite?", build an interval around the paired lift $B - A$.

Comparing two separate pass-rate intervals can hide the pairing. The evaluation report needs a single interval for the row-by-row difference.

The 95% label describes the intended long-run coverage of the interval procedure over repeated experiments. It doesn't mean this specific realized interval has a 95% probability of containing the true parameter. Classical textbook formulas (such as the Wald interval $\hat{p} \pm 1.96\sqrt{\hat{p}(1-\hat{p})/N}$) fail on small or discrete benchmarks: they assume symmetric normal sampling distributions, can generate impossible bounds outside $[0, 1]$, and deliver poor coverage when success rates approach the boundaries.^{[5]Reference 5NIST/SEMATECH e-Handbook: What Are Confidence Intervals?https://www.itl.nist.gov/div898/handbook/prc/section1/prc14.htm}

In the estimation chapter, bootstrap resampling placed an interval around a single metric by drawing observations with replacement.^{[6]Reference 6Bootstrap Methods: Another Look at the Jackknife.https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods--Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full} Here the unit of resampling is the paired task row: keep Model A's outcome beside Model B's outcome, then recompute the mean difference. The percentile bootstrap interval takes the 2.5th and 97.5th percentiles of those resampled lifts as its bounds. This task bootstrap reflects sampling variation across a broader population of comparable tasks.^{[7]Reference 7scipy.stats.bootstraphttps://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html}

Prediction check: Six tasks help develop intuition but are too few for a stable interval. Consider a slightly larger fixture with 40 paired tasks. Before running the resampler, predict whether a +7.5-point lift from only 13 disagreements will remain strictly positive:

The observed lift is $(8 - 5) / 40 = 0.075$, or $+7.5$ percentage points. Bootstrap the paired differences to inspect how unstable that lift remains. The next cell resamples complete paired differences and reports the percentile interval.

Resampling the differences array is equivalent to resampling complete rows for this binary example because each row contributes one difference. With richer multi-metric evals, keep the full row together so the analysis preserves the pairing. The task row, not an isolated model score, remains the fundamental unit of resampling.

The printed 24% is a fraction of bootstrap resamples, not a p-value and not a Bayesian posterior probability that Model B is worse. These resamples center around the observed sample lift; they weren't generated under the zero-lift null hypothesis.

Bootstrap intervals are approximate, especially with small or discrete samples. For instance, resampling six observed ties produces only zero lifts and a [0, 0] interval, even though six ties don't prove the models are identical on future tasks. Running more bootstrap iterations can't conjure missing evidence out of thin air. For our 40-task fixture, the actionable conclusion is: "Model B gained 7.5 points in this paired sample, but the approximate interval spans losses as well as gains."

Set a minimum useful lift threshold before examining results. If deploying Model B requires at least two percentage points of improvement to justify higher hosting costs, an interval of [+1, +14] points excludes zero but doesn't establish the required threshold. The next function formalizes those decisions:

So far each task evaluated a single completion from each model. Coding evaluations often permit several candidate attempts. That measures a different capability, and it demands its own evaluation protocol.

pass@k measures attempts

A coding assistant can generate multiple candidate functions, allowing an automated test harness to verify whether at least one candidate passes hidden tests. That is the capability measured by pass@k in functional code-generation benchmarks like HumanEval.^{[8]Reference 8Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374} The metric fixes a budget of $k$ candidate attempts per task. An alternative completion may solve a task that the model's first attempt missed.

HumanEval comprises 164 handwritten Python coding problems with an average of 7.7 unit tests each. In that paper, the authors generated $n = 200$ samples per task ($n \ge k$) and reported $k$ up to $100$. Those parameters reflect their specific experimental design, not a rigid requirement. The estimator below works for any predeclared $n \ge k$ that you run.

Consider this illustrative sample of three platform-helper tasks with three completions each. Follow the retry-state row: only the third candidate passes.

first_attempt_hit records whether Attempt 1 passed. any_of_3_hit records whether any of the three completions passed. Extra attempts discover additional solutions. These realized trials illustrate the underlying concept; a HumanEval-style harness evaluates all sampled candidates to compute an unbiased estimate.

Count each task once in both perspectives:

The ordered table answered a product question about a known first completion. For reporting standard HumanEval metrics, treat the $n$ candidates as an unordered pool and evaluate the combinatorial estimator on the count of correct completions. For the same three tasks with $n = 3$ samples each, the correct counts are $c = [1, 1, 0]$. The unbiased pass@1 score is the mean of $c/n$:

$$\frac{1}{3}\left(\frac{1}{3} + \frac{1}{3} + \frac{0}{3}\right) \approx 0.222$$

When $k = 1$, the combinatorial formula simplifies to $c/n$, regardless of which completion was generated first. When $n = k = 3$, the combinatorial estimator equals the any_of_3_hit column. These aren't conflicting definitions: one is a single realized draw, while the estimator computes the expected value over all possible subsets drawn from the generated pool.

Keep two budgets distinct: $k$ specifies the operational attempt budget being evaluated; $n$ is the number of samples drawn to estimate that metric. You can estimate pass@1 using 200 samples without giving the end product 200 attempts.

When a task has n=10 candidates and c=2 correct, what is HumanEval-style pass@1?

Answer

0.2. For k=1 the combinatorial estimator is c/n. It doesn't depend on which of the ten samples came first in generation order.

Publish $k$, the sample pool size $n$, the decoding parameters, and the test suites used to verify functional correctness. A higher score under a larger candidate budget doesn't imply stronger single-completion performance. Chen and colleagues also observed that higher sampling temperature often benefits larger $k$, because more diverse candidates increase the chance that at least one completion solves the problem.^{[8]Reference 8Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Derive the HumanEval estimator by hand

Suppose the assistant generates $n = 10$ candidate implementations for a function, and hidden tests accept $c = 2$ of them. You want to compute the expected pass@5 score if you pick five candidates from that pool. Before calculating, predict which event is easier to count: at least one candidate passing, or all five candidates failing?

Counting successful subsets directly is tedious. Count the failure case instead. The event "at least one candidate passes" is the complement of "all five candidates fail":

1. There are $10 - 2 = 8$ failing candidates.
2. There are $\binom{10}{5} = 252$ total ways to pick five candidates from the pool.
3. There are $\binom{8}{5} = 56$ ways to pick five candidates that all fail.
4. The probability that at least one candidate passes is $1 - 56 / 252 = 0.778$.

In general notation:

$$\text{pass@k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$$

The target parameter is generative: the probability that at least one of $k$ independent model completions succeeds, written $1 - (1-p)^k$ for an unknown per-token solve probability $p$. The combinatorial formula is an unbiased minimum-variance estimator (a U-statistic) of that generative quantity. It calculates the exact fraction of $k$-sized subsets of the $n$ completions that contain at least one passing solution. The HumanEval protocol samples $n \ge k$ and reports this combinatorial estimator rather than a naive plug-in.^{[8]Reference 8Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Here i.i.d. signifies independent draws from the same decoding distribution for this task. Fix $n$ in advance and don't discard duplicate completions: identical outputs still carry probability mass. Halting early after the first passing completion, deduplicating outputs, or feeding test errors back into subsequent attempts alters the experimental distribution. A multi-turn agent that inspects error traces requires an evaluation of that complete iterative harness, not this single-turn formula.

Hidden tests serve as an evaluation oracle: pass@k credits a candidate pool whenever it contains a passing solution. It doesn't assess whether an unguided production system, without access to hidden unit tests, can pick out that passing candidate. Evaluate your reranker or selection model separately if the product must return a single answer.

The next snippet verifies the hand calculation with comb:

How the naive plug-in underestimates pass@k

A tempting shortcut calculates the empirical success rate and substitutes it into the generative formula:

$$\text{pass@k}_{\text{naive}} = 1 - (1 - \hat{p})^k$$

where $\hat{p} = c / n$ is the observed single-sample pass rate. Chen and colleagues demonstrated that $1 - (1 - \hat{p})^k$ is a biased underestimate of the generative pass@k probability $1 - (1 - p)^k$.^{[8]Reference 8Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Two distinct mathematical principles explain this shortfall:

1. Jensen's inequality on concave functions: Consider the function $g(p) = 1 - (1 - p)^k$. For $k > 1$ and $p \in (0, 1)$, the second derivative with respect to $p$ is strictly negative: $$g''(p) = -k(k - 1)(1 - p)^{k - 2} < 0$$ Because $g(p)$ is strictly concave, Jensen's inequality guarantees that the expected value of the plug-in estimator is strictly smaller than the true function evaluated at the mean: $$\mathbb{E}\left[1 - (1 - \hat{p})^k\right] < 1 - \left(1 - \mathbb{E}[\hat{p}]\right)^k = 1 - (1 - p)^k$$ Plugging a noisy empirical proportion into a concave curve inevitably pushes the estimate downward.
2. Sampling with replacement versus without replacement: The plug-in expression $(1 - c/n)^k$ represents drawing $k$ samples with replacement from the finite pool of $n$ completions. Each draw assumes an unchanged failure rate of $(n - c) / n$. But in an actual without-replacement draw, pulling a failing completion removes that failure from the denominator. The proportion of successes among the remaining candidates rises from $c/n$ to $c / (n - 1)$. Drawing without replacement depletes the failure pool, making a success more probable on subsequent picks.

For $0 < c < n$ and $k > 1$, the naive plug-in is strictly smaller than the unbiased combinatorial estimate on the same $(n, c, k)$.

For $n = 10$, $c = 2$, and $k = 5$, compare both calculations directly:

Two boundary checks confirm the logic:

• When $c = 0$, no selected group can pass.
• When fewer than $k$ failures exist ($n - c < k$), every $k$-sized group contains at least one passing candidate.

For larger $n$, assembling gigantic combinations can trigger floating-point overflow. The HumanEval paper provides an equivalent product formulation that stays numerically well-behaved:^{[8]Reference 8Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

$$\frac{\binom{n-c}{k}}{\binom{n}{k}} = \prod_{i=n-c+1}^{n} \left(1 - \frac{k}{i}\right)$$

It evaluates the failure probability factor by factor, avoiding factorials of large numbers.

Average pass@k across tasks

A benchmark score macro-averages task-level results. A complex permission-verification function counts as one task; it shouldn't be drowned out by hundreds of completions from an easy string helper. The estimator runs per task before the benchmark computes its overall mean.

Suppose four tasks each produce $n = 10$ candidates, with the following counts of correct completions:

Compute each task's estimate first, then average those four estimates. A task with zero passing completions contributes zero across every value of $k$.

Notice that pass@5 climbs to $0.563$ while pass@1 sits at $0.175$. That extra search capacity helps if your production product can verify multiple candidates. It doesn't indicate that single-completion quality improved. The metric grew because the attempt budget expanded, not because the underlying tasks became easier.

Changed protocols create fake wins

To compare models fairly under a fixed sampling policy, keep the task suite, unit tests, candidate budget $k$, and decoding rules matched. Record token cutoffs, execution timeouts, and candidate selection rules. A timeout shouldn't quietly vanish from the denominator.

Record $n$ as well, but don't confuse it with $k$. Two independently sampled pools with different predeclared $n \ge k$ estimate the same underlying pass@k; their precision differs. Matching $n$ is a convenient balanced setup, not a mathematical necessity for a valid fixed-$k$ benchmark. Don't omit challenging tasks simply because they generated fewer completions than requested: finish the protocol or report that the score can't be computed.^{[9]Reference 9HumanEval: Functional Correctness Evaluation and pass@k Estimatorhttps://github.com/openai/human-eval/blob/master/human_eval/evaluation.py}

A deterministic decoder illustrates the risk. If every generated candidate is identical, extra sampling slots can't uncover an alternative correct solution. When temperature is zero, pass@5 collapses completely to pass@1.

Serving engines can introduce nondeterminism from external system factors, so record the exact runtime flags. A degenerate deterministic output distribution remains compatible with independent draws: it simply yields the identical completion on every call. Duplicate completions alone don't prove a protocol violation; they remain in $n$ and $c$. A larger $k$ represents a larger compute expenditure, not an unearned model upgrade.

Another risk is subtler: code that passes basic unit tests can still harbor edge-case bugs or malicious side effects. Unit-test evaluation establishes functional correctness under that specific suite. It doesn't authorize executing untrusted code directly on host infrastructure.

HumanEval's authors executed model completions inside an isolated sandbox for that exact reason.^{[8]Reference 8Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374} Treat functional correctness testing and execution isolation as separate, indispensable layers.

Build an evaluation report

Pull the pieces together for a model-review scorecard. The 40-task paired fixture evaluates single-attempt lift. A separate four-task pool evaluates Model B's multi-candidate search behavior. Both require reported uncertainty and a documented protocol; the second fixture isn't a direct comparison with Model A.

This self-contained script runs the calculations and reports each fixture alongside its protocol metadata:

For $k > 1$, pass@k is a nonlinear function of task-level completion counts, which is subsequently averaged across tasks. Resampling tasks and recalculating macro pass@k models task-sampling uncertainty. Comparing Model A and Model B at the same $k$ requires per-task estimates from both models: resample the paired differences between them, not two disconnected sets of tasks.

With four toy tasks the bootstrap interval is wide, but width alone doesn't guarantee nominal coverage. This demonstrates the calculation rather than providing enough evidence for a launch decision. This task bootstrap holds the generated completions fixed when resampling rows; it doesn't model how an entirely new batch of completions on the same tasks would vary. For grouped benchmark sets, resample independent clusters rather than pretending every prompt is independent.

Before approving a production rollout, verify that your evaluation report contains:

• An interval for the paired lift rather than two disconnected headline numbers
• The macro pass@k point estimate alongside an empirical bootstrap interval
• The planned hypothesis direction chosen before inspecting test results
• The total count of comparisons evaluated and any family-wise error adjustments
• Explicit checkpoint hashes, prompt templates, decoding seeds, and timeout configs
• Sandboxing guarantees and test suite coverage audits
• Latency and cost ceilings for candidate generation in production

Statistical significance and operational value address different questions. Even ironclad statistical proof can't tell you whether a performance gain justifies additional candidate generation, evaluation overhead, execution latency, or security exposure.

Practice: review a benchmark claim

A teammate submits this update: "Model B is better because its pass@5 reached 64%, whereas Model A's pass@1 was 58%."

Write a code review comment addressing three specific issues:

1. Request matched attempt budgets $k$, prompt suites, test suites, and decoding settings, while noting the sample size $n \ge k$.
2. Ask for the paired lift and its confidence interval under that matched protocol.
3. Inquire whether the extra generation budget, verification latency, and compute cost are viable for production serving.

A rigorous rewritten claim should read:

Under the same 200 paired platform-helper tasks and a matched pass@1 protocol, Model B's observed pass rate was 2.5 percentage points higher. The paired 95% interval and serving cost guardrails are reported below; the point estimate alone doesn't justify a production swap. pass@5 is documented separately because it reflects a larger candidate search budget.

Why is an interval for B - A preferable to comparing two separate intervals here?

Answer

The tasks are paired: both models solve the same prompts under the same tests. An interval for B - A preserves that pairing and directly measures the quantity the decision needs, which is the model lift.

Why can pass@5 be much higher than pass@1 without proving better single-candidate behavior?

Answer

pass@5 permits five candidate attempts per task. It answers whether at least one of five samples passes, while pass@1 evaluates one sample. The added search budget can raise the score.