LearnMath & StatisticsHypothesis Tests, Intervals, and pass@k

📊EasyEvaluation & Benchmarks

Hypothesis Tests, Intervals, and pass@k

Compare an order-operations coding assistant with paired evidence, uncertainty for lift, and pass@k under a fixed sampling budget.

12 min read

Learning path

Step 14 of 155 in the full curriculum

Distributions and Sampling Neural Networks from Scratch

In the previous lesson, you simulated resolution outcomes and saw why random samples wobble. Now imagine an e-commerce engineering team evaluating a coding assistant. It must write small functions for refunds, return labels, delivery promises, and carrier scans. Hidden tests mark each generated function as pass or fail.

Model B passes one more task than Model A on a six-task evaluation. Is that enough evidence to replace the old model? This chapter teaches how to answer without confusing a promising result with a proven improvement.

Evaluation flow for an order-operations coding assistant: paired tasks, an interval for Model B minus Model A, and pass at k under a fixed sampling budget. — A trustworthy comparison holds tasks and decoding rules fixed, measures the paired lift, and reports each multi-attempt score with its sampling budget.

The numbers below are deliberately small so you can calculate them by hand. They teach the method, not a production launch threshold.

Start with paired pass or fail outcomes

Each prompt asks for one Python helper used in an order workflow. Both models receive the same prompt and are checked by the same hidden tests.

Task	Hidden-test requirement	Model A	Model B
1	refund deadline is inclusive	pass	pass
2	damaged-item return label routing	fail	pass
3	late-delivery credit cap	pass	pass
4	carrier scan deduplication	fail	fail
5	partial-refund rounding	pass	pass
6	backorder cancellation window	fail	fail

Convert pass to 1 and fail to 0. Then Model A has a pass rate of 3 / 6 = 0.500, Model B has 4 / 6 = 0.667, and the observed lift is 1 / 6 = 0.167, or 16.7 percentage points.

Run the calculation rather than trusting a headline.

paired-pass-rates.py

model_a = [1, 0, 1, 0, 1, 0]
model_b = [1, 1, 1, 0, 1, 0]

rate_a = sum(model_a) / len(model_a)
rate_b = sum(model_b) / len(model_b)
lift = rate_b - rate_a

print(f"Model A pass@1: {rate_a:.3f}")
print(f"Model B pass@1: {rate_b:.3f}")
print(f"observed lift:   {lift:+.3f} ({lift * 100:+.1f} percentage points)")

Output

Model A pass@1: 0.500
Model B pass@1: 0.667
observed lift:   +0.167 (+16.7 percentage points)

A paired evaluation preserves more information than two totals. Subtract A from B for each task:

Result on one task	Difference `B - A`	Count
both pass	`0`	3
both fail	`0`	2
B passes, A fails	`+1`	1
A passes, B fails	`-1`	0

Only disagreement tasks tell you which model won. Five ties make the benchmark look larger without giving any directional evidence.

count-disagreements.py

model_a = [1, 0, 1, 0, 1, 0]
model_b = [1, 1, 1, 0, 1, 0]

differences = [b - a for a, b in zip(model_a, model_b)]
b_wins = differences.count(1)
a_wins = differences.count(-1)
ties = differences.count(0)

print("paired differences:", differences)
print(f"B wins={b_wins}, A wins={a_wins}, ties={ties}")
print("directional evidence comes from disagreements:", b_wins + a_wins)

Output

paired differences: [0, 1, 0, 0, 0, 0]
B wins=1, A wins=0, ties=5
directional evidence comes from disagreements: 1

A hypothesis test asks how surprising the win is

The null hypothesis says that, on disagreement tasks, Model A and Model B are equally likely to win. The directional alternative says Model B wins more often.

Under that null hypothesis, each disagreement is like a fair coin: B wins or A wins. Our six-task benchmark has one disagreement, and it went to B. A one-sided p-value for the planned claim "B is better" asks:

If both models were equally likely to win a disagreement, how often would B win at least this many of the disagreements?

With one disagreement, B wins it with probability 0.5. That result isn't rare. The sample moved upward, but the evidence is weak.

Paired hypothesis-test view with five ties, one win for Model B, zero wins for Model A, and a directional p-value of 0.50. — For paired binary outcomes, ties don't separate the models. One win out of one disagreement leaves a directional p-value of 0.50, far too weak for a model-replacement claim.

This exact coin calculation is easy to implement with the binomial coefficients you already know.

exact-directional-p-value.py

from math import comb

def b_wins_one_sided_p_value(b_wins: int, a_wins: int) -> float:
    if b_wins < 0 or a_wins < 0:
        raise ValueError("win counts must be nonnegative")
    disagreements = b_wins + a_wins
    if disagreements == 0:
        return 1.0
    tail_count = sum(comb(disagreements, wins) for wins in range(b_wins, disagreements + 1))
    return tail_count / (2 ** disagreements)

print(f"one B win, zero A wins: p={b_wins_one_sided_p_value(1, 0):.3f}")
print(f"thirteen B wins, three A wins: p={b_wins_one_sided_p_value(13, 3):.4f}")

Output

one B win, zero A wins: p=0.500
thirteen B wins, three A wins: p=0.0106

The second line shows why more disagreement evidence matters. A result with 13 B wins and 3 A wins under a predeclared directional question is much harder to explain with an equal-win coin.

If you had planned to detect a difference in either direction, you would use a two-sided version instead. Pick the question before seeing the winning direction.

Put an interval around the quantity you care about

A confidence interval should target the decision. If your question is "How much better is B than A on the same tasks?", calculate an interval for the paired lift B - A. Comparing two separate pass-rate intervals can hide the pairing and isn't the right decision rule.

A paired bootstrap interval repeatedly resamples complete task rows with replacement. Each resample keeps Model A's outcome beside Model B's outcome, then recomputes the mean difference. The bootstrap is a general resampling method for estimating sampling uncertainty from observed data.^[1]

Six tasks are useful for intuition but painfully sparse. Use a slightly larger illustrative evaluation with 40 paired tasks:

Paired outcome	Tasks
both pass	17
both fail	10
B passes, A fails	8
A passes, B fails	5

The observed lift is (8 - 5) / 40 = 0.075, or +7.5 percentage points. Bootstrap the paired differences to see how unstable that lift remains.

paired-bootstrap-interval.py

import numpy as np

differences = np.array([0] * 27 + [1] * 8 + [-1] * 5)
rng = np.random.default_rng(7)

resamples = rng.choice(differences, size=(20_000, differences.size), replace=True)
bootstrap_lifts = resamples.mean(axis=1)
low, high = np.quantile(bootstrap_lifts, [0.025, 0.975])

print(f"observed paired lift: {differences.mean() * 100:+.1f} percentage points")
print(f"approximate 95% bootstrap interval: {low * 100:+.1f} to {high * 100:+.1f} points")
print("interval includes zero:", low <= 0 <= high)

Output

observed paired lift: +7.5 percentage points
approximate 95% bootstrap interval: -10.0 to +25.0 points
interval includes zero: True

Paired lift interval for Model B minus Model A, centered at plus 7.5 percentage points and crossing zero. — Measure the lift directly. An interval that crosses zero says this paired evaluation still supports both a small loss and a meaningful gain, so deployment language must stay cautious.

Bootstrap intervals are approximate, especially with tiny or highly discrete samples. They help you see uncertainty; they don't turn weak evidence into certainty. Here the correct statement is: "B gained 7.5 points in this paired sample, and the interval still includes zero."

Write that conclusion as a rule your evaluation report can enforce.

language-from-lift-interval.py

def comparison_claim(observed_lift: float, interval: tuple[float, float]) -> str:
    low, high = interval
    if low > high:
        raise ValueError("interval low must not exceed high")
    if low > 0:
        return f"evidence of improvement: estimated lift {observed_lift:+.3f}"
    if high < 0:
        return f"evidence of regression: estimated lift {observed_lift:+.3f}"
    return f"inconclusive: estimated lift {observed_lift:+.3f}, interval crosses zero"

print(comparison_claim(0.075, (-0.100, 0.250)))
print(comparison_claim(0.075, (0.010, 0.140)))

Output

inconclusive: estimated lift +0.075, interval crosses zero
evidence of improvement: estimated lift +0.075

`pass@k` asks a different question

So far, each task used one candidate completion from each model. A coding assistant can also generate several candidate functions and let an evaluator check whether at least one passes hidden tests. That is the question measured by pass@k in functional code-generation evaluations such as HumanEval.^[2]

Consider three order-operations tasks with three sampled completions apiece:

Task	Attempt 1	Attempt 2	Attempt 3	`pass@1`	`pass@3`
return eligibility	pass	fail	fail	1	1
carrier ETA fallback	fail	fail	pass	0	1
refund split rounding	fail	fail	fail	0	0

In this ordered toy table, pass@1 records whether Attempt 1 passed. pass@3 asks whether any of three completions passed. In a larger benchmark pool, pass@1 estimates success for one randomly selected candidate under the fixed sampling protocol. The model isn't being awarded the same budget in the two columns.

pass-at-k-from-attempt-table.py

attempts = [
    [1, 0, 0],
    [0, 0, 1],
    [0, 0, 0],
]

pass_at_1 = sum(row[0] for row in attempts) / len(attempts)
pass_at_3 = sum(any(row[:3]) for row in attempts) / len(attempts)

print(f"pass@1: {pass_at_1:.3f}")
print(f"pass@3: {pass_at_3:.3f}")
print("extra solved tasks from extra attempts:", int((pass_at_3 - pass_at_1) * len(attempts)))

Output

pass@1: 0.333
pass@3: 0.667
extra solved tasks from extra attempts: 1

This is why a report must publish k, the number of generated samples, the decoding policy, and the tests used to judge correctness. A higher score under a larger attempt budget isn't evidence of stronger single-candidate behavior.

Derive the HumanEval estimator by hand

Suppose the assistant generates n = 10 candidate implementations for one function and hidden tests accept c = 2 of them. You want the expected pass@5 result if you select five candidates from that pool.

Counting success cases is tedious. Count the failure case instead:

There are 10 - 2 = 8 failing candidates.
There are C(10, 5) = 252 ways to choose five candidates.
There are C(8, 5) = 56 all-failing choices.
The chance of at least one pass is 1 - 56 / 252 = 0.778.

In notation:

\text{pass@k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}

The formula estimates the chance that a random group of k samples includes at least one correct completion. Chen and colleagues use this unbiased estimator for HumanEval evaluation after generating more samples per task than the reported k.^[2]

pass-at-k-hand-calculation.py

from math import comb

n = 10
c = 2
k = 5
all_groups = comb(n, k)
all_failing_groups = comb(n - c, k)
score = 1 - all_failing_groups / all_groups

print("all groups:", all_groups)
print("all-failing groups:", all_failing_groups)
print(f"pass@5: {score:.3f}")

Output

all groups: 252
all-failing groups: 56
pass@5: 0.778

Two boundary checks should feel right:

When c = 0, no selected group can pass.
When fewer than k failures exist, every k-sized group contains at least one passing candidate.

For larger values of n, avoid assembling enormous combinations. The HumanEval paper gives an equivalent product implementation that stays numerically well behaved.^[2]

stable-pass-at-k.py

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    failure_probability = np.prod(1.0 - k / np.arange(n - c + 1, n + 1))
    return float(1.0 - failure_probability)

print(f"n=10, c=2, k=1: {pass_at_k(10, 2, 1):.3f}")
print(f"n=10, c=2, k=5: {pass_at_k(10, 2, 5):.3f}")
print(f"no passing candidates: {pass_at_k(10, 0, 5):.3f}")
print(f"not enough failures: {pass_at_k(10, 8, 5):.3f}")

try:
    pass_at_k(10, 11, 5)
except ValueError as error:
    print(error)

Output

n=10, c=2, k=1: 0.200
n=10, c=2, k=5: 0.778
no passing candidates: 0.000
not enough failures: 1.000
require n > 0, 0 <= c <= n, and 1 <= k <= n

Average per-task estimates, not raw completions

A benchmark score averages task-level results. One difficult refund function counts as one task; it shouldn't disappear beneath hundreds of samples from an easier address formatter.

Assume four tasks each produce n = 10 candidates, with the following correct-count vector:

text

[0, 1, 2, 4]

Compute each task's estimate first, then average those four estimates.

macro-average-pass-at-k.py

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    return float(1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1)))

correct_counts = [0, 1, 2, 4]
for k in (1, 3, 5):
    task_scores = [pass_at_k(10, correct, k) for correct in correct_counts]
    print(f"pass@{k}: {np.mean(task_scores):.3f}  per-task={[round(score, 3) for score in task_scores]}")

Output

pass@1: 0.175  per-task=[0.0, 0.1, 0.2, 0.4]
pass@3: 0.417  per-task=[0.0, 0.3, 0.533, 0.833]
pass@5: 0.563  per-task=[0.0, 0.5, 0.778, 0.976]

Notice that pass@5 can rise dramatically while pass@1 remains modest. That is useful information if your product can test several generated candidates, but it isn't a substitute for measuring single-candidate behavior under the same protocol.

Failure case: changed protocols create fake wins

pass@k only supports a comparison when the protocol matches. Keep the same task set, hidden tests, number of generated samples n, retained attempt budget k, and decoding rule.

A deterministic decoder illustrates the trap. If every generated candidate is identical for a task, extra attempt slots can't discover a different correct solution. In a controlled deterministic fixture, pass@5 collapses to pass@1.

deterministic-candidates-add-no-search.py

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    return float(1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1)))

identical_failed_candidates = [0] * 10
identical_passing_candidates = [1] * 10

for name, outcomes in [
    ("same failed candidate", identical_failed_candidates),
    ("same passing candidate", identical_passing_candidates),
]:
    correct = sum(outcomes)
    print(name, f"pass@1={pass_at_k(10, correct, 1):.1f}", f"pass@5={pass_at_k(10, correct, 5):.1f}")

Output

same failed candidate pass@1=0.0 pass@5=0.0
same passing candidate pass@1=1.0 pass@5=1.0

Real serving stacks can introduce nondeterminism from outside the sampling policy, so record the actual decoder and run settings. The lesson is operational: multiple attempts only buy search when they produce meaningfully different candidates.

A second failure is subtler: generated code that passes weak hidden tests can still be wrong on missing cases or unsafe to execute. Unit-test passing measures functional correctness under that test suite. It doesn't authorize running untrusted code in a production environment. HumanEval's authors evaluated generated code in a sandbox for that reason.^[2]

Build an evaluation report

Put the pieces together into one report for a model-review meeting. The first comparison measures a paired pass@1 lift. The second metric measures multi-attempt capability under an explicitly recorded stochastic sampling setup.

evaluation-report.py

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    return float(1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1)))

model_a = np.array([1] * 17 + [0] * 10 + [0] * 8 + [1] * 5)
model_b = np.array([1] * 17 + [0] * 10 + [1] * 8 + [0] * 5)
paired_lift = float((model_b - model_a).mean())

correct_counts_b = [0, 1, 2, 4]
pass5 = float(np.mean([pass_at_k(10, correct, 5) for correct in correct_counts_b]))
protocol = {
    "paired_tasks": 40,
    "metric": "hidden-test functional correctness",
    "samples_per_task": 10,
    "reported_k": 5,
    "decoding": "stochastic sampling, fixed settings for both models",
}

print(f"paired pass@1 lift: {paired_lift * 100:+.1f} percentage points")
print(f"Model B pass@5 on candidate pool: {pass5:.3f}")
for key, value in protocol.items():
    print(f"{key}: {value}")

Output

paired pass@1 lift: +7.5 percentage points
Model B pass@5 on candidate pool: 0.563
paired_tasks: 40
metric: hidden-test functional correctness
samples_per_task: 10
reported_k: 5
decoding: stochastic sampling, fixed settings for both models

Before anyone calls a winner, your report still needs:

an interval for the paired lift, not two unrelated headline intervals
the directional or two-sided hypothesis question chosen before inspecting results
task and hidden-test quality checks
latency, token cost, and sandboxing constraints for any deployment decision

Statistical significance and product value answer different questions. Even convincing statistical evidence can't tell you whether a gain justifies additional candidate generation, test execution, latency, or risk.

Practice: review a benchmark claim

A teammate writes: "Model B is better because its pass@5 is 64%, while Model A's pass@1 is 58%."

Write a review comment with three corrections:

Request the same k, n, task set, tests, and decoding policy for both models.
Ask for the paired lift and its uncertainty interval under that matched protocol.
Ask whether added generation and test-execution budget is acceptable for the product.

A good rewritten claim would sound like this:

Under the same 200 paired order-operations tasks and fixed pass@1 protocol, Model B improved hidden-test pass rate by 2.5 percentage points; the paired interval and cost guardrails are reported below. pass@5 is listed separately because it measures a larger candidate-search budget.

Mastery check

Key concepts

null hypothesis and planned alternative
paired task differences
directional p-value intuition
paired bootstrap interval
functional correctness
unbiased pass@k estimator
sampling protocol and practical significance

Evaluation rubric

Foundational: Computes paired pass rates and explains why disagreement tasks carry directional evidence.
Intermediate: Implements a paired bootstrap interval and the numerically stable pass@k estimator.
Advanced: Designs a comparison report that fixes task set, hidden tests, decoding policy, n, and k, then separates statistical evidence from deployment value.

Common pitfalls

Symptom: A one-task win becomes a model-replacement announcement. Cause: ties and sample size were ignored. Fix: count paired disagreements and report inference conservatively.
Symptom: Two model intervals are compared by eye. Cause: paired task structure was discarded. Fix: bootstrap or test the paired lift directly.
Symptom: pass@5 is compared with pass@1. Cause: candidate-search budgets differ. Fix: compare matched protocols and label multi-attempt capability separately.
Symptom: Extra attempts never improve results. Cause: the candidates are duplicates under a deterministic setup. Fix: record and inspect sampling diversity before interpreting higher k.
Symptom: Passing generated code is treated as production-safe code. Cause: unit-test correctness was confused with execution safety. Fix: keep generated code sandboxed and review test coverage.

Follow-up questions

Next Step

Continue to Neural Networks from Scratch

You can now reason about gradients, vectors, and uncertainty, then judge whether evaluation results support a model claim. Next you will turn that math into a trainable neural network, so tensors, loss, and prediction errors start feeling like one connected system.

PreviousDistributions and Sampling

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Bootstrap Methods: Another Look at the Jackknife.

Efron, B. · 1979 · Annals of Statistics

Evaluating Large Language Models Trained on Code (HumanEval).

Chen, M., et al. · 2021 · arXiv preprint

Back to Topics

LearnMath & StatisticsHypothesis Tests, Intervals, and pass@k

📊EasyEvaluation & Benchmarks

Hypothesis Tests, Intervals, and pass@k

Compare an order-operations coding assistant with paired evidence, uncertainty for lift, and pass@k under a fixed sampling budget.

12 min read

Learning path

Step 14 of 155 in the full curriculum

Distributions and Sampling Neural Networks from Scratch

The numbers below are deliberately small so you can calculate them by hand. They teach the method, not a production launch threshold.

Start with paired pass or fail outcomes

Each prompt asks for one Python helper used in an order workflow. Both models receive the same prompt and are checked by the same hidden tests.

Task	Hidden-test requirement	Model A	Model B
1	refund deadline is inclusive	pass	pass
2	damaged-item return label routing	fail	pass
3	late-delivery credit cap	pass	pass
4	carrier scan deduplication	fail	fail
5	partial-refund rounding	pass	pass
6	backorder cancellation window	fail	fail

Convert pass to 1 and fail to 0. Then Model A has a pass rate of 3 / 6 = 0.500, Model B has 4 / 6 = 0.667, and the observed lift is 1 / 6 = 0.167, or 16.7 percentage points.

Run the calculation rather than trusting a headline.

paired-pass-rates.py

model_a = [1, 0, 1, 0, 1, 0]
model_b = [1, 1, 1, 0, 1, 0]

rate_a = sum(model_a) / len(model_a)
rate_b = sum(model_b) / len(model_b)
lift = rate_b - rate_a

print(f"Model A pass@1: {rate_a:.3f}")
print(f"Model B pass@1: {rate_b:.3f}")
print(f"observed lift:   {lift:+.3f} ({lift * 100:+.1f} percentage points)")

Output

Model A pass@1: 0.500
Model B pass@1: 0.667
observed lift:   +0.167 (+16.7 percentage points)

A paired evaluation preserves more information than two totals. Subtract A from B for each task:

Result on one task	Difference `B - A`	Count
both pass	`0`	3
both fail	`0`	2
B passes, A fails	`+1`	1
A passes, B fails	`-1`	0

Only disagreement tasks tell you which model won. Five ties make the benchmark look larger without giving any directional evidence.

count-disagreements.py

model_a = [1, 0, 1, 0, 1, 0]
model_b = [1, 1, 1, 0, 1, 0]

differences = [b - a for a, b in zip(model_a, model_b)]
b_wins = differences.count(1)
a_wins = differences.count(-1)
ties = differences.count(0)

print("paired differences:", differences)
print(f"B wins={b_wins}, A wins={a_wins}, ties={ties}")
print("directional evidence comes from disagreements:", b_wins + a_wins)

Output

paired differences: [0, 1, 0, 0, 0, 0]
B wins=1, A wins=0, ties=5
directional evidence comes from disagreements: 1

A hypothesis test asks how surprising the win is

The null hypothesis says that, on disagreement tasks, Model A and Model B are equally likely to win. The directional alternative says Model B wins more often.

If both models were equally likely to win a disagreement, how often would B win at least this many of the disagreements?

With one disagreement, B wins it with probability 0.5. That result isn't rare. The sample moved upward, but the evidence is weak.

This exact coin calculation is easy to implement with the binomial coefficients you already know.

exact-directional-p-value.py

from math import comb

def b_wins_one_sided_p_value(b_wins: int, a_wins: int) -> float:
    if b_wins < 0 or a_wins < 0:
        raise ValueError("win counts must be nonnegative")
    disagreements = b_wins + a_wins
    if disagreements == 0:
        return 1.0
    tail_count = sum(comb(disagreements, wins) for wins in range(b_wins, disagreements + 1))
    return tail_count / (2 ** disagreements)

print(f"one B win, zero A wins: p={b_wins_one_sided_p_value(1, 0):.3f}")
print(f"thirteen B wins, three A wins: p={b_wins_one_sided_p_value(13, 3):.4f}")

Output

one B win, zero A wins: p=0.500
thirteen B wins, three A wins: p=0.0106

The second line shows why more disagreement evidence matters. A result with 13 B wins and 3 A wins under a predeclared directional question is much harder to explain with an equal-win coin.

If you had planned to detect a difference in either direction, you would use a two-sided version instead. Pick the question before seeing the winning direction.

Put an interval around the quantity you care about

Six tasks are useful for intuition but painfully sparse. Use a slightly larger illustrative evaluation with 40 paired tasks:

Paired outcome	Tasks
both pass	17
both fail	10
B passes, A fails	8
A passes, B fails	5

The observed lift is (8 - 5) / 40 = 0.075, or +7.5 percentage points. Bootstrap the paired differences to see how unstable that lift remains.

paired-bootstrap-interval.py

import numpy as np

differences = np.array([0] * 27 + [1] * 8 + [-1] * 5)
rng = np.random.default_rng(7)

resamples = rng.choice(differences, size=(20_000, differences.size), replace=True)
bootstrap_lifts = resamples.mean(axis=1)
low, high = np.quantile(bootstrap_lifts, [0.025, 0.975])

print(f"observed paired lift: {differences.mean() * 100:+.1f} percentage points")
print(f"approximate 95% bootstrap interval: {low * 100:+.1f} to {high * 100:+.1f} points")
print("interval includes zero:", low <= 0 <= high)

Output

observed paired lift: +7.5 percentage points
approximate 95% bootstrap interval: -10.0 to +25.0 points
interval includes zero: True

Write that conclusion as a rule your evaluation report can enforce.

language-from-lift-interval.py

def comparison_claim(observed_lift: float, interval: tuple[float, float]) -> str:
    low, high = interval
    if low > high:
        raise ValueError("interval low must not exceed high")
    if low > 0:
        return f"evidence of improvement: estimated lift {observed_lift:+.3f}"
    if high < 0:
        return f"evidence of regression: estimated lift {observed_lift:+.3f}"
    return f"inconclusive: estimated lift {observed_lift:+.3f}, interval crosses zero"

print(comparison_claim(0.075, (-0.100, 0.250)))
print(comparison_claim(0.075, (0.010, 0.140)))

Output

inconclusive: estimated lift +0.075, interval crosses zero
evidence of improvement: estimated lift +0.075

`pass@k` asks a different question

Consider three order-operations tasks with three sampled completions apiece:

Task	Attempt 1	Attempt 2	Attempt 3	`pass@1`	`pass@3`
return eligibility	pass	fail	fail	1	1
carrier ETA fallback	fail	fail	pass	0	1
refund split rounding	fail	fail	fail	0	0

pass-at-k-from-attempt-table.py

attempts = [
    [1, 0, 0],
    [0, 0, 1],
    [0, 0, 0],
]

pass_at_1 = sum(row[0] for row in attempts) / len(attempts)
pass_at_3 = sum(any(row[:3]) for row in attempts) / len(attempts)

print(f"pass@1: {pass_at_1:.3f}")
print(f"pass@3: {pass_at_3:.3f}")
print("extra solved tasks from extra attempts:", int((pass_at_3 - pass_at_1) * len(attempts)))

Output

pass@1: 0.333
pass@3: 0.667
extra solved tasks from extra attempts: 1

Derive the HumanEval estimator by hand

Counting success cases is tedious. Count the failure case instead:

There are 10 - 2 = 8 failing candidates.
There are C(10, 5) = 252 ways to choose five candidates.
There are C(8, 5) = 56 all-failing choices.
The chance of at least one pass is 1 - 56 / 252 = 0.778.

In notation:

\text{pass@k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}

pass-at-k-hand-calculation.py

from math import comb

n = 10
c = 2
k = 5
all_groups = comb(n, k)
all_failing_groups = comb(n - c, k)
score = 1 - all_failing_groups / all_groups

print("all groups:", all_groups)
print("all-failing groups:", all_failing_groups)
print(f"pass@5: {score:.3f}")

Output

all groups: 252
all-failing groups: 56
pass@5: 0.778

Two boundary checks should feel right:

When c = 0, no selected group can pass.
When fewer than k failures exist, every k-sized group contains at least one passing candidate.

For larger values of n, avoid assembling enormous combinations. The HumanEval paper gives an equivalent product implementation that stays numerically well behaved.^[2]

stable-pass-at-k.py

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    failure_probability = np.prod(1.0 - k / np.arange(n - c + 1, n + 1))
    return float(1.0 - failure_probability)

print(f"n=10, c=2, k=1: {pass_at_k(10, 2, 1):.3f}")
print(f"n=10, c=2, k=5: {pass_at_k(10, 2, 5):.3f}")
print(f"no passing candidates: {pass_at_k(10, 0, 5):.3f}")
print(f"not enough failures: {pass_at_k(10, 8, 5):.3f}")

try:
    pass_at_k(10, 11, 5)
except ValueError as error:
    print(error)

Output

n=10, c=2, k=1: 0.200
n=10, c=2, k=5: 0.778
no passing candidates: 0.000
not enough failures: 1.000
require n > 0, 0 <= c <= n, and 1 <= k <= n

Average per-task estimates, not raw completions

A benchmark score averages task-level results. One difficult refund function counts as one task; it shouldn't disappear beneath hundreds of samples from an easier address formatter.

Assume four tasks each produce n = 10 candidates, with the following correct-count vector:

text

[0, 1, 2, 4]

Compute each task's estimate first, then average those four estimates.

macro-average-pass-at-k.py

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    return float(1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1)))

correct_counts = [0, 1, 2, 4]
for k in (1, 3, 5):
    task_scores = [pass_at_k(10, correct, k) for correct in correct_counts]
    print(f"pass@{k}: {np.mean(task_scores):.3f}  per-task={[round(score, 3) for score in task_scores]}")

Output

pass@1: 0.175  per-task=[0.0, 0.1, 0.2, 0.4]
pass@3: 0.417  per-task=[0.0, 0.3, 0.533, 0.833]
pass@5: 0.563  per-task=[0.0, 0.5, 0.778, 0.976]

Failure case: changed protocols create fake wins

pass@k only supports a comparison when the protocol matches. Keep the same task set, hidden tests, number of generated samples n, retained attempt budget k, and decoding rule.

deterministic-candidates-add-no-search.py

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    return float(1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1)))

identical_failed_candidates = [0] * 10
identical_passing_candidates = [1] * 10

for name, outcomes in [
    ("same failed candidate", identical_failed_candidates),
    ("same passing candidate", identical_passing_candidates),
]:
    correct = sum(outcomes)
    print(name, f"pass@1={pass_at_k(10, correct, 1):.1f}", f"pass@5={pass_at_k(10, correct, 5):.1f}")

Output

same failed candidate pass@1=0.0 pass@5=0.0
same passing candidate pass@1=1.0 pass@5=1.0

Build an evaluation report

evaluation-report.py

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    return float(1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1)))

model_a = np.array([1] * 17 + [0] * 10 + [0] * 8 + [1] * 5)
model_b = np.array([1] * 17 + [0] * 10 + [1] * 8 + [0] * 5)
paired_lift = float((model_b - model_a).mean())

correct_counts_b = [0, 1, 2, 4]
pass5 = float(np.mean([pass_at_k(10, correct, 5) for correct in correct_counts_b]))
protocol = {
    "paired_tasks": 40,
    "metric": "hidden-test functional correctness",
    "samples_per_task": 10,
    "reported_k": 5,
    "decoding": "stochastic sampling, fixed settings for both models",
}

print(f"paired pass@1 lift: {paired_lift * 100:+.1f} percentage points")
print(f"Model B pass@5 on candidate pool: {pass5:.3f}")
for key, value in protocol.items():
    print(f"{key}: {value}")

Output

paired pass@1 lift: +7.5 percentage points
Model B pass@5 on candidate pool: 0.563
paired_tasks: 40
metric: hidden-test functional correctness
samples_per_task: 10
reported_k: 5
decoding: stochastic sampling, fixed settings for both models

Before anyone calls a winner, your report still needs:

an interval for the paired lift, not two unrelated headline intervals
the directional or two-sided hypothesis question chosen before inspecting results
task and hidden-test quality checks
latency, token cost, and sandboxing constraints for any deployment decision

Practice: review a benchmark claim

A teammate writes: "Model B is better because its pass@5 is 64%, while Model A's pass@1 is 58%."

Write a review comment with three corrections:

Request the same k, n, task set, tests, and decoding policy for both models.
Ask for the paired lift and its uncertainty interval under that matched protocol.
Ask whether added generation and test-execution budget is acceptable for the product.

A good rewritten claim would sound like this:

Under the same 200 paired order-operations tasks and fixed pass@1 protocol, Model B improved hidden-test pass rate by 2.5 percentage points; the paired interval and cost guardrails are reported below. pass@5 is listed separately because it measures a larger candidate-search budget.

Mastery check

Key concepts

null hypothesis and planned alternative
paired task differences
directional p-value intuition
paired bootstrap interval
functional correctness
unbiased pass@k estimator
sampling protocol and practical significance

Evaluation rubric

Foundational: Computes paired pass rates and explains why disagreement tasks carry directional evidence.
Intermediate: Implements a paired bootstrap interval and the numerically stable pass@k estimator.
Advanced: Designs a comparison report that fixes task set, hidden tests, decoding policy, n, and k, then separates statistical evidence from deployment value.

Common pitfalls

Symptom: A one-task win becomes a model-replacement announcement. Cause: ties and sample size were ignored. Fix: count paired disagreements and report inference conservatively.
Symptom: Two model intervals are compared by eye. Cause: paired task structure was discarded. Fix: bootstrap or test the paired lift directly.
Symptom: pass@5 is compared with pass@1. Cause: candidate-search budgets differ. Fix: compare matched protocols and label multi-attempt capability separately.
Symptom: Extra attempts never improve results. Cause: the candidates are duplicates under a deterministic setup. Fix: record and inspect sampling diversity before interpreting higher k.
Symptom: Passing generated code is treated as production-safe code. Cause: unit-test correctness was confused with execution safety. Fix: keep generated code sandboxed and review test coverage.

Follow-up questions

Next Step

Continue to Neural Networks from Scratch

PreviousDistributions and Sampling

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Bootstrap Methods: Another Look at the Jackknife.

Efron, B. · 1979 · Annals of Statistics

Evaluating Large Language Models Trained on Code (HumanEval).

Chen, M., et al. · 2021 · arXiv preprint

Hypothesis Tests, Intervals, and pass@k

Start with paired pass or fail outcomes

A hypothesis test asks how surprising the win is

Put an interval around the quantity you care about

`pass@k` asks a different question

Derive the HumanEval estimator by hand

Average per-task estimates, not raw completions

Failure case: changed protocols create fake wins

Build an evaluation report

Practice: review a benchmark claim

Mastery check

Key concepts

Evaluation rubric

Common pitfalls

Follow-up questions

Mastery Check

Hypothesis Tests, Intervals, and pass@k

Start with paired pass or fail outcomes

A hypothesis test asks how surprising the win is

Put an interval around the quantity you care about

`pass@k` asks a different question

Derive the HumanEval estimator by hand

Average per-task estimates, not raw completions

Failure case: changed protocols create fake wins

Build an evaluation report

Practice: review a benchmark claim

Mastery check

Key concepts

Evaluation rubric

Common pitfalls

Follow-up questions

Mastery Check

Hypothesis Tests, Intervals, and pass@k

Start with paired pass or fail outcomes

A hypothesis test asks how surprising the win is

Model B passes four tasks and Model A passes three, but there is only one disagreement. What is the safe conclusion?

Put an interval around the quantity you care about

pass@k asks a different question

Derive the HumanEval estimator by hand

Average per-task estimates, not raw completions

Failure case: changed protocols create fake wins

Build an evaluation report

Practice: review a benchmark claim

Why is an interval for B - A preferable to comparing two separate intervals here?

Why can pass@5 be much higher than pass@1 without proving better single-candidate behavior?

Mastery check

Key concepts

Evaluation rubric

Common pitfalls

Follow-up questions

What evidence separates Model B from Model A in the six-task paired table?

What does it mean when the paired-lift interval crosses zero?

Why should a benchmark compute pass@k per task before averaging?

What must ship beside a pass@k score?

Why isn't a statistically convincing win enough to deploy a coding assistant?

Mastery Check

Hypothesis Tests, Intervals, and pass@k

Start with paired pass or fail outcomes

A hypothesis test asks how surprising the win is

Model B passes four tasks and Model A passes three, but there is only one disagreement. What is the safe conclusion?

Put an interval around the quantity you care about

pass@k asks a different question

Derive the HumanEval estimator by hand

Average per-task estimates, not raw completions

Failure case: changed protocols create fake wins

Build an evaluation report

Practice: review a benchmark claim

Why is an interval for B - A preferable to comparing two separate intervals here?

Why can pass@5 be much higher than pass@1 without proving better single-candidate behavior?

Mastery check

Key concepts

Evaluation rubric

Common pitfalls

Follow-up questions

What evidence separates Model B from Model A in the six-task paired table?

What does it mean when the paired-lift interval crosses zero?

Why should a benchmark compute pass@k per task before averaging?

What must ship beside a pass@k score?

Why isn't a statistically convincing win enough to deploy a coding assistant?

Mastery Check

`pass@k` asks a different question

`pass@k` asks a different question