LearnMath & StatisticsHypothesis Tests, Intervals, and pass@k

📊EasyEvaluation & Benchmarks

Hypothesis Tests, Intervals, and pass@k

Compare a code-generation model with paired evidence, uncertainty for lift, and pass@k under a fixed sampling budget.

13 min read

Learning path

Step 17 of 177 in the full curriculum

Distributions and Sampling Neural Networks from Scratch

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Sampling rates in Distributions and Sampling moved from sample to sample. Hypothesis tests build on that wobble: a one-task win isn't yet a model replacement. It matters when a platform engineering team evaluates a coding assistant that writes small functions for timestamp parsing, retry limits, permission checks, and cache invalidation. Hidden tests mark each generated function as pass or fail.

Model B passes one more task than Model A on a six-task evaluation. Is that enough evidence to replace the old model? Learn how to answer without confusing a promising result with a proven improvement.

Paired evaluation pipeline: three task rows with Model A and B pass or fail, a bootstrap interval for B minus A from minus 10 to plus 25 points with observed lift plus 7.5 that still crosses zero, and a four-task macro pass at k series rising from 0.175 at one try to 0.563 at five tries with ten candidates per task. — A trustworthy comparison holds tasks and decoding rules fixed, measures the paired lift, and reports each multi-attempt score with its sampling budget.

The numbers below are deliberately small so you can calculate them by hand. They teach the method, not a production launch threshold.

Start with paired pass or fail outcomes

Each prompt asks for one Python helper used in an internal developer workflow. Both models receive the same prompt and are checked by the same hidden tests.

Task	Hidden-test requirement	Model A	Model B
1	timestamp range is inclusive	pass	pass
2	permission fallback preserves denial	fail	pass
3	retry budget caps at configured limit	pass	pass
4	event deduplication keeps latest version	fail	fail
5	quota rounding is deterministic	pass	pass
6	cache invalidation skips expired keys	fail	fail

Convert pass to 1 and fail to 0. Then Model A has a pass rate of 3 / 6 = 0.500, Model B has 4 / 6 = 0.667, and the observed lift is 1 / 6 = 0.167, or 16.7 percentage points.

Run the calculation rather than trusting a headline.

paired-pass-rates.py

model_a = [1, 0, 1, 0, 1, 0]
model_b = [1, 1, 1, 0, 1, 0]

rate_a = sum(model_a) / len(model_a)
rate_b = sum(model_b) / len(model_b)
lift = rate_b - rate_a

print(f"Model A pass@1: {rate_a:.3f}")
print(f"Model B pass@1: {rate_b:.3f}")
print(f"observed lift:   {lift:+.3f} ({lift * 100:+.1f} percentage points)")

Output

Model A pass@1: 0.500
Model B pass@1: 0.667
observed lift:   +0.167 (+16.7 percentage points)

A paired evaluation preserves more information than two totals. Subtract A from B for each task:

Result on one task	Difference `B - A`	Count
both pass	`0`	3
both fail	`0`	2
B passes, A fails	`+1`	1
A passes, B fails	`-1`	0

Only disagreement tasks tell you which model won. Five ties make the benchmark look larger without giving any directional evidence.

count-disagreements.py

model_a = [1, 0, 1, 0, 1, 0]
model_b = [1, 1, 1, 0, 1, 0]

differences = [b - a for a, b in zip(model_a, model_b)]
b_wins = differences.count(1)
a_wins = differences.count(-1)
ties = differences.count(0)

print("paired differences:", differences)
print(f"B wins={b_wins}, A wins={a_wins}, ties={ties}")
print("directional evidence comes from disagreements:", b_wins + a_wins)

Output

paired differences: [0, 1, 0, 0, 0, 0]
B wins=1, A wins=0, ties=5
directional evidence comes from disagreements: 1

A hypothesis test asks how surprising the win is

The null hypothesis says that, on disagreement tasks, Model A and Model B are equally likely to win. The directional alternative says Model B wins more often.

Under that null hypothesis, each disagreement is like a fair coin: B wins or A wins. Our six-task benchmark has one disagreement, and it went to B. A one-sided p-value for the planned claim "B is better" asks:

If both models were equally likely to win a disagreement, how often would B win at least this many of the disagreements?

With one disagreement, B wins it with probability 0.5. That result isn't rare. The sample moved upward, but the evidence is weak.

Six paired tasks yield five ties and one Model B win. A donut chart shows ties dominate; the single disagreement produces a one-sided p-value of 0.50, compared with a stronger 13-to-3 sample at p approximately 0.01. — For paired binary outcomes, ties don't separate the models. One win out of one disagreement leaves a directional p-value of 0.50, far too weak for a model-replacement claim.

This exact coin calculation is easy to implement with the binomial coefficients you already know.

exact-directional-p-value.py

from math import comb

def b_wins_one_sided_p_value(b_wins: int, a_wins: int) -> float:
    if b_wins < 0 or a_wins < 0:
        raise ValueError("win counts must be nonnegative")
    disagreements = b_wins + a_wins
    if disagreements == 0:
        return 1.0
    tail_count = sum(comb(disagreements, wins) for wins in range(b_wins, disagreements + 1))
    return tail_count / (2 ** disagreements)

print(f"one B win, zero A wins: p={b_wins_one_sided_p_value(1, 0):.3f}")
print(f"thirteen B wins, three A wins: p={b_wins_one_sided_p_value(13, 3):.4f}")

Output

one B win, zero A wins: p=0.500
thirteen B wins, three A wins: p=0.0106

The second line shows why more disagreement evidence matters. A result with 13 B wins and 3 A wins under a predeclared directional question is much harder to explain with an equal-win coin.

If you had planned to detect a difference in either direction, you would use a two-sided version instead. Pick the question before seeing the winning direction.

Put an interval around the quantity you care about

A confidence interval should target the decision. If your question is "How much better is B than A on the same tasks?", calculate an interval for the paired lift B - A. Comparing two separate pass-rate intervals can hide the pairing and isn't the right decision rule.

A paired bootstrap interval repeatedly resamples complete task rows with replacement. Each resample keeps Model A's outcome beside Model B's outcome, then recomputes the mean difference. The bootstrap is a general resampling method for estimating sampling uncertainty from observed data.^{[1]Reference 1Bootstrap Methods: Another Look at the Jackknife.https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods--Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full}

Six tasks are useful for intuition but painfully sparse. Use a slightly larger illustrative evaluation with 40 paired tasks:

Paired outcome	Tasks
both pass	17
both fail	10
B passes, A fails	8
A passes, B fails	5

The observed lift is (8 - 5) / 40 = 0.075, or +7.5 percentage points. Bootstrap the paired differences to see how unstable that lift remains.

paired-bootstrap-interval.py

import numpy as np

differences = np.array([0] * 27 + [1] * 8 + [-1] * 5)
rng = np.random.default_rng(7)

resamples = rng.choice(differences, size=(20_000, differences.size), replace=True)
bootstrap_lifts = resamples.mean(axis=1)
low, high = np.quantile(bootstrap_lifts, [0.025, 0.975])

print(f"observed paired lift: {differences.mean() * 100:+.1f} percentage points")
print(f"approximate 95% bootstrap interval: {low * 100:+.1f} to {high * 100:+.1f} points")
print("interval includes zero:", low <= 0 <= high)

Output

observed paired lift: +7.5 percentage points
approximate 95% bootstrap interval: -10.0 to +25.0 points
interval includes zero: True

Paired bootstrap interval for Model B minus Model A spanning minus 10 to plus 25 percentage points, with observed lift plus 7.5 points. Because zero lies inside the interval, the model comparison remains inconclusive. — Measure the paired lift directly. The observed `+7.5` points look promising, but zero lies inside the `−10.0` to `+25.0` interval, so this evaluation can't yet distinguish improvement from no gain or regression.

Bootstrap intervals are approximate, especially with tiny or highly discrete samples. They help you see uncertainty; they don't turn weak evidence into certainty. Here the correct statement is: "B gained 7.5 points in this paired sample, and the interval still includes zero."

Write that conclusion as a rule your evaluation report can enforce.

language-from-lift-interval.py

def comparison_claim(observed_lift: float, interval: tuple[float, float]) -> str:
    low, high = interval
    if low > high:
        raise ValueError("interval low must not exceed high")
    if low > 0:
        return f"evidence of improvement: estimated lift {observed_lift:+.3f}"
    if high < 0:
        return f"evidence of regression: estimated lift {observed_lift:+.3f}"
    return f"inconclusive: estimated lift {observed_lift:+.3f}, interval crosses zero"

print(comparison_claim(0.075, (-0.100, 0.250)))
print(comparison_claim(0.075, (0.010, 0.140)))

Output

inconclusive: estimated lift +0.075, interval crosses zero
evidence of improvement: estimated lift +0.075

pass@k measures attempts

So far, each task used one candidate completion from each model. A coding assistant can also generate several candidate functions and let an evaluator check whether at least one passes hidden tests. That's the question measured by pass@k in functional code-generation evaluations such as HumanEval.^{[2]Reference 2Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Consider this illustrative ordered sample of three platform-helper tasks with three completions apiece. It shows one realized batch, not the HumanEval estimator:

Task	Attempt 1	Attempt 2	Attempt 3	first_attempt_hit	any_of_3_hit
parse timestamp window	pass	fail	fail	1	1
retry-state fallback	fail	fail	pass	0	1
quota split rounding	fail	fail	fail	0	0

Here first_attempt_hit records whether Attempt 1 passed, and any_of_3_hit asks whether any of three completions passed. These columns illustrate how a larger attempt budget can solve more tasks. They are not what a HumanEval-style harness reports.

pass-at-k-from-attempt-table.py

attempts = [
    [1, 0, 0],
    [0, 0, 1],
    [0, 0, 0],
]

first_attempt_hit = sum(row[0] for row in attempts) / len(attempts)
any_of_3_hit = sum(any(row[:3]) for row in attempts) / len(attempts)

print(f"first_attempt_hit: {first_attempt_hit:.3f}")
print(f"any_of_3_hit: {any_of_3_hit:.3f}")
print("extra solved tasks from extra attempts:", int((any_of_3_hit - first_attempt_hit) * len(attempts)))

Output

first_attempt_hit: 0.333
any_of_3_hit: 0.667
extra solved tasks from extra attempts: 1

For reporting HumanEval-style numbers, treat the n candidates as an unordered bag and use the combinatorial estimator on the count of correct outcomes. For the same three tasks with n=3 per task, correct counts are c = [1, 1, 0], so unbiased pass@1 is the average of c/n:

\frac{1}{3}\left(\frac{1}{3} + \frac{1}{3} + \frac{0}{3}\right) \approx 0.222

That is not "column 0 of an ordered table." When k=1, the combinatorial formula always reduces to c/n, regardless of which sample happened to appear first. The model isn't being awarded the same budget at k=1 and k=3.

For pass@k, a report must publish k, the number of generated samples, the decoding policy, and the tests used to judge correctness. A higher score under a larger attempt budget isn't evidence of stronger single-candidate behavior.

Derive the HumanEval estimator by hand

Suppose the assistant generates n = 10 candidate implementations for one function and hidden tests accept c = 2 of them. You want the expected pass@5 result if you select five candidates from that pool.

Counting success cases is tedious. Count the failure case instead:

There are 10 - 2 = 8 failing candidates.
There are C(10, 5) = 252 ways to choose five candidates.
There are C(8, 5) = 56 all-failing choices.
The chance of at least one pass is 1 - 56 / 252 = 0.778.

In notation:

\text{pass@k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}

The quantity we care about is still generative: the probability that at least one of k i.i.d. model samples passes, often written $1 - (1-p)^k$ for unknown per-sample success rate $p$ . The combinatorial formula is an unbiased U-statistic for that generative quantity; combinatorially it equals the fraction of $k$ -subsets of the $n$ draws that contain a success. The paper therefore samples $n \ge k$ and reports the U-statistic, not a with-replacement plug-in. Chen and colleagues use this unbiased estimator for HumanEval evaluation after generating more samples per task than the reported k.^{[2]Reference 2Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

pass-at-k-hand-calculation.py

from math import comb

n = 10
c = 2
k = 5
all_groups = comb(n, k)
all_failing_groups = comb(n - c, k)
score = 1 - all_failing_groups / all_groups

print("all groups:", all_groups)
print("all-failing groups:", all_failing_groups)
print(f"pass@5: {score:.3f}")

Output

all groups: 252
all-failing groups: 56
pass@5: 0.778

Naive estimator bias: with replacement vs. without replacement

A tempting plug-in estimate for pass@k is:

\text{pass@k}_{\text{naive}} = 1 - (1 - \hat{p})^k

where $\hat{p} = \frac{c}{n}$ is the observed single-sample success rate. The plug-in $1-(1-\hat{p})^k$ is a biased estimator of the generative pass@k probability $1-(1-p)^k$ . The combinatorial HumanEval form is unbiased for that same generative quantity; combinatorially it also equals the chance that at least one of $k$ distinct selections from the frozen pool of $n$ candidates succeeds.^{[2]Reference 2Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

After drawing a failing candidate without replacement, correct candidates make up a larger share of the remaining pool. For $0 < c < n$ and $k > 1$ , the plug-in calculation is strictly smaller than the unbiased combinatorial estimate on the same $(n,c,k)$ .

For $n=10$ , $c=2$ , and $k=5$ , compare both calculations:

naive-vs-unbiased-passk.py

p_hat = c / n
naive_score = 1.0 - (1.0 - p_hat) ** k
unbiased_score = score  # From previous cell: 1 - comb(8, 5) / comb(10, 5)

shortfall = unbiased_score - naive_score
print(f"Plug-in pass@5 1-(1-c/n)^k:            {naive_score:.4f}")
print(f"Unbiased combinatorial pass@5:         {unbiased_score:.4f}")
print(f"Plug-in shortfall vs unbiased estimate: {shortfall:.4f}")

assert abs(naive_score - 0.67232) < 1e-5

Output

Plug-in pass@5 1-(1-c/n)^k:            0.6723
Unbiased combinatorial pass@5:         0.7778
Plug-in shortfall vs unbiased estimate: 0.1055

Two boundary checks should feel right:

When c = 0, no selected group can pass.
When fewer than k failures exist, every k-sized group contains at least one passing candidate.

For larger values of n, avoid assembling enormous combinations. The HumanEval paper gives an equivalent product implementation that stays numerically well behaved.^{[2]Reference 2Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

stable-pass-at-k.py

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    failure_probability = np.prod(1.0 - k / np.arange(n - c + 1, n + 1))
    return float(1.0 - failure_probability)

print(f"n=10, c=2, k=1: {pass_at_k(10, 2, 1):.3f}")
print(f"n=10, c=2, k=5: {pass_at_k(10, 2, 5):.3f}")
print(f"no passing candidates: {pass_at_k(10, 0, 5):.3f}")
print(f"not enough failures: {pass_at_k(10, 8, 5):.3f}")

try:
    pass_at_k(10, 11, 5)
except ValueError as error:
    print(error)

Output

n=10, c=2, k=1: 0.200
n=10, c=2, k=5: 0.778
no passing candidates: 0.000
not enough failures: 1.000
require n > 0, 0 <= c <= n, and 1 <= k <= n

Average per-task estimates, not raw completions

A benchmark score averages task-level results. One difficult permission-check function counts as one task; it shouldn't disappear beneath hundreds of samples from an easier string formatter.

Assume four tasks each produce n = 10 candidates, with the following correct-count vector:

text

[0, 1, 2, 4]

Compute each task's estimate first, then average those four estimates.

macro-average-pass-at-k.py

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    return float(1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1)))

correct_counts = [0, 1, 2, 4]
for k in (1, 3, 5):
    task_scores = [pass_at_k(10, correct, k) for correct in correct_counts]
    print(f"pass@{k}: {np.mean(task_scores):.3f}  per-task={[round(score, 3) for score in task_scores]}")

Output

pass@1: 0.175  per-task=[0.0, 0.1, 0.2, 0.4]
pass@3: 0.417  per-task=[0.0, 0.3, 0.533, 0.833]
pass@5: 0.563  per-task=[0.0, 0.5, 0.778, 0.976]

Notice that pass@5 can rise dramatically while pass@1 remains modest. That's useful information if your product can test several generated candidates, but it isn't a substitute for measuring single-candidate behavior under the same protocol.

Failure case: changed protocols create fake wins

pass@k only supports a comparison when the protocol matches. Keep the same task set, hidden tests, number of generated samples n, retained attempt budget k, and decoding rule.

A deterministic decoder illustrates the trap. If every generated candidate is identical for a task, extra attempt slots can't discover a different correct solution. In a controlled deterministic fixture, pass@5 collapses to pass@1.

deterministic-candidates-add-no-search.py

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    return float(1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1)))

identical_failed_candidates = [0] * 10
identical_passing_candidates = [1] * 10

for name, outcomes in [
    ("same failed candidate", identical_failed_candidates),
    ("same passing candidate", identical_passing_candidates),
]:
    correct = sum(outcomes)
    print(name, f"pass@1={pass_at_k(10, correct, 1):.1f}", f"pass@5={pass_at_k(10, correct, 5):.1f}")

Output

same failed candidate pass@1=0.0 pass@5=0.0
same passing candidate pass@1=1.0 pass@5=1.0

Real serving stacks can introduce nondeterminism from outside the sampling policy, so record the actual decoder and run settings. Operationally, multiple attempts only buy search when they produce meaningfully different candidates.

A second failure is subtler: generated code that passes weak hidden tests can still be wrong on missing cases or unsafe to execute. Unit-test passing measures functional correctness under that test suite. It doesn't authorize running untrusted code in a production environment. HumanEval's authors evaluated generated code in a sandbox for that reason.^{[2]Reference 2Evaluating Large Language Models Trained on Code (HumanEval).https://arxiv.org/abs/2107.03374}

Build an evaluation report

Put the pieces together into one report for a model-review meeting. The first comparison measures a paired pass@1 lift. The second metric measures multi-attempt capability under an explicitly recorded stochastic sampling setup.

evaluation-report.py

import numpy as np

def pass_at_k(n: int, c: int, k: int) -> float:
    if n <= 0 or not 0 <= c <= n or not 1 <= k <= n:
        raise ValueError("require n > 0, 0 <= c <= n, and 1 <= k <= n")
    if n - c < k:
        return 1.0
    return float(1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1)))

model_a = np.array([1] * 17 + [0] * 10 + [0] * 8 + [1] * 5)
model_b = np.array([1] * 17 + [0] * 10 + [1] * 8 + [0] * 5)
paired_lift = float((model_b - model_a).mean())

correct_counts_b = [0, 1, 2, 4]
task_scores = [pass_at_k(10, correct, 5) for correct in correct_counts_b]
pass5 = float(np.mean(task_scores))
# Task-level bootstrap of macro pass@k: every with-replacement resample of the
# four task scores (4^4 = 256), then percentile interval of the resample means.
# With only four toy tasks the interval is huge; the point is the reporting contract.
from itertools import product

boot_means = [
    float(np.mean([task_scores[i] for i in idxs]))
    for idxs in product(range(len(task_scores)), repeat=len(task_scores))
]
pass5_lo, pass5_hi = np.quantile(boot_means, [0.025, 0.975])
protocol = {
    "paired_tasks": 40,
    "metric": "hidden-test functional correctness",
    "samples_per_task": 10,
    "reported_k": 5,
    "decoding": "stochastic sampling, fixed settings for both models",
}

print(f"paired pass@1 lift: {paired_lift * 100:+.1f} percentage points")
print(f"Model B pass@5 on candidate pool: {pass5:.3f}")
print(f"pass@5 95% task-bootstrap interval: [{pass5_lo:.3f}, {pass5_hi:.3f}]")
for key, value in protocol.items():
    print(f"{key}: {value}")

Output

paired pass@1 lift: +7.5 percentage points
Model B pass@5 on candidate pool: 0.563
pass@5 95% task-bootstrap interval: [0.194, 0.877]
paired_tasks: 40
metric: hidden-test functional correctness
samples_per_task: 10
reported_k: 5
decoding: stochastic sampling, fixed settings for both models

pass@k is a nonlinear U-statistic averaged over tasks. A bare point estimate reintroduces the overconfidence habit this chapter fights. Resample tasks, recompute macro pass@k on each draw, and report the percentile interval beside the point. With four toy tasks the interval is enormous; that is honest, not a bug.

Before anyone calls a winner, your report still needs:

an interval for the paired lift, not two unrelated headline intervals
the macro pass@k point estimate plus an interval or another uncertainty summary
the directional or two-sided hypothesis question chosen before inspecting results
task and hidden-test quality checks
latency, token cost, and sandboxing constraints for any deployment decision

Statistical significance and product value answer different questions. Even convincing statistical evidence can't tell you whether a gain justifies additional candidate generation, test execution, latency, or risk.

Practice: review a benchmark claim

A teammate writes: "Model B is better because its pass@5 is 64%, while Model A's pass@1 is 58%."

Write a review comment with three corrections:

Request the same k, n, task set, tests, and decoding policy for both models.
Ask for the paired lift and its uncertainty interval under that matched protocol.
Ask whether added generation and test-execution budget is acceptable for the product.

A good rewritten claim would sound like this:

Under the same 200 paired platform-helper tasks and fixed pass@1 protocol, Model B improved hidden-test pass rate by 2.5 percentage points; the paired interval and cost guardrails are reported below. pass@5 is listed separately because it measures a larger candidate-search budget.

Next Step

Continue to Neural Networks from Scratch

You can now reason about <span data-glossary="gradient">gradients</span>, vectors, and uncertainty, then judge whether evaluation results support a model claim. Next you'll turn that math into a trainable neural network, so <span data-glossary="tensor">tensors</span>, loss, and prediction errors start feeling like one connected system.

PreviousDistributions and Sampling

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Bootstrap Methods: Another Look at the Jackknife.

Efron, B. · 1979 · Annals of Statistics

https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods--Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full

Evaluating Large Language Models Trained on Code (HumanEval).

Chen, M., et al. · 2021 · arXiv preprint

https://arxiv.org/abs/2107.03374

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.

Hypothesis Tests, Intervals, and pass@k

Start with paired pass or fail outcomes

A hypothesis test asks how surprising the win is

Model B passes four tasks and Model A passes three, but there is only one disagreement. What is the safe conclusion?

Put an interval around the quantity you care about

pass@k measures attempts

When a task has `n=10` candidates and `c=2` correct, what is HumanEval-style pass@1?

Derive the HumanEval estimator by hand

Naive estimator bias: with replacement vs. without replacement

Average per-task estimates, not raw completions

Failure case: changed protocols create fake wins

Build an evaluation report

Practice: review a benchmark claim

Why is an interval for `B - A` preferable to comparing two separate intervals here?

Why can `pass@5` be much higher than `pass@1` without proving better single-candidate behavior?

Mastery Check

Discussion

Start with paired pass or fail outcomes

A hypothesis test asks how surprising the win is

Model B passes four tasks and Model A passes three, but there is only one disagreement. What is the safe conclusion?

Model B passes four tasks and Model A passes three, but there is only one disagreement. What is the safe conclusion?

Put an interval around the quantity you care about

pass@k measures attempts

When a task has n=10 candidates and c=2 correct, what is HumanEval-style pass@1?

When a task has n=10 candidates and c=2 correct, what is HumanEval-style pass@1?

Derive the HumanEval estimator by hand

Naive estimator bias: with replacement vs. without replacement

Average per-task estimates, not raw completions

Failure case: changed protocols create fake wins

Build an evaluation report

Practice: review a benchmark claim

Why is an interval for B - A preferable to comparing two separate intervals here?

Why is an interval for B - A preferable to comparing two separate intervals here?

Why can pass@5 be much higher than pass@1 without proving better single-candidate behavior?

Why can pass@5 be much higher than pass@1 without proving better single-candidate behavior?

Mastery Check

Discussion

When a task has `n=10` candidates and `c=2` correct, what is HumanEval-style pass@1?

Why is an interval for `B - A` preferable to comparing two separate intervals here?

Why can `pass@5` be much higher than `pass@1` without proving better single-candidate behavior?