Bayesian Inference from Evidence

A coding agent passes eight of ten held-out tasks. Is its success rate above 70%? The observed rate is 80%, but ten tasks leave room for luck. We need to distinguish a promising estimate from strong evidence that the agent clears the threshold.

Probability for Machine Learning introduced base rates and conditioning. Bayesian inference applies the same reasoning to an unknown model parameter: state which values were plausible before the evaluation, then update those beliefs using the results. The answer is a distribution of plausible success rates, not just one adjusted score.^{[1]Reference 1Machine Learning: A Probabilistic Perspective.https://probml.github.io/pml-book/book0.html}

We'll use those same eight passes and two failures throughout. Start with a mild preference for middle success rates, then change that preference to see how much the conclusion depends on it.

Formulating the unknown success parameter

Let $p$ mean the probability that this agent passes a randomly drawn task from the intended deployment population. Pin the agent's model, prompt, tools, retry budget, and pass criterion. One observation is one task evaluated under that fixed policy.

Our simple model treats the outcomes as independent conditional on $p$, with the same success probability for each draw. That means: if we knew $p$, knowing one result wouldn't change the probability of another. Tasks can differ in difficulty, but their sampling process must stay the same. Ten handpicked easy tasks won't support a claim about a different production mix.

The direction of conditioning matters. A likelihood asks about the data given a proposed rate. A posterior asks about the rate given the data. Bayes' rule connects them; they aren't interchangeable.

Binomial likelihood scores candidate success rates

At a candidate rate of 0.8, each pass contributes a factor of 0.8 and each failure contributes 0.2. One particular sequence of eight passes and two failures has probability $0.8^8 \times 0.2^2$. There are 45 possible positions for the two failures, so the probability of exactly eight passes, regardless of order, is 45 times that value.

Python's comb(10, 8) counts the 45 unique arrangements of eight passes across ten trials. Evaluating this binomial formula across candidate rates reveals how sharply the observed run separates hypotheses:

from math import comb

for rate in (0.2, 0.5, 0.8):
    likelihood = comb(10, 8) * rate**8 * (1 - rate)**2
    print(f"rate={rate:.1f}: P(exactly 8 passes)={likelihood:.6f}")

rate=0.2: P(exactly 8 passes)=0.000074
rate=0.5: P(exactly 8 passes)=0.043945
rate=0.8: P(exactly 8 passes)=0.301990

The eight-pass result is about 6.9 times as likely at rate 0.8 as at rate 0.5. That favors 0.8 relative to 0.5, but it doesn't mean there's a 30.2% posterior probability that the rate is 0.8. We haven't combined the evidence with a prior.

For a general candidate rate, the likelihood is $45p^8(1-p)^2$. The factor 45 is the same for every candidate, so it cancels when we normalize the posterior. We can therefore work with a likelihood proportional to $p^8(1-p)^2$. The symbol $\propto$ means "equal up to a constant factor."

Beta distributions quantify prior beliefs

A beta distribution describes uncertainty about a probability between 0 and 1. Its two positive shape parameters, $\alpha$ and $\beta$, control where that uncertainty sits. We'll choose Beta(2, 2): it favors middle rates over near-zero or near-one rates, without ruling out a broad range.

This is an explicit modeling choice, not a claim that we observed two historical passes and two failures. A uniform Beta(1, 1) prior would give equal density to all rates. Increasing both parameters equally concentrates the distribution around 0.5. These alternatives make it possible to test how much the prior matters.

For a continuous rate, probability belongs to a range of values, represented by area under a density curve. The height at one point isn't a probability and can exceed 1. Beta(2, 2) has density $6p(1-p)$; the constant 6 makes the total area from 0 to 1 equal to 1.^{[2]Reference 2scipy.stats.betahttps://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.beta.html}

Under these continuous beta priors and posteriors, any single exact rate, including $p=0.8$, has probability zero. A question such as "what is the probability the rate lies between 0.75 and 0.85?" has a nonzero range to integrate over. A discrete prior on a few candidate rates would be a different model.

Bayes' rule multiplies that prior shape by the likelihood, then rescales the result so its total area is 1:

For our run, multiplying $p(1-p)$ by $p^8(1-p)^2$ gives $p^9(1-p)^3$. A beta density has the shape $p^{\alpha-1}(1-p)^{\beta-1}$, so the new exponents identify Beta(10, 4). The prior's exponents and the data's exponents add; the observations don't replace the prior.^{[3]Reference 3Gittins Index for Bayesian Bandits: Beta-Binomial Conjugacyhttps://www.stat.berkeley.edu/~bartlett/courses/2014fall-cs294stat260/lectures/gittins-notes.pdf}

The normalizing denominator is the evidence, $P(D)$: the probability of the observed data averaged over the prior. It differs from the likelihood $P(D\mid p)$ at one candidate rate. Here the full prior-times-likelihood product is $270p^9(1-p)^3$. Its area is $270/2860\approx0.0944$; dividing by that area gives the normalized posterior density $2860p^9(1-p)^3$.

Beta-binomial conjugacy produces an exact posterior

Now the arithmetic has a reason behind it: $2+8=10$ and $2+2=4$. A beta prior combined with a binomial count likelihood stays in the beta family. This makes beta a conjugate prior for that likelihood. For any observed data $D$ containing $s$ passes and $f$ failures:

Here $\alpha$ and $\beta$ are the prior parameters, $s$ counts observed passes, and $f$ counts observed failures. Substitute the actual numbers:

Posterior expectation is given by:

Observed alone, the empirical pass fraction is $8/10=0.800$. Before seeing any runs, the prior mean was $2/4=0.500$. After the update, the posterior mean resolves to an exact weighted average of those two quantities:

Here the prior holds weight 4 while the empirical runs contribute weight 10. These weights explain the gravitational pull toward 0.5; they don't mean anyone actually ran four historical benchmark tasks.

The next cell repeats that arithmetic and checks the parameter totals before printing the updated rate.

prior_success = 2
prior_failure = 2
observed_pass = 8
observed_fail = 2

posterior_success = prior_success + observed_pass
posterior_failure = prior_failure + observed_fail
posterior_mean = posterior_success / (posterior_success + posterior_failure)

assert (posterior_success, posterior_failure) == (10, 4)
print(f"observed rate: {observed_pass / (observed_pass + observed_fail):.3f}")
print(f"posterior: Beta({posterior_success}, {posterior_failure})")
print(f"next-task pass probability: {posterior_mean:.3f}")

observed rate: 0.800
posterior: Beta(10, 4)
next-task pass probability: 0.714

Forecasting the next run with the posterior predictive

If the next task is comparable, the posterior predictive probability gives a forecast before it runs. To forecast the outcome of the eleventh task, average each candidate success rate weighted by its posterior credibility. At rate 0.6, the next task has pass probability 0.6; at rate 0.8, it has probability 0.8. Integrating over the entire Beta(10, 4) distribution yields an expected future pass probability equal to the posterior mean, $10/14\approx0.714$.^{[3]Reference 3Gittins Index for Bayesian Bandits: Beta-Binomial Conjugacyhttps://www.stat.berkeley.edu/~bartlett/courses/2014fall-cs294stat260/lectures/gittins-notes.pdf}

This reflects uncertainty about a future outcome, not a claim that $p$ is exactly 0.714. If the next task passes, the posterior becomes Beta(11, 4), and the following task's forecast rises to $11/15\approx0.733$. Sequential updates and one combined update give the same parameters when they use the same observations and model.

Forecast several tasks without freezing the rate

Before either of two new tasks runs, the probability that both pass is the posterior average of $p^2$, not the square of the posterior average of $p$:

Squaring the one-task forecast gives $(10/14)^2\approx0.510$. The difference is exactly the posterior variance of $p$. The outcomes are independent given $p$, but after averaging over our uncertainty about that shared rate, they're dependent in the predictive distribution. A pass is evidence for higher rates, which raises the forecast for the following task. That dependence doesn't mean the tasks were duplicated.

For ten new comparable tasks, average a binomial count distribution over the beta posterior. The result is a beta-binomial predictive distribution. It describes the number of future passes; the beta posterior describes the unknown rate.^{[4]Reference 4scipy.stats.betabinomhttps://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.betabinom.html} Compare it with a binomial that treats the posterior mean as a known rate:

from scipy.stats import betabinom, binom

a, b, future_tasks = 10, 4, 10
mean_rate = a / (a + b)
print(f"both pass, shared uncertain rate: {a * (a + 1) / ((a + b) * (a + b + 1)):.3f}")
print(f"both pass, rate frozen at mean: {mean_rate**2:.3f}")
for name, distribution in [
    ("posterior predictive", betabinom(future_tasks, a, b)),
    ("rate frozen at mean", binom(future_tasks, mean_rate)),
]:
    print(f"{name}: expected passes={distribution.mean():.3f}, variance={distribution.var():.3f}")
    print(f"P(all ten pass)={distribution.pmf(future_tasks):.3f}")

both pass, shared uncertain rate: 0.524
both pass, rate frozen at mean: 0.510
posterior predictive: expected passes=7.143, variance=3.265
P(all ten pass)=0.081
rate frozen at mean: expected passes=7.143, variance=2.041
P(all ten pass)=0.035

Both count models have the same expected number of passes. The beta-binomial has greater variance because it includes uncertainty about $p$ as well as randomness of individual outcomes. Use its count probabilities when forecasting a batch before observing its results. A credible interval for $p$ is not an interval for how many tasks that batch will pass.

Prior sensitivity: when assumptions outweigh small samples

Two engineers can see the same ten outcomes and reach different posterior means because they chose different priors. Hold the observations fixed and vary only the starting assumption.

The next cell reuses the pass and failure counts and compares a weak Beta(1, 1) prior, a moderate Beta(2, 2) prior, and a strong Beta(20, 20) prior. Predict what should happen before reading the numbers: the strong prior should stay closer to 0.5.

for success_count, failure_count in [(1, 1), (2, 2), (20, 20)]:
    updated_success = success_count + observed_pass
    updated_total = success_count + failure_count + observed_pass + observed_fail
    print(
        f"Beta({success_count:>2}, {failure_count:>2}) -> "
        f"posterior mean {updated_success / updated_total:.3f}"
    )

Beta( 1,  1) -> posterior mean 0.750
Beta( 2,  2) -> posterior mean 0.714
Beta(20, 20) -> posterior mean 0.560

Beta(20, 20) gives its starting mean a weight of 40, compared with 10 for the new data. The updated mean is therefore $0.8\times0.5+0.2\times0.8=0.56$. That strength needs justification, such as substantial evidence that genuinely applies to this agent and task population.

Relative to a uniform Beta(1, 1) baseline, reaching Beta(20, 20) would require 19 historical passes and 19 historical failures, not 40 observed tasks. Parameter weights function as bookkeeping; they aren't provenance records. Keep those two facts separate.

Report several defensible priors when the sample is small. If the deployment decision flips between them, collect more representative evidence instead of quietly choosing the prior that makes the preferred outcome win.

Credible intervals and tail probabilities versus point averages

A mean of 0.714 leaves our original question unanswered: how much evidence says the rate exceeds 0.7? Beta(10, 4) assigns probability both below and above that threshold.

A 90% equal-tailed credible interval leaves 5% of the posterior area below its lower endpoint and 5% above its upper endpoint. The remaining 90% lies between them. These endpoints are the 5th and 95th quantiles, or percentile cutoffs, of the posterior.

SciPy's beta.ppf returns those cutoffs. Its beta.sf(0.7, a, b) returns the area above 0.7, which directly answers our threshold question.^{[2]Reference 2scipy.stats.betahttps://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.beta.html} Use the same Beta(10, 4) posterior for both calculations:

from scipy.stats import beta

a, b = 10, 4
lower, upper = beta.ppf([0.05, 0.95], a, b)
prob_above_threshold = beta.sf(0.7, a, b)

print(f"90% credible interval: [{lower:.3f}, {upper:.3f}]")
print(f"P(rate > 0.7 | data): {prob_above_threshold:.3f}")
assert abs((beta.cdf(upper, a, b) - beta.cdf(lower, a, b)) - 0.9) < 1e-10

90% credible interval: [0.505, 0.887]
P(rate > 0.7 | data): 0.579

The interval is [0.505, 0.887], while the posterior probability above 0.7 is 0.579. Under this prior and model, the remaining 42.1% of posterior probability lies below the release bar. A mean above 0.7 doesn't say how much probability clears that bar.

This interval reflects parameter uncertainty under the chosen prior and binomial model. Tasks themselves don't yield fractional outputs: each run either succeeds or fails. Frequentist procedures interpret intervals differently, evaluating long-run coverage under repeated sampling. Subsequent lessons explore those distinctions in detail.

Another summary is the posterior standard deviation, which measures spread around the mean. Its square, the variance, has a closed form for Beta($a,b$):

At Beta(10, 4), the variance is $40/(196\cdot15)\approx0.0136$, and its square root is about 0.117. Use beta quantiles for the interval above; a shortcut such as mean plus or minus two standard deviations needn't give the intended probability, especially near 0 or 1.

Sample size scaling: comparing 8/10 against 80/100

Hold the Beta(2, 2) prior fixed and compare two alternative sample sizes: eight passes out of ten and eighty passes out of one hundred. Both observed rates are 0.8. The larger run yields Beta(82, 22), not Beta(90, 24): we're replacing the ten-task example with a hundred-task example, not adding the runs together.

The next independent cell computes both means, standard deviations, and equal-tailed intervals. Predict which interval will be narrower before running it:

from math import sqrt
from scipy.stats import beta

for observed_pass, observed_fail in [(8, 2), (80, 20)]:
    a, b = 2 + observed_pass, 2 + observed_fail
    total = a + b
    mean = a / total
    variance = a * b / (total**2 * (total + 1))
    lower, upper = beta.ppf([0.05, 0.95], a, b)
    print(f"Beta({a}, {b}): mean={mean:.3f}, sd={sqrt(variance):.3f}")
    print(f"90% interval: [{lower:.3f}, {upper:.3f}]")
    print(f"P(rate > 0.7 | data): {beta.sf(0.7, a, b):.3f}")

Beta(10, 4): mean=0.714, sd=0.117
90% interval: [0.505, 0.887]
P(rate > 0.7 | data): 0.579
Beta(82, 22): mean=0.788, sd=0.040
90% interval: [0.720, 0.851]
P(rate > 0.7 | data): 0.981

With one hundred tasks, the posterior mean moves to 0.788, and the 90% credible interval contracts to [0.720, 0.851]. Posterior probability above 0.7 rises from 57.9% to 98.1%. The observed fractions are both 80%, but the larger sample leaves less parameter uncertainty under the same model.

For the figure, suppose an example release policy requires at least 95% posterior probability that $p>0.7$. This is a chosen decision rule, not a theorem that defines when an agent is safe to deploy. Compare each tail probability with 0.95.

More data doesn't remove randomness from individual future tasks. Even if we learned that $p$ was exactly 0.8, a new task could still fail. It also doesn't remove uncertainty caused by a wrong model or an unrepresentative sample.

Posterior dispersion versus empirical calibration

Posterior spread and calibration answer different questions. The posterior describes uncertainty about a success rate under a chosen model. Calibration asks whether forecasts line up with later outcomes: among many comparable tasks assigned a pass probability near 0.7, do about 70% pass?

A frozen 0.714 forecast can't establish calibration from the same ten tasks that produced it. Save forecasts before later outcomes arrive. Across enough predictions, group similar probabilities and compare each group's average forecast with its observed pass fraction.^{[5]Reference 5On Calibration of Modern Neural Networkshttps://arxiv.org/abs/1706.04599} With only one frozen forecast, this amounts to comparing that forecast with the pass fraction on a fresh batch.

Small batches fluctuate, so one mismatch isn't proof of a bad model. Persistent discrepancies deserve investigation: did the task mix shift, were retries counted as new tasks, or did the agent's policy change? More data can narrow a wrong model's posterior without making its predictions reliable.

Sampling failure modes that corrupt the posterior

The arithmetic can be correct while the inference is wrong.

Keep the collection and reporting process visible. Omitted runs, selected agents, or an undisclosed change of target population can make the reported analysis answer a different question from the one a reader assumes. Updating can't turn duplicated or biased rows into representative observations.

Reweight a measured task mix explicitly

Suppose the hundred-task example consists of 80 easy tasks with 72 passes and 20 hard tasks with eight passes. The pooled result is still 80/100. Now deployment traffic is only 20% easy and 80% hard. The benchmark's task weights don't describe that population.

For this new model, give each family's unknown rate its own Beta(2, 2) prior and update using that family's observations. The family posteriors are Beta(74, 10) and Beta(10, 14). To forecast one task from the deployment mix, multiply each family's predictive pass probability by its traffic share and add them, using the law of total probability from the previous lesson.

easy_mean = (2 + 72) / (4 + 80)
hard_mean = (2 + 8) / (4 + 20)
pooled_mean = (2 + 80) / (4 + 100)
deployment_mean = 0.2 * easy_mean + 0.8 * hard_mean

print(f"pooled single-rate forecast: {pooled_mean:.3f}")
print(f"easy-family forecast: {easy_mean:.3f}")
print(f"hard-family forecast: {hard_mean:.3f}")
print(f"deployment-mix forecast: {deployment_mean:.3f}")

pooled single-rate forecast: 0.788
easy-family forecast: 0.881
hard-family forecast: 0.417
deployment-mix forecast: 0.510

The two-family model uses two priors, so it isn't another way to compute the original single-rate posterior. It answers a different population question. The adjustment assumes that each family's results remain representative within that family after deployment and that the traffic shares are known. Reweighting doesn't repair hidden bias inside a family.

The deployment rate is now $p_{\text{deploy}}=0.2p_{\text{easy}}+0.8p_{\text{hard}}$. Its mean is about 0.510, but it isn't generally a beta-distributed rate. A release tail probability requires propagating both family posteriors through that weighted sum; don't insert the weighted mean into beta.sf or treat it as a new count of passes.

Production release gates with posterior decision rules

For this exercise, choose the rule before seeing results: increase traffic only if the posterior probability that $p>0.7$ is at least 95%. This is an illustrative evidence threshold, not a universal release policy. Real decisions also depend on the cost of failures, traffic exposure, and other safety checks.

Under Beta(2, 2), the ten-task result doesn't pass this rule: 0.579 < 0.95. The hundred-task result does: 0.981 > 0.95, provided its tasks and policy satisfy the model assumptions. Neither decision follows from the raw 80% rate alone.

Try these changes before reading the answers:

1. One more comparable task fails after the original eight passes and two failures. What are the new posterior and next-task forecast?
2. Apply the 95% evidence rule to the original ten outcomes with Beta(1, 1) and Beta(20, 20) priors. Use beta.sf(0.7, a, b), not just the mean. Does either pass?
3. Could the hundred-task result justify the same claim if every task came from an unusually easy benchmark family?

Decision and sensitivity analysis checks

1. Add the failure once: Beta(10, 5), with forecast $10/15\approx0.667$. Don't add the original ten observations again when starting from their posterior.
2. Posteriors update to Beta(9, 3) and Beta(28, 22). Their probabilities above 0.7 evaluate to 0.687 and 0.019, respectively. Neither meets the 0.95 requirement. Although the estimates differ, all three priors produce the exact same rejection decision under this rule.
3. No. That posterior describes the sampled task family. A narrow interval doesn't correct the mismatch with deployment traffic; sample from the intended population or explicitly model and weight its task families.

Report the observations, prior justification, posterior interval, threshold probability, and decision rule together. If reasonable priors do lead to opposite decisions, make that dependence visible rather than choosing the most convenient prior.