Statistics and Uncertainty

This week the review queue is larger than the team can label. Reviewers can draw 100 flagged signups at random and confirm 16 abuse cases. The dashboard wants to report 16%, while the review lead needs to decide how much that number could move before changing capacity.

Bayesian Inference from Evidence combined ten coding-agent outcomes with an explicit prior. Here we'll ask how a sample-based estimate moves across repeated audits, before returning to the role of a prior.

The earlier probability lesson counted 95 abusive flags among all 590 flags in a fully labeled signup snapshot. That was a population calculation. This week's queue is much larger and mostly unlabeled: 16 / 100 is evidence about an unknown rate, not an exact count of the whole queue.

Start with the target behind 16%

Before doing arithmetic, finish this sentence: "16 of 100 what?" Both numbers refer to flagged signups, so the sample has a precise target:

The population is the full set you want to describe. This week's queue is that population.

The sample is the 100 signups you managed to label. It's evidence about the queue, not a smaller name for the queue itself.

Now name the target quantity: the estimand is the queue's unknown abuse rate, $p = P(\text{abusive} \mid \text{flagged})$.

The estimator is the rule applied to labels, here $\hat{p} = k/n$. This sample produces one realized estimate, $16/100 = 0.16$. A different random sample could produce a different estimate even if the queue's true rate stayed fixed.

$$\hat{p} = \frac{16}{100} = 0.16$$

The hat on $\hat{p}$ says "estimate." It doesn't say the full queue is exactly 16 percent abuse. For a Bernoulli rate, $k/n$ is also the maximum-likelihood estimate: the value of $p$ that makes these observed labels most likely under the model.

This is deliberately not overall accuracy. If only 1 percent of all signups are abusive, a useless model that labels every signup clean is 99 percent accurate. The review decision needs a different denominator: how much of the flagged queue is worth a human review?

Run that smallest possible report before adding any uncertainty machinery:

Why is 16 / 100 a better running metric here than detector accuracy over all signups?

Answer

You are deciding whether flagged signups deserve review, so the useful quantity is the abusive share inside the flagged queue. With rare abuse, overall accuracy can look high even for a detector that never catches abuse.

Let the next hundred challenge the point estimate

The first report gave us a center, not a promise. If the queue's true risk were 16 percent, what should the next random 100 labels contain? Not always 16. One batch might contain 12, another 19, another 17.

The simulation holds the rate fixed at 0.16 and changes only which signups get drawn. Each draw is an independent coin flip with success probability 0.16, called a Bernoulli outcome. The number of successes in 100 such draws has a binomial distribution. Will all eight estimates stay close to 0.16?

The batches don't all return 16 percent. That movement isn't model drift because the simulated rate never changed. It's sampling variation.

One measured rate answers a question about one sample. A production claim needs both the center and the amount that center can move.

The simulation knew the true rate because we wrote it into the code. A real audit doesn't. It has to estimate sampling movement from the labels already in hand.

Replay sampling with the labels in hand

Our real audit has sixteen 1 values and eighty-four 0 values. The true queue risk remains unknown.

The bootstrap repeatedly samples 100 rows with replacement from those observed labels. Each resample gives another value of $k/n$, so its distribution approximates how the estimate might move if we could redraw the audit. It treats the measured sample as the best available stand-in for the population, not as new evidence.^{[1]Reference 1Bootstrap Methods: Another Look at the Jackknife.https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods--Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full[2]Reference 2The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/}

For example, one resample might draw a particular abusive signup three times and omit another entirely. It needn't preserve exactly 16 successes. rng.choice(..., replace=True) performs that draw; each of its 5,000 rows is a resampled audit. Taking the 2.5th and 97.5th percentiles gives a percentile bootstrap interval.^{[3]Reference 3scipy.stats.bootstraphttps://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html}

The central bootstrap range is much wider than a single point. It approximates sampling uncertainty by treating these labels as the population. That approximation can fail: with ten zero labels, every resample also contains ten zeros, so the percentile interval collapses to [0, 0]. Resampling can't invent an unobserved abuse case. We'll need a different interval for that boundary case.

Extra bootstrap repetitions only reduce Monte Carlo jitter in those percentile cut points. They don't add reviewed signups or repair a biased sample.

Row-by-row resampling also assumes the rows act like independent draws. If the audit selects whole groups of linked signups, keep those groups together when estimating uncertainty; ten related rows aren't necessarily ten independent pieces of evidence. Match the resampling unit to the sampling design.^{[4]Reference 4Statistics Canada Quality Guidelines: Weighting and Estimationhttps://www150.statcan.gc.ca/n1/pub/12-539-x/2009001/weighting-ponderation-eng.htm}

Bootstrap gives us an empirical picture. For a binary rate, we can also estimate its typical movement with a short formula.

Put a scale on the movement

Bootstrap gave us a distribution of possible estimates. For this binary rate, we can also get a quick scale for that movement from the 16 successes in 100 labels.

With 16 / 100, the estimated variance of one binary label is 0.16 × 0.84 = 0.1344. Averaging 100 independent labels divides that variance by 100. Take the square root to put it back on the rate's scale:

$$\widehat{SE} = \sqrt{\frac{0.16 \times 0.84}{100}} \approx 0.0367$$

That's about 3.7 percentage points of typical sampling movement, not a 3.7 percent relative error. Under the independent-binomial model, the estimator's variance is $p(1-p)/n$. Substituting the observed rate $\hat{p}$ for the unknown $p$ gives the general standard-error estimate:

$$\widehat{SE} = \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$

Standard error is the estimated standard deviation of the estimator across repeated samples. It isn't the spread of individual labels.

Notice the $\sqrt{n}$ in the denominator: standard error scales as $1/\sqrt{n}$. Cutting your typical sampling margin in half requires $4\times$ as many labels. Cutting it by a factor of 10 requires reviewing $100\times$ as many signups. This square-root penalty governs labeling budgets across machine learning: statistical certainty gets progressively more expensive as you demand narrower margins.

The central limit theorem explains why a sample mean can approach a normal shape as enough independent observations accumulate. It doesn't rescue tiny samples, dependent rows, or rates near 0 and 1.

These examples treat the live queue as much larger than the sample. Sampling a substantial fraction of a fixed queue without replacement needs a finite-population correction, because inspecting more of the queue leaves fewer unknown labels. A census of all its labels has no sampling uncertainty about that queue, though future traffic and label errors remain uncertain.

A familiar first approximation turns that scale into the Wald interval:

$$\hat{p} \pm 1.96 \widehat{SE} = 0.16 \pm 0.0719 \approx [0.088,\ 0.232]$$

Look at the midpoint before running the code. The Wald midpoint is the observed rate, 0.16; the standard error supplies the margin on each side.

The output makes the scale visible: 0.16 plus or minus about 0.072. A larger n shrinks standard error because the same kind of estimator moves less across repeated samples.

Keep this calculation as a useful first pass, not as the default report. Its symmetric endpoints can become impossible or collapse at the edges.

Wald collapses at 0/10; Wilson doesn't

The Wald interval breaks when the rate is near 0 or 1. If reviewers inspect ten flagged signups and find zero abuse, $\hat{p}=0$, the standard error is zero, and the interval becomes [0, 0]. That result looks decisive. Should ten clean labels count as proof that abuse risk is exactly zero? They shouldn't.

A Wilson score confidence interval handles that case better. Instead of treating the observed zero as certainty, it asks which candidate population rates remain compatible with the count. Its two-sided 95% interval for 0 / 10 reaches from 0 to about 0.278.

Wilson's construction inverts a score test for a binomial proportion: the test uses each candidate rate to calculate its standard error, not the observed $\hat{p}$.^{[5]Reference 5NIST/SEMATECH e-Handbook: Confidence Intervals for Proportionshttps://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm} For 16 / 100, the interval's adjusted center is about 0.173 and its margin is about 0.072, giving [0.101, 0.244]. The general calculation is:

$$\text{center} = \frac{\hat{p} + z^2/(2n)}{1 + z^2/n}$$ $$\text{margin} = \frac{z}{1 + z^2/n} \times \sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}$$

For a rough 95 percent interval, use $z = 1.96$. The center isn't $\hat{p}$ because the score construction adds a small adjustment toward 0.5.

With $n=100$ and $\hat{p}=0.16$, $z^2 \approx 3.84$, so the center is about $(0.16 + 3.84/200) / (1 + 3.84/100) \approx 0.173$. For 0 / 10, the same adjustment moves the center off zero and keeps the margin positive. The code below uses $z=1.96$ exactly and returns center - margin and center + margin, clipped to $[0, 1]$.

Predict the three rows before running it: Wald should collapse at both endpoints, while Wilson should leave room above 0 after zero successes and below 1 after ten successes.

A 95% confidence level describes repeated use of the procedure: draw a sample, calculate an interval, and check whether it contains the true rate. Wilson targets 95% coverage under the binomial model and typically improves on Wald, but its coverage isn't exactly 95% for every sample size and rate.^{[6]Reference 6NIST/SEMATECH e-Handbook: What Are Confidence Intervals?https://www.itl.nist.gov/div898/handbook/prc/section1/prc14.htm[5]Reference 5NIST/SEMATECH e-Handbook: Confidence Intervals for Proportionshttps://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm}

That guarantee is about the procedure across replications. After seeing one realized interval, don't read it as "probability 0.95 that the true rate sits inside these two numbers." It isn't a posterior probability on these fixed endpoints.

For the review decision, if a threshold lies inside this two-sided 95% interval, the interval doesn't establish that the rate is above or below it. Inclusion isn't proof of equality. Keep the sample size and confidence level fixed in advance; repeatedly checking until an interval clears a threshold requires a different analysis.

For this binary report, use the Wilson interval. The immediate question is now practical: how much does a larger, still-representative sample narrow it?

Why should a report avoid the quick Wald interval for 0 / 10 reviewed abuse cases?

Answer

It returns [0, 0], which treats ten observed non-abuse cases as proof that abuse risk is exactly zero. A score interval preserves uncertainty and correctly leaves a nonzero upper bound.

More reviews only help if they match the queue

The Wilson interval tells us how much random movement remains. Suppose later labeling keeps the same center: 16 percent of reviewed flagged signups are abuse. If the added labels are still representative, increasing the sample size should narrow the interval.

Call the direction before running the next example: the estimate should stay at 0.16, while the interval should contract as n grows.

16 / 100 is an early read. 1,600 / 10,000 is much tighter evidence about the population, assuming both samples represent the queue you intend to serve. More labels reduced random movement without changing the center.

Here, bias means the estimator's average over repeated samples minus the target rate. For simple random sampling, $E[\hat{p}] = p$, so the sample fraction is unbiased. One particular sample can still miss $p$. Its variance shrinks as $n$ grows, which is why these intervals tighten. This is greater statistical precision, distinct from the classifier metric also called precision.

But n only buys precision when the labels target the same population. A large sample from one slice can be confidently wrong about the queue.

A domestic-only sample is precise about the wrong pile

Suppose the next labeling budget goes to 2,000 routine domestic flags. Those labels describe that slice well, but they don't establish the rate in parts of the queue they never cover.

Selection bias appears when the selection process systematically shifts the estimate away from its target. To make the missing information visible, suppose we know the true rates in this separate, hypothetical queue. The slices must not overlap: classify high-value signups first, then divide the remainder into domestic and international signups.

A review project that samples only routine domestic flags can return a very tight estimate near 12 percent while missing higher-risk parts of the queue. That 12 percent is an estimate for the domestic slice.

Picture 1,000 flags with this mix: 700 domestic flags contribute 84 abuse cases, 200 international flags contribute 50, and 100 high-value flags contribute 42. The total is 176 / 1,000 = 17.6%. Equivalently, weight each rate by its share: 0.70 × 0.12 + 0.20 × 0.25 + 0.10 × 0.42 = 0.176. Domestic-only sampling has bias 12% - 17.6% = -5.6 percentage points for this queue target. More domestic labels don't remove it.

To estimate overall queue precision, sample each important slice or sample randomly from the actual queue, then account for the slice mix.

In a real audit, the slice rates are estimates too. Confirm that the shares describe the target queue and that labels were selected randomly within each slice. If you deliberately review equal numbers from unequal-sized slices, don't pool their raw counts as though the result were a simple random sample of the queue. Weight the rates by population shares, and calculate uncertainty using the sampling design rather than applying the unweighted Wilson formula to the pooled counts.^{[4]Reference 4Statistics Canada Quality Guidelines: Weighting and Estimationhttps://www150.statcan.gc.ca/n1/pub/12-539-x/2009001/weighting-ponderation-eng.htm}

This trade-off has a clean mathematical foundation: the bias-variance decomposition of Mean Squared Error ($\text{MSE}$). For any estimator $\hat{p}$ targeting the unknown queue rate $p$:

$$\text{MSE}(\hat{p}) = \mathbb{E}[(\hat{p} - p)^2] = \text{Bias}(\hat{p})^2 + \text{Var}(\hat{p})$$

where $\text{Bias}(\hat{p}) = \mathbb{E}[\hat{p}] - p$.

When you sample randomly from the actual queue, your estimator is unbiased: $\text{Bias}(\hat{p}) = 0$. Mean squared error equals sampling variance, $\text{Var}(\hat{p}) = p(1-p)/n$. Gathering more labels drives variance down as $1/n$ (and standard error down as $1/\sqrt{n}$), converging toward zero error.

When you sample only routine domestic flags, your estimator converges to $\mathbb{E}[\hat{p}] = 0.120$ instead of the true queue rate $p = 0.176$. That leaves an irreducible bias of $\text{Bias} = 0.120 - 0.176 = -0.056$. Even if you label 10,000 domestic signups so variance collapses near zero:

$$\text{MSE}(\hat{p}) \to (-0.056)^2 + 0 \approx 0.00314$$

The squared bias term creates an error floor that extra labels can't shrink. Spending budget on more domestic reviews only buys high-confidence delusion about the broader queue.

Random uncertainty, sampling design, and label quality need different fixes. Use the symptom to choose the repair:

A domestic-only sample of 2,000 flags yields a tight interval around 12%. The queue mix is 70/20/10 at 12%, 25%, and 42%. Can you treat 12% as queue precision?

Answer

No. The queue-weighted rate is 17.6%. Extra domestic labels shrink variance around the wrong center.

A prior pulls the center for a different reason

With one abuse case in five reviews, MLE gives 1 / 5 = 0.20. With 160 in 1,000, it gives 0.16. It uses only the observed labels. More generally, k positive and n - k negative labels contribute likelihood proportional to $p^k(1-p)^{n-k}$. This expression is maximized at $k/n$, the same sample fraction we've used throughout.

Maximum a posteriori estimation (MAP) picks the posterior's highest-density point, or mode. With the $\text{Beta}(2,2)$ prior from the Bayesian lesson, one abuse case in five reviews gives a $\text{Beta}(3,6)$ posterior. Its mode is (3 - 1) / (3 + 6 - 2) = 2/7, about 0.286. That differs from its mean, 3/9, about 0.333.^{[7]Reference 7Gittins Index for Bayesian Bandits: Beta-Binomial Conjugacyhttps://www.stat.berkeley.edu/~bartlett/courses/2014fall-cs294stat260/lectures/gittins-notes.pdf}

For k successes in n labels, this prior gives posterior $\text{Beta}(k+2,n-k+2)$. Substituting those parameters into the mode formula gives:

$$\hat{p}_{\text{MAP}} = \frac{k + 1}{n + 2}$$

The mode formula acts as if one success and one failure were added to the observed counts. The posterior-mean formula instead uses (k + 2) / (n + 4). Neither rule adds actual reviews. This symmetric prior is an illustration, not a recommendation for a low-abuse queue: a defensible prior should reflect relevant knowledge, and its influence should be reported.

Run the same three counts through both rules. The labels stay fixed; only the source of the center changes.

The output makes the scale of the prior's influence visible: 1 / 5 moves from 0.200 to 0.286, while 160 / 1,000 moves from 0.160 to about 0.161.

Wilson's center also blends $\hat{p}$ toward 0.5, but that's a confidence-interval construction, not a posterior mode. Don't hide weak evidence behind a convenient prior. State the prior and show how much it changes the result.^{[8]Reference 8Machine Learning: A Probabilistic Perspective.https://probml.github.io/pml-book/book0.html[9]Reference 9Pattern Recognition and Machine Learning.https://www.microsoft.com/en-us/research/publication/pattern-recognition-machine-learning/}

So far the target has been a rate inside the flagged queue. A score adds another question: when it says 20 percent, do outcomes in that score bucket really average 20 percent?

A four-point calibration gap can still be noise

The Bayesian lesson asked whether forecasts line up with outcomes on new tasks. That's calibration: among many comparable cases assigned a probability near 20 percent, do about 20 percent have positive outcomes?

Use a separate audit of 100 randomly selected signups all assigned risk 0.20 before reviewers see their outcomes. Suppose 16 are abusive. These are new labels, chosen from one score bucket; the same 16 / 100 arithmetic applies. The observed gap is four percentage points. Does this bucket have enough evidence to rule out 20 percent?

The interval includes 20 percent, so this preselected bucket hasn't established a calibration failure by the two-sided 95% comparison. It hasn't established exact calibration either. A wide interval can hide an operationally important gap.

Real buckets contain nearby, not identical, predictions. Compare their average prediction with their observed rate and preserve the number of labels in each bucket. Use held-out outcomes, choose bins before inspecting them, and don't treat one compatible bucket as evidence that the whole model is calibrated. Searching many buckets for a mismatch also needs a multiple-comparison plan.^{[10]Reference 10NIST/SEMATECH e-Handbook: Bonferroni's Methodhttps://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm}

Deep networks can be miscalibrated even when accuracy looks strong, so calibration has to be measured rather than assumed.^{[11]Reference 11On Calibration of Modern Neural Networkshttps://arxiv.org/abs/1706.04599} Accuracy and calibration answer different questions, and neither replaces representative labels.

We now have the pieces of an honest audit: a rate, an interval, a sampling plan, and a calibration comparison. Put those pieces into a report without letting the dashboard hide context.

A risk bucket predicts 20 percent, while 16 of 100 reviewed cases are positive and the observed Wilson interval is 10.1 to 24.4 percent. Can this sample establish miscalibration?

Answer

No. The interval still includes 20 percent, so the four-point observed gap is compatible with finite-sample variation. Gather more representative labels before declaring a calibration failure.

Print the counts, the interval, and the sampling plan

A report should let someone reconstruct the estimate before they act. This function prints the numerator, denominator, estimate, interval width, and sampling method.

It doesn't decide whether to launch. That decision still needs costs, thresholds, and slice coverage.

There's no universal label count that makes an audit adequate. A 14-percentage-point interval may be enough for rough staffing and too wide for a close policy decision. Show the width and let the decision's tolerance determine whether more labels are needed. A slice name alone also isn't a sampling plan: say how its reviewed cases were selected.

The international line is a separate sample of 100 international flags, not a subset of the mixed 16/100 draw. Neither line should claim to measure next month's traffic without evidence that the population is comparable.

Also record which important slices still lack labels. A number without its collection process invites overconfidence.

Tighter intervals don't fix the wrong labels

An interval around queue precision measures estimation uncertainty: how much a finite reviewed sample can move. It doesn't identify every reason a detector can fail.

Ask what happens when the interval is narrow but reviewers saw model predictions before resolving ambiguous cases. The arithmetic can be precise while the labels no longer represent an independent review process.

Some literature groups irreducible input or label ambiguity under aleatoric uncertainty, and lack of knowledge about unfamiliar cases under epistemic uncertainty. Those names can be useful, but the repair matters more: don't use an interval for sampling noise as if it solved biased sampling, unclear labels, or unfamiliar traffic.

Practice with 24/120 high-value flags

Transfer the audit to one new slice. You review 24 / 120 flagged high-value signups and confirm abuse. Before calculating, predict whether 120 labels can settle a 25 percent policy threshold.

1. Compute the point estimate and run the wilson_interval function for 24, 120.
2. Decide whether this sample alone proves the high-value slice meets a 25 percent minimum precision requirement.
3. Your routine-domestic slice has 120 / 1000 abuse. Explain why you can't replace one slice with the other.
4. A predicted-risk bucket averages 25 percent risk and contains 24 / 120 abuse. Decide whether this is enough to declare it miscalibrated.

Solution checks:

The original 16 / 100 wasn't wrong; it was incomplete. Counts plus an interval describe sampling uncertainty. The sampling method and label process determine which population that description deserves to represent.