Statistics and Uncertainty

An imagined detector population can give you the exact abuse risk for flagged signups. A live API platform doesn't hand you that answer. You see a queue of flagged signups, pay reviewers to label a sample, and estimate the risk for the queue you haven't reviewed yet.

That difference is statistics: probability reasons from known rates; statistics reasons from measured samples back toward unknown rates.

Estimate the quantity you need

Keep the same event from the probability chapter:

Suppose reviewers confirm abuse in 16 of the 100 sampled flagged signups.

$$\hat{p} = \frac{16}{100} = 0.16$$

The hat on $\hat{p}$ says "estimate." It doesn't say the full queue is exactly 16 percent abuse. It says 16 percent is the point estimate computed from this sample.

This is deliberately not overall accuracy. If only 1 percent of all signups are abusive, a useless model that labels every signup clean is 99 percent accurate. For rare events, report the metric attached to the product decision: here, how much of the costly review queue is worthwhile.

A new sample gives a new estimate

If the true queue risk were 16 percent, random batches of 100 reviewed flags wouldn't all contain exactly 16 abusive signups. One might contain 12, another 19, another 17. That movement isn't model drift. It's sampling variation.

Use a seeded simulation only to make the idea visible. The rate is fixed at 0.16; only the sampled signups change.

This is the central problem: one measured rate is an answer about one sample. A production claim needs both the center and how much that center can move.

Bootstrap makes the movement visible

In real work you don't know the true queue risk used by the simulation above. You only have the reviewed labels: sixteen 1 values and eighty-four 0 values.

The bootstrap repeatedly samples 100 rows with replacement from those observed labels. It asks how the estimate moves when your measured sample is treated as the best available stand-in for the population. It's not new evidence; it's a way to inspect sampling variability from evidence you already collected.^{[1]Reference 1Bootstrap Methods: Another Look at the Jackknife.https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods--Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full[2]Reference 2The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/}

The bootstrap range is much wider than one point estimate. A queue that measured 16 percent abuse in a 100-signup sample could plausibly have a materially lower or higher precision.

Standard error gives a quick first calculation

For a binary rate, the plug-in standard error estimate is:

$$\widehat{SE} = \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$

Here, $\hat{p}$ is the observed rate and $n$ is the number of reviewed flagged signups. With 16 / 100:

$$\widehat{SE} = \sqrt{\frac{0.16 \times 0.84}{100}} \approx 0.0367$$

A quick interval sometimes taught first is the Wald interval:

$$\hat{p} \pm 1.96 \widehat{SE} = 0.16 \pm 0.0719 \approx [0.088,\ 0.232]$$

The formula teaches why sample size matters: divide by a larger n, and the standard error shrinks. But this quick interval isn't reliable enough to make your default.

Use an interval that behaves at the edges

The quick Wald interval breaks badly when the rate is near 0 or 1. If reviewers inspect ten flagged signups and find zero abuse, the standard error formula returns zero and the interval becomes [0, 0]. That would claim certainty from ten observations. It's plainly the wrong engineering conclusion.

A Wilson score confidence interval handles that case better. Its formula is longer, but the code is still small:

$$\text{center} = \frac{\hat{p} + z^2/(2n)}{1 + z^2/n}$$ $$\text{margin} = \frac{z}{1 + z^2/n} \times \sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}$$

For a rough 95 percent interval, use $z = 1.96$, then return center - margin and center + margin. This score interval stays within the probability range and doesn't collapse to certainty when a small sample contains zero positives.

A 95% Wilson interval is built so that, under the sampling model, the procedure covers the true rate in about 95% of repeated samples. After you see one realized interval, never read it as "probability 0.95 that the true rate sits inside these two numbers." Coverage is a property of the procedure across replications, not a posterior probability on the fixed endpoints. Operationally: if a decision threshold or calibration target lies inside the interval, this sample can't establish that the target was missed or hit.

For binary reports here, this lesson uses the Wilson interval. The next interval-and-testing chapter examines how interval choice and decision rules connect to formal comparison.

Larger representative samples sharpen the claim

Suppose later labeling keeps the same center: 16 percent of reviewed flagged signups are abuse. Increasing only the representative sample size narrows the interval.

Same center, different strength. 16 / 100 is an early read. 1,600 / 10,000 is far tighter evidence about the population, assuming both samples represent the queue you intend to serve.

More labels can't fix the wrong slice

Precision about the wrong population is still wrong. Imagine your queue contains several slices with different abuse rates:

A review project that samples only routine domestic flags can return a very tight estimate near 12 percent while missing the higher-risk parts of the queue. To estimate overall queue precision, sample each important slice or sample randomly from the actual queue, then account for the slice mix.

Random uncertainty and sampling bias require different fixes:

Build it: emit an evaluation report

A small reporting function makes the contract concrete. It prints numerator, denominator, estimate, interval, and a low-data warning. It doesn't decide whether to launch; that decision requires costs, thresholds, and slice coverage.

An evaluation report should pair that line with how the sample was chosen and which important slices have enough labels. A number without its collection process invites overconfidence.

MLE and MAP name two estimation choices

The sample rate did not come from habit alone. Treat each reviewed flag as a Bernoulli outcome with unknown abuse probability $p$. If k of n reviewed flags are abuse, the likelihood of that observation is proportional to:

$$L(p) = p^k(1-p)^{n-k}$$

Maximum likelihood estimation (MLE) chooses the value of $p$ that makes the observed labels most likely:

$$\hat{p}_{\text{MLE}} = \frac{k}{n}$$

For 16 / 100, MLE is 0.16.

Maximum a posteriori estimation (MAP) makes the prior explicit. With a mild $\text{Beta}(2,2)$ prior, centered at 0.5 but weak relative to a large labeled sample, the posterior mode is:

$$\hat{p}_{\text{MAP}} = \frac{k + 1}{n + 2}$$

The prior moves a five-label estimate more than a thousand-label estimate. This isn't permission to hide weak evidence behind a convenient prior. It's a reminder to state the prior and show how much it changes the result.^{[3]Reference 3Machine Learning: A Probabilistic Perspective.https://probml.github.io/pml-book/book0.html[4]Reference 4Pattern Recognition and Machine Learning.https://www.microsoft.com/en-us/research/publication/pattern-recognition-machine-learning/}

Calibration needs its own evidence

The previous chapter introduced calibration: a model score near 0.20 should correspond to abuse roughly 20 percent of the time across similarly scored signups. Now statistics changes what you can claim from a finite bucket.

Suppose 100 flagged signups had predicted risk near 20 percent and reviewers confirm abuse in 16. The observed gap is four percentage points, but the Wilson interval for the observed rate still includes 20 percent. With this sample, you have not established a calibration failure.

Calibration matters for classifiers, and modern neural networks can be miscalibrated, so it must be evaluated rather than assumed.^{[5]Reference 5On Calibration of Modern Neural Networkshttps://arxiv.org/abs/1706.04599} This example adds the statistical discipline: a bucket gap without enough labels isn't strong evidence either.

Name uncertainty before proposing a fix

An interval around queue precision measures estimation uncertainty: how much a finite reviewed sample can move. It doesn't identify every reason a detector can fail.

Some literature groups irreducible input or label ambiguity under aleatoric uncertainty, and lack of knowledge about unfamiliar cases under epistemic uncertainty. The practical move matters more than the label: don't use an interval for sampling noise as if it solved biased sampling, unclear labels, or unfamiliar traffic.

Practice: write the report before the conclusion

You review 24 / 120 flagged high-value signups and confirm abuse.

1. Compute the point estimate.
2. Run the wilson_interval function for 24, 120.
3. Explain whether this sample alone proves the high-value slice meets a 25 percent minimum precision requirement.
4. Your routine-domestic slice has 120 / 1000 abuse. Explain why you can't replace one slice with the other.
5. A predicted-risk bucket averages 25 percent risk and contains 24 / 120 abuse. Is this enough to declare it miscalibrated?

Solution checks:

The habit is intentionally conservative. Report what your labels support; don't convert weak evidence into a confident product claim.