Statistics and Uncertainty

The last chapter counted every order in an imagined detector population and found that a flagged order had about 16 percent fraud risk. A live e-commerce system doesn't hand you that answer. You see a queue of flagged orders, pay reviewers to label a sample, and estimate the risk for the queue you haven't reviewed yet.

That difference is statistics: probability reasons from known rates; statistics reasons from measured samples back toward unknown rates.

Estimate the quantity you actually need

Keep the same event from the probability chapter:

Suppose reviewers confirm fraud in 16 of the 100 sampled flagged orders.

$$\hat{p} = \frac{16}{100} = 0.16$$

The hat on $\hat{p}$ says "estimate." It doesn't say the full queue is exactly 16 percent fraud. It says 16 percent is the point estimate computed from this sample.

This is deliberately not overall accuracy. If only 1 percent of all orders are fraudulent, a useless model that labels every order legitimate is 99 percent accurate. For rare events, report the metric attached to the product decision: here, how much of the costly review queue is worthwhile.

A new sample gives a new estimate

If the true queue risk were 16 percent, random batches of 100 reviewed flags wouldn't all contain exactly 16 fraudulent orders. One might contain 12, another 19, another 17. That movement isn't model drift. It is sampling variation.

Use a seeded simulation only to make the idea visible. The rate is fixed at 0.16; only the sampled orders change.

This is the central problem: one measured rate is an answer about one sample. A production claim needs both the center and how much that center can move.

Bootstrap makes the movement visible

In real work you don't know the true queue risk used by the simulation above. You only have the reviewed labels: sixteen 1 values and eighty-four 0 values.

The bootstrap repeatedly samples 100 rows with replacement from those observed labels. It asks how the estimate moves when your measured sample is treated as the best available stand-in for the population. It is not new evidence; it is a way to inspect sampling variability from evidence you already collected.^[1][2]

The bootstrap range is much wider than one point estimate. A queue that measured 16 percent fraud in a 100-order sample could plausibly have a materially lower or higher precision.

Standard error gives a quick first calculation

For a binary rate, the plug-in standard error estimate is:

$$\widehat{SE} = \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$

Here, $\hat{p}$ is the observed rate and $n$ is the number of reviewed flagged orders. With 16 / 100:

$$\widehat{SE} = \sqrt{\frac{0.16 \times 0.84}{100}} \approx 0.0367$$

A quick interval sometimes taught first is the Wald interval:

$$\hat{p} \pm 1.96 \widehat{SE} = 0.16 \pm 0.0719 \approx [0.088,\ 0.232]$$

The formula teaches why sample size matters: divide by a larger n, and the standard error shrinks. But this quick interval isn't robust enough to make your default.

Use an interval that behaves at the edges

The quick Wald interval breaks badly when the rate is near 0 or 1. If reviewers inspect ten flagged orders and find zero fraud, the standard error formula returns zero and the interval becomes [0, 0]. That would claim certainty from ten observations. It is plainly the wrong engineering conclusion.

A Wilson score confidence interval handles that case better. Its formula is longer, but the code is still small:

$$\text{center} = \frac{\hat{p} + z^2/(2n)}{1 + z^2/n}$$ $$\text{margin} = \frac{z}{1 + z^2/n} \times \sqrt{\frac{\hat{p}(1-\hat{p})}{n} + \frac{z^2}{4n^2}}$$

For a rough 95 percent interval, use $z = 1.96$, then return center - margin and center + margin. This score interval stays within the probability range and doesn't collapse to certainty when a small sample contains zero positives.

For this chapter's binary reports, we will use the Wilson interval. The next interval-and-testing chapter will examine how interval choice and decision rules connect to formal comparison.

Larger representative samples sharpen the claim

Suppose later labeling keeps the same center: 16 percent of reviewed flagged orders are fraud. Increasing only the representative sample size narrows the interval.

Same center, different strength. 16 / 100 is an early read. 1,600 / 10,000 is far tighter evidence about the population, assuming both samples represent the queue you intend to serve.

More labels cannot fix the wrong slice

Precision about the wrong population is still wrong. Imagine your queue contains several slices with different fraud rates:

A review project that samples only routine domestic flags can return a very tight estimate near 12 percent while missing the higher-risk parts of the queue. To estimate overall queue precision, sample each important slice or sample randomly from the actual queue, then account for the slice mix.

Random uncertainty and sampling bias require different fixes:

Build it: emit an evaluation report

A small reporting function makes the contract concrete. It prints numerator, denominator, estimate, interval, and a low-data warning. It doesn't decide whether to launch; that decision requires costs, thresholds, and slice coverage.

An evaluation report should pair that line with how the sample was chosen and which important slices have enough labels. A number without its collection process invites overconfidence.

MLE and MAP name two estimation choices

The sample rate did not come from habit alone. Treat each reviewed flag as a Bernoulli outcome with unknown fraud probability $p$. If k of n reviewed flags are fraud, the likelihood of that observation is proportional to:

$$L(p) = p^k(1-p)^{n-k}$$

Maximum likelihood estimation (MLE) chooses the value of $p$ that makes the observed labels most likely:

$$\hat{p}_{\text{MLE}} = \frac{k}{n}$$

For 16 / 100, MLE is 0.16.

Maximum a posteriori estimation (MAP) makes the prior explicit. With a mild $\text{Beta}(2,2)$ prior, centered at 0.5 but weak relative to a large labeled sample, the posterior mode is:

$$\hat{p}_{\text{MAP}} = \frac{k + 1}{n + 2}$$

The prior moves a five-label estimate more than a thousand-label estimate. This isn't permission to hide weak evidence behind a convenient prior. It is a reminder to state the prior and show how much it changes the result.^[3][4]

Calibration needs its own evidence

The previous chapter introduced calibration: a model score near 0.20 should correspond to fraud roughly 20 percent of the time across similarly scored orders. Now statistics changes what you can claim from a finite bucket.

Suppose 100 flagged orders had predicted risk near 20 percent and reviewers confirm fraud in 16. The observed gap is four percentage points, but the Wilson interval for the observed rate still includes 20 percent. With this sample, you have not established a calibration failure.

Calibration matters for classifiers, and modern neural networks can be miscalibrated, so it must be evaluated rather than assumed.^[5] This example adds the statistical discipline: a bucket gap without enough labels isn't strong evidence either.

Name uncertainty before proposing a fix

An interval around queue precision measures estimation uncertainty: how much a finite reviewed sample can move. It does not identify every reason a detector can fail.

Some literature groups irreducible input or label ambiguity under aleatoric uncertainty, and lack of knowledge about unfamiliar cases under epistemic uncertainty. For this lesson, the practical move matters more than the label: don't use an interval for sampling noise as if it solved biased sampling, unclear labels, or unfamiliar traffic.

Practice: write the report before the conclusion

You review 24 / 120 flagged high-value orders and confirm fraud.

1. Compute the point estimate.
2. Run the wilson_interval function for 24, 120.
3. Explain whether this sample alone proves the high-value slice meets a 25 percent minimum precision requirement.
4. Your routine-domestic slice has 120 / 1000 fraud. Explain why you cannot replace one slice with the other.
5. A predicted-risk bucket averages 25 percent risk and contains 24 / 120 fraud. Is this enough to declare it miscalibrated?

Solution checks:

The habit is intentionally conservative. Report what your labels support; do not convert weak evidence into a confident product claim.

Mastery check

Key concepts

• A sample rate estimates a population rate; it is not the rate itself.
• For rare fraud, queue precision is more informative than overall accuracy for manual-review usefulness.
• Bootstrap resampling visualizes finite-sample movement without creating new evidence.
• Standard error describes movement in an estimator, while a score interval makes the report operational.
• A Wilson score interval behaves sensibly when a binary sample has zero or all successes.
• More representative labels reduce random uncertainty; they don't correct sampling bias.
• MLE and MAP differ because MAP states an explicit prior.
• Calibration gaps also need enough labeled evidence before they justify action.

Evaluation rubric

• Foundational: Names population, sample, event, and point estimate for a flagged review queue.
• Intermediate: Implements bootstrap resampling and a Wilson interval, then interprets a wide interval without overclaiming.
• Applied: Produces a report with numerator, denominator, interval, sampling method, and slice coverage.
• Advanced: Distinguishes sample noise, biased sampling, label ambiguity, unfamiliar traffic, and calibration error before proposing a fix.

Follow-up questions

Common pitfalls

• Symptom: A rare-event detector is celebrated for 99 percent accuracy. Cause: accuracy ignored the review decision and class imbalance. Fix: report review-queue precision and fraud recall for the action being taken.
• Symptom: 0 / 10 is reported as zero risk. Cause: the quick normal interval collapsed at an edge. Fix: use a binary-rate interval that preserves uncertainty, such as the Wilson score interval.
• Symptom: A narrow interval fails in production. Cause: only a routine traffic slice was labeled. Fix: sample from deployment traffic and report slice coverage.
• Symptom: A four-point calibration gap triggers a model rollback from 100 labels. Cause: point estimates were compared without uncertainty. Fix: interval-check the observed bucket rate and gather more labels if the decision remains unclear.
• Symptom: MAP is reported without naming a prior. Cause: shrinkage was hidden in a formula. Fix: state the prior distribution and show both MLE and MAP estimates.