Conformal Prediction and Uncertainty

A latency model predicts that the next request will finish in 40 milliseconds. That single number doesn't tell an on-call engineer whether a 50-millisecond deadline is safe. A prediction interval such as [32,48] makes the decision visible, but the range earns trust only when its coverage evidence comes from requests the model never saw during fitting.

Statistical Learning and Generalization established why untouched evidence matters. Here, conformal prediction uses a separate calibration split to turn those model errors into finite-sample prediction sets.^{[1]Reference 1A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationhttps://arxiv.org/abs/2107.07511}

Freeze the model before measuring its errors

Before calculating a radius, decide which rows are allowed to influence it. Three slices play different roles:

For a 40-millisecond prediction, an observed 44 milliseconds gives an error of 4; an observed 53 gives an error of 13. A simple regression nonconformity score records that miss as the absolute residual $s_i=|y_i-\widehat y_i|$. Larger scores mean the point prediction missed by more.

The diagram turns those held-out errors into one radius. Its guarantee needs exchangeable calibration and future examples: their joint distribution stays the same when their positions are swapped. Shuffling files doesn't repair a changed deployment distribution.

A model was repeatedly tuned using the same nine examples later labeled as its calibration split. Why can't their residuals support the ordinary split-conformal guarantee?

Answer

Those outcomes influenced the model before calibration, so their residuals are no longer held-out measurements of a frozen predictor. Their apparent errors can be optimistically small, and the ordinary exchangeability argument for fresh request outcomes no longer follows.

Let nine errors choose the radius

Nine held-out requests produce these sorted absolute errors, measured in milliseconds. The list gives us a small decision to make: which position should set the radius for the next request?

$$[2,3,3,4,5,6,7,8,12]$$

For target coverage $1-\alpha=0.8$ and calibration size $n=9$, split conformal selects a corrected residual quantile. Its order statistic, the value at a numbered position in the sorted sample, has rank:^{[1]Reference 1A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationhttps://arxiv.org/abs/2107.07511}

$$k=\left\lceil(n+1)(1-\alpha)\right\rceil=\lceil10\times0.8\rceil=8$$

Rank eight points to a residual of 8 milliseconds. A new prediction of 40 milliseconds therefore gets:

$$[\widehat y-q,\widehat y+q]=[40-8,40+8]=[32,48]$$

The next cell repeats that arithmetic, checks that this calibration size can reach the requested coverage, and prints the interval. Predict its rank and bounds before running it.

Now raise the target to 95% while keeping nine calibration rows. The rank is $\lceil10\times0.95\rceil=10$, but no tenth observed residual exists. A distribution-free finite interval at that level isn't available from this tiny sample; gather more calibration data or explicitly admit an unbounded set.

Why does the example choose the eighth residual instead of simply multiplying nine by 0.8 and rounding down?

Answer

The finite-sample conformal rank is ceil((n+1)(1-alpha)). For nine calibration rows and 80% coverage, that is ceil(10 times 0.8), which selects the eighth order statistic.

Watch coverage appear in new requests

The interval [32,48] covers an actual 44-millisecond outcome and misses 53 milliseconds. That single request tells us what the rule does, not whether the rule is reliable. Coverage is measured across a stream of new comparable examples.

Keep radius 8 fixed and inspect four new request outcomes. Before running the cell, predict how many will land inside the interval.

A four-row observed coverage of 75% doesn't contradict an 80% marginal coverage guarantee. Four requests are a noisy sample, so the observed fraction can move around the target.

The guarantee averages over repeated exchangeable draws and the calibration sample's randomness. It isn't an exact fraction in every short deployment window, and it isn't a guarantee conditional on one fixed realized calibration set.

Find the traffic the average hides

The 75% result is one population summary. Under the exchangeability condition above, standard split conformal provides marginal coverage over the overall population.^{[1]Reference 1A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationhttps://arxiv.org/abs/2107.07511[2]Reference 2Conformal Prediction Beyond Exchangeabilityhttps://arxiv.org/abs/2202.13415}

That guarantee isn't conditional coverage for each subgroup or individual feature value. If short prompts receive 96% coverage while long prompts receive 54%, an aggregate passing threshold can hide unsafe long-request behavior.

An 80% conformal target passes across all traffic, but long requests are covered only 54% of the time. Does the population-wide guarantee refute the long-request measurement?

Answer

No. Standard split conformal guarantees marginal coverage across the exchangeable population, not equal conditional coverage for every prompt-length subgroup. A common well-covered group can keep the aggregate above target while a smaller long-request group remains unsafe.

The failure patterns below turn those distinctions into operational diagnoses:

Turn coverage into an operating check

A coverage number without interval width is incomplete. An interval spanning every plausible latency never misses, but it can't guide a 50-millisecond deadline.

Start the audit with one row per held-out request: frozen prediction, observed latency, absolute residual, sorted residual rank, and approved calibration-slice membership.

For the nine worked residuals, verify corrected rank eight, radius 8 milliseconds, and interval [32,48] for a 40-millisecond prediction. Then check overall coverage, long-request coverage, interval width, training/calibration separation, and the unsupported 95% target that would require a nonexistent tenth residual. Keep the deployment gate closed when exchangeability, subgroup safety, or a finite requested quantile isn't established.

For one final run, build a coverage scorecard from held-out rows. Verify each interval, calculate overall and long-request coverage, record interval width, and note whether exchangeability and the requested finite rank hold. Keep that scorecard with the deployment gate so a later traffic shift has evidence to trigger recalibration.