Validation and Leakage

In our fictional access gateway, request REQ-10234 arrives on July 1 at 09:00:00. By 10:22, a manager approves it. On July 15, a compliance audit discovers a policy violation and revokes the grant. Finished database rows make those later fields tempting predictors of review need. At 09:00:00, however, the router couldn't read them. An impressive score built from them answers a different question from the one the deployed model faces.

Even with honest fields, fitting still chases the error on the rows it can see. To fit this access-review model, empirical risk minimization chooses a hypothesis $h$ with the lowest average loss on training sample $S$ of $n$ observations:

Because the parameters adapt to the training sample, its error is generally optimistic as an estimate of future error. Under an i.i.d. sample and exact empirical risk minimization over a fixed hypothesis class, this optimism holds in expectation; an individual sample can have a negative generalization gap. The population risk for a specified distribution $\mathcal{D}$ is:

The difference between what our model scores on training data and how it performs in the wild is the generalization gap, $G(h) = R(h) - \hat{R}_S(h)$.^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/}

Offline evaluation estimates performance on the population and decision conditions your split represents. It doesn't promise performance after an arbitrary distribution shift. Data leakage crosses an information boundary that the evaluation claim requires: unavailable future information enters inputs, or held-out data influences fitting or selection. It can inflate a score and hide errors; it needn't produce a perfect score or a zero measured gap.

View larger

Start at the decision moment

You're evaluating a guardrail that predicts review need at request opening. For these constructed labs, the later audit supplies needs_review under the policy in effect at opening. Real audits need checks for label mistakes, selective coverage, and policy-version consistency. Independently enforce mandatory authorization rules; a learned review score isn't permission to bypass them. At prediction time, later approvals, triage logs, and revocations aren't available.

For REQ-10234, the decision timestamp is 2026-07-01 09:00:00.

Write this feature contract before fitting. Each request's features, prompt context, and retrieval snapshot must use information available at its decision moment. Separately, learned artifacts must obey their model-fit cutoff. A scaler fitted on June history can serve July requests even though some of that history occurred after a January training request. It cannot use August data to simulate a July deployment.

Predict the audit before running it: the two 08:59 fields should be allowed, while manager approval and audit revocation must be blocked.

from datetime import datetime

decision_at = datetime.fromisoformat("2026-07-01T09:00:00+00:00")
fields = [
    ("ambiguity_score", "2026-07-01T08:59:00+00:00"),
    ("prior_exceptions_90d", "2026-07-01T08:59:00+00:00"),
    ("manager_approval_at", "2026-07-01T10:22:00+00:00"),
    ("access_revoked_at", "2026-07-15T12:00:00+00:00"),
]

for field, observed_at in fields:
    known = datetime.fromisoformat(observed_at) <= decision_at
    print(f"{field:22} {'ALLOW' if known else 'BLOCK'}")

ambiguity_score        ALLOW
prior_exceptions_90d   ALLOW
manager_approval_at    BLOCK
access_revoked_at      BLOCK

The two blocked fields help forensic analysis, but aren't opening-time inputs. Observation time means when the system could actually read the value, not merely when the event happened. A role change recorded in HR at 08:50 but synced to the router's cache at midnight isn't available for a 09:00 decision. The code uses explicit UTC offsets; production timestamps must use comparable time zones and the actual serving datastore's availability times.

Timestamp audits catch obvious future fields, but derived features need the same scrutiny. Rebuild them from strict as-of snapshots and inspect their aggregation windows. An offline database join can stamp a record at 08:59 while aggregating log rows that arrived at 11:00.

Labels demand an independent audit. A June request audited in August can't train a model deployed on July 1, even though the request itself is weeks old. Training requires both the request features and the ground-truth label to be available prior to the model-fit cutoff. Held-out validation labels can arrive later, because they score predictions frozen at the decision moment.

Train, validation, and test have different jobs

Feature availability is one boundary. Fitting, selecting, and independently evaluating also need distinct roles. They needn't always be three physical datasets: cross-validation and nested evaluation can rotate roles within development data.^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/}

To estimate performance on future access requests, use a chronological train, validation, and test split:

• January through June: training.
• July: validation for selecting features and choosing the review threshold. Fit each candidate's learned parameters on January-June.
• August: locked test for the frozen candidate. Report prespecified metrics and slices without using them to select a new candidate.

The small fixture below isolates month assignment. It uses two requests per month: 12 training requests, 2 validation requests, and 2 locked test requests. The counts demonstrate partitioning logic; evaluate on a production-scale sample to estimate model performance.

episodes = [
    {"request": f"{month}-{index}", "month": month}
    for month in range(1, 9)
    for index in range(2)
]

train = [row for row in episodes if row["month"] <= 6]
validation = [row for row in episodes if row["month"] == 7]
test = [row for row in episodes if row["month"] == 8]

assert len(train) == 12
assert len(validation) == 2
assert len(test) == 2
print("train requests =", len(train), "months = 1..6")
print("valid requests =", len(validation), "months = 7")
print("test  requests =", len(test), "months = 8 (locked)")

train requests = 12 months = 1..6
valid requests = 2 months = 7
test  requests = 2 months = 8 (locked)

View larger

January through June fits parameters, July selects thresholds, and August estimates frozen performance. July's ground-truth labels must be available before the August system is locked; otherwise, move the test window later.

A common failure is test-set hill-climbing. An engineer evaluates candidate A on August, revises features and the threshold after inspecting its errors, then reports candidate B's higher August score as independent release evidence.

The revised model's August score is no longer an independent final estimate. August feedback now informs selection, which can make the reported winner optimistic.^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/} You can still inspect and report errors for the original frozen candidate. The boundary concerns how feedback changes choices, not how many times a file is opened. After revising the system from August results, verify it on a fresh window with labels ready before making the release claim.

Cross-validation rotates the validation job

A single validation month can be unusually forgiving or harsh. Cross-validation (CV) refits and evaluates across multiple development partitions, exposing sensitivity to the split without spending the final test. Averaging can stabilize an estimate, but a universal variance reduction isn't guaranteed: folds share training rows and may represent different populations.^{[2]Reference 2Cross-validation: evaluating estimator performancehttps://scikit-learn.org/stable/modules/cross_validation.html}

In standard $K$-fold CV, development rows form disjoint folds $V_1,\ldots,V_K$ of nearly equal size. Iteration $k$ fits a fresh pipeline $h_{-k}$ on the other folds and evaluates on $V_k$. An equally weighted fold summary is:

The notation denotes a refitting procedure, not one model evaluated $K$ times. With unequal fold sizes, pooled per-row loss weights each fold by $|V_k|/n$ instead. Choosing $K$ changes training size, computational cost, and dependence between fitted models:

• $K=5$ or $10$: Each fit uses roughly 80% or 90% of development rows. If additional training data improves the procedure, this can understate its full-data performance. Neither the size of that bias nor a low variance is guaranteed.
• Leave-One-Out ($K=n$): Each fit uses $n-1$ rows, close to the final training size. It can require $n$ expensive fits, though some estimators have efficient shortcuts. Two training sets share $n-2$ rows, a fraction $(n-2)/(n-1)$ of either set. That overlap alone doesn't determine prediction similarity or the variance of the final score. Bias and variance depend on the learner, loss, and data.^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/}

With rare positive labels, a random fold can contain none. Stratified K-Fold preserves class proportions approximately, subject to integer counts. It cannot put a positive in all five folds when only three positives exist. Stratification helps avoid unusable folds; it neither enforces time/group boundaries nor makes fold spread a population uncertainty estimate.^{[2]Reference 2Cross-validation: evaluating estimator performancehttps://scikit-learn.org/stable/modules/cross_validation.html}

Try the impossible allocation. Before running, count how many folds must lack a positive. The warning is part of the lesson, so the lab captures and reports it rather than ignoring it silently.

import warnings
import numpy as np
from sklearn.model_selection import StratifiedKFold

X = np.zeros((10, 1))
y = np.array([0] * 7 + [1] * 3)
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always", UserWarning)
    folds = list(StratifiedKFold(n_splits=5).split(X, y))
counts = [int(y[validation].sum()) for _, validation in folds]
assert sum(counts) == 3 and counts.count(0) == 2
assert any("least populated class" in str(item.message) for item in caught)
print("positive counts per fold:", counts)
print("folds with no positive:", counts.count(0))
print("rare-class warning captured")

positive counts per fold: [0, 0, 1, 1, 1]
folds with no positive: 2
rare-class warning captured

Suppose five earlier-history folds give these F1 scores for our review guardrail: 0.67, 0.75, 0.58, 0.83, and 0.71. F1 combines precision and recall: how many flagged tickets genuinely needed review, and how many violative tickets were caught.

For the first fold, suppose the guardrail catches 4 needed reviews, sends 3 routine requests to review, and misses 1 needed review. That gives 4 true positives, 3 false positives, and 1 false negative:

The harmonic mean severely penalizes either low precision or low recall. It treats false positives and false negatives symmetrically, which rarely matches business reality: missing a critical policy violation is usually far more dangerous than sending an extra ticket to triage. Keep raw confusion counts beside F1 when choosing production operating points.

import statistics

fold_f1 = [0.67, 0.75, 0.58, 0.83, 0.71]
mean_f1 = statistics.mean(fold_f1)
std_f1 = statistics.stdev(fold_f1)

assert round(mean_f1, 2) == 0.71
assert round(std_f1, 2) == 0.09
print("fold F1 =", fold_f1)
print(f"mean F1 = {mean_f1:.2f}")
print(f"std  F1 = {std_f1:.2f}")

fold F1 = [0.67, 0.75, 0.58, 0.83, 0.71]
mean F1 = 0.71
std  F1 = 0.09

The mean 0.71 summarizes these development folds. The sample standard deviation 0.09 describes their spread. A standard error or confidence interval needs additional assumptions about fold dependence and the population the folds represent.

Don't select 0.83 as the deployment estimate. Inspect the 0.58 fold's traffic and label support. Mean fold F1 and pooled out-of-fold F1 can differ because F1 is a nonlinear ratio, but equality is possible. Two folds with identical confusion counts have identical fold and pooled F1. State which summary you report.

Shuffled CV is appropriate when its held-out rows represent the intended prediction population and information conditions. It can answer a different question from future-time or new-user evaluation. Stratification balances labels, but doesn't prevent a user or future record from crossing a boundary your claim requires.^{[2]Reference 2Cross-validation: evaluating estimator performancehttps://scikit-learn.org/stable/modules/cross_validation.html}

Target leakage: when a feature steals the answer key

Suppose access_revoked_after_audit sits in your historical table. It may be predictive of review need, but isn't available at opening. Its usefulness for retrospective analysis doesn't make it a valid opening-time feature, and revocation isn't necessarily equivalent to a missed required review.

Here target leakage means exposing the target or outcome-derived information unavailable under the prediction contract. A strong proxy known legitimately before the decision isn't automatically leakage. Audit its provenance and timing instead of banning every feature correlated with the label.

In the lab below, eight held-out requests are scored under two configurations. The honest rule uses only opening-time fields: predict review if ambiguity - 0.9 * policy_support > 0. The leaky rule includes the post-close audit field, which in this dataset directly mirrors the label. Predict which score will look impressive before running the comparison.

rows = [
    {"ambiguity": 1.4, "policy": 0, "y": 1},
    {"ambiguity": -0.2, "policy": 1, "y": 0},
    {"ambiguity": 0.8, "policy": 0, "y": 1},
    {"ambiguity": 0.1, "policy": 1, "y": 0},
    {"ambiguity": 1.1, "policy": 0, "y": 1},
    {"ambiguity": -0.5, "policy": 1, "y": 0},
    {"ambiguity": 0.3, "policy": 0, "y": 0},
    {"ambiguity": 0.6, "policy": 1, "y": 1},
]

def accuracy(pred, y):
    return sum(a == b for a, b in zip(pred, y)) / len(y)

y = [row["y"] for row in rows]
honest = [int(row["ambiguity"] - 0.9 * row["policy"] > 0) for row in rows]
leaked = list(y)

print(f"decision-time accuracy={accuracy(honest, y):.2f} preds={honest}")
print(f"future-field  accuracy={accuracy(leaked, y):.2f}")
assert accuracy(honest, y) == 0.75
assert accuracy(leaked, y) == 1.0

decision-time accuracy=0.75 preds=[1, 0, 1, 0, 1, 0, 1, 0]
future-field  accuracy=1.00

The last two constructed rows expose this rule's errors: at ambiguity=0.3 it predicts review with label 0; at ambiguity=0.6 with policy support it predicts no review with label 1. Six of eight predictions match, giving 0.75 fixture accuracy. The constructed rows report a teaching result; estimate deployment accuracy on representative real requests.

The leaked rule scores 1.00 because the lab explicitly copies y. A surprising real score should trigger checks for leakage, duplicates, label construction, sample size, and task difficulty. Legitimate tasks or small samples can also be perfectly separable, so inspect these alternatives before attributing the score to leakage.

Clustered entities change the generalization claim

A random row split is commonly used for i.i.d. evaluation. Many datasets instead cluster around users, patients, organizations, or machines. Decide whether the serving claim concerns new entities, returning entities, or both before choosing the split.

If your product promise is "this model generalizes to brand-new users," but each user's earlier requests appear in the training split while later requests appear in validation, your evaluation is answering the wrong question. The model doesn't have to learn a rule that would transfer to a new user. It can memorize this user's request times, resource IDs, or writing style.

The fixture below isolates this failure mode. Eight users each submit two identical requests, and the label is a user-specific attribute. The model is a simple lookup table: if a user appeared in training, repeat their known label; otherwise fall back to 0.

The first split puts one row from every user on each side. The second split holds out entire users using a group boundary.

rows = []
for user_index in range(8):
    user_id = f"u{user_index:02d}"
    label = user_index % 2
    rows.extend([(user_id, label), (user_id, label)])

def lookup_accuracy(train_idx, val_idx):
    memory = {rows[i][0]: rows[i][1] for i in train_idx}
    correct = 0
    for i in val_idx:
        user_id, label = rows[i]
        pred = memory.get(user_id, 0)
        correct += pred == label
    return correct / len(val_idx)

row_train = list(range(0, 16, 2))
row_val = list(range(1, 16, 2))
group_train = [i for i, (user_id, _) in enumerate(rows) if int(user_id[1:]) < 4]
group_val = [i for i, (user_id, _) in enumerate(rows) if int(user_id[1:]) >= 4]

row_acc = lookup_accuracy(row_train, row_val)
group_acc = lookup_accuracy(group_train, group_val)
assert row_acc == 1.0
assert group_acc == 0.5
print(f"paired-row accuracy = {row_acc:.2f}")
print(f"held-out-user accuracy = {group_acc:.2f}")

paired-row accuracy = 1.00
held-out-user accuracy = 0.50

The 1.00 score measures recognition of users already represented in training. The 0.50 score is the honest unseen-user result for this balanced fixture and its fallback rule.

For a new-entity claim, a group splitter keeps each entity out of its corresponding training fold. Dropping user_id doesn't establish independence: other fields may identify the user. GroupKFold alone also doesn't enforce future-time ordering; combine boundaries when both are part of the claim.^{[2]Reference 2Cross-validation: evaluating estimator performancehttps://scikit-learn.org/stable/modules/cross_validation.html}

Preprocessing can learn from validation rows

Data leakage doesn't require a future column. A scaler, imputer, target encoder, PCA projection, or feature selector learns parameters. Fitting across held-out rows violates an inductive evaluation boundary, even without using their labels. Supervised selection also exposes held-out labels. Fixed operations such as a predefined unit conversion have no data-dependent fit.^{[3]Reference 3Common Pitfalls and Recommended Practices: Data Leakagehttps://scikit-learn.org/stable/common_pitfalls.html#data-leakage}

View larger

Consider eight rows and two candidate features: T and L. Column T perfectly mirrors training labels and is noisy on validation. Column L is weak on training and copies validation labels.

Our selector picks the column with the larger absolute difference between class means: $\Delta = |\mu_1 - \mu_0|$. On training rows, T achieves $\Delta = 1.00$ while L achieves $\Delta = 0.50$. But across all eight rows, validation labels join the calculation, inflating L's separation to $0.75$ while dragging T down to $0.50$.

Both paths fit a nearest-mean classifier on training rows only. Only the feature selector's input changes: all eight rows, or training rows alone. Predict which column each path chooses before running the script.

T = [1, 1, 0, 0, 1, 1, 0, 0]
L = [1, 0, 0, 0, 1, 0, 1, 0]
y = [1, 1, 0, 0, 1, 0, 1, 0]
train, val = range(4), range(4, 8)

def mean_gap(col, idx):
    ones = [col[i] for i in idx if y[i] == 1]
    zeros = [col[i] for i in idx if y[i] == 0]
    return abs(sum(ones) / len(ones) - sum(zeros) / len(zeros))

def nearest_mean_accuracy(col, train_idx, val_idx):
    ones = [col[i] for i in train_idx if y[i] == 1]
    zeros = [col[i] for i in train_idx if y[i] == 0]
    mean_one = sum(ones) / len(ones)
    mean_zero = sum(zeros) / len(zeros)
    correct = 0
    for i in val_idx:
        pred = 1 if abs(col[i] - mean_one) < abs(col[i] - mean_zero) else 0
        correct += pred == y[i]
    return correct / len(val_idx)

print(f"train gap T={mean_gap(T, train):.2f} L={mean_gap(L, train):.2f}")
print(f"all   gap T={mean_gap(T, range(8)):.2f} L={mean_gap(L, range(8)):.2f}")
columns = {"T": T, "L": L}
leaky_name = max(columns, key=lambda name: mean_gap(columns[name], range(8)))
safe_name = max(columns, key=lambda name: mean_gap(columns[name], train))
leaky_acc = nearest_mean_accuracy(columns[leaky_name], train, val)
safe_acc = nearest_mean_accuracy(columns[safe_name], train, val)
print(f"select {leaky_name} on all rows val acc={leaky_acc:.2f}")
print(f"select {safe_name} on train    val acc={safe_acc:.2f}")
assert (mean_gap(T, train), mean_gap(L, train)) == (1.0, 0.5)
assert (mean_gap(T, range(8)), mean_gap(L, range(8))) == (0.5, 0.75)
assert leaky_acc == 1.0
assert safe_acc == 0.5

train gap T=1.00 L=0.50
all   gap T=0.50 L=0.75
select L on all rows val acc=1.00
select T on train    val acc=0.50

On training rows, T wins (1.00 versus 0.50 separation). Across all rows, L wins (0.75 versus 0.50) because validation labels influence selection. Its train-fitted classifier then scores perfectly on those reused rows. The train-only path gets two of four validation predictions right. Calling that 0.50 a measured population chance rate would overstate what four constructed rows tell us.

The current scikit-learn documentation gives a larger random-label example: 200 rows, 10,000 features, and selection of 25 features. Its displayed leaky holdout accuracy is 0.76; train-only selection gives 0.50. These are seeded example results, not constants that every random-label experiment must reproduce.^{[3]Reference 3Common Pitfalls and Recommended Practices: Data Leakagehttps://scikit-learn.org/stable/common_pitfalls.html#data-leakage}

Put learned transforms inside the object passed to cross-validation. This runnable example checks the scaler's fitted means, rather than treating an easy synthetic accuracy as evidence of deployment quality:

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X = np.column_stack((np.arange(20, dtype=float), np.tile([0, 1], 10)))
y = np.array([0] * 10 + [1] * 10)
folds = list(StratifiedKFold(n_splits=5).split(X, y))
pipeline = Pipeline([
    ("scaler", StandardScaler()),
    ("selector", SelectKBest(score_func=f_classif, k=1)),
    ("classifier", LogisticRegression()),
])
result = cross_validate(pipeline, X, y, cv=folds, return_estimator=True)
means = []
for (train, _), fitted in zip(folds, result["estimator"]):
    mean = fitted.named_steps["scaler"].mean_
    assert np.allclose(mean, X[train].mean(axis=0))
    means.append(float(mean[0]))
assert np.allclose(means, [10.5, 10.0, 9.5, 9.0, 8.5])
print("global first-feature mean:", float(X[:, 0].mean()))
print("training-fold fitted means:", means)
print("each scaler matches only its training rows")

global first-feature mean: 9.5
training-fold fitted means: [10.5, 10.0, 9.5, 9.0, 8.5]
each scaler matches only its training rows

Cross-validation clones and fits the pipeline within each training fold. The fitted transforms then process its held-out rows. This protects the declared fit flow, not arbitrary code: a custom transformer that reads a global full-data table, or precomputed leaked features, can still cross the boundary. Target encoders also need protection against using a training row's own target when constructing that row's encoded input, often through internal cross-fitting. In current scikit-learn, TargetEncoder.fit_transform performs that internal split; fit(...).transform(...) on the same training rows doesn't.^{[4]Reference 4TargetEncoder: internal cross-fitting contracthttps://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.TargetEncoder.html}

Temporal validation: future requests, ready labels, and usable history

Access routing faces seasonal traffic, policy shifts, infrastructure migrations, and evolving user habits. A model fitted on August can't predict June in any honest simulation of future performance. Rolling-origin evaluation (walk-forward time-series cross-validation) advances the training cutoff chronologically, ensuring that every validation fold evaluates requests that occurred strictly after its training data.^{[5]Reference 5Forecasting: Principles and Practice, Third Edition.https://otexts.com/fpp3/}

Expanding historical windows mirror production retraining: you retrain periodically on accumulated historical data. In our access routing timeline, we keep development folds strictly within January through June: train on months 1-2 and validate on 3-4; then expand training to months 1-4 and validate on 5-6. July is the dedicated threshold-selection month, while August remains locked.

View larger

Chronological row order doesn't establish every information boundary:

1. Label readiness: A June 25 request audited on July 9 can't be a labeled training example for a July 1 model. Filter by label availability as well as request time. A fixed gap is insufficient when label delays vary.
2. Feature readiness: A July 1 request can legitimately use June events in a backward-looking seven-day count. Those events appearing in training history isn't itself leakage. The count becomes invalid if it includes events or backfilled records the router couldn't read at that decision moment.
3. Overlapping label intervals: Some tasks define labels over future windows. Financial CV designs may purge training examples whose label intervals overlap held-out labels, and embargo training observations immediately after a test block. Such designs can train on both sides of a held-out block; they aren't automatically simulations of a model deployed before that block.^{[6]Reference 6CombinatorialPurgedCV: purging and embargo definitionshttps://skfolio.org/generated/skfolio.model_selection.CombinatorialPurgedCV.html} Specify the intervals and claim. An arbitrary seven-day buffer doesn't prove independence or repair a future-aware join.

TimeSeriesSplit(gap=...) excludes a number of samples immediately before each test block. That is a sample count, not automatically seven calendar days in irregular request traffic, and it doesn't inspect label timestamps.^{[7]Reference 7TimeSeriesSplit: sample gaps and temporal orderinghttps://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html} The monthly lab below demonstrates ordering only; the next audit checks label readiness.

months = [month for month in range(1, 7) for _ in range(2)]
folds = [(1, 2, 3, 4), (1, 4, 5, 6)]

for fold, (train_lo, train_hi, val_lo, val_hi) in enumerate(folds, start=1):
    train = [m for m in months if train_lo <= m <= train_hi]
    validation = [m for m in months if val_lo <= m <= val_hi]
    assert max(train) < min(validation)
    assert max(validation) <= 6  # July selects; August stays locked.
    print(
        f"fold {fold}: train <= {max(train)} "
        f"validate = {min(validation)}..{max(validation)}"
    )

fold 1: train <= 2 validate = 3..4
fold 2: train <= 4 validate = 5..6

Each fold keeps validation after training. Refit the pipeline within each fold. This fixture doesn't construct feature histories, filter delayed labels, or prove a particular gap is necessary.

Now inspect a seven-day feature at 00:05 on July 1. Two June events are usable history. A third also has a June event timestamp, but arrives after the decision. Which timestamp does a naive event-time-only join miss?

from datetime import datetime, timedelta

def utc(text):
    return datetime.fromisoformat(text + "+00:00")

decision = utc("2026-07-01T00:05:00")
# Event time, then time the serving system could read it.
events = [
    (utc("2026-06-29T10:00:00"), utc("2026-06-29T10:00:01")),
    (utc("2026-06-30T10:00:00"), utc("2026-06-30T10:00:01")),
    (utc("2026-06-30T18:00:00"), utc("2026-07-01T12:00:00")),
]
window_start = decision - timedelta(days=7)
naive = sum(window_start <= event < decision for event, _ in events)
available = sum(window_start <= event < decision and observed <= decision
                for event, observed in events)
assert (naive, available) == (3, 2)
print("event-time-only count:", naive)
print("count available at decision:", available)
print("two June events remain legitimate July history")

event-time-only count: 3
count available at decision: 2
two June events remain legitimate July history

Time and group boundaries combined

A future validation month contains both returning and genuinely new users. If your business cares about both, evaluate and report them as distinct slices.

Repeated users aren't automatically leakage: prior user history is legitimate context for a returning-user prediction if that history existed before the decision moment. But a claim about performance on new users requires both a temporal boundary and a group boundary: no user in that validation slice can have prior history in training.

The fixture below freezes training at July 1. Request r2 opened in June, but its label arrives in August, so it can't enter July training. July traffic contains returning user u1 and new user u4.

from datetime import datetime

# request, user, opening time, time its label becomes available
rows = [
    ("r1", "u1", "2026-01-03", "2026-01-10"),
    ("r2", "u2", "2026-06-20", "2026-08-02"),
    ("r3", "u3", "2026-06-10", "2026-06-20"),
    ("r4", "u1", "2026-07-03", "2026-07-10"),
    ("r5", "u4", "2026-07-05", "2026-07-20"),
]
rows = [(r, u, datetime.fromisoformat(t), datetime.fromisoformat(l))
        for r, u, t, l in rows]
fit_at = datetime(2026, 7, 1)
end_at = datetime(2026, 8, 1)
history = [row for row in rows if row[2] < fit_at]
train = [row for row in history if row[3] <= fit_at]
seen_users = {row[1] for row in history}
july = [row for row in rows if fit_at <= row[2] < end_at]
new_users = [row for row in july if row[1] not in seen_users]
returning_users = [row for row in july if row[1] in seen_users]

assert {row[1] for row in train}.isdisjoint(row[1] for row in new_users)
print("training requests:", [row[0] for row in train])
print("returning-user July requests:", [row[0] for row in returning_users])
print("new-user July requests:", [row[0] for row in new_users])

training requests: ['r1', 'r3']
returning-user July requests: ['r4']
new-user July requests: ['r5']

User u2 is marked as previously seen even though its label isn't ready for training. Newness depends on past activity, not on whether an earlier ticket made it into the training split.

Nested selection and validation overuse

Validation lets you select features, regularizers, and operating thresholds. Repeatedly selecting the best of noisy scores can favor configurations that happened to suit this sample. The optimism depends on score noise, candidate dependence, and search behavior; 5,000 candidates don't mathematically guarantee a spurious winner.

Reporting the winning validation score as independent production evidence ignores selection. Nested cross-validation separates selection from outer evaluation, provided the outer results don't themselves drive another search:^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/}

1. Outer loop: Partitions data into $K$ outer folds (or walk-forward temporal blocks) that never participate in tuning or model selection.
2. Inner loop: For each outer fold, searches hyperparameters, features, and thresholds using internal cross-validation on the remaining development folds only.
3. Outer evaluation: Fits the winning inner configuration on the development partition and scores it once on the untouched outer fold.

The outer average evaluates the selection-and-fitting procedure at the outer training sizes under the chosen split design. It doesn't certify a configuration selected after inspecting those outer scores. If nested fitting is too expensive, preserve an independent temporal holdout. You can publish exploratory validation results when their selection role and uncertainty are explicit.

Threshold selection: validation chooses, test reports

A classifier can expose both a class prediction and scores. Here we explicitly threshold a score $p\in[0,1]$: predict review when $p\ge\theta$. A score in that range isn't necessarily a calibrated probability. Threshold selection chooses an action boundary; probability calibration checks or adjusts the meaning of probability values. They are distinct tasks.^{[8]Reference 8Tuning the decision threshold for class predictionhttps://scikit-learn.org/stable/modules/classification_threshold.html}

Choosing a threshold from test labels is selection/evaluation leakage: the same labels choose and evaluate the decision rule. It differs from putting a future outcome column into each request's features.

In the lab below, July validation data evaluates three candidate thresholds: $0.30$, $0.50$, and $0.70$. Only after selection is locked does August report one final test score. F1 for threshold $\theta$ uses predictions $p \ge \theta$. Predict which threshold July selects before opening August.

thresholds = [0.30, 0.50, 0.70]
validation_probability = [0.82, 0.62, 0.58, 0.42, 0.31, 0.12]
validation_label = [1, 1, 1, 0, 0, 0]
test_probability = [0.77, 0.49, 0.39, 0.52, 0.66, 0.14]
test_label = [1, 1, 0, 0, 1, 0]

def f1_score(y, pred):
    tp = sum(a == 1 and b == 1 for a, b in zip(y, pred))
    fp = sum(a == 0 and b == 1 for a, b in zip(y, pred))
    fn = sum(a == 1 and b == 0 for a, b in zip(y, pred))
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    return 0.0 if precision + recall == 0 else 2 * precision * recall / (precision + recall)

validation_f1 = {
    threshold: f1_score(
        validation_label, [int(p >= threshold) for p in validation_probability]
    )
    for threshold in thresholds
}
chosen = max(validation_f1, key=validation_f1.get)
test_f1 = f1_score(test_label, [int(p >= chosen) for p in test_probability])

for threshold, score in validation_f1.items():
    print(f"threshold={threshold:.2f} valid_f1={score:.2f}")
print(f"chosen threshold={chosen:.2f} test_f1={test_f1:.2f}")
assert validation_f1[0.50] == 1.0
assert round(test_f1, 2) == 0.67

threshold=0.30 valid_f1=0.75
threshold=0.50 valid_f1=1.00
threshold=0.70 valid_f1=0.50
chosen threshold=0.50 test_f1=0.67

Threshold 0.50 classifies all six July validation requests perfectly, scoring 1.00. Applying that frozen threshold to August yields two true positives, one false positive, and one false negative, producing an F1 of 4 / (4 + 1 + 1) = 0.67.

This calculation assumes audit labels for all six requests, including the three automatically processed ones. If only routed requests get labels, you observe TP=2 and FP=1, so queue precision is 2/3. You don't observe the missed review at score 0.49. The same three reviewed labels are compatible with zero, one, two, or three positives among automatic requests, giving recalls of 1, 2/3, 1/2, or 2/5. Missing labels aren't negative labels. This is selective label coverage: the routing action controls which targets become observable.^{[9]Reference 9The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservableshttps://cs.stanford.edu/~jure/pubs/contraction-kdd17.pdf} An audit that also samples automatic requests can supply evidence about misses; report its sampling scheme and account for unequal sampling when estimating population metrics.

July's score participated in selection; August's didn't. That procedural difference matters even if both scores happen to match. The drop here doesn't isolate selection bias: small-sample variation or different traffic can also cause it. Six constructed rows establish the arithmetic, not statistical certainty about users. Report the frozen threshold, feature contract, and both scores with their roles. A revision based on August needs fresh verification.

Modern relevance: benchmark contamination in large language models

The same question applies to LLM evaluation: what information and development feedback would the evaluated system legitimately have?

For an enterprise access-policy assistant, determine exactly what you're evaluating. If your claim is generalization to new policy manuals, keep each document and its derived queries together. If your claim is answering new questions about an existing, static knowledge base, retrieving from shared documents is intended production behavior, not leakage. Leaking occurs when evaluation queries or reference answers contaminate prompt tuning or embedding models.

The fixture below verifies document-level isolation. Alternating chunks shares every source document across train and test, whereas partitioning complete source pages first shares zero documents.

documents = {
    "policy_access": [
        "temporary admin roles need review",
        "manager approval may support access grant",
    ],
    "policy_incidents": [
        "sev2 incidents need incident commander approval",
        "service-owner approval precedes production access",
    ],
    "policy_admins": [
        "privileged role appeal needs audit",
        "audit stores decision reason",
    ],
}
chunks = [(document, chunk) for document, texts in documents.items() for chunk in texts]

bad_train = chunks[::2]
bad_test = chunks[1::2]
bad_overlap = sorted(
    {document for document, _ in bad_train} & {document for document, _ in bad_test}
)

safe_train_documents = {"policy_access", "policy_incidents"}
safe_test_documents = {"policy_admins"}
safe_train = [(doc, text) for doc, texts in documents.items()
              if doc in safe_train_documents for text in texts]
safe_test = [(doc, text) for doc, texts in documents.items()
             if doc in safe_test_documents for text in texts]
safe_overlap = sorted({doc for doc, _ in safe_train} & {doc for doc, _ in safe_test})

print("split chunks first shared documents =", bad_overlap)
print("split documents first shared documents =", safe_overlap)
assert bad_overlap == ["policy_access", "policy_admins", "policy_incidents"]
assert safe_overlap == []

split chunks first shared documents = ['policy_access', 'policy_admins', 'policy_incidents']
split documents first shared documents = []

At foundation scale, pretraining corpora include trillions of tokens from Common Crawl, GitHub, and books. Those web scrapes frequently contain the exact test splits of standard benchmarks: MMLU, HumanEval, GSM8K, and MATH. A model that reaches 90% on a coding benchmark may be recalling repository files from training, rather than writing a program it has not seen.

Three checks look for different kinds of overlap:

1. Exact n-gram audits: Consecutive-word overlap flags candidates, not proven memorization. GPT-3's initial filtering used normalized 13-word sequences; its subsequent analysis chose a dataset-specific length, generally 8-13 words for nonsynthetic tasks. A filtering bug left overlaps in the trained corpus.^{[10]Reference 10Language Models are Few-Shot Learners.https://arxiv.org/abs/2005.14165} Llama 3 scored token coverage by matching 8-grams, then selected dataset-specific thresholds using performance differences. Table 15 flags 85% of HellaSwag and reports a 14.8-point full-versus-clean difference for 8B. This paper estimator reports the observed difference. A controlled retraining experiment would be needed to attribute the gain to contamination. The selected subsets can also differ in difficulty, which can produce false positives and negatives.^{[11]Reference 11The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783}
2. MinHash and LSH: Hash-based candidate search scales lexical near-duplicate detection. MinHash estimates resemblance between sets, such as word shingles. A semantic paraphrase with little shared wording can have low shingle similarity and escape detection. Embedding search is a separate semantic screen, also requiring review and thresholds.^{[12]Reference 12On the Resemblance and Containment of Documents.https://www.cs.princeton.edu/courses/archive/spring13/cos598C/broder97resemblance.pdf}
3. Loss-based diagnostics: Unusually low loss can motivate investigation, but common wording or easier items can also lower loss. It doesn't identify an exact training member by itself. One code-completion study found its tested perplexity and membership metrics poorly separated its groups. That study assumed pre-cutoff code was contaminated and called later/refactored code cleansed; it didn't establish actual training membership for every item. Its result cautions against a universal loss threshold.^{[13]Reference 13Concerned with Data Contamination? Assessing Countermeasures in Code Language Modelhttps://arxiv.org/abs/2403.16898}

LiveBench refreshes tasks and uses recent source material to limit exposure.^{[14]Reference 14LiveBench: A Challenging, Contamination-Limited LLM Benchmarkhttps://arxiv.org/abs/2406.19314[15]Reference 15LiveBench maintained benchmark repositoryhttps://github.com/LiveBench/LiveBench} LiveCodeBench records contest-problem dates and supports filtering by an evaluation time window.^{[16]Reference 16LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codehttps://arxiv.org/abs/2403.07974[17]Reference 17LiveCodeBench maintained repository: dataset versions and date-window evaluationhttps://github.com/LiveCodeBench/LiveCodeBench} Their full releases aren't necessarily newer than every evaluated model. Pin the benchmark release, date range, model version, and tool access; compare problem dates with the model's documented training cutoff where known. A model release date is only a proxy. Subsequent training, public solutions, browsing, or benchmark-driven tuning can reintroduce exposure. Refresh cadence also changes, so inspect the selected release rather than assuming all questions are new.

The local smoke test below searches for exact 5-gram overlap between training and evaluation prompts:

training_prompts = [
    "route privileged access request with required review to human queue",
    "approve access grant after clear manager approval arrives",
]
evaluation_prompts = [
    "route privileged access request with required review to human queue",
    "send an ambiguous admin request to a reviewer",
]

def ngrams(text, size=5):
    words = text.lower().split()
    return {
        " ".join(words[start : start + size])
        for start in range(len(words) - size + 1)
    }

training_grams = set().union(*(ngrams(prompt) for prompt in training_prompts))
for prompt in evaluation_prompts:
    overlaps = ngrams(prompt) & training_grams
    status = "FLAG" if overlaps else "NO EXACT MATCH"
    print(f"{status}: {prompt}")

FLAG: route privileged access request with required review to human queue
NO EXACT MATCH: send an ambiguous admin request to a reviewer

Exact matching flags verbatim copy-paste. It can't catch paraphrases, translations, reworded questions, or syntactically altered code. Boilerplate phrases like "please review the attached log" can trigger false positives. Use n-gram filters as a first screen, then review the candidates with semantic embeddings and provenance.

Overlap checks ask whether the evaluation text already appeared in training. They don't ask what would have happened if the router had taken a different action on the same request.

Historical logs don't reveal unchosen actions

Our labs evaluate a supervised guardrail: given request context available at opening, predict whether compliance review was required. That setup doesn't estimate the business return of a brand-new sequential decision policy.

Historical server logs tell you what happened under the action the legacy router took. They can't tell you what would have happened if the router took a different action for that same request. Those missing results are counterfactual outcomes.

In reinforcement learning, off-policy evaluation (OPE) uses historical-policy logs to estimate another policy's value. Importance-sampling methods require action support, valid behavior probabilities, and suitable state/dynamics assumptions. Recorded propensities are useful; some methods estimate them, adding modeling assumptions. For sequential policies, trajectory or cumulative per-decision ratios multiply probabilities across actions, as in the previous lesson. Neither a time split nor recorded propensities recovers an action with zero behavior-policy probability.^{[18]Reference 18Reinforcement Learning: An Introductionhttp://incompleteideas.net/book/the-book-2nd.html}

A simulator or fitted outcome model can predict counterfactuals under additional assumptions. Validate those assumptions and distinguish predictions from observed outcomes. Staged deployments monitor effects; appropriately designed randomized experiments can estimate causal policy impact.

Write the production evaluation contract

Before publishing any benchmark or shipping an access guardrail, document your boundaries in a concrete contract:

Decision moment: request opening, before evidence request or reviewer action
Allowed features: ambiguity_score, policy_support, prior_policy_exceptions_90d
Forbidden features: manager_approval_received_at, reviewer_action, access_revoked_after_audit
Train: January-June requests
Training labels: available by the model-fit cutoff, not merely attached later
Validation: July requests; select candidates/thresholds, with candidate transforms fitted on training
Test: August requests; prespecified report for the frozen candidate without feedback-driven selection
Label timing: July labels must be ready before freezing the August system
Label coverage: include automatic requests in the audit; missing labels aren't negatives
Entity promise: report future returning-user and genuinely new-user slices separately
Preprocessing: declared fit steps use training-fold rows; audit custom transforms and target cross-fitting
Feature snapshot: as-of request opening; aggregation windows terminate before decision time
Metrics: review-required F1, missed-review count, and false-alarm triage load

Write down the decision time, the allowed fields, which users are new, and which window was allowed to choose the threshold. Without that contract, a higher number can describe a different deployment than the one you are about to ship.

Stress-test the contract

The contract is only useful if a broken assumption shows up in the harness. Break one rule at a time and predict the symptom before you read the analysis:

1. Add reviewer_action to the feature availability audit and assign an observation time. Explain why it's label-adjacent target leakage.
2. Modify group-leakage-from-repeat-users.py so the product promise is performance on returning users rather than new users. Does row-wise splitting answer that question, and what time boundary remains critical?
3. Add a neutral column N = [0, 1, 1, 0, 0, 1, 1, 0] to the feature selector's dictionary. Calculate its class-mean gaps and predict whether either selected column changes.
4. Extend the document-chunk lab with a paraphrased evaluation chunk. Why does the exact n-gram matcher miss it, and how would you detect it?
5. Write an evaluation contract for the reinforcement learning router. Mark which outcomes are directly observed and which are counterfactual.

What each mutation exposes

1. reviewer_action is logged only after human triage begins. It belongs in forensic outcome analysis, never in opening-time feature vectors.
2. Overlapping user membership can fit a returning-user claim. The paired-row fixture doesn't establish future-time performance; enforce prediction/fit chronology and label readiness, with earlier user history available at serving time.
3. Feature N has class mean 0.5 for both label classes across all rows. Its separation gap is exactly zero: column T still wins honest training selection, and column L still wins leaky global selection.
4. A paraphrase may have little lexical overlap and evade both exact matching and shingle-based MinHash. Embedding search can surface semantic candidates; audit those pairs and their provenance rather than certifying everything below a similarity threshold as clean.
5. Historical rewards are observed for the logging policy's actions. Alternatives need counterfactual inference with explicit coverage and modeling assumptions, a validated simulator, or new experimental observations. Importance sampling is one approach, not the only approach and not a cure for absent support.