Validation and Leakage

The reinforcement-learning routing policy for temporary admin-access requests earned more expected reward for human_review once abandonment risk was included. That's not enough to ship a policy. A scientific claim needs evidence from requests that did not shape the decision rule.

Validation asks whether a fitted rule works on new examples. Leakage occurs when evaluation lets the rule see information it couldn't have at decision time, or lets test data influence choices. Both mistakes make weak systems look strong.^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/}

Start at the Decision Moment

Suppose you're fitting a guardrail model for the access-review router. When a request opens, it predicts whether the request must go to human review. The label comes later from an audit, but the prediction must be made before manager approvals, reviewer actions, and access outcomes arrive.

For request REQ-10234, the decision moment is 2026-07-01 09:00.

Write this feature contract before training a model. If a field doesn't exist when the action is chosen, it can't enter features, prompt context, retrieval results, or preprocessing statistics.

The two blocked fields may be useful for later outcome analysis. They aren't legal inputs for the opening decision.

A timestamp audit is necessary, but it's not sufficient. Rebuild derived features from an as-of snapshot and inspect their source windows too. A bad offline join can stamp a feature before the decision while still aggregating records that arrived later.

Train, Validation, and Test Have Different Jobs

Every dataset split controls a different kind of influence:

Access requests arrive over time, so the first honest train, validation, and test split is chronological:

• January through June requests: training data.
• July requests: validation data for choosing the review threshold.
• August requests: locked test data, opened once after choices are done.

If you inspect August failures and revise the pipeline, that work may be excellent engineering. It also spends the August test set. You then need a later untouched window for a final estimate.

Cross-Validation Rotates the Validation Job

One validation month can be unusual. Cross-validation (CV) rotates the held-out role through earlier data: fit on some folds, score on one unseen fold, repeat, and average the scores. It's a tool for model development, not permission to open the final test set repeatedly.^{[1]Reference 1The Elements of Statistical Learning.https://hastie.su.domains/ElemStatLearn/}

Five earlier-history folds give these F1 scores for the review guardrail: 0.67, 0.75, 0.58, 0.83, and 0.71. The F1 score (harmonic mean of precision and recall) is useful here because review-required requests may be less frequent than routine ones, and both missed reviews and excessive reviews matter.

The mean of 0.71 summarizes performance across folds. The standard deviation of 0.09 says the estimate isn't equally reliable everywhere. Investigate the weak fold before automating high-cost actions.

Ordinary shuffled CV assumes rows are exchangeable: any row could plausibly have arrived in any fold. That assumption breaks when the future differs from the past or when several rows come from one user, repository, service, document, or conversation.

Leak 1: A Future Outcome Looks Like an Amazing Feature

Suppose access_revoked_after_audit is set after a request closes. It almost reveals whether review was required. Adding it to an opening-time guardrail can produce a spectacular metric and an unusable model.

This experiment creates eight months of access requests. It fits on months 1 through 6 and scores months 7 and 8. The only difference between the two feature sets is a post-close audit field that copies the target.

1.00 isn't evidence of a brilliant guardrail. Here it proves that the evaluation admitted its answer key. The 0.73 result asks a real question: how well can opening-time information identify later review decisions?

Leak 2: Repeated Users Change the Promise

A random row split can be correct when production will repeatedly serve known users. It's wrong when you claim the model generalizes to new users while each user's earlier rows appear in training.

The next dataset is intentionally stark. A user-specific pattern determines every label. A model with user_id memorizes users perfectly when their rows are scattered through folds, then fails on user IDs it never encountered.

Neither metric is universally correct. 1.00 answers, "Can I recognize users already represented in history?" 0.50 answers, "Can user identity alone help on unseen users?" Write the intended deployment promise beside the score.

Leak 3: Preprocessing Can Learn From Validation Data

Leakage doesn't require a suspicious column. A scaler, imputer, vocabulary, principal component analysis (PCA) transform, or feature selector learns parameters during fit. If it sees all rows before a split, the validation data already affected the trained pipeline.

The experiment below uses scikit-learn's Pipeline with random labels.^{[2]Reference 2Common Pitfalls and Recommended Practices: Data Leakagehttps://scikit-learn.org/stable/common_pitfalls.html#data-leakage} If a feature selector sees all labels before splitting, it can select chance correlations that happen to fit the test rows. Fitting selection inside the pipeline removes that advantage.

Because the labels are random, the true attainable accuracy is around chance. The inflated 0.74 score comes entirely from leakage. In a real access model, you often won't know the true attainable score, which is why pipeline discipline matters.

Time-ordered validation preserves causality

Access routing faces seasonal traffic, policy changes, and changing user behavior. A model fitted on August shouldn't be used to predict June in an experiment that claims future performance. TimeSeriesSplit creates expanding history: each validation window comes after its training window.

If one user can appear over many months and you need a promise about unseen users, time ordering alone is insufficient. Hold out future months and keep the relevant user boundary intact inside the evaluation design.

Time and group together

Production access-review often needs both constraints: evaluate on future months and on users the model hasn't trained on. A pure TimeSeriesSplit can still leak through repeated users who appear before and after the cut. A pure GroupKFold can still score earlier months after later ones. Sketch the joint rule before you code it:

Don't stop at "I used a time split" if the product claim is about new users, or at "I used GroupKFold" if the product claim is about next-month traffic.

Nested selection and validation overuse

Validation may choose thresholds, features, and model families, but unlimited search has a cost. Trying many models, many feature subsets, and many thresholds on the same validation window, then reporting the best validation score as if it were a locked estimate, is another leak class: selection bias / validation overuse. The cure is nested evaluation or a final locked window:

1. Outer split: reserve a true test period (or outer CV fold) that never guides choices.
2. Inner loop: on the remaining data only, search hyperparameters, features, and thresholds.
3. Freeze the winner, then open the outer test once.

If you lack nested CV compute, still keep one untouched final window and treat heavy validation search as exploratory, not as the published number.

Validation Chooses; Test Reports

A probability model still needs an action threshold. Choosing that threshold from the test set is leakage, because the test labels then influence the deployed decision rule.

Below, July validation data chooses among three review thresholds. Only after selection is frozen does August report one test score.

The test score may be lower than validation. That's not a reason to retune on test; it's information about how uncertain your claimed performance is. Log the threshold, split definition, feature contract, and final metric together.

LLM Evaluation Has the Same Boundaries

Language-model systems add three common ways to leak information:

For an access-policy assistant, splitting chunks after document creation is too late if chunks from the same policy page reach both sides. The retriever then appears to generalize while it's retrieving near-copies of documents seen during development.

Public LLM benchmarks have a harder version of this problem: training corpora can include published test material. LiveBench was designed around frequently updated questions from recent information sources with objective scoring to reduce contamination risk, not to make contamination impossible for all future uses.^{[3]Reference 3LiveBench: A Challenging, Contamination-Limited LLM Benchmarkhttps://arxiv.org/abs/2406.19314}

A cheap local smoke test searches for exact phrase overlap between your own training and evaluation artifacts:

This flags copied wording. It doesn't prove the second prompt is safe: paraphrases, translated items, and memorized solution patterns can evade exact matching. Treat overlap checks as a reason to investigate, not as a certificate of clean evaluation.

A Necessary Caveat for the Reinforcement Learning (RL) Policy

These labs evaluate a supervised guardrail: given information available when a request opens, predict whether human review is required. They don't by themselves estimate the reward of a brand-new sequential policy.

Historical trajectories reveal the outcome of the action that was taken. They usually don't reveal what would have happened if the router had selected a different action for the same user. A time split prevents future leakage, but it doesn't fill in those missing counterfactual outcomes. For an RL policy, pair split discipline with a validated simulator, carefully logged exploration or propensities for off-policy evaluation, human review, or a staged online experiment before automation.^{[4]Reference 4Reinforcement Learning: An Introductionhttp://incompleteideas.net/book/the-book-2nd.html}

Write the Evaluation Contract

Before publishing a score, create a short, reviewable artifact:

The metric is only the last line of reasoning. The contract states what the metric means.

Practice tasks

1. Add reviewer_action to the feature-audit lab and assign an observation time. Explain why it's label-adjacent leakage.
2. Change group-leakage-from-repeat-users.py so prediction is for returning users rather than new users. State whether row-wise splitting now answers a relevant product question and what time boundary is still needed.
3. Replace SelectKBest with StandardScaler or PCA inside a Pipeline. Identify which learned values must come only from training rows.
4. Extend the document-chunk lab with a paraphrased evaluation chunk. Explain why the n-gram check misses it and what manual or semantic review you would add.
5. Write an evaluation contract for the RL policy itself. Mark which outcomes are observed and which are counterfactual.

Practice guidance

1. reviewer_action exists only after routing and review work begin. It belongs in outcome analysis, not opening-time features.
2. A row-wise split can answer a returning-user question if prior user history is legitimately available at decision time. Keep the evaluation chronological so later activity doesn't leak backward.
3. StandardScaler must learn means and variances from training rows. PCA must learn its centering values and components from training rows. A Pipeline keeps those fit calls inside the split.
4. A paraphrase may share no exact five-word sequence with its source. Add document provenance, semantic near-duplicate search, and manual review of flagged pairs.
5. An RL contract should log action, propensity, reward, and decision-time context. Historical logs observe the chosen action's outcome; alternative-action outcomes remain counterfactual and need stronger evaluation before automation.