Validation and Leakage

The reinforcement-learning lesson ended with a routing policy for damaged returns. On its tiny table, human_review earned more expected reward once abandonment risk was included. That is not enough to ship a policy. A scientific claim needs evidence from claims that did not shape the decision rule.

Validation asks whether a fitted rule works on new examples. Leakage occurs when evaluation lets the rule see information it could not have at decision time, or lets test data influence choices. Both mistakes make weak systems look strong.^[1]

Start at the Decision Moment

Suppose you are fitting a guardrail model for the return router. When a claim opens, it predicts whether the claim must go to human review. The label comes later from an audit, but the prediction must be made before photos, reviewer actions, and refund outcomes arrive.

For claim A10234, the decision moment is 2026-07-01 09:00.

Write this feature contract before training a model. If a field does not exist when the action is chosen, it cannot enter features, prompt context, retrieval results, or preprocessing statistics.

The two blocked fields may be useful for later outcome analysis. They are not legal inputs for the opening decision.

A timestamp audit is necessary, but it is not sufficient. Rebuild derived features from an as-of snapshot and inspect their source windows too. A bad offline join can stamp a feature before the decision while still aggregating records that arrived later.

Train, Validation, and Test Have Different Jobs

Every dataset split controls a different kind of influence:

Claims arrive over time, so the first honest train, validation, and test split is chronological:

• January through June claims: training data.
• July claims: validation data for choosing the review threshold.
• August claims: locked test data, opened once after choices are done.

If you inspect August failures and revise the pipeline, that work may be excellent engineering. It also spends the August test set. You then need a later untouched window for a final estimate.

Cross-Validation Rotates the Validation Job

One validation month can be unusual. Cross-validation (CV) rotates the held-out role through earlier data: fit on some folds, score on one unseen fold, repeat, and average the scores. It is a tool for model development, not permission to open the final test set repeatedly.^[1]

Imagine five earlier-history folds give these F1 scores for the review guardrail: 0.67, 0.75, 0.58, 0.83, and 0.71. F1 is useful here because review-required claims may be less frequent than routine ones, and both missed reviews and excessive reviews matter.

The mean of 0.71 summarizes performance across folds. The standard deviation of 0.09 says the result is not equally reliable everywhere. Investigate the weak fold before automating high-cost actions.

Ordinary shuffled CV assumes rows are exchangeable: any row could plausibly have arrived in any fold. That assumption breaks when the future differs from the past or when several rows come from one customer, order, merchant, document, or conversation.

Leak 1: A Future Outcome Looks Like an Amazing Feature

Suppose refund_reversed is set after a claim closes. It almost reveals whether review was required. Adding it to an opening-time guardrail can produce a spectacular metric and an unusable model.

This experiment creates eight months of return claims. It fits on months 1 through 6 and scores months 7 and 8. The only difference between the two feature sets is a post-close audit field that copies the target.

1.00 is not evidence of a brilliant guardrail. Here it proves that the evaluation admitted its answer key. The 0.73 result asks a real question: how well can opening-time information identify later review decisions?

Leak 2: Repeated Customers Change the Promise

A random row split can be correct when production will repeatedly serve known customers. It is wrong when you claim the model generalizes to new customers while each customer's earlier rows appear in training.

The next dataset is intentionally stark. A customer-specific pattern determines every label. A model with customer_id memorizes customers perfectly when their rows are scattered through folds, then fails on customer IDs it never encountered.

Neither metric is universally correct. 1.00 answers, "Can I recognize customers already represented in history?" 0.47 answers, "Can customer identity alone help on unseen customers?" Write the intended deployment promise beside the score.

Leak 3: Preprocessing Can Learn From Validation Data

Leakage does not require a suspicious column. A scaler, imputer, vocabulary, principal component analysis (PCA) transform, or feature selector learns parameters during fit. If it sees all rows before a split, the validation data already affected the trained pipeline.

The experiment below uses scikit-learn's Pipeline with random labels.^[2] If a feature selector sees all labels before splitting, it can select chance correlations that happen to fit the test rows. Fitting selection inside the pipeline removes that advantage.

Because the labels are random, the true attainable accuracy is around chance. The inflated 0.74 score comes entirely from leakage. In a real claim model, you often will not know the true attainable score, which is why pipeline discipline matters.

Time-Ordered Validation Preserves Causality

Return routing faces seasonal traffic, policy changes, and changing customer behavior. A model fitted on August should not be used to predict June in an experiment that claims future performance. TimeSeriesSplit creates expanding history: each validation window comes after its training window.

If one customer can appear over many months and you need a promise about unseen customers, time ordering alone is insufficient. Hold out future months and keep the relevant customer boundary intact inside the evaluation design.

Validation Chooses; Test Reports

A probability model still needs an action threshold. Choosing that threshold from the test set is leakage, because the test labels then influence the deployed decision rule.

Below, July validation data chooses among three review thresholds. Only after selection is frozen does August report one test score.

The test score may be lower than validation. That is not a reason to retune on test; it is information about how uncertain your claimed performance is. Log the threshold, split definition, feature contract, and final metric together.

LLM Evaluation Has the Same Boundaries

Language-model systems add three common ways to leak information:

For a return-policy assistant, splitting chunks after document creation is too late if chunks from the same policy page reach both sides. The retriever then appears to generalize while it is retrieving near-copies of documents seen during development.

Public LLM benchmarks have a harder version of this problem: training corpora can include published test material. LiveBench was designed around frequently updated questions from recent information sources with objective scoring to reduce contamination risk, not to make contamination impossible for all future uses.^[3]

A cheap local smoke test searches for exact phrase overlap between your own training and evaluation artifacts:

This flags copied wording. It does not prove the second prompt is safe: paraphrases, translated items, and memorized solution patterns can evade exact matching. Treat overlap checks as a reason to investigate, not as a certificate of clean evaluation.

A Necessary Caveat for the RL Policy

These labs evaluate a supervised guardrail: given information available when a claim opens, predict whether human review is required. They do not by themselves estimate the reward of a brand-new sequential policy.

Historical trajectories reveal the outcome of the action that was taken. They usually do not reveal what would have happened if the router had selected a different action for the same customer. A time split prevents future leakage, but it does not fill in those missing counterfactual outcomes. For an RL policy, pair split discipline with a validated simulator, carefully logged exploration or propensities for off-policy evaluation, human review, or a staged online experiment before automation.^[4]

Write the Evaluation Contract

Before publishing a score, create a short, reviewable artifact:

The metric is only the last line of reasoning. The contract states what the metric means.

Practice Tasks

1. Add reviewer_action to the feature-audit lab and assign an observation time. Explain why it is label-adjacent leakage.
2. Change group-leakage-from-repeat-customers.py so prediction is for returning customers rather than new customers. State whether row-wise splitting now answers a relevant product question and what time boundary is still needed.
3. Replace SelectKBest with StandardScaler or PCA inside a Pipeline. Identify which learned values must come only from training rows.
4. Extend the document-chunk lab with a paraphrased evaluation chunk. Explain why the n-gram check misses it and what manual or semantic review you would add.
5. Write an evaluation contract for the RL policy itself. Mark which outcomes are observed and which are counterfactual.

Practice guidance

1. reviewer_action exists only after routing and review work begin. It belongs in outcome analysis, not opening-time features.
2. A row-wise split can answer a returning-customer question if prior customer history is legitimately available at decision time. Keep the evaluation chronological so later activity does not leak backward.
3. StandardScaler must learn means and variances from training rows. PCA must learn its centering values and components from training rows. A Pipeline keeps those fit calls inside the split.
4. A paraphrase may share no exact five-word sequence with its source. Add document provenance, semantic near-duplicate search, and manual review of flagged pairs.
5. An RL contract should log action, propensity, reward, and decision-time context. Historical logs observe the chosen action's outcome; alternative-action outcomes remain counterfactual and need stronger evaluation before automation.

Key Concepts

• decision moment and feature availability
• training, validation, and locked test roles
• cross-validation and score variation
• future-outcome leakage
• group leakage
• preprocessing fit boundaries
• time-ordered evaluation
• threshold selection
• RAG corpus leakage
• LLM benchmark contamination
• policy counterfactuals

Evaluation rubric

• Foundational: Identifies fields unavailable at decision time and keeps a locked test set separate from model choices.
• Intermediate: Selects time, group, and pipeline boundaries that match a stated deployment promise and reproduces the leak demonstrations.
• Advanced: Designs an evaluation contract for an LLM or sequential policy system, including contamination and counterfactual limitations.

Follow-up questions

Common pitfalls

• Symptom: An opening-time model becomes nearly perfect after adding outcome fields. Cause: Later events reveal the label. Fix: Maintain and test a decision-time feature allowlist.
• Symptom: Random-fold performance collapses on new customers or documents. Cause: Repeated entities crossed folds. Fix: Match group boundaries to the deployment claim.
• Symptom: A null-data experiment beats chance after feature selection. Cause: Preprocessing fit on held-out labels. Fix: Split first and fit transforms inside a Pipeline.
• Symptom: A team keeps improving after reading test failures but still reports the original test as final. Cause: The test set became development data. Fix: Lock a later untouched window.
• Symptom: A RAG or LLM benchmark score looks excellent, but source provenance is unknown. Cause: Document overlap or benchmark contamination may be present. Fix: Preserve provenance, split before chunking, scan overlaps, and prefer fresh held-out evaluation material.