Personalize this lesson
Adapt explanations and teaching visuals to your background and preferred voice.
The reinforcement-learning routing policy for temporary admin-access requests earned more expected reward for human_review once abandonment risk was included. That's not enough to ship a policy. A scientific claim needs evidence from requests that did not shape the decision rule.
Validation asks whether a fitted rule works on new examples. Leakage occurs when evaluation lets the rule see information it couldn't have at decision time, or lets test data influence choices. Both mistakes make weak systems look strong.[1]
Start at the Decision Moment
Suppose you're fitting a guardrail model for the access-review router. When a request opens, it predicts whether the request must go to human review. The label comes later from an audit, but the prediction must be made before manager approvals, reviewer actions, and access outcomes arrive.
For request REQ-10234, the decision moment is 2026-07-01 09:00.
| Field | When it exists | Valid model feature at opening? | Why |
|---|---|---|---|
ambiguity_score | Before opening decision | Yes | Extracted from submitted request text |
prior_policy_exceptions_90d | Before opening decision | Yes | User history already recorded |
manager_approval_received_at | After evidence request | No | The router's action can create this event |
access_revoked_after_audit | After resolution | No | It reveals a later outcome |
needs_review | After audit | Label only | It's what the guardrail predicts |
Write this feature contract before training a model. If a field doesn't exist when the action is chosen, it can't enter features, prompt context, retrieval results, or preprocessing statistics.
1from datetime import datetime
2
3decision_at = datetime.fromisoformat("2026-07-01T09:00")
4fields = [
5 ("ambiguity_score", "2026-07-01T08:59"),
6 ("prior_policy_exceptions_90d", "2026-07-01T08:59"),
7 ("manager_approval_received_at", "2026-07-01T10:22"),
8 ("access_revoked_after_audit", "2026-07-15T12:00"),
9]
10
11for field, observed_at in fields:
12 known_in_time = datetime.fromisoformat(observed_at) <= decision_at
13 decision = "ALLOW" if known_in_time else "BLOCK"
14 print(f"{field:18} {decision}")1ambiguity_score ALLOW
2prior_policy_exceptions_90d ALLOW
3manager_approval_received_at BLOCK
4access_revoked_after_audit BLOCKThe two blocked fields may be useful for later outcome analysis. They aren't legal inputs for the opening decision.
A timestamp audit is necessary, but it's not sufficient. Rebuild derived features from an as-of snapshot and inspect their source windows too. A bad offline join can stamp a feature before the decision while still aggregating records that arrived later.
Why is manager_approval_received_at invalid even though it may appear in the final request record?
Answer
The prediction is made before a manager approval request is issued. Using manager_approval_received_at would reveal an event caused by later actions, so offline evaluation would give the model information unavailable when it must act.
Train, Validation, and Test Have Different Jobs
Every dataset split controls a different kind of influence:
| Split | Allowed use | Must not do |
|---|---|---|
| Train | Fit coefficients, trees, encoders, scalers, or learned features | Claim this score estimates deployment quality |
| Validation | Choose features, thresholds, model families, prompts, or reward settings | Reuse it as an untouched final estimate |
| Test | Estimate final performance after choices are frozen | Inspect errors, revise system, then report same test score as final |
Access requests arrive over time, so the first honest train, validation, and test split is chronological:
- January through June requests: training data.
- July requests: validation data for choosing the review threshold.
- August requests: locked test data, opened once after choices are done.
1episodes = [
2 {"request": f"{month}-{index}", "month": month}
3 for month in range(1, 9)
4 for index in range(2)
5]
6
7train = [row for row in episodes if row["month"] <= 6]
8validation = [row for row in episodes if row["month"] == 7]
9test = [row for row in episodes if row["month"] == 8]
10
11print("train requests =", len(train), "months = 1..6")
12print("valid requests =", len(validation), "months = 7")
13print("test requests =", len(test), "months = 8 (locked)")1train requests = 12 months = 1..6
2valid requests = 2 months = 7
3test requests = 2 months = 8 (locked)
If you inspect August failures and revise the pipeline, that work may be excellent engineering. It also spends the August test set. You then need a later untouched window for a final estimate.
Cross-Validation Rotates the Validation Job
One validation month can be unusual. Cross-validation (CV) rotates the held-out role through earlier data: fit on some folds, score on one unseen fold, repeat, and average the scores. It's a tool for model development, not permission to open the final test set repeatedly.[1]
Five earlier-history folds give these F1 scores for the review guardrail: 0.67, 0.75, 0.58, 0.83, and 0.71. The F1 score (harmonic mean of precision and recall) is useful here because review-required requests may be less frequent than routine ones, and both missed reviews and excessive reviews matter.
1import numpy as np
2
3fold_f1 = np.array([0.67, 0.75, 0.58, 0.83, 0.71])
4
5print("fold F1 =", fold_f1.tolist())
6print(f"mean F1 = {fold_f1.mean():.2f}")
7print(f"std F1 = {fold_f1.std(ddof=1):.2f}")1fold F1 = [0.67, 0.75, 0.58, 0.83, 0.71]
2mean F1 = 0.71
3std F1 = 0.09The mean of 0.71 summarizes performance across folds. The standard deviation of 0.09 says the estimate isn't equally reliable everywhere. Investigate the weak fold before automating high-cost actions.
Ordinary shuffled CV assumes rows are exchangeable: any row could plausibly have arrived in any fold. That assumption breaks when the future differs from the past or when several rows come from one user, repository, service, document, or conversation.
Leak 1: A Future Outcome Looks Like an Amazing Feature
Suppose access_revoked_after_audit is set after a request closes. It almost reveals whether review was required. Adding it to an opening-time guardrail can produce a spectacular metric and an unusable model.
This experiment creates eight months of access requests. It fits on months 1 through 6 and scores months 7 and 8. The only difference between the two feature sets is a post-close audit field that copies the target.
1import numpy as np
2from sklearn.linear_model import LogisticRegression
3from sklearn.metrics import accuracy_score
4
5rng = np.random.default_rng(9)
6month = np.repeat(np.arange(1, 9), 50)
7ambiguity = rng.normal(0, 1, len(month))
8policy_support = rng.integers(0, 2, len(month))
9needs_review = (
10 ambiguity
11 - 0.9 * policy_support
12 + rng.normal(0, 1.1, len(month))
13 > 0
14).astype(int)
15
16# This field is recorded after resolution, so it is forbidden at request opening.
17post_close_audit = needs_review.copy()
18train_mask = month <= 6
19future_mask = month >= 7
20
21decision_time_X = np.column_stack([ambiguity, policy_support])
22future_field_X = np.column_stack([ambiguity, policy_support, post_close_audit])
23
24for name, features in [
25 ("decision-time", decision_time_X),
26 ("future-field", future_field_X),
27]:
28 model = LogisticRegression().fit(features[train_mask], needs_review[train_mask])
29 predictions = model.predict(features[future_mask])
30 score = accuracy_score(needs_review[future_mask], predictions)
31 print(f"{name:13} accuracy={score:.2f}")1decision-time accuracy=0.73
2future-field accuracy=1.001.00 isn't evidence of a brilliant guardrail. Here it proves that the evaluation admitted its answer key. The 0.73 result asks a real question: how well can opening-time information identify later review decisions?
Leak 2: Repeated Users Change the Promise
A random row split can be correct when production will repeatedly serve known users. It's wrong when you claim the model generalizes to new users while each user's earlier rows appear in training.
The next dataset is intentionally stark. A user-specific pattern determines every label. A model with user_id memorizes users perfectly when their rows are scattered through folds, then fails on user IDs it never encountered.
1import numpy as np
2import pandas as pd
3from sklearn.linear_model import LogisticRegression
4from sklearn.model_selection import GroupKFold, StratifiedKFold, cross_val_score
5from sklearn.pipeline import make_pipeline
6from sklearn.preprocessing import OneHotEncoder
7
8users = np.repeat([f"c{i:02}" for i in range(60)], 4)
9needs_review = np.repeat([i % 2 for i in range(60)], 4)
10features = pd.DataFrame({"user_id": users})
11model = make_pipeline(
12 OneHotEncoder(handle_unknown="ignore"),
13 LogisticRegression(max_iter=1000),
14)
15
16row_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)
17group_cv = GroupKFold(n_splits=5)
18
19row_score = cross_val_score(
20 model, features, needs_review, cv=row_cv, scoring="accuracy"
21).mean()
22group_score = cross_val_score(
23 model,
24 features,
25 needs_review,
26 groups=users,
27 cv=group_cv,
28 scoring="accuracy",
29).mean()
30
31print(f"random rows mean accuracy = {row_score:.2f}")
32print(f"new users mean accuracy = {group_score:.2f}")1random rows mean accuracy = 1.00
2new users mean accuracy = 0.50Neither metric is universally correct. 1.00 answers, "Can I recognize users already represented in history?" 0.50 answers, "Can user identity alone help on unseen users?" Write the intended deployment promise beside the score.
Why should every row from one user stay in the same fold when the deployment promise is performance on unseen users?
Answer
Rows from one user share history and behavior. Splitting that identity across train and validation lets the model benefit from information about an entity it is supposed to treat as unseen.
Leak 3: Preprocessing Can Learn From Validation Data
Leakage doesn't require a suspicious column. A scaler, imputer, vocabulary, principal component analysis (PCA) transform, or feature selector learns parameters during fit. If it sees all rows before a split, the validation data already affected the trained pipeline.
The experiment below uses scikit-learn's Pipeline with random labels.[2] If a feature selector sees all labels before splitting, it can select chance correlations that happen to fit the test rows. Fitting selection inside the pipeline removes that advantage.
1import numpy as np
2from sklearn.feature_selection import SelectKBest
3from sklearn.metrics import accuracy_score
4from sklearn.model_selection import train_test_split
5from sklearn.pipeline import make_pipeline
6from sklearn.tree import DecisionTreeClassifier
7
8rng = np.random.RandomState(42)
9features = rng.standard_normal((200, 2_000))
10random_labels = rng.choice(2, 200)
11
12selected = SelectKBest(k=25).fit_transform(features, random_labels)
13leaky_train_X, leaky_test_X, leaky_train_y, leaky_test_y = train_test_split(
14 selected, random_labels, random_state=42
15)
16leaky_model = DecisionTreeClassifier(max_depth=4, random_state=1).fit(
17 leaky_train_X, leaky_train_y
18)
19
20train_X, test_X, train_y, test_y = train_test_split(
21 features, random_labels, random_state=42
22)
23safe_model = make_pipeline(
24 SelectKBest(k=25),
25 DecisionTreeClassifier(max_depth=4, random_state=1),
26).fit(train_X, train_y)
27
28print(
29 "fit selector before split accuracy =",
30 f"{accuracy_score(leaky_test_y, leaky_model.predict(leaky_test_X)):.2f}",
31)
32print(
33 "pipeline after split accuracy =",
34 f"{accuracy_score(test_y, safe_model.predict(test_X)):.2f}",
35)1fit selector before split accuracy = 0.74
2pipeline after split accuracy = 0.52Because the labels are random, the true attainable accuracy is around chance. The inflated 0.74 score comes entirely from leakage. In a real access model, you often won't know the true attainable score, which is why pipeline discipline matters.
Where should a scaler or imputer be fitted during cross-validation?
Answer
Fit it on each fold's training partition, then apply that fitted transform to the fold's validation partition. Fitting once on all rows leaks validation statistics into training.
Time-ordered validation preserves causality
Access routing faces seasonal traffic, policy changes, and changing user behavior. A model fitted on August shouldn't be used to predict June in an experiment that claims future performance. TimeSeriesSplit creates expanding history: each validation window comes after its training window.
1import numpy as np
2from sklearn.model_selection import TimeSeriesSplit
3
4months = np.repeat(np.arange(1, 9), 3)
5splitter = TimeSeriesSplit(n_splits=3)
6
7for fold, (train_index, validation_index) in enumerate(splitter.split(months), start=1):
8 train_months = months[train_index]
9 validation_months = months[validation_index]
10 print(
11 f"fold {fold}: train <= {train_months.max()} "
12 f"validate = {validation_months.min()}..{validation_months.max()}"
13 )1fold 1: train <= 2 validate = 3..4
2fold 2: train <= 4 validate = 5..6
3fold 3: train <= 6 validate = 7..8If one user can appear over many months and you need a promise about unseen users, time ordering alone is insufficient. Hold out future months and keep the relevant user boundary intact inside the evaluation design.
Time and group together
Production access-review often needs both constraints: evaluate on future months and on users the model hasn't trained on. A pure TimeSeriesSplit can still leak through repeated users who appear before and after the cut. A pure GroupKFold can still score earlier months after later ones. Sketch the joint rule before you code it:
1Promise: score future months on users unseen in training.
21. Sort requests by decision time.
32. Pick a future validation window (e.g. July).
43. Training rows: decision_time < July AND user_id not in July's user set
5 (or: train only on users who never appear in the validation window).
64. Report metrics on the July window for those held-out users.
75. If you also need returning-user quality, define a second contract
8 that allows prior user history but still freezes the time cut.Don't stop at "I used a time split" if the product claim is about new users, or at "I used GroupKFold" if the product claim is about next-month traffic.
Nested selection and validation overuse
Validation may choose thresholds, features, and model families, but unlimited search has a cost. Trying many models, many feature subsets, and many thresholds on the same validation window, then reporting the best validation score as if it were a locked estimate, is another leak class: selection bias / validation overuse. The cure is nested evaluation or a final locked window:
- Outer split: reserve a true test period (or outer CV fold) that never guides choices.
- Inner loop: on the remaining data only, search hyperparameters, features, and thresholds.
- Freeze the winner, then open the outer test once.
If you lack nested CV compute, still keep one untouched final window and treat heavy validation search as exploratory, not as the published number.
Validation Chooses; Test Reports
A probability model still needs an action threshold. Choosing that threshold from the test set is leakage, because the test labels then influence the deployed decision rule.
Below, July validation data chooses among three review thresholds. Only after selection is frozen does August report one test score.
1import numpy as np
2from sklearn.metrics import f1_score
3
4thresholds = [0.30, 0.50, 0.70]
5validation_probability = np.array([0.82, 0.62, 0.58, 0.42, 0.31, 0.12])
6validation_label = np.array([1, 1, 1, 0, 0, 0])
7
8test_probability = np.array([0.77, 0.49, 0.39, 0.52, 0.66, 0.14])
9test_label = np.array([1, 1, 0, 0, 1, 0])
10
11validation_f1 = {
12 threshold: f1_score(validation_label, validation_probability >= threshold)
13 for threshold in thresholds
14}
15chosen = max(validation_f1, key=validation_f1.get)
16
17for threshold, score in validation_f1.items():
18 print(f"threshold={threshold:.2f} valid_f1={score:.2f}")
19print(
20 f"chosen threshold={chosen:.2f} "
21 f"test_f1={f1_score(test_label, test_probability >= chosen):.2f}"
22)1threshold=0.30 valid_f1=0.75
2threshold=0.50 valid_f1=1.00
3threshold=0.70 valid_f1=0.50
4chosen threshold=0.50 test_f1=0.67The test score may be lower than validation. That's not a reason to retune on test; it's information about how uncertain your claimed performance is. Log the threshold, split definition, feature contract, and final metric together.
LLM Evaluation Has the Same Boundaries
Language-model systems add three common ways to leak information:
| System | Leakage path | Honest boundary |
|---|---|---|
| Retrieval-augmented generation (RAG) | Chunks from one source document appear in both tuning and test corpora | Split documents before chunking |
| Fine-tuning or prompting | Eval questions, answers, or worked traces enter training examples or prompt demonstrations | Maintain provenance and exclude eval material |
| Public benchmark evaluation | A benchmark item may be present in model training data | Prefer fresh or contamination-limited test material and report uncertainty |
For an access-policy assistant, splitting chunks after document creation is too late if chunks from the same policy page reach both sides. The retriever then appears to generalize while it's retrieving near-copies of documents seen during development.
1documents = {
2 "policy_access": [
3 "temporary admin roles need review",
4 "manager approval may support access grant",
5 ],
6 "policy_incidents": [
7 "sev2 incidents need incident commander approval",
8 "service-owner approval precedes production access",
9 ],
10 "policy_admins": [
11 "privileged role appeal needs audit",
12 "audit stores decision reason",
13 ],
14}
15chunks = [(document, chunk) for document, texts in documents.items() for chunk in texts]
16
17bad_train = chunks[::2]
18bad_test = chunks[1::2]
19bad_overlap = sorted(
20 {document for document, _ in bad_train}
21 & {document for document, _ in bad_test}
22)
23
24safe_train_documents = {"policy_access", "policy_incidents"}
25safe_test_documents = {"policy_admins"}
26safe_overlap = sorted(safe_train_documents & safe_test_documents)
27
28print("split chunks first shared documents =", bad_overlap)
29print("split documents first shared documents =", safe_overlap)1split chunks first shared documents = ['policy_access', 'policy_admins', 'policy_incidents']
2split documents first shared documents = []Public LLM benchmarks have a harder version of this problem: training corpora can include published test material. LiveBench was designed around frequently updated questions from recent information sources with objective scoring to reduce contamination risk, not to make contamination impossible for all future uses.[3]
A cheap local smoke test searches for exact phrase overlap between your own training and evaluation artifacts:
1training_prompts = [
2 "route privileged access request with required review to human queue",
3 "approve access grant after clear manager approval arrives",
4]
5evaluation_prompts = [
6 "route privileged access request with required review to human queue",
7 "send an ambiguous admin request to a reviewer",
8]
9
10def ngrams(text, size=5):
11 words = text.lower().split()
12 return {
13 " ".join(words[start : start + size])
14 for start in range(len(words) - size + 1)
15 }
16
17training_grams = set().union(*(ngrams(prompt) for prompt in training_prompts))
18for prompt in evaluation_prompts:
19 overlaps = ngrams(prompt) & training_grams
20 status = "FLAG" if overlaps else "clear"
21 print(f"{status}: {prompt}")1FLAG: route privileged access request with required review to human queue
2clear: send an ambiguous admin request to a reviewerThis flags copied wording. It doesn't prove the second prompt is safe: paraphrases, translated items, and memorized solution patterns can evade exact matching. Treat overlap checks as a reason to investigate, not as a certificate of clean evaluation.
A Necessary Caveat for the Reinforcement Learning (RL) Policy
These labs evaluate a supervised guardrail: given information available when a request opens, predict whether human review is required. They don't by themselves estimate the reward of a brand-new sequential policy.
Historical trajectories reveal the outcome of the action that was taken. They usually don't reveal what would have happened if the router had selected a different action for the same user. A time split prevents future leakage, but it doesn't fill in those missing counterfactual outcomes. For an RL policy, pair split discipline with a validated simulator, carefully logged exploration or propensities for off-policy evaluation, human review, or a staged online experiment before automation.[4]
Write the Evaluation Contract
Before publishing a score, create a short, reviewable artifact:
1Decision moment: request opening, before evidence request or reviewer action
2Allowed features: ambiguity_score, policy_support, prior_policy_exceptions_90d
3Forbidden features: manager_approval_received_at, reviewer_action, access_revoked_after_audit
4Train: January-June requests
5Validation: July requests; choose threshold and features here
6Test: August requests; open once after choices are frozen
7Entity promise: new-user quality, so no user crosses split boundary
8Preprocessing: every fit step is inside training fold Pipeline
9Feature snapshot: as of request opening; aggregation windows stop before decision time
10Metrics: review-required F1 plus unsupported-access countThe metric is only the last line of reasoning. The contract states what the metric means.
Practice tasks
- Add
reviewer_actionto the feature-audit lab and assign an observation time. Explain why it's label-adjacent leakage. - Change
group-leakage-from-repeat-users.pyso prediction is for returning users rather than new users. State whether row-wise splitting now answers a relevant product question and what time boundary is still needed. - Replace
SelectKBestwithStandardScaleror PCA inside aPipeline. Identify which learned values must come only from training rows. - Extend the document-chunk lab with a paraphrased evaluation chunk. Explain why the n-gram check misses it and what manual or semantic review you would add.
- Write an evaluation contract for the RL policy itself. Mark which outcomes are observed and which are counterfactual.
Practice guidance
reviewer_actionexists only after routing and review work begin. It belongs in outcome analysis, not opening-time features.- A row-wise split can answer a returning-user question if prior user history is legitimately available at decision time. Keep the evaluation chronological so later activity doesn't leak backward.
StandardScalermust learn means and variances from training rows. PCA must learn its centering values and components from training rows. APipelinekeeps thosefitcalls inside the split.- A paraphrase may share no exact five-word sequence with its source. Add document provenance, semantic near-duplicate search, and manual review of flagged pairs.
- An RL contract should log action, propensity, reward, and decision-time context. Historical logs observe the chosen action's outcome; alternative-action outcomes remain counterfactual and need stronger evaluation before automation.