Personalize this lesson
Adapt explanations and teaching visuals to your background and preferred voice.
A shopper types label-printer. In our ten-listing catalog, the two strongest raw retrieval matches can't be shown: P8 is policy-blocked and out of stock, while P11 can't ship to the shopper's region. A soft score penalty can be outweighed by other features. How will you keep these products out of the response even when their match scores are excellent?
The displayed list also influences which products receive shopper attention. Suppose weak item P4 receives more clicks because an older heuristic pins it at rank 1. Treating those clicks as direct relevance judgments can reinforce the old ordering. The same log may contain no evidence about a better item that was never shown. We'll separate ranking quality from the mechanism that produced the feedback.
Ranking and Recommendation Systems introduced candidate recall and NDCG. In this capstone, you'll implement eligibility rules, check a ranking fixture by hand, fit a pairwise model on dated synthetic requests, and retain proposed response rows before outcomes arrive. The data, judgments, scores, latency samples, and online counts are all authored examples. They test implementation mechanics, data contracts, and gates.
label-printer fixture, P8 is policy-blocked (and out of stock) and P11 can't ship to the shopper's region. The hand-entered rank scores only reorder the survivors. Control puts weak P4 first (NDCG@3 0.736); treatment restores the authored order (NDCG@3 1.000).Specify the ranking surface and multi-stage funnel
Before picking model architectures, define what the serving surface is allowed to do. For label-printer, a high similarity score isn't permission to show an item that can't ship.
For large catalogs, an expensive query-item model may be impractical to run over every listing on each request. Retrieval followed by ranking is a common solution; TensorFlow Recommenders teaches that separation using a two-tower retrieval model.[1] The stages below are a design you can extend toward, rather than implementations supplied by this lab. Candidate counts and latency budgets require measurements on your catalog, hardware, and traffic.
- Retrieve candidates. Choose a manageable set using lexical search, embedding search, or a combination. This fixture sorts authored retrieval scores and filters eligibility first. It doesn't build an ANN index or embedding model.
- Score survivors. A ranker can use query-item interactions and features that a cheaper retriever omitted. One option is LambdaMART, a boosted-tree method whose pairwise updates are weighted by changes in a ranking metric.[2] Here you'll fit a much smaller linear pairwise model with two features.
- Construct the slate. A high-scoring list can still contain near-duplicates or too many listings from one seller. Maximal Marginal Relevance (MMR) is one way to trade query relevance against redundancy:[3] Here is the candidate set, contains already selected items, and trades relevance against similarity to . For the first selection, define the redundancy term as zero. Specify compatible score scales and evaluate the tradeoff on your surface. This lab doesn't implement MMR or sponsored placement. It checks a fictional cap of two listings per seller in a three-item slate and rechecks the current in-memory eligibility state before creating response rows. A real service needs an authoritative eligibility source and a policy for changes between serving and rendering.
| Contract | Decision |
|---|---|
| surface | marketplace search results |
| query set | frozen judged search queries plus online experiment traffic |
| eligibility | in-stock, deliverable region, policy-approved listing |
| offline metric | recall@K and NDCG@K; use for visible arithmetic, then choose separate retrieval/display cutoffs for the actual surface |
| online primary metric | purchase conversion per search |
| guardrails | returns, latency, unsafe listings, seller concentration |
Candidate generation and reranking stay separate artifacts. First, the candidate component has to recover relevant eligible items. Only then can the ranker change their order.
The earlier ranking lesson used pairwise preferences the way RankNet does: a relevant product should outscore a weaker candidate.[4] That modeling choice still doesn't let a learned score override eligibility.
Filter first, then score, and pin the eligibility version on every impression log. The same item can be eligible under one catalog snapshot and blocked under the next, so a score without that policy context can't be replayed.
Build offline evidence first
A live shopper log can't grade candidate coverage by itself. It records what a ranking surface showed, not which relevant items were absent before scoring. Start with a judged fixture. Real evaluations need independently collected judgments; the grades below are authored so you can verify each calculation.
Run the marked Python cells in order. The first fixture uses the standard library; the later model-fitting exercise also uses NumPy and the downloadable CSV. The directory below is a suggested project layout, not a claim that a deployed service or all these files already exist.
ranking-product/
data/
catalog_snapshot.jsonl
judged_queries.jsonl
eligibility_policy.json
retrieval/
candidate_generator.py
ranking/
ranker.py
evaluate_ndcg.py
experiments/
impression_schema.json
ab_plan.md
tests/
test_blocked_listing_never_surfaces.py
test_ndcg_regression_gate.pyThe local checks score NDCG@3 and recall@3. A larger system might retrieve 100 candidates and display 10; that would mean different retrieval and display cutoffs. Pin each cutoff and its judgment universe instead of treating a metric name as a complete specification.
Turn the offline contract into independent checks:
| Check | Why it blocks release |
|---|---|
| eligible-only results | a relevant prohibited listing still can't display |
| candidate recall@K (fixture @3) | the ranker can't repair missing products |
| NDCG@K by query or category (fixture: two queries, @3) | overall improvement may hide poor slices |
| scoring latency | a slower list damages shopping experience |
| diversity/seller concentration | one seller with many feedback events shouldn't crowd out catalog |
Eligibility comes first in this implementation. Another design may score a larger set internally, then filter and refill. The hard requirement is that the displayed response obeys the current eligibility rules. Measure eligible-candidate recall after filtering so an empty or incomplete survivor set doesn't look healthy.
Filter policy before retrieval
A prohibited or sold-out product can still have an excellent text match and a high learned score. Our fixture rejects it before ranking.
Eligibility is an explicit rule in this lab. A model shouldn't have to infer permission from clicks.
The compact fixture keeps two judged searches in memory and retrieves three products per query. Its baseline and candidate scores are hand-entered regression cases, not fitted models. The fitting exercise follows after the metric checks. Before running this fixture, predict which high-score products disappear and which eligible products fill the budget.
from collections import Counter
from dataclasses import dataclass
from datetime import datetime
from math import ceil, isfinite, log2
def nonblank(value: object) -> bool:
return type(value) is str and bool(value.strip())
def finite_number(value: object) -> float | None:
if type(value) not in (int, float):
return None
try:
number = float(value)
except OverflowError:
return None
return number if isfinite(number) else None
@dataclass(frozen=True)
class Listing:
product_id: str
query: str
relevance: int
retrieval_score: float
baseline_rank_score: float
candidate_rank_score: float
in_stock: bool
policy_approved: bool
deliverable_region: bool
seller_id: str
def __post_init__(self):
if not all(nonblank(value) for value in (self.product_id, self.query, self.seller_id)):
raise ValueError("listing IDs and query must be nonblank strings")
if type(self.relevance) is not int or not 0 <= self.relevance <= 3:
raise ValueError("fixture grade must be an integer from 0 to 3")
if any(type(flag) is not bool for flag in (self.in_stock, self.policy_approved, self.deliverable_region)):
raise ValueError("eligibility flags must be booleans")
if any(finite_number(score) is None for score in (self.retrieval_score, self.baseline_rank_score, self.candidate_rank_score)):
raise ValueError("listing scores must be finite numbers")
CATALOG = [
Listing("P1", "insulated-bag", 3, 0.96, 0.82, 0.97, True, True, True, "S1"),
Listing("P2", "insulated-bag", 1, 0.88, 0.75, 0.62, True, True, True, "S2"),
Listing("P3", "insulated-bag", 2, 0.84, 0.65, 0.90, True, True, True, "S3"),
Listing("P9", "insulated-bag", 3, 0.99, 0.99, 0.99, True, False, True, "S9"),
Listing("P10", "insulated-bag", 2, 0.95, 0.98, 0.96, False, True, True, "S1"),
Listing("P4", "label-printer", 1, 0.74, 0.90, 0.45, True, True, True, "S4"),
Listing("P5", "label-printer", 3, 0.93, 0.72, 0.96, True, True, True, "S5"),
Listing("P6", "label-printer", 2, 0.86, 0.65, 0.84, True, True, True, "S6"),
Listing("P8", "label-printer", 3, 0.98, 0.99, 0.99, False, False, True, "S8"),
Listing("P11", "label-printer", 2, 0.97, 0.97, 0.97, True, True, False, "S11"),
]
QUERIES = ("insulated-bag", "label-printer")
def exclusion_reason(listing: Listing) -> str | None:
if not listing.policy_approved:
return "policy_block"
if not listing.in_stock:
return "out_of_stock"
if not listing.deliverable_region:
return "unavailable_region"
return None
print("catalog listings:", len(CATALOG))
print("queries:", QUERIES)catalog listings: 10
queries: ('insulated-bag', 'label-printer')def retrieve(query: str, budget: int = 3) -> list[Listing]:
if not nonblank(query) or query not in QUERIES:
raise ValueError("unknown fixture query")
if type(budget) is not int or budget < 1:
raise ValueError("candidate budget must be a positive integer")
eligible = [
listing for listing in CATALOG
if listing.query == query and exclusion_reason(listing) is None
]
return sorted(eligible, key=lambda listing: (-listing.retrieval_score, listing.product_id))[:budget]
def relevant_ids(query: str) -> set[str]:
return {
listing.product_id for listing in CATALOG
if listing.query == query and exclusion_reason(listing) is None and listing.relevance >= 2
}
def candidate_recall(query: str, candidates: list[Listing]) -> float:
if len({row.product_id for row in candidates}) != len(candidates):
raise ValueError("duplicate candidate")
if any(row.query != query or exclusion_reason(row) is not None for row in candidates):
raise ValueError("candidate violates query or eligibility")
relevant = relevant_ids(query)
if not relevant:
raise ValueError(f"{query}: no relevant eligible judgments")
return len(relevant & {listing.product_id for listing in candidates}) / len(relevant)
retrieved = {query: retrieve(query) for query in QUERIES}
rejected = {
listing.product_id: exclusion_reason(listing)
for listing in CATALOG
if exclusion_reason(listing) is not None
}
for query, candidates in retrieved.items():
print(query, [listing.product_id for listing in candidates], f"recall@3={candidate_recall(query, candidates):.3f}")
print("rejected:", rejected)insulated-bag ['P1', 'P2', 'P3'] recall@3=1.000
label-printer ['P5', 'P6', 'P4'] recall@3=1.000
rejected: {'P9': 'policy_block', 'P10': 'out_of_stock', 'P8': 'policy_block', 'P11': 'unavailable_region'}Products P9 and P8 have the strongest scores for their searches, but policy excludes them. P8 is also out of stock; exclusion_reason() logs the first failing check, so the receipt shows policy_block.
P10 is approved but sold out, and P11 can't ship to the shopper's region. Filtering first keeps the ranker from treating a business invariant as a preference it can trade away. The next question is whether the survivors are complete before we judge their order.
Measure retrieval and ordering separately
Once the eligible set is fixed, ask two different questions. Candidate recall asks whether relevant eligible products reached the ranker. NDCG asks whether stronger judgments sit near the top of the returned list. A ranker can't repair a relevant product that retrieval dropped.
Evaluating a multi-stage ranking pipeline with a single end-to-end metric masks critical component failures. If candidate generation drops a relevant product, an NDCG metric computed only over returned survivors can still look flawless. A comprehensive ranking evaluation uses distinct metrics tailored to each stage:
- Candidate recall@K: Evaluates Stage 1 (retrieval). Did the candidate set capture the relevant, eligible products?
If a relevant product
P3never reaches the ranker, no downstream scoring model can recover it. - NDCG@K (Normalized Discounted Cumulative Gain): Evaluates Stage 2 (ranker ordering). It uses graded relevance () with logarithmic position discounting:
Manning, Raghavan, and Schütze document this exponential-gain form; some libraries use linear gain instead, so the evaluation setup has to pin one.[5] In the helper below,
enumerateis 0-based, making the denominatorlog2(rank + 2), which equals . - MRR (Mean Reciprocal Rank): Summarizes the position of the first relevant result under a specified relevance criterion: Here is 1-based; define reciprocal rank as zero when no relevant result appears. This can suit an exact SKU lookup where finding one acceptable item matters most. Moving that item from position 1 to 3 changes reciprocal rank from 1.000 to 0.333. Click-based MRR instead measures observed clicking behavior, which is affected by exposure. Neither metric directly measures shopper satisfaction.
- Probability calibration, when probabilities are needed: RankNet and LambdaMART produce ordering scores, not purchase probabilities.[2] Our fitted score also has no probability interpretation. If a separate model estimates click and purchase probabilities, a possible calculation is: This assumes at most one attributed purchase, purchase only through the measured click path, and a fixed item price. It excludes fees, refunds, and other purchase paths. Reliability diagrams compare binned probability estimates with observed frequencies on held-out data; they don't prove calibration for future traffic.[6] ECE summarizes binned discrepancies and depends on binning and sample size.[7] Ordering-only evaluation doesn't require calibrated probabilities.
Work label-printer by hand before looking at code. Control shows P4, P5, P6 with grades , , and . Which position is costing the list most?
The ideal order is P5, P6, P4 with grades , , and :
Control NDCG@3 is . Treatment is already the ideal order, so its NDCG@3 is .
Recall@3 stays at on both arms because P5 and P6 (the eligible grades ) are in the candidate set either way. Ordering quality moved; retrieval didn't.
The next cells make that distinction executable: rerank each eligible set, print NDCG, and recheck that scoring didn't sneak a blocked listing back in.
def dcg(rows: list[Listing], k: int = 3) -> float:
if type(k) is not int or k < 1:
raise ValueError("cutoff must be a positive integer")
if any(type(row.relevance) is not int or not 0 <= row.relevance <= 3 for row in rows):
raise ValueError("fixture grades must be integers from 0 to 3")
return sum((2**listing.relevance - 1) / log2(rank + 2) for rank, listing in enumerate(rows[:k]))
def ndcg(rows: list[Listing], universe: list[Listing] | None = None, k: int = 3) -> float:
universe = rows if universe is None else universe
judgments = {listing.product_id: listing for listing in universe}
if len(judgments) != len(universe) or len({row.product_id for row in rows}) != len(rows):
raise ValueError("duplicate product in evaluation")
if any(row.product_id not in judgments or row.relevance != judgments[row.product_id].relevance for row in rows):
raise ValueError("returned item must match fixed judgments")
ideal = sorted(universe, key=lambda listing: (-listing.relevance, listing.product_id))
ideal_dcg = dcg(ideal, k)
return dcg(rows, k) / ideal_dcg if ideal_dcg else 0.0
def rerank(rows: list[Listing], score_field: str) -> list[Listing]:
if score_field not in {"baseline_rank_score", "candidate_rank_score"}:
raise ValueError("unapproved score field")
if any(exclusion_reason(row) is not None for row in rows):
raise ValueError("ineligible candidate reached ranker")
if any(not isfinite(getattr(row, score_field)) for row in rows):
raise ValueError("rank scores must be finite")
return sorted(rows, key=lambda listing: (-getattr(listing, score_field), listing.product_id))
baseline_ranked = {
query: rerank(candidates, "baseline_rank_score")
for query, candidates in retrieved.items()
}
candidate_ranked = {
query: rerank(candidates, "candidate_rank_score")
for query, candidates in retrieved.items()
}
print("queries ranked:", list(candidate_ranked))for query in QUERIES:
print(
query,
f"baseline_ndcg@3={ndcg(baseline_ranked[query]):.3f}",
f"candidate_ndcg@3={ndcg(candidate_ranked[query]):.3f}",
)
blocked_hits = [
listing.product_id
for rows in candidate_ranked.values()
for listing in rows
if exclusion_reason(listing) is not None
]
print("blocked hits:", blocked_hits)insulated-bag baseline_ndcg@3=0.972 candidate_ndcg@3=1.000
label-printer baseline_ndcg@3=0.736 candidate_ndcg@3=1.000
blocked hits: []The candidate improves ordering on both fixture queries, and no blocked listing returns. Two edge policies stay explicit: a query with no relevant eligible judgments is a fixture error here, and NDCG returns 0.0 when a ranked set has no positive gain.
That still isn't a launch decision. Offline judgments qualify a controlled experiment; they don't tell you what shoppers saw or whether later clicks are unbiased. The displayed slate is the data you'll later be tempted to train on.
Before trusting the metric, remove P3 deliberately. Perfect ordering of the remaining candidates still scores 1.0 under the candidate-conditional denominator. Fixed-universe NDCG and recall expose the loss:
complete = retrieved["insulated-bag"]
missing_p3 = [row for row in candidate_ranked["insulated-bag"] if row.product_id != "P3"]
print(f"candidate-conditional NDCG@3={ndcg(missing_p3):.3f}")
print(f"fixed-universe NDCG@3={ndcg(missing_p3, complete):.3f}")
print(f"candidate recall={candidate_recall('insulated-bag', missing_p3):.3f}")
assert ndcg(missing_p3) == 1.0 and ndcg(missing_p3, complete) < 1.0
assert candidate_recall("insulated-bag", missing_p3) == 0.5candidate-conditional NDCG@3=1.000
fixed-universe NDCG@3=0.812
candidate recall=0.500Fit a ranker without borrowing tomorrow's labels
Hand-entered scores make good regression fixtures but don't train a model. The downloadable temporal dataset supplies 24 authored rows across eight search requests. Each request has three eligible products, two numeric features, and a relevance grade. text_match is a synthetic query-match score; quality_score is a synthetic listing-quality feature already available at serving time. Neither is derived from the later relevance grade.
For the target here, a model fitted at a historical cutoff must predict later requests. Random folds can put later requests into the training data for an earlier evaluation, answering a different question. Random splitting can be appropriate for independent examples from the target distribution; it isn't inherently wrong for ranking. This lab uses chronological windows, request grouping, and feature/label availability checks to match its stated target.[8]
Dates have three distinct responsibilities:
served_at: Locates the search request in physical time and defines feature hydration.feature_available_at: Prevents future feature values (like 30-day rolling conversion rates computed after the query) from leaking into earlier predictions.label_available_at: Marks when the outcome or human judgment became accessible for training.
Outcome windows depend on the metric and reporting pipeline. A click may occur quickly but be ingested late; a return may need a much longer follow-up. Define both the observation window and when a finalized label becomes available. A purchase with no return observed yet isn't automatically a negative return label. Excluding immature examples can also favor faster outcomes, so report that cohort selection rather than assuming an availability check removes every source of bias.
Freeze training on May 5, select regularization on May 5-6 requests whose labels are available before May 7, and evaluate May 7-8 requests once their labels are available before May 10. Keep every request's product rows together. PENDING-4 occurred before May 5, but its labels arrived afterward, so it can't enter the frozen training set.
The same query text can recur across windows. That's appropriate for testing later traffic on a known search surface, not evidence of generalization to new queries or products. Report separate cold-start slices when those are deployment requirements.
import csv
from pathlib import Path
import numpy as np
def utc_time(value: str) -> datetime:
if not nonblank(value):
raise ValueError("an explicit UTC timestamp is required")
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
if parsed.tzinfo is None or parsed.utcoffset().total_seconds() != 0:
raise ValueError("an explicit UTC timestamp is required")
return parsed
with Path("assets/temporal-ranking.csv").open() as source:
temporal_rows = list(csv.DictReader(source))
groups: dict[str, list[dict]] = {}
for row in temporal_rows:
if any(not nonblank(row[field]) for field in ("request_id", "query", "product_id")):
raise ValueError("temporal IDs and query must be nonblank strings")
for field in ("served_at", "feature_available_at", "label_available_at"):
row[field] = utc_time(row[field])
row["features"] = np.array([float(row["text_match"]), float(row["quality_score"])])
row["grade"] = int(row["relevance"])
if not np.isfinite(row["features"]).all() or not ((0 <= row["features"]) & (row["features"] <= 1)).all():
raise ValueError("features must be finite scores in [0, 1]")
if not 0 <= row["grade"] <= 3:
raise ValueError("invalid relevance grade")
if not row["feature_available_at"] <= row["served_at"] <= row["label_available_at"]:
raise ValueError("future feature or impossible judgment time")
groups.setdefault(row["request_id"], []).append(row)
def window(start: str, end: str, labels_before: str) -> dict[str, list[dict]]:
start_at, end_at, freeze_at = map(utc_time, (start, end, labels_before))
if not start_at < end_at <= freeze_at:
raise ValueError("window must end no later than its label freeze")
result = {}
for request_id, rows in groups.items():
if len({(row["served_at"], row["query"]) for row in rows}) != 1:
raise ValueError("one request must have one serving time and query")
if len({row["product_id"] for row in rows}) != len(rows):
raise ValueError("duplicate product within a request")
served = rows[0]["served_at"]
if start_at <= served < end_at and all(row["label_available_at"] < freeze_at for row in rows):
result[request_id] = rows
return result
train = window("2026-05-01T00:00:00Z", "2026-05-05T00:00:00Z", "2026-05-05T00:00:00Z")
validation = window("2026-05-05T00:00:00Z", "2026-05-07T00:00:00Z", "2026-05-07T00:00:00Z")
test = window("2026-05-07T00:00:00Z", "2026-05-09T00:00:00Z", "2026-05-10T00:00:00Z")
assert not (set(train) & set(validation) or set(train) & set(test) or set(validation) & set(test))
assert "PENDING-4" not in train
print("request counts:", {"train": len(train), "validation": len(validation), "test": len(test)})
print("excluded from frozen training: PENDING-4 (labels arrived too late)")request counts: {'train': 3, 'validation': 2, 'test': 2}
excluded from frozen training: PENDING-4 (labels arrived too late)For each training request, form a pair whenever one product's grade exceeds another's. A2 has features (0.85, 0.90) and should beat A1 at (0.95, 0.30). Their feature difference is (-0.10, 0.60): match alone prefers the wrong product, while listing quality points toward the judged preference.
Let the score be . For a preferred pair, the margin is . Minimize , which falls as the preferred product pulls ahead. This is a linear pairwise logistic ranker using the preference objective behind RankNet, not RankNet's neural architecture.[4] Regularization penalizes large weights. All features already share the authored scale; any fitted scaler for real data must learn its statistics from training only.
To check the update's sign, write each preferred feature difference as . With pairs and L2 penalty , the code minimizes
wrong_order_probability computes stably. For the A2 versus A1 pair alone at zero weights, it equals 0.5; the gradient is (0.05, -0.30). Subtracting 0.3 times that gradient gives weights (-0.015, 0.09), increasing this pair's margin to 0.0555. The actual cell averages all nine pair gradients before updating. Each of these three requests supplies three pairs, so they have equal weight here. Requests with more judged pairs would contribute more under this pair average; use a per-request average if equal request weighting is your intended objective.
The next cell runs 600 fixed gradient steps for each of two L2 penalties using training pairs, chooses the penalty on validation NDCG, then computes the test comparison. It doesn't check optimizer convergence or search every model setting. The baseline ranks by text_match alone. Both methods receive exactly the same products and judgments. Re-running the notebook is possible; using test results to make another modeling choice would turn this test set into tuning data.
def preference_differences(grouped: dict[str, list[dict]]) -> np.ndarray:
differences = [
better["features"] - worse["features"]
for rows in grouped.values() for better in rows for worse in rows
if better["grade"] > worse["grade"]
]
if not differences:
raise ValueError("training needs at least one judged preference")
return np.stack(differences)
def fit_ranker(differences: np.ndarray, penalty: float) -> np.ndarray:
penalty = finite_number(penalty)
if penalty is None or penalty < 0:
raise ValueError("L2 penalty must be finite and nonnegative")
if (not isinstance(differences, np.ndarray) or differences.ndim != 2
or differences.shape[0] < 1 or differences.shape[1] != 2 or differences.dtype.kind not in "fi"
or not np.isfinite(differences).all() or np.any((differences < -1) | (differences > 1))):
raise ValueError("expected a nonempty two-feature difference matrix in [-1, 1]")
weights = np.zeros(differences.shape[1])
for _ in range(600):
margins = differences @ weights
wrong_order_probability = np.exp(-np.logaddexp(0, margins))
gradient = -(differences.T @ wrong_order_probability) / len(differences) + penalty * weights
weights -= 0.3 * gradient
if not np.isfinite(weights).all():
raise ValueError("optimizer produced nonfinite weights")
return weights
def ranking_score(grouped: dict[str, list[dict]], weights: np.ndarray) -> float:
if (not isinstance(weights, np.ndarray) or weights.shape != (2,)
or weights.dtype.kind not in "fi" or not np.isfinite(weights).all()):
raise ValueError("expected two finite weights")
values = []
for rows in grouped.values():
if not rows or len({row["product_id"] for row in rows}) != len(rows):
raise ValueError("evaluation request must contain unique products")
features = np.stack([row["features"] for row in rows])
if (features.shape != (len(rows), 2) or features.dtype.kind not in "fi"
or not np.isfinite(features).all() or np.any((features < 0) | (features > 1))):
raise ValueError("expected two feature scores in [0, 1] per product")
if any(type(row["grade"]) is not int or not 0 <= row["grade"] <= 3 for row in rows):
raise ValueError("invalid evaluation grade")
with np.errstate(over="ignore", invalid="ignore"):
scores = features @ weights
if not np.isfinite(scores).all():
raise ValueError("model produced nonfinite scores")
ordered = [row for _, row in sorted(zip(scores, rows), key=lambda pair: (-float(pair[0]), pair[1]["product_id"]))]
ideal = sorted((row["grade"] for row in rows), reverse=True)
gain = lambda grades: sum((2**grade - 1) / log2(i + 2) for i, grade in enumerate(grades[:3]))
denominator = gain(ideal)
values.append(gain([row["grade"] for row in ordered]) / denominator if denominator else 0.0)
if not values:
raise ValueError("evaluation window is empty")
return float(np.mean(values))
differences = preference_differences(train)
models = {penalty: fit_ranker(differences, penalty) for penalty in (0.0, 0.1)}
# Prefer stronger regularization when validation scores tie.
chosen_penalty = max(models, key=lambda penalty: (ranking_score(validation, models[penalty]), penalty))
weights = models[chosen_penalty]
baseline_weights = np.array([1.0, 0.0])
test_baseline = ranking_score(test, baseline_weights)
test_candidate = ranking_score(test, weights)
print(f"training pairs={len(differences)} selected_penalty={chosen_penalty}")
print("weights:", np.round(weights, 3).tolist())
print(f"validation NDCG@3={ranking_score(validation, weights):.3f}")
print(f"test baseline NDCG@3={test_baseline:.3f}; fitted NDCG@3={test_candidate:.3f}")training pairs=9 selected_penalty=0.1
weights: [-0.109, 1.229]
validation NDCG@3=1.000
test baseline NDCG@3=0.854; fitted NDCG@3=1.000The fitted weights give quality more influence and even assign a small negative coefficient to text match. That's a consequence of this tiny authored dataset, where some strong text matches have weak grades, not a recommendation to penalize matching products. Scores order candidates; they aren't calibrated purchase probabilities.
This execution fits weights and evaluates held-out rows. Its tiny, deliberately separable synthetic data still can't establish a useful production ranker. The earlier two-query hand-entered fixture remains separate: it tests eligibility and logging regressions, while this temporal exercise tests fitting and holdout discipline. Don't label either as live shopper evidence.
Log exposure before learning from outcomes
Offline ranking tells you how a candidate orders judged items. The next question is what a shopper actually saw. If that slate is going to become training data, freeze it before any click arrives.
Every displayed slate should log enough context to replay the decision:
| Field | Why |
|---|---|
request_id, query, served_at | group one displayed decision |
shopper_id, experiment_id, experiment_arm | replay stable experiment assignment |
catalog_snapshot, eligibility_version | show available choices |
candidate_version, ranker_version | identify scoring path |
product_id, position | preserve exposure |
position_propensity, propensity_model_version | record an examination-propensity estimate when a validated model exists; otherwise leave unset |
Clicks are affected by position bias: the ranking changes the chance that an item is examined. A missing click doesn't establish irrelevance. Joachims, Swaminathan, and Schnabel derive an unbiased ranking-risk estimator under explicit observation and support assumptions, rather than treating every non-click as a negative relevance judgment.[9]
Work with a deliberately simplified model. Let indicate examination, let indicate relevance, and assume a noise-free click . At a fixed context, assume relevance is independent of examination and examination depends only on position. Then:
Here and . The independence and noise-free assumptions are modeling choices. The probability chain rule itself doesn't imply independence, and real clicks can also depend on attractiveness, trust, or accidental interaction.[9]
Suppose two equally relevant items have , with examination probabilities and . Their expected click rates are and . The fourfold difference comes entirely from our authored attention assumptions. Inverse weights and make both expected weighted clicks equal . One finite log won't necessarily equal those expectations.
Now clip each weight at . The less examined item's expected weighted click becomes , while the first stays at . Clipping bounds weights but introduces bias; the bias needn't be small or the remaining variance negligible. Predict these outputs before running:
relevance_probability = 0.50
for position, propensity in ((1, 0.80), (3, 0.20)):
click_probability = propensity * relevance_probability
weight = 1 / propensity
print(f"position={position} click={click_probability:.2f} "
f"IPS expectation={click_probability * weight:.2f} "
f"clipped expectation={click_probability * min(weight, 2):.2f}")position=1 click=0.40 IPS expectation=0.50 clipped expectation=0.50
position=3 click=0.10 IPS expectation=0.50 clipped expectation=0.20Two probabilities must stay separate. Examination propensity describes looking at a shown item. Action-selection probability describes the logging policy choosing an item or slate. A 50/50 experiment allocation supplies neither item's examination probability. If an item has zero probability of being exposed under the logging policy, examination weights can't create feedback about it. Exploration or independent judgments can supply additional evidence, subject to their own sampling assumptions.
Estimating propensities needs an identifiable model and suitable data, not a guessed curve. Randomized position swaps are one approach. Agarwal and colleagues also show how to harvest position variation from multiple historical rankers without new intrusive swap experiments, under their model and assignment assumptions.[10] State whether you estimate absolute or relative examination propensities. A common positive scaling factor preserves an unregularized loss's minimizer; with regularization, account for that scaling in the objective. Relative propensities alone don't identify an absolute relevance rate like above.
The fixture has no propensity study. Its fields stay None and its rows are marked evaluation_only. Before using real clicks for training, specify the estimator and its assumptions, collect evidence for the appropriate propensity, and audit outcome attribution. Presence of a model-version string isn't proof that the estimate is valid.
Experiment check: Use the judged fixture to catch regressions, then use a controlled A/B experiment to compare shopper outcomes. Experiment Design and A/B Testing already made the assignment unit, primary metric, and guardrails explicit; Kohavi, Tang, and Xu treat that online comparison as the way to measure a product change.[11]
Keep each shopper in one arm and analyze uncertainty at that assignment unit, not as if repeated searches from one shopper were independent. Pick the primary outcome, attribution windows, minimum detectable effect, sample-size calculation, and sequential stopping method before exposure. Check sample-ratio mismatch and missing logs. A higher conversion estimate doesn't cancel safety or return-rate concerns.
Event streaming architecture decouples exposure from later outcomes:
- Response and rendering records: Retain the server-returned slate and its decision context. Record client rendering or viewability separately, with stable identifiers to join the records. A returned item isn't necessarily rendered, visible, or examined. Append-only storage can retain the original records and explicit corrections.
- Asynchronous outcome stream: Clicks, add-to-carts, purchases, and returns arrive asynchronously over streaming message buses (like Kafka) seconds, hours, or days later. Each outcome event carries
event_id,occurred_at,request_id,product_id, andoutcome. - Join correctness and hygiene:
- Idempotence: At-least-once streaming delivery can emit duplicate events. Ingestion pipelines must deduplicate by
event_idbefore joins. - Causal order: Require
occurred_at >= served_atunder the chosen clock contract. Earlier timestamps need investigation for clock skew or identifier errors; timestamp order alone doesn't prove attribution. - Attribution windows: Direct search purchases must be distinguished from conversions occurring days later through retargeting emails.
- Preserve non-clicked rows: Keep the exposure denominator and observation-window status. A finalized non-click can be a target for predicting clicks under a specified serving policy; it doesn't prove negative relevance or examination.
- Idempotence: At-least-once streaming delivery can emit duplicate events. Ingestion pipelines must deduplicate by
The code uses ordinary in-memory dictionaries. Append-only storage, durable deduplication, privacy-aware retention, and transaction boundaries are deployment work, not properties created by calling a row immutable. Preserve the candidate set and feature snapshot as well when you need to reproduce scores, not just reconstruct the displayed slate.
The remaining cells serialize proposed response rows, index them, and check whether two outcome records match that index and follow serving time. They don't collect client-render acknowledgments, materialize finalized labels, implement attribution windows, deduplicate a stream, or deploy a rollback. Their local report keeps real-experiment readiness false.
CATALOG_SNAPSHOT = "catalog-2026-05-01"
ELIGIBILITY_VERSION = "market-eligibility-v1"
CANDIDATE_VERSION = "retrieval-v1"
RANKER_VERSION = "market-ranker-v1"
PREVIOUS_RANKER = "market-ranker-v0"
EXPERIMENT_ID = "ranking-ab-2026-05"
PROPENSITY_MODEL_VERSION = None
POSITION_PROPENSITY = {1: None, 2: None, 3: None}
REPLAY_CATALOG = tuple(CATALOG) # Retained fixture snapshot, not today's stock.
import hashlib
def assigned_arm(shopper_id: str) -> str:
if not nonblank(shopper_id):
raise ValueError("shopper ID must be a nonblank string")
digest = hashlib.sha256(f"{EXPERIMENT_ID}:{shopper_id}".encode()).digest()
bucket = int.from_bytes(digest[:4], "big") % 2
return ("treatment", "control")[bucket]
ARM_CONFIG = {
"control": (PREVIOUS_RANKER, baseline_ranked),
"treatment": (RANKER_VERSION, candidate_ranked),
}
def impression_rows(
request_id: str,
shopper_id: str,
served_at: str,
query: str,
arm: str,
) -> list[dict[str, object]]:
if (not all(nonblank(value) for value in (request_id, query, arm))
or query not in QUERIES or arm not in ARM_CONFIG):
raise ValueError("invalid request ID, query, or arm")
if arm != assigned_arm(shopper_id):
raise ValueError("arm differs from stable shopper assignment")
utc_time(served_at)
ranker_version, ranked_by_query = ARM_CONFIG[arm]
# Simulate an eligibility recheck against the current in-memory catalog.
current_catalog = {listing.product_id: listing for listing in CATALOG}
if len(current_catalog) != len(CATALOG):
raise ValueError("duplicate product ID in catalog")
if any(listing != current_catalog.get(listing.product_id) or exclusion_reason(current_catalog[listing.product_id]) is not None
for listing in ranked_by_query[query]):
raise ValueError("eligibility changed before display; rebuild the slate")
return [
{
"request_id": request_id,
"shopper_id": shopper_id,
"served_at": served_at,
"query": query,
"catalog_snapshot": CATALOG_SNAPSHOT,
"eligibility_version": ELIGIBILITY_VERSION,
"candidate_version": CANDIDATE_VERSION,
"ranker_version": ranker_version,
"experiment_id": EXPERIMENT_ID,
"experiment_arm": arm,
"product_id": listing.product_id,
"position": position,
"position_propensity": POSITION_PROPENSITY[position],
"propensity_model_version": PROPENSITY_MODEL_VERSION,
"slate_size": len(ranked_by_query[query]),
"training_use": "evaluation_only",
}
for position, listing in enumerate(ranked_by_query[query], start=1)
]
print("experiment:", EXPERIMENT_ID, "arms:", list(ARM_CONFIG))REPLAY_CATALOG retains this fixture's frozen listing records for historical checks. New response rows check the current CATALOG too. Version fields declare names; they don't verify content or store fitted weights. For a real replay, retain the identified catalog, candidate set, features, model, and rule definitions. Record client rendering separately from these server-side rows.
control_impressions = impression_rows("REQ-500", "SHOP-100", "2026-05-01T12:00:00Z", "label-printer", "control")
treatment_impressions = impression_rows("REQ-501", "SHOP-200", "2026-05-01T12:00:05Z", "label-printer", "treatment")
impressions = control_impressions + treatment_impressions
FIXTURE_REQUESTS = {
"REQ-500": ("SHOP-100", "label-printer", "control", "2026-05-01T12:00:00Z"),
"REQ-501": ("SHOP-200", "label-printer", "treatment", "2026-05-01T12:00:05Z"),
}
outcomes = [
{
"event_id": "EVT-900",
"occurred_at": "2026-05-01T12:00:08Z",
"request_id": "REQ-501",
"product_id": "P5",
"outcome": "click",
},
{
"event_id": "EVT-901",
"occurred_at": "2026-05-01T12:04:18Z",
"request_id": "REQ-501",
"product_id": "P5",
"outcome": "purchase",
},
]
print("impressions:", len(impressions), "outcomes:", len(outcomes))required_impression_fields = {
"request_id", "shopper_id", "served_at", "query", "catalog_snapshot",
"eligibility_version", "candidate_version", "ranker_version",
"experiment_id", "experiment_arm", "product_id", "position",
"position_propensity", "propensity_model_version", "slate_size", "training_use",
}
required_outcome_fields = {"event_id", "occurred_at", "request_id", "product_id", "outcome"}
# Synthetic values exercise percentile arithmetic, not measured runtime.
latencies_ms = [11, 13, 12, 15, 14, 16, 13, 12, 17, 18]
# Synthetic online counts exercise a conservative stop rule, not significance.
arm_searches = {"control": 100, "treatment": 100}
arm_purchases = {"control": 4, "treatment": 6}
arm_returns = {"control": 0, "treatment": 1}
conversion = {
arm: arm_purchases[arm] / arm_searches[arm]
for arm in arm_searches
}
return_rate = {
arm: (arm_returns[arm] / arm_purchases[arm]) if arm_purchases[arm] else None
for arm in arm_searches
}
return_comparison_defined = all(rate is not None for rate in return_rate.values())
stop_returns = return_comparison_defined and return_rate["treatment"] > return_rate["control"] + 0.01
def nearest_rank(values: list[float], percentile: float) -> float:
if not values:
raise ValueError("latency sample must not be empty")
percentile = finite_number(percentile)
if percentile is None or not 0 < percentile <= 1:
raise ValueError("percentile must be in (0, 1]")
if any(finite_number(value) is None or value < 0 for value in values):
raise ValueError("latencies must be finite and nonnegative")
rank = max(1, ceil(len(values) * percentile))
return sorted(values)[rank - 1]
baseline_ndcg = sum(ndcg(rows) for rows in baseline_ranked.values()) / len(QUERIES)
candidate_ndcg = sum(ndcg(rows) for rows in candidate_ranked.values()) / len(QUERIES)
max_seller_count = max(
max(Counter(listing.seller_id for listing in rows).values())
for rows in candidate_ranked.values()
)
def parse_utc(value: str) -> datetime:
return utc_time(value)
def outcome_follows_impression(row: dict[str, object]) -> bool:
if not valid_outcome(row):
return False
impression = impression_index.get((row["request_id"], row["product_id"]))
try:
return impression is not None and parse_utc(impression["served_at"]) <= parse_utc(row["occurred_at"])
except (KeyError, ValueError, TypeError, AttributeError):
return False
def valid_impression(row: dict[str, object]) -> bool:
if not isinstance(row, dict) or not required_impression_fields <= row.keys():
return False
string_fields = required_impression_fields - {"position", "slate_size", "position_propensity", "propensity_model_version"}
if any(not nonblank(row[field]) for field in string_fields):
return False
if type(row["position"]) is not int or type(row["slate_size"]) is not int:
return False
if not 1 <= row["position"] <= row["slate_size"]:
return False
arm = row["experiment_arm"]
if arm not in ARM_CONFIG or row["ranker_version"] != ARM_CONFIG[arm][0]:
return False
query = row["query"]
if query not in QUERIES:
return False
expected_slate = ARM_CONFIG[arm][1][query]
if (row["slate_size"] != len(expected_slate)
or row["product_id"] != expected_slate[row["position"] - 1].product_id):
return False
pinned_listing = next((listing for listing in REPLAY_CATALOG if listing.product_id == row["product_id"]), None)
if pinned_listing is None or pinned_listing.query != query or exclusion_reason(pinned_listing) is not None:
return False
versions = {"catalog_snapshot": CATALOG_SNAPSHOT, "eligibility_version": ELIGIBILITY_VERSION,
"candidate_version": CANDIDATE_VERSION, "experiment_id": EXPERIMENT_ID}
if any(row[field] != expected for field, expected in versions.items()):
return False
try:
parse_utc(row["served_at"])
except (ValueError, TypeError, AttributeError):
return False
propensity = row["position_propensity"]
model_version = row["propensity_model_version"]
if propensity is None:
if model_version is not None:
return False
elif finite_number(propensity) is None or not 0 < propensity <= 1 or not nonblank(model_version):
return False
return (row["training_use"] == "evaluation_only"
or (row["training_use"] == "implicit" and propensity is not None
and nonblank(PROPENSITY_MODEL_VERSION) and model_version == PROPENSITY_MODEL_VERSION))
def valid_outcome(row: dict[str, object]) -> bool:
if (not isinstance(row, dict) or not required_outcome_fields <= row.keys()
or any(not nonblank(row[field]) for field in required_outcome_fields)
or row["outcome"] not in {"click", "purchase", "return"}):
return False
try:
parse_utc(row["occurred_at"])
except ValueError:
return False
return True
def complete_slates(rows: list[dict[str, object]]) -> bool:
by_request: dict[str, list[dict]] = {}
for row in rows:
if not valid_impression(row):
return False
by_request.setdefault(row["request_id"], []).append(row)
for slate in by_request.values():
if sorted(row["position"] for row in slate) != list(range(1, len(slate) + 1)):
return False
if len({row["product_id"] for row in slate}) != len(slate):
return False
same_fields = required_impression_fields - {"product_id", "position", "position_propensity"}
if any(len({row[field] for row in slate}) != 1 for field in same_fields):
return False
if slate[0]["slate_size"] != len(slate):
return False
return bool(by_request)
valid_rows = [row for row in impressions if valid_impression(row)]
impression_index = {(row["request_id"], row["product_id"]): row for row in valid_rows}
outcome_event_ids = [row["event_id"] for row in outcomes if valid_outcome(row)]
shopper_arms: dict[str, set[str]] = {}
for row in valid_rows:
shopper_arms.setdefault(row["shopper_id"], set()).add(row["experiment_arm"])
print("baseline_ndcg:", round(baseline_ndcg, 3), "candidate_ndcg:", round(candidate_ndcg, 3))release_gates = {
"candidate_recall_complete": all(candidate_recall(query, retrieved[query]) == 1.0 for query in QUERIES),
"candidate_ndcg_improves": candidate_ndcg > baseline_ndcg,
"ndcg_slices_at_least_0_95": all(ndcg(rows) >= 0.95 for rows in candidate_ranked.values()),
"ranker_p95_latency_ms_at_most_20": nearest_rank(latencies_ms, 0.95) <= 20,
"blocked_items_absent": not blocked_hits,
"impression_schema": bool(impressions) and all(valid_impression(row) for row in impressions),
"slates_complete_and_unique": complete_slates(impressions),
"fixture_requests_retained": (
{row["request_id"] for row in valid_rows} == set(FIXTURE_REQUESTS)
and all((row["shopper_id"], row["query"], row["experiment_arm"], row["served_at"])
== FIXTURE_REQUESTS.get(row["request_id"]) for row in valid_rows)
),
"implicit_feedback_has_propensity": all(
row.get("training_use") != "implicit"
or (
type(row.get("position_propensity")) in (int, float)
and finite_number(row["position_propensity"]) is not None
and 0 < row["position_propensity"] <= 1
and PROPENSITY_MODEL_VERSION is not None
and row.get("propensity_model_version") == PROPENSITY_MODEL_VERSION
and isinstance(row.get("slate_size"), int)
and row["slate_size"] >= 1
)
for row in valid_rows
),
"outcome_schema": bool(outcomes) and all(valid_outcome(row) for row in outcomes),
"outcome_event_ids_unique": len(outcomes) == len(outcome_event_ids) == len(set(outcome_event_ids)),
"fixture_outcomes_retained": set(outcome_event_ids) == {"EVT-900", "EVT-901"},
"outcomes_join_impressions": all(valid_outcome(row) and (row["request_id"], row["product_id"]) in impression_index for row in outcomes),
"outcomes_follow_impressions": all(outcome_follows_impression(row) for row in outcomes),
"one_arm_per_shopper": all(len(arms) == 1 for arms in shopper_arms.values()),
"assignment_matches_hash": bool(impressions) and all(valid_impression(row) and row["experiment_arm"] == assigned_arm(row["shopper_id"]) for row in impressions),
"seller_concentration_at_most_2_per_slate": max_seller_count <= 2,
}
print("release_gates_pass:", all(release_gates.values()))required_gates = (
"candidate_recall_complete", "candidate_ndcg_improves", "ndcg_slices_at_least_0_95",
"ranker_p95_latency_ms_at_most_20", "blocked_items_absent", "impression_schema",
"slates_complete_and_unique", "fixture_requests_retained", "implicit_feedback_has_propensity", "outcome_schema",
"outcome_event_ids_unique", "fixture_outcomes_retained", "outcomes_join_impressions", "outcomes_follow_impressions",
"one_arm_per_shopper", "assignment_matches_hash", "seller_concentration_at_most_2_per_slate",
)
mechanical_pass = all(release_gates.get(name) is True for name in required_gates)
receipt = {
"bundle_id": RANKER_VERSION,
"previous_ranker": PREVIOUS_RANKER,
"catalog_snapshot": CATALOG_SNAPSHOT,
"eligibility_version": ELIGIBILITY_VERSION,
"candidate_version": CANDIDATE_VERSION,
"offline": {
"baseline_ndcg_at_3": round(baseline_ndcg, 3),
"candidate_ndcg_at_3": round(candidate_ndcg, 3),
"ranker_p95_latency_ms": nearest_rank(latencies_ms, 0.95),
"latency_evidence": "synthetic_percentile_fixture_not_runtime_measurement",
"ranking_evidence": "hand_entered_regression_scores",
},
"temporal_fitting_exercise": {
"dataset": "authored_synthetic_temporal-ranking.csv",
"training_requests": len(train), "validation_requests": len(validation), "test_requests": len(test),
"chosen_penalty": chosen_penalty,
"test_baseline_ndcg_at_3": round(test_baseline, 3),
"test_fitted_ndcg_at_3": round(test_candidate, 3),
},
"release_gates": release_gates,
"experiment": {
"experiment_id": EXPERIMENT_ID,
"assignment_unit": "shopper_id",
"control": PREVIOUS_RANKER,
"treatment": RANKER_VERSION,
"allocation": {"control": 0.5, "treatment": 0.5},
"minimum_eligible_searches": None, # calculate power from actual traffic
"primary_metric": "purchase_conversion_per_search",
"guardrails": ["returns_per_purchase", "search_p95_latency_ms", "blocked_listing_impressions", "seller_concentration_at_3"],
"stop_conditions": [
"blocked_listing_impressions > 0",
"search_p95_latency_ms > 120",
"returns_per_purchase > control + 0.01",
],
},
"illustrative_online_review": {
"status": ("hold_undefined_return_comparison" if not return_comparison_defined else
"stop_treatment_return_guardrail" if stop_returns else "review_under_predeclared_plan"),
"control_conversion": conversion["control"],
"treatment_conversion": conversion["treatment"],
"control_return_rate": return_rate["control"],
"treatment_return_rate": return_rate["treatment"],
"stop_returns": stop_returns,
"release_evidence": False,
},
"candidate_decision": "fixture_checks_pass" if mechanical_pass else "hold",
"live_experiment_ready": False,
"missing_live_evidence": ["representative judgments", "measured serving latency", "powered experiment plan", "tested rollback",
"render acknowledgments", "finalized attributed outcomes"],
}
print("candidate_decision:", receipt["candidate_decision"])
print("live_experiment_ready:", receipt["live_experiment_ready"])print("control slate:", [row["product_id"] for row in control_impressions])
print("treatment slate:", [row["product_id"] for row in treatment_impressions])
print("offline:", receipt["offline"])
print("release_gates_pass:", mechanical_pass)
print("online_review:", receipt["illustrative_online_review"]["status"])
print("candidate_decision:", receipt["candidate_decision"])
print("live_experiment_ready:", receipt["live_experiment_ready"])control slate: ['P4', 'P5', 'P6']
treatment slate: ['P5', 'P6', 'P4']
offline: {'baseline_ndcg_at_3': 0.854, 'candidate_ndcg_at_3': 1.0, 'ranker_p95_latency_ms': 18, 'latency_evidence': 'synthetic_percentile_fixture_not_runtime_measurement', 'ranking_evidence': 'hand_entered_regression_scores'}
release_gates_pass: True
online_review: stop_treatment_return_guardrail
candidate_decision: fixture_checks_pass
live_experiment_ready: Falsefixture_checks_pass means these authored examples passed their mechanical checks. live_experiment_ready remains false. A synthetic success can't authorize real exposure, and a previous-version string doesn't prove that rollback was tested.
The log checks bind each product and position to the configured fixture slate and retained catalog snapshot. They also require both named fixture requests and both named outcome events, so deleting one can't turn this report green. Those IDs define this exercise's coverage, not a general rule that every search must receive an outcome. The 0.95 NDCG threshold and two-listing seller cap are exercise settings. They aren't universal launch criteria. The evaluation_only tag doesn't enforce access control; a downstream training pipeline must enforce the permitted use and validate its feedback assumptions.
The local 20 millisecond threshold is exercised with synthetic scoring samples; no timer produced the reported 18 ms p95. A real benchmark needs warmup, representative candidate counts, concurrency, hardware/runtime details, and repeated end-to-end measurements. The draft experiment's 120 millisecond stop applies to full search latency, not scoring alone. The tiny online table is stop-rule practice, not live evidence:
| Arm | Searches | Purchases | Conversion | Returns / purchase |
|---|---|---|---|---|
| control | 100 | 4 | 0.04 | 0.00 |
| treatment | 100 | 6 | 0.06 | 0.17 |
Treatment's point estimate rises, but the illustrative conservative stop rule still fires: one return out of six purchases is about 0.167, above zero plus 0.01. These counts are authored separately, not aggregated from the six response rows or two outcome records. That's a reason to stop under this chosen rule, not statistically reliable evidence of a population increase. Return outcomes need comparable follow-up windows. With zero purchases, returns per purchase is undefined; the code reports None and holds the comparison. A real experiment must distinguish immediate safety stops from statistically designed business-metric monitoring.
Conversion doesn't measure whether buyers keep their purchases or whether eligible sellers receive useful exposure. Specify which failures should stop treatment and what recovery entails:
- Serving failures: The draft plan treats a blocked listing or full-search p95 above 120 ms as stop conditions. Detection, alert latency, traffic control, and recovery remain to be implemented and tested.
- Outcome monitoring: This fictional rule stops at a one-percentage-point return-rate increase. A real monitoring rule needs its denominator, follow-up window, uncertainty, and stopping procedure specified in advance.
- Seller concentration: The fixture checks at most two listings per seller in each three-item treatment slate. This limits concentration under that particular rule; it doesn't establish fairness or legal compliance.
Recovery requires loading a compatible prior ranker and feature bundle, stopping or rerouting treatment traffic, and verifying the served version and current eligibility checks. A pointer or ranker name alone isn't recovery. Already displayed products and recorded outcomes remain part of the experiment history.
Turn the exercise into a portfolio project
Replace the authored CSV with a documented, permissible dataset and independently collected judgments. Keep the text-match baseline. Freeze a time-based train/validation/test protocol before tuning, save the feature definitions and fitted weights together, and report per-query or category results with uncertainty. Add cold-start and low-traffic slices instead of letting popular queries dominate every conclusion.
Serve one ranked response from a pinned catalog snapshot, then change stock between retrieval and display and demonstrate the fail-closed or refill behavior. Save one full candidate/feature snapshot, one displayed slate, and later deduplicated outcomes. Demonstrate that restoring the previous model also restores its compatible feature code, without restoring an obsolete eligibility rule.
Benchmark the actual serving path and write an experiment plan with defensible sample size and stop rules. The project is ready for review when another engineer can reproduce the held-out comparison, explain why an excluded product stayed excluded, and run the rollback exercise. Claim only the evidence you collected.
Practice: break the ranking contract
Try these failures that can make a ranking report look healthier than the system is. Run from the first cell after each change so cached intermediate values don't hide the mutation.
Lower P3's retrieval score to 0.10 and retrieve only two candidates. How do recall and the two NDCG denominators respond? Then restore retrieval and demote P6's rank score to 0.20. Is that the same failure?
Answer
Missing P3 lowers recall to 0.5 and fixed-universe NDCG@3 to about 0.812; ideal ordering of the survivors still scores 1.0 conditionally. Demoting P6 changes ordering instead: recall stays complete and NDCG falls to about 0.972, still above the 0.95 gate. Missing candidates and imperfect ordering need different checks.
Move PENDING-4's label availability to May 4 at 18:00. How should training change? What if you instead move a feature's availability after serving?
Answer
Training now contains four requests instead of three because those labels were available at the freeze. A feature available only after serving must fail loading: it couldn't have been used for that prediction. The first change alters a valid historical training set; the second introduces future information.
Duplicate EVT-900, then change the duplicate's product to P9. Which two problems must outcome ingestion distinguish?
Answer
An identical repeated event can be deduplicated; the same event ID with a conflicting payload must be quarantined. Independently, P9 fails the displayed-product join for REQ-501. Neither error changes offline judgments, but both would corrupt later behavioral evidence.
In the IPS exercise, keep relevance at 0.50 but lower the weight cap from 2 to 1. What are the two expected weighted-click values? Can either recover an item the logging policy never shows?
Answer
Both weights become 1, so the expectations are the raw click probabilities: 0.40 at position 1 and 0.10 at position 3. The cap no longer corrects this toy attention difference. No finite weight creates feedback for an item with zero exposure support. Extra judgments or a suitable exploration policy would supply new evidence, rather than reweighting missing observations.
For an additional deployment check, mark P5 out of stock after ranking and before calling impression_rows(). The display-time recheck must stop the slate. Re-running the ranker with stale candidates isn't a substitute for refreshing eligibility.