Capstone: Product Ranking

A shopper types label-printer. The ranker decides which products sit near the top, and those positions then collect the clicks you'll be tempted to treat as labels.

Capstone: Delivery ETA Prediction shipped a delay warning from time-safe features and a receipt that only qualified shadow traffic. A search ranker is a different kind of service. Ranking and Recommendation Systems already separated eligibility, candidate recall, and NDCG on a docs corpus. Package that funnel into a marketplace artifact. Stock, region, and policy have to win before any learned score. Impressions have to freeze before outcomes arrive. An offline NDCG win still isn't a launch. Ship a candidate that may reorder eligible, in-stock listings, and that may not surface blocked listings or call clicks an unbiased truth signal.

Specify the ranking surface

Pin the product before you pick a model:

Candidate generation and reranking stay separate artifacts. The candidate component has to recover relevant eligible items. The ranker can only change order inside that set.

The earlier ranking lesson used pairwise preferences the way RankNet does: a relevant product should outscore a weaker candidate.^{[1]Reference 1Learning to Rank using Gradient Descent.https://www.microsoft.com/en-us/research/publication/learning-to-rank-using-gradient-descent/} That modeling choice still doesn't let a learned score override eligibility. Filter first, then score, and pin the eligibility version on every impression log.

Build offline evidence first

A live shopper log can't grade that contract. Start with a judged fixture: each query has eligible candidates and graded relevance from human review.

Offline gates in this lesson score NDCG@3 / recall@3 on the small judged fixture. Production contracts often pin NDCG@10 and recall@100; keep the K explicit so the receipt and the surface table match.

Required offline checks:

Those gates only mean something if blocked listings never enter scoring. Start there.

Filter policy before retrieval

A prohibited or sold-out product can still have an excellent text match and a high learned score. It still can't reach the ranker. Eligibility is the boundary a model can't learn safely from clicks.

The compact fixture below keeps two judged searches in memory. Production might retrieve 100 candidates per query; this local receipt uses 3 so each listing stays visible.

Products P9 and P8 have the strongest scores for their searches, but policy excludes them. P8 is also out of stock; exclusion_reason() logs the first failing check, so the receipt shows policy_block. Product P10 is approved but sold out. Product P11 can't ship to the shopper's region. Filtering first stops the ranker from treating a business invariant as a preference it can trade away.

Measure retrieval and ordering separately

Candidate recall asks whether relevant eligible products reached the ranker. NDCG then asks whether stronger judgments sit near the top of the returned list. A ranker can't repair a relevant product that retrieval dropped.

Use the same exponential-gain form as the ranking chapter. At cutoff $K$, discounted cumulative gain is

$\mathrm{DCG}@K = \sum_{i=1}^{K} \frac{2^{rel_i}-1}{\log_2(i+1)}$

where $rel_i$ is the judged grade at 1-based position $i$. Position 1 has $\log_2 2 = 1$, so it keeps full weight. NDCG divides by the DCG of the ideal ordering of that same candidate set. Manning, Raghavan, and Schütze document this exponential-gain form; some libraries use linear gain $rel_i / \log_2(i+1)$ instead, so the eval setup has to pin one.^{[2]Reference 2Introduction to Information Retrieval.https://nlp.stanford.edu/IR-book/} In the helper below, enumerate is 0-based, so the denominator is log2(rank + 2), which equals $\log_2(i+1)$.

Work label-printer by hand before looking at code. Control shows P4, P5, P6 with grades $1$, $3$, and $2$:

$\mathrm{DCG} = \frac{1}{1} + \frac{7}{\log_2 3} + \frac{3}{2} \approx 1 + 4.417 + 1.500 = 6.917$

The ideal order is P5, P6, P4 with grades $3$, $2$, and $1$:

$\mathrm{IDCG} = 7 + \frac{3}{\log_2 3} + 0.5 \approx 7 + 1.893 + 0.500 = 9.393$

So control NDCG@3 is $6.917 / 9.393 \approx 0.736$. Treatment is already the ideal order, so its NDCG@3 is $1.000$. Recall@3 is still $1.000$ on both arms because P5 and P6 (the eligible grades $\ge 2$) are in the candidate set either way. Ordering quality moved; retrieval didn't.

The next cells rerank each eligible set, print those NDCG values, and recheck that scoring didn't sneak a blocked listing back in.

The candidate improves ordering on both fixture queries. Two edge policies stay explicit: a query with no relevant eligible judgments is a fixture error here, and NDCG returns 0.0 when a ranked set has no positive gain. That still isn't a launch decision. Offline judgments only qualify a controlled experiment, and the displayed slate is the data you'll later be tempted to train on.

Log exposure before learning from outcomes

If that slate is going to become training data, freeze what was shown before any click arrives.

Every displayed slate should log:

Clicks alone aren't reliable targets. Top-ranked items get attention because they're top-ranked. That's position bias. Joachims, Swaminathan, and Schnabel show that using raw clicks as learning-to-rank labels is biased, and they derive a propensity-weighted IPS correction for that implicit feedback.^{[3]Reference 3Unbiased Learning-to-Rank with Biased Feedbackhttps://arxiv.org/abs/1608.04468} Their method still needs a real propensity model, often from a small randomized logging study, not a made-up curve. A formula such as 1 / position isn't a measured propensity. When training_use=implicit, refuse to train unless the impression row carries the actual randomized action probability or a calibrated examination propensity with a pinned model version.

Use the judged fixture to catch regressions. Use a controlled A/B experiment to compare shopper outcomes. Experiment Design and A/B Testing already made the assignment unit, primary metric, and guardrails explicit; Kohavi, Tang, and Xu treat that online comparison as the way to measure a product change.^{[4]Reference 4Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/} Keep each shopper in one arm. A conversion lift that raises returns or policy violations isn't a successful ranking release.

Clicks, purchases, and returns arrive later. Append them as outcome events with event_id, occurred_at, request_id, product_id, and outcome. Don't overwrite the served impression row.

The remaining cells publish replayable impression rows, two joined outcome events, and one candidate receipt. They don't invent live lift. They prove offline evidence, exposure schema, stable assignment, outcome join, and a rollback pointer are ready before traffic starts.

candidate_for_ab_test is narrower than launch approval. The local 20 millisecond gate measures ranker scoring only; the experiment's 120 millisecond stop applies to end-to-end search latency. The tiny online table is stop-rule practice, not live evidence:

Treatment conversion rises, and the return-rate guardrail still fires (0.17 is far above control plus 0.01). The receipt says which ranker deserves controlled exposure, which immutable logs make later outcomes interpretable, and which previous alias stays available if a guardrail fails.

Submission checklist

Practice: break the ranking contract

Use the runnable examples as a verification testbed. Change one condition at a time, predict the result, then rerun the examples.

1. Change P3 retrieval score from 0.84 to 0.10, then change retrieve() default budget from 3 to 2. Which metric exposes missing relevant product?
2. Set P9.policy_approved to True. Why isn't that a harmless relevance change?
3. Change candidate score for P6 from 0.84 to 0.20. Does ndcg_slices_at_least_0_95 fail?
4. Remove position from impression_rows(). Which receipt gate blocks experiment readiness?
5. Change all three eligible insulated-bag listings to seller S1. Which marketplace guardrail fails?
6. Use SHOP-100 for both control and treatment impressions. Which assignment gate fails?
7. Change first outcome's product_id from P5 to P9. Which replay gate fails?
8. Append a retry with the same event_id as EVT-900. Which deduplication gate fails?

Practice answer sketches

Which metric catches dropped P3 after retrieval budget falls to 2?

Answer

candidate_recall_complete fails because the ranker never receives one relevant eligible product. NDCG on surviving candidates can't repair that omission.

Why isn't approving P9 a harmless relevance edit?

Answer

P9 was excluded by marketplace policy. Changing that field changes eligibility contract, not ranking quality, and requires its own reviewed policy release.

What happens when P6 candidate score falls from 0.84 to 0.20?

Answer

label-printer is no longer ideal (P5, P4, P6 instead of P5, P6, P4), but NDCG@3 only falls to about 0.972, which still clears 0.95. A worse-than-ideal list isn't the same as a failed slice gate. Demoting relevance-3 P5 to the bottom is what drops the slice below 0.95.

Which gate fails when impression rows drop position?

Answer

impression_schema fails. Without displayed position, later click and purchase data can't be interpreted against exposure.

Which gate fails when three products in one displayed slate share seller S1?

Answer

seller_concentration_at_most_2_per_slate fails, blocking an experiment that would crowd too much exposure into one seller.

Which gate fails when SHOP-100 appears in both experiment arms?

Answer

one_arm_per_shopper fails. A shopper-level experiment must keep each shopper in one arm so treatment effects stay interpretable.

Which gate fails when outcome product P9 was never displayed for REQ-501?

Answer

outcomes_join_impressions fails. A click or purchase can't become ranking feedback unless it joins a product exposed by the same request.

Which gate fails when a retry reuses EVT-900?

Answer

outcome_event_ids_unique fails. Event IDs distinguish one outcome from a replayed delivery, so a duplicate must be deduplicated before training or experiment analysis.