Capstone: Product Ranking

A search ranker makes a decision that shapes its own future data: which products shoppers see near the top of results. Clicks and purchases then reflect that exposure, so they can't be treated as unbiased relevance labels.

Your task is to ship a ranking candidate for searches such as insulated delivery bag. It may reorder eligible, in-stock listings. It may not surface blocked listings or call clicks an unbiased truth signal.

Specify the Ranking Surface

Define the scope tightly:

Candidate generation and reranking are separate artifacts. The candidate component must retrieve relevant items reliably. The ranker can only adjust order inside that eligible set.

Learning-to-rank methods can learn pairwise preferences so a relevant product receives a higher score than a less relevant candidate.^{[1]Reference 1Learning to Rank using Gradient Descent.https://www.microsoft.com/en-us/research/publication/learning-to-rank-using-gradient-descent/} That modeling detail doesn't excuse missing policy: eligibility filtering must run before scoring, and its version belongs in every impression log.

Build offline evidence first

Create a judged fixture set. Each query has eligible candidates and graded relevance from human review:

Offline gates in this lesson score NDCG@3 / recall@3 on the small judged fixture. Production contracts often pin NDCG@10 and recall@100; keep the K explicit so the receipt and the surface table match.

Required offline checks:

Filter policy before retrieval

Start with the boundary that a model can't learn safely: listing eligibility. A prohibited or sold-out product may have an excellent text match and a high learned score. It still can't reach the ranker.

The compact fixture below keeps two judged searches in memory. Production might retrieve 100 candidates per query; this local receipt uses 3 so each listing stays visible.

Products P9 and P8 have the strongest scores for their searches, but policy excludes them. Product P10 is approved but out of stock. Product P11 can't ship to the shopper's region. Filtering first prevents the ranker from treating a business invariant as a preference it can trade away.

Measure Retrieval and Ordering Separately

Candidate recall asks whether relevant eligible products reached the ranker. Normalized discounted cumulative gain (NDCG) then asks whether stronger judgments appear near the top of the returned list. A ranker can't repair a relevant product that retrieval dropped.

The next cell reranks each eligible candidate set and compares baseline and candidate NDCG. It also rechecks the invariant after scoring.

The candidate improves ordering for both query fixtures. The metric helpers also make two edge policies explicit: a query with no relevant eligible judgments is a fixture error for this capstone, while NDCG returns 0.0 when a ranked set has no positive gain. This still isn't a launch decision. Offline judgments support a controlled experiment, and the displayed slate creates the future data used to evaluate or retrain the system.

Log exposure before learning from outcomes

Every displayed slate should log:

Clicks alone aren't reliable targets: top-ranked items receive attention because they are top-ranked. Joachims, Swaminathan, and Schnabel show why position bias makes direct click training suboptimal and derive a counterfactual correction for biased implicit feedback.^{[2]Reference 2Unbiased Learning-to-Rank with Biased Feedbackhttps://arxiv.org/abs/1608.04468} Use judged offline sets for regression detection, then use a controlled A/B experiment to compare user outcomes. Kohavi et al. describe online experimentation as a disciplined way to measure product changes.^{[3]Reference 3Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/} When training_use=implicit, refuse to train if impression rows lack the actual randomized action probability or a calibrated examination propensity with a pinned model version. A convenient formula such as 1 / position isn't a measured propensity.

Clicks, purchases, and returns arrive later. Append them as outcome events with event_id, occurred_at, request_id, product_id, and outcome. Don't overwrite the served impression row. An A/B plan should state allocation, duration or sample-size rule, primary metric, latency and return-rate guardrails, prohibited-listing invariant, and stop conditions. A conversion lift that increases returns or policy violations isn't a successful ranking release.

The final cell publishes replayable impression rows, illustrative joined outcome events, and one candidate receipt. It doesn't invent live lift. It proves offline evidence, exposure schema, stable assignment, outcome join, and rollback pointer are ready before traffic begins.

candidate_for_ab_test is intentionally narrower than launch approval. A local 20 millisecond gate measures ranker scoring only; the experiment's 120 millisecond stop applies to end-to-end search latency. Its tiny online table is stop-rule practice, not live release evidence: treatment conversion rises, but the return-rate guardrail fires. This receipt says which ranker deserves controlled exposure, which immutable logs make later outcomes interpretable, and which previous alias remains available if guardrails fail.

Submission checklist

Practice: break the ranking contract

Use the runnable examples as a release harness. Change one condition at a time, predict the result, then rerun the examples.

1. Change P3 retrieval score from 0.84 to 0.10, then change retrieve() default budget from 3 to 2. Which metric exposes missing relevant product?
2. Set P9.policy_approved to True. Why isn't that a harmless relevance change?
3. Change candidate score for P6 from 0.84 to 0.20. Which NDCG slice fails?
4. Remove position from impression_rows(). Which receipt gate blocks experiment readiness?
5. Change all three eligible insulated-bag listings to seller S1. Which marketplace guardrail fails?
6. Use SHOP-100 for both control and treatment impressions. Which assignment gate fails?
7. Change first outcome's product_id from P5 to P9. Which replay gate fails?
8. Append a retry with the same event_id as EVT-900. Which deduplication gate fails?

Practice answer sketches

Which metric catches dropped P3 after retrieval budget falls to 2?

Answer

candidate_recall_complete fails because the ranker never receives one relevant eligible product. NDCG on surviving candidates can't repair that omission.

Why isn't approving P9 a harmless relevance edit?

Answer

P9 was excluded by marketplace policy. Changing that field changes eligibility contract, not ranking quality, and requires its own reviewed policy release.

What happens when P6 candidate score falls from 0.84 to 0.20?

Answer

label-printer no longer receives ideal ordering, so ndcg_slices_at_least_0_95 fails.

Which gate fails when impression rows drop position?

Answer

impression_schema fails. Without displayed position, later click and purchase data can't be interpreted against exposure.

Which gate fails when three products in one displayed slate share seller S1?

Answer

seller_concentration_at_most_2_per_slate fails, blocking an experiment that would crowd too much exposure into one seller.

Which gate fails when SHOP-100 appears in both experiment arms?

Answer

one_arm_per_shopper fails. A shopper-level experiment must keep each shopper in one arm so treatment effects stay interpretable.

Which gate fails when outcome product P9 was never displayed for REQ-501?

Answer

outcomes_join_impressions fails. A click or purchase can't become ranking feedback unless it joins a product exposed by the same request.

Which gate fails when a retry reuses EVT-900?

Answer

outcome_event_ids_unique fails. Event IDs distinguish one outcome from a replayed delivery, so a duplicate must be deduplicated before training or experiment analysis.