Capstone: Product Ranking

The ETA project produced one guarded prediction for one parcel. This project makes a decision that shapes its own future data: which products shoppers see near the top of search results.

Your task is to ship a ranking candidate for searches such as insulated delivery bag. It may reorder eligible, in-stock listings. It may not surface blocked listings or call clicks an unbiased truth signal.

Specify the Ranking Surface

Define the scope tightly:

Candidate generation and reranking are separate artifacts. The candidate component must retrieve relevant items reliably. The ranker can only adjust order inside that eligible set.

Learning-to-rank methods can learn pairwise preferences so a relevant product receives a higher score than a less relevant candidate.^{[1]Reference 1Learning to Rank using Gradient Descent.https://www.microsoft.com/en-us/research/publication/learning-to-rank-using-gradient-descent/} That modeling detail doesn't excuse missing policy: eligibility filtering must run before scoring, and its version belongs in every impression log.

Build offline evidence first

Create a judged fixture set. Each query has eligible candidates and graded relevance from human review:

Required offline checks:

Filter policy before retrieval

Start with the boundary that a model can't learn safely: listing eligibility. A prohibited or sold-out product may have an excellent text match and a high learned score. It still can't reach the ranker.

The compact fixture below keeps two judged searches in memory. Production might retrieve 100 candidates per query; this local receipt uses 3 so each listing stays visible.

Products P9 and P8 have the strongest scores for their searches, but policy excludes them. Product P10 is approved but out of stock. Product P11 can't ship to the shopper's region. Filtering first prevents the ranker from treating a business invariant as a preference it can trade away.

Measure Retrieval and Ordering Separately

Candidate recall asks whether relevant eligible products reached the ranker. Normalized discounted cumulative gain (NDCG) then asks whether stronger judgments appear near the top of the returned list. A ranker can't repair a relevant product that retrieval dropped.

The next cell reranks each eligible candidate set and compares baseline and candidate NDCG. It also rechecks the invariant after scoring.

The candidate improves ordering for both query fixtures. The metric helpers also make two edge policies explicit: a query with no relevant eligible judgments is a fixture error for this capstone, while NDCG returns 0.0 when a ranked set has no positive gain. This still isn't a launch decision. Offline judgments support a controlled experiment, and the displayed slate creates the future data used to evaluate or retrain the system.

Log exposure before learning from outcomes

Every displayed slate should log:

Clicks alone aren't reliable targets: top-ranked items receive attention because they are top-ranked. Joachims, Swaminathan, and Schnabel show why position bias makes direct click training suboptimal and derive a counterfactual correction for biased implicit feedback.^{[2]Reference 2Unbiased Learning-to-Rank with Biased Feedbackhttps://arxiv.org/abs/1608.04468} Use judged offline sets for regression detection, then use a controlled A/B experiment to compare user outcomes. Kohavi et al. describe online experimentation as a disciplined way to measure product changes.^{[3]Reference 3Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/}

Clicks, purchases, and returns arrive later. Append them as outcome events with event_id, occurred_at, request_id, product_id, and outcome. Don't overwrite the served impression row. An A/B plan should state allocation, duration or sample-size rule, primary metric, latency and return-rate guardrails, prohibited-listing invariant, and stop conditions. A conversion lift that increases returns or policy violations isn't a successful ranking release.

The final cell publishes replayable impression rows, illustrative joined outcome events, and one candidate receipt. It doesn't invent live lift. It proves offline evidence, exposure schema, stable assignment, outcome join, and rollback pointer are ready before traffic begins.

candidate_for_ab_test is intentionally narrower than launch approval. The local 20 millisecond gate measures ranker scoring only; the experiment's 120 millisecond stop applies to end-to-end search latency. The receipt says which ranker deserves controlled exposure, which immutable logs make later outcomes interpretable, and which previous alias remains available if guardrails fail.

Submission checklist

Practice: break the ranking contract

Use the runnable examples as a release harness. Change one condition at a time, predict the result, then rerun the examples.

1. Change P3 retrieval score from 0.84 to 0.10, then change retrieve() default budget from 3 to 2. Which metric exposes missing relevant product?
2. Set P9.policy_approved to True. Why isn't that a harmless relevance change?
3. Change candidate score for P6 from 0.84 to 0.20. Which NDCG slice fails?
4. Remove position from impression_rows(). Which receipt gate blocks experiment readiness?
5. Change all three eligible insulated-bag listings to seller S1. Which marketplace guardrail fails?
6. Use SHOP-100 for both control and treatment impressions. Which assignment gate fails?
7. Change first outcome's product_id from P5 to P9. Which replay gate fails?
8. Append a retry with the same event_id as EVT-900. Which deduplication gate fails?

Practice answer sketches

Mastery check

Evaluation rubric

Common failures