LearnPortfolio CapstonesCapstone: Product Ranking

📊HardEvaluation & Benchmarks

Capstone: Product Ranking

Ship a marketplace ranking candidate with eligible retrieval, NDCG evidence, exposure logging, and A/B guardrails.

9 min read

Learning path

Step 77 of 151 in the full curriculum

Capstone: Delivery ETA Prediction Capstone: Demand Forecasting

Capstone: Product Ranking

The ETA project produced one guarded prediction for one parcel. This project makes a decision that shapes its own future data: which products shoppers see near the top of search results.

Your task is to ship a ranking candidate for searches such as insulated delivery bag. It may reorder eligible, in-stock listings. It may not surface blocked listings, disguise sponsored placement, or call clicks an unbiased truth signal.

Product-ranking capstone showing eligible catalog filtering, candidate retrieval, learned ordering, impression logging, offline relevance judging, and guarded online experiment. — The ranker earns exposure only after offline ordering evidence and an impression schema make future feedback measurable rather than self-confirming.

Specify the Ranking Surface

Define the scope tightly:

Contract	Decision
surface	marketplace search results
query set	frozen judged search queries plus online experiment traffic
eligibility	in-stock, deliverable region, policy-approved listing
offline metric	candidate recall@100 and NDCG@10
online primary metric	purchase conversion per search
guardrails	returns, latency, unsafe listings, seller concentration

Candidate generation and reranking are separate artifacts. The candidate component must retrieve relevant items reliably. The ranker can only adjust order inside that eligible set.

Learning-to-rank methods can learn pairwise preferences so a relevant product receives a higher score than a less relevant candidate.^[1] That modeling detail doesn't excuse missing policy: eligibility filtering must run before scoring, and its version belongs in every impression log.

Diagram showing Catalog snapshot inventory + policy, Eligible candidates recall@100, Ranker v1 NDCG@10, and Displayed slate + impression log purchases and returns. — Catalog snapshot inventory + policy, Eligible candidates recall@100, Ranker v1 NDCG@10, and Displayed slate + impression log purchases and returns.

Build Offline Evidence First

Create a judged fixture set. Each query has eligible candidates and graded relevance from human review:

text

ranking-product/
  data/
    catalog_snapshot.jsonl
    judged_queries.jsonl
    eligibility_policy.json
  retrieval/
    candidate_generator.py
  ranking/
    ranker.py
    evaluate_ndcg.py
  experiments/
    impression_schema.json
    ab_plan.md
  tests/
    test_blocked_listing_never_surfaces.py
    test_ndcg_regression_gate.py

Required offline checks:

Check	Why it blocks release
eligible-only results	a relevant prohibited listing still can't display
candidate recall@100	the ranker can't repair missing products
NDCG@10 by query category	overall improvement may hide poor critical categories
scoring latency	a slower list damages shopping experience
diversity/seller concentration	one feedback-rich seller shouldn't crowd out catalog

Make the Ranking Gate Runnable

This compact evaluator ensures the candidate improves ordering for two queries while never surfacing a blocked product. In the full artifact, judgments would live in versioned data files.

ranking-release-gate.py

from math import log2

queries = [
    {"id": "insulated-bag", "blocked": {"P9"}, "baseline": [("P1", 3), ("P2", 1), ("P3", 2)], "candidate": [("P1", 3), ("P3", 2), ("P2", 1)]},
    {"id": "label-printer", "blocked": {"P8"}, "baseline": [("P4", 1), ("P5", 3), ("P6", 2)], "candidate": [("P5", 3), ("P6", 2), ("P4", 1)]},
]

def dcg(rows):
    return sum((2**rel - 1) / log2(rank + 2) for rank, (_, rel) in enumerate(rows))

def ndcg(rows):
    ideal = sorted(rows, key=lambda row: row[1], reverse=True)
    return dcg(rows) / dcg(ideal)

baseline = sum(ndcg(query["baseline"]) for query in queries) / len(queries)
candidate = sum(ndcg(query["candidate"]) for query in queries) / len(queries)
blocked_hits = [product for query in queries for product, _ in query["candidate"] if product in query["blocked"]]

decision = "eligible_for_ab_review" if candidate > baseline and not blocked_hits else "hold"
print("baseline:", round(baseline, 3))
print("candidate:", round(candidate, 3))
print("blocked hits:", blocked_hits)
print("decision:", decision)

Output

baseline: 0.854
candidate: 1.0
blocked hits: []
decision: eligible_for_ab_review

This decision is intentionally limited. Offline judged queries support an online experiment, not broad promotion. Live users may respond differently from judges, and new ranking exposure changes the future outcome data.

Log Exposure Before Learning From Outcomes

Every displayed slate should log:

Field	Why
`request_id`, `query`	group one decision
`catalog_snapshot`, `eligibility_version`	show available choices
`candidate_version`, `ranker_version`	identify scoring path
`product_id`, `position`	preserve exposure
`clicked`, `purchased`, `returned`	distinguish shallow and useful outcomes
experiment arm	estimate treatment effect

Clicks alone are not reliable targets: top-ranked items receive attention because they are top-ranked. Use judged offline sets for regression detection, and use a controlled A/B experiment to compare user outcomes. Kohavi et al. describe online experimentation as a disciplined way to measure changes while monitoring guardrails and unexpected harm.^[2]

An A/B plan should state allocation, duration or sample-size rule, primary metric, latency and return-rate guardrails, prohibited-listing invariant, and stop conditions. A conversion lift that increases returns or policy violations is not a successful ranking release.

Submission Checklist

Artifact	Evidence
eligibility policy	blocked items can't enter ranking
candidate evaluation	recall measured before reranking
ranker evaluation	NDCG slices and latency recorded
impression schema	positions and versions logged
experiment plan	metrics, guardrails, stop rules
rollback	stable ranker alias remains available

Mastery Check

Evaluation rubric

Artifact	Strong submission demonstrates
ranking stack	explicit eligibility, candidate generation, reranking inputs, and deterministic fallback
evaluation	NDCG-style offline result with policy guardrails and slice review
experiment plan	exposure logging, primary metric, guardrails, stopping rule, and rollback

Common Failures

Symptom	Cause	Fix
Great NDCG while prohibited item appears	ranking runs before eligibility	filter first and test invariant
New model increasingly favors former winners	clicks used without exposure evidence	log impressions and experiment arms
Conversion increases while customer value falls	returns absent from guardrails	gate promotion on downstream outcomes

Next Step

Continue to Capstone: Demand Forecasting

You can now ship a measured ranking surface whose outputs influence future data. Next you will ship time-ordered forecasts and alerts for inventory and warehouse capacity.

PreviousCapstone: Delivery ETA Prediction

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Learning to Rank using Gradient Descent.

Burges, C. J. C., et al. · 2005 · ICML 2005

Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.

Kohavi, R., Tang, D., Xu, Y. · 2020

Back to Topics

LearnPortfolio CapstonesCapstone: Product Ranking

📊HardEvaluation & Benchmarks

Capstone: Product Ranking

Ship a marketplace ranking candidate with eligible retrieval, NDCG evidence, exposure logging, and A/B guardrails.

9 min read

Learning path

Step 77 of 151 in the full curriculum

Capstone: Delivery ETA Prediction Capstone: Demand Forecasting

Capstone: Product Ranking

The ETA project produced one guarded prediction for one parcel. This project makes a decision that shapes its own future data: which products shoppers see near the top of search results.

Specify the Ranking Surface

Define the scope tightly:

Contract	Decision
surface	marketplace search results
query set	frozen judged search queries plus online experiment traffic
eligibility	in-stock, deliverable region, policy-approved listing
offline metric	candidate recall@100 and NDCG@10
online primary metric	purchase conversion per search
guardrails	returns, latency, unsafe listings, seller concentration

Candidate generation and reranking are separate artifacts. The candidate component must retrieve relevant items reliably. The ranker can only adjust order inside that eligible set.

Build Offline Evidence First

Create a judged fixture set. Each query has eligible candidates and graded relevance from human review:

text

ranking-product/
  data/
    catalog_snapshot.jsonl
    judged_queries.jsonl
    eligibility_policy.json
  retrieval/
    candidate_generator.py
  ranking/
    ranker.py
    evaluate_ndcg.py
  experiments/
    impression_schema.json
    ab_plan.md
  tests/
    test_blocked_listing_never_surfaces.py
    test_ndcg_regression_gate.py

Required offline checks:

Check	Why it blocks release
eligible-only results	a relevant prohibited listing still can't display
candidate recall@100	the ranker can't repair missing products
NDCG@10 by query category	overall improvement may hide poor critical categories
scoring latency	a slower list damages shopping experience
diversity/seller concentration	one feedback-rich seller shouldn't crowd out catalog

Make the Ranking Gate Runnable

This compact evaluator ensures the candidate improves ordering for two queries while never surfacing a blocked product. In the full artifact, judgments would live in versioned data files.

ranking-release-gate.py

from math import log2

queries = [
    {"id": "insulated-bag", "blocked": {"P9"}, "baseline": [("P1", 3), ("P2", 1), ("P3", 2)], "candidate": [("P1", 3), ("P3", 2), ("P2", 1)]},
    {"id": "label-printer", "blocked": {"P8"}, "baseline": [("P4", 1), ("P5", 3), ("P6", 2)], "candidate": [("P5", 3), ("P6", 2), ("P4", 1)]},
]

def dcg(rows):
    return sum((2**rel - 1) / log2(rank + 2) for rank, (_, rel) in enumerate(rows))

def ndcg(rows):
    ideal = sorted(rows, key=lambda row: row[1], reverse=True)
    return dcg(rows) / dcg(ideal)

baseline = sum(ndcg(query["baseline"]) for query in queries) / len(queries)
candidate = sum(ndcg(query["candidate"]) for query in queries) / len(queries)
blocked_hits = [product for query in queries for product, _ in query["candidate"] if product in query["blocked"]]

decision = "eligible_for_ab_review" if candidate > baseline and not blocked_hits else "hold"
print("baseline:", round(baseline, 3))
print("candidate:", round(candidate, 3))
print("blocked hits:", blocked_hits)
print("decision:", decision)

Output

baseline: 0.854
candidate: 1.0
blocked hits: []
decision: eligible_for_ab_review

Log Exposure Before Learning From Outcomes

Every displayed slate should log:

Field	Why
`request_id`, `query`	group one decision
`catalog_snapshot`, `eligibility_version`	show available choices
`candidate_version`, `ranker_version`	identify scoring path
`product_id`, `position`	preserve exposure
`clicked`, `purchased`, `returned`	distinguish shallow and useful outcomes
experiment arm	estimate treatment effect

Submission Checklist

Artifact	Evidence
eligibility policy	blocked items can't enter ranking
candidate evaluation	recall measured before reranking
ranker evaluation	NDCG slices and latency recorded
impression schema	positions and versions logged
experiment plan	metrics, guardrails, stop rules
rollback	stable ranker alias remains available

Mastery Check

Evaluation rubric

Artifact	Strong submission demonstrates
ranking stack	explicit eligibility, candidate generation, reranking inputs, and deterministic fallback
evaluation	NDCG-style offline result with policy guardrails and slice review
experiment plan	exposure logging, primary metric, guardrails, stopping rule, and rollback

Common Failures

Symptom	Cause	Fix
Great NDCG while prohibited item appears	ranking runs before eligibility	filter first and test invariant
New model increasingly favors former winners	clicks used without exposure evidence	log impressions and experiment arms
Conversion increases while customer value falls	returns absent from guardrails	gate promotion on downstream outcomes

Next Step

Continue to Capstone: Demand Forecasting

You can now ship a measured ranking surface whose outputs influence future data. Next you will ship time-ordered forecasts and alerts for inventory and warehouse capacity.

PreviousCapstone: Delivery ETA Prediction

Share this article

X Facebook LinkedIn Bluesky Reddit Hacker News Email

References

Learning to Rank using Gradient Descent.

Burges, C. J. C., et al. · 2005 · ICML 2005

Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.

Kohavi, R., Tang, D., Xu, Y. · 2020

Capstone: Product Ranking

Capstone: Product Ranking

Specify the Ranking Surface

Build Offline Evidence First

Make the Ranking Gate Runnable

Log Exposure Before Learning From Outcomes

Submission Checklist

Mastery Check

Why does this capstone require candidate recall and NDCG separately?

Why is eligible_for_ab_review stronger than a demo but weaker than a launch?

Which log field protects later training from treating placement as relevance?

Evaluation rubric

Common Failures

Capstone: Product Ranking

Capstone: Product Ranking

Specify the Ranking Surface

Build Offline Evidence First

Make the Ranking Gate Runnable

Log Exposure Before Learning From Outcomes

Submission Checklist

Mastery Check

Why does this capstone require candidate recall and NDCG separately?

Why is eligible_for_ab_review stronger than a demo but weaker than a launch?

Which log field protects later training from treating placement as relevance?

Evaluation rubric

Common Failures