Ranking and Recommendation Systems

A developer searches for retry idempotency key. A search ranker doesn't emit one yes-or-no action. Given that query, which documents should appear first, and in what order?

Gradient boosted trees scored one job at a time: intervene or wait. Ranked order also changes the training data. Classifier labels are usually collected independently of the last prediction. Yesterday's ranking becomes today's clicks. A recommendation feed uses the same funnel without a typed query: developer and session context replace query features.

Separate eligibility, retrieval, and ranking

The corpus for that retry idempotency key query has one million documents. A feature-heavy model shouldn't score every document on every request.

A document that never enters the candidate set can't be rescued by a perfect ranker. A forbidden document shouldn't enter the candidate set at all. Candidate recall, policy filtering, and ranking quality are separate obligations.

Build a seven-document fixture. The relevance value is an offline human judgment for one frozen query. It belongs in evaluation data, not in online model features.

The corpus includes a private draft and a stale guide. Remove both before retrieval. A learned score can't override access policy.

Measure candidate recall before ordering quality

Candidate generation runs a cheap retrieval stage. This lab uses a precomputed retrieval score so you can focus on the contract between stages. In Core Retrieval Algorithms that cheap stage was BM25, dense similarity, or a fused shortlist. Here the same contract shows up in a production ranker: recover the right documents first, then spend compute ordering them.

The judged relevant eligible documents are api-guide, retry-runbook, and sdk-example. Measure how many survive candidate generation.

At k=3, retrieval keeps only one of three relevant eligible documents. No downstream ranker can recover the missing runbook or SDK example. Raising this tiny lab's candidate budget to 5 restores candidate recall before the expensive scorer runs.

The relevant item never enters the top-100 candidate set. Can a more accurate reranker place it first?

Answer

No. Ranking can reorder only retrieved candidates. Fix candidate recall before interpreting downstream ordering metrics.

Rerank the surviving candidates

Keep five candidates. The first-stage retrieval order is fast but imprecise: blog-post and legacy-changelog sit near the top on retrieval alone, yet their query_fit stays low for the idempotency intent.

The scorer below is hand-written so each feature remains visible. query_fit represents a richer query-document match feature. reliability represents source trust. Neither uses the offline relevance label.

A production scorer learns weights or nonlinear interactions from labeled examples. The request path stays the same: retrieve cheaply, score the bounded candidate set, then apply policy. Those two orders of the same five documents are ready to compare. Which one is actually better?

A metric that values the top slots

A classifier's accuracy doesn't care which mistakes you made first. A ranked list does. Putting the best document first is a bigger win than sliding it from position 40 to position 39. Discounted Cumulative Gain (DCG) gives graded relevance near the top more weight:

$$\text{DCG@k} = \sum_{i=1}^{k} \frac{2^{rel_i} - 1}{\log_2(i+1)}$$

Here $rel_i$ is the judged relevance at position $i$, counting from 1. Position 1 has denominator $\log_2(2)=1$, so it keeps full weight. Lower positions receive a logarithmic discount. This is the exponential-gain form Manning uses for NDCG.^{[1]Reference 1Introduction to Information Retrieval.https://nlp.stanford.edu/IR-book/} Common library helpers often use linear gain $rel_i / \log_2(i+1)$ instead. The two agree on binary labels and disagree on grades such as 0, 2, 3, so pin the gain function in the evaluation suite.

Normalized DCG (NDCG) divides by the score of the ideal ordering, producing a value between zero and one for a query.

Work the retrieval order [3, 0, 0, 2, 2] by hand before looking at code. Gain is $2^{rel}-1$, so a grade-3 document contributes 7 and a grade-2 document contributes 3:

$$\begin{align*} \text{DCG@5} &= \frac{7}{\log_2 2} + 0 + 0 + \frac{3}{\log_2 5} + \frac{3}{\log_2 6} \\ &\approx 7 + 1.292 + 1.161 = 9.453 \end{align*}$$

The ideal order of these five grades is [3, 2, 2, 0, 0]:

$$\text{IDCG@5} = 7 + \frac{3}{\log_2 3} + \frac{3}{\log_2 4} \approx 7 + 1.893 + 1.500 = 10.393$$

So retrieval-order NDCG@5 is $9.453 / 10.393 \approx 0.910$. Build that ideal list from all candidate relevances before truncating. Otherwise a relevant candidate below the evaluated slate could disappear from the denominator. The guard for best == 0 handles a query with no judged relevant documents instead of dividing by zero.

This lab computes reranker NDCG over documents that survived retrieval. That boundary is deliberate: candidate recall measures misses before ranking, while reranker NDCG measures ordering quality inside the bounded candidate set. A whole-system evaluation can also compute NDCG against the eligible judged corpus so retrieval misses reduce the final score.

The scorer lifts reranker NDCG@5 from 0.910 to 1.000. That doesn't prove the model will improve a live docs portal. It only proves this frozen judged query improved after reranking.

Reranker NDCG can still look perfect when retrieval drops relevant documents. Compare it with whole-system NDCG, which builds the ideal list from the eligible judged corpus so missing relevants lower the score:

At k=3, the perfect reranker order among survivors still yields reranker NDCG 1.000, because the ideal list is built only from those three documents. System NDCG falls to 0.674 because the ideal top three from the eligible corpus is [3, 2, 2] and two relevant documents never entered the slate. Report both when you care about end-to-end quality; keep candidate recall as the explicit retrieval diagnostic.

One query isn't an evaluation set. Average reranker NDCG over a frozen query set, report slices such as new documents and languages, and inspect failed queries. Keep candidate recall separate so a strong reranker score can't hide retrieval loss.

Why does normalized discounted cumulative gain (NDCG) penalize moving a relevant item from rank 1 to rank 10?

Answer

Discounted cumulative gain gives earlier positions more weight through a logarithmic discount. NDCG then divides by the ideal ordering, making the score comparable across queries with different relevance sets.

Two-tower retrieval and the precomputation constraint

The lab just ranked five rows. A live corpus has closer to a million. Candidate retrieval often uses a two-tower architecture so that million-row scan never happens on the request path. One encoder maps the live query or session context to a vector $u(q) \in \mathbb{R}^d$. A separate encoder maps each document or item to a vector $v(i) \in \mathbb{R}^d$. The search system compares those vectors with a dot product or cosine similarity. Huang et al. trained deep structured semantic models that map queries and documents into one shared space and score them with cosine similarity.^{[2]Reference 2Learning Deep Structured Semantic Models for Web Search Using Clickthrough Data.https://www.microsoft.com/en-us/research/publication/learning-deep-structured-semantic-models-for-web-search-using-clickthrough-data/}

The separation changes serving work. Because the item tower doesn't depend on the current query, item vectors can be computed offline and loaded into an approximate nearest neighbor (ANN) index. A request runs the query tower once, retrieves a small candidate set, and reserves expensive pairwise scoring for those candidates. Exact complexity and latency depend on the index, recall target, hardware, and corpus, so ANN search shouldn't be described with one universal $O(\log N)$ bound.

The cross-feature bottleneck

An early cross-feature mixes query and item information before each side has produced its independent vector. Query-document cross-attention is one example. The resulting item representation changes with every query, so it can't be precomputed.

Using such a model as first-stage retrieval requires pairwise model work across the corpus. Using it after retrieval requires pairwise work only across the bounded candidate set. That makes cross-features useful for reranking even when they're too expensive for candidate generation.

The miniature example uses exact dot products so every step stays visible. A production ANN index changes how candidates are found, but it preserves the same separation between precomputed item vectors and live query work.

Precomputation removes item-encoder work from the request path. Candidate retrieval narrows the pairwise work by a factor of 200 in this fixture, from 1,000 query-item rows to 5. Measure ANN recall and latency on your own data because the index adds an approximation that this exact-search lab doesn't model.

Turn judgments into pair preferences

The hand-written scorer is pointwise: each document gets an independent score. Pairwise learning-to-rank methods such as RankNet train from preferences so a relevant item scores above a less relevant one.^{[3]Reference 3Learning to Rank using Gradient Descent.https://www.microsoft.com/en-us/research/publication/learning-to-rank-using-gradient-descent/} Listwise methods try to optimize a list metric directly. This lab stays with a transparent pointwise score and pairwise labels built from the same frozen query.

The pair ("api-guide", "blog-post") says the API guide should score higher for this query. Equal grades don't form a pair, so retry-runbook vs sdk-example is absent. A learned pairwise model uses features to reduce preference mistakes across many queries. The hand-written scorer stays useful as a transparent baseline.

Log exposure before reading clicks

The ranked slate is now ready to display. Persist one immutable impression row per exposed document before reading clicks. Later click and resolved events join back through request and document IDs. Without the slate, position, eligibility-policy version, candidate-set version, and ranker version, you can't reproduce what the developer saw.

The first row doesn't mean api-guide is universally best. It means this request exposed it at position 1, and one developer clicked it.

Log enough context to interpret later outcomes:

Clicks are biased labels

When a model places a document first, it receives more attention and therefore more clicks. Training directly on clicks rewards past ranking position as if it were document relevance. This is a feedback loop.

Take two documents with equal underlying appeal. Historical ranker always places same-quality-a first and same-quality-b second. Slot one is examined more often.

Naive training labels say document a is twice as attractive. The log doesn't justify that conclusion because document and position never vary independently.

A simple counterfactual correction weights each click by the inverse of its position's examination probability. This example fixes slot-one propensity at 1.0 and slot-two propensity at 0.5.

Both corrected signals become 0.40. The arithmetic exposes the idea, not a production estimator. Real propensities need careful estimation, often through randomized interventions or a validated click model. Very small propensities also create large, noisy weights, so a production estimator needs variance controls and evaluation. Inverse-propensity learning-to-rank treats examination as the missing piece: you can only supervise a click on documents the user actually saw.^{[4]Reference 4Unbiased Learning-to-Rank with Biased Feedbackhttps://arxiv.org/abs/1608.04468}

Why can't raw click rate serve as an unbiased relevance label?

Answer

Users can click only items they were shown, and higher-ranked items receive more attention. Log exposure and position, then use controlled exploration or bias-aware estimation before treating clicks as preference evidence.

Protect users and the dataset

Ranking can narrow what developers see. Eligibility constraints for stale documents, private drafts, and source policy run before retrieval because those items must never compete. Diversity among otherwise eligible items belongs after scoring: if product policy permits at most two results from one source in the top three, adjust the final slate deterministically and gate that invariant in release checks. Monitor slices such as new documents, small doc sets, languages, and stale categories rather than accepting one portal-wide NDCG.

Policy must stay outside the learned score. Reproduce a bad integration that ranks the raw corpus without applying eligibility first.

The private draft ranks first because its query match is strong. Better model training won't fix this bug. Eligibility filtering must run before retrieval and remain covered by release checks.

Gate an offline candidate

A useful release packet includes offline query judgments, candidate recall, reranker NDCG, p95 scoring latency, diversity checks, impression and outcome schemas, and an A/B stopping rule. A list that looks good offline isn't allowed to silently write its own future training labels.

Use concrete gates for this candidate:

• candidate recall at 5 must be 1.0
• reranker NDCG@5 must be at least 0.95
• p95 scorer latency must be at most 20 milliseconds
• blocked documents must stay absent
• impression rows must include fields needed for replay
• outcome events must join a logged impression

Passing offline checks earns a controlled online experiment, not an immediate global rollout. Engagement changes can reflect latency, layout, freshness, source trust, and ranking behavior. Randomized A/B assignment isolates the candidate change more reliably than comparing two time windows, which is the same discipline as Experiment Design and A/B Testing.^{[5]Reference 5Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/} Predeclare minimum sample size, minimum duration, and safety stops before exposure starts so the team doesn't stop when a noisy result happens to look favorable. The miniature receipt below uses illustrative commitments; real values come from traffic, power analysis, and risk policy.

Publish a receipt before exposure begins.

The receipt binds eligibility policy, candidate generator, ranker, offline evidence, release checks, assignment unit, primary metric, minimum exposure, and guardrails. Your later analysis must keep these versions pinned.

When ranking breaks