Ranking and Recommendation Systems

An incident classifier chooses one action for one alert. A search ranker makes a different prediction: among thousands of documents, which few should appear first for a query or developer?

This is a ranking problem. Retrieval finds plausible candidates quickly; a ranking model orders them using query, document, freshness, and source-trust features. A recommendation feed uses the same shape without a typed query: developer and session context replace query features. In both cases, the displayed order becomes part of the future training data.

Separate eligibility, retrieval, and ranking

Suppose a developer searches for retry idempotency key. The corpus contains one million documents. A feature-heavy model shouldn't score every document on every request.

A document absent from the candidate set can't be rescued by a perfect ranker. A forbidden document shouldn't enter the candidate set at all. Candidate recall, policy filtering, and ranking quality are separate obligations.

Build a seven-document fixture. The relevance value is an offline human judgment for one frozen query. It belongs in evaluation data, not in online model features.

The corpus includes a private draft and a stale guide. Remove both before retrieval. A learned score can't override access policy.

Measure candidate recall before ordering quality

Candidate generation runs a cheap retrieval stage. This lab uses a precomputed retrieval score so you can focus on the contract between stages. A real system might combine lexical retrieval, embeddings, popularity, and personalized retrieval.

The judged relevant eligible documents are api-guide, retry-runbook, and sdk-example. Measure how many survive candidate generation.

At k=3, retrieval keeps only one of three relevant eligible documents. No downstream ranker can recover the missing runbook or SDK example. Raising this tiny lab's candidate budget to 5 restores candidate recall before the expensive scorer runs.

The relevant item never enters the top-100 candidate set. Can a more accurate reranker place it first?

Answer

No. Ranking can reorder only retrieved candidates. Fix candidate recall before interpreting downstream ordering metrics.

Rerank the surviving candidates

Keep five candidates. The first-stage retrieval order is fast but imprecise: blog-post and stale-changelog sit near the top on retrieval alone, yet their query_fit stays low for the idempotency intent.

The scorer below is hand-written so each feature remains visible. query_fit represents a richer query-document match feature. reliability represents source trust. Neither uses the offline relevance label.

A production scorer learns weights or nonlinear interactions from labeled examples. The request path remains the same: retrieve cheaply, score the bounded candidate set, then apply policy.

Two-tower architectures and the precomputation constraint

Candidate retrieval often uses a two-tower architecture. One encoder maps the live query or session context to a vector $u(q) \in \mathbb{R}^d$. A separate encoder maps each document or item to a vector $v(i) \in \mathbb{R}^d$. The search system compares those vectors with a dot product or cosine similarity. This pattern dates back at least to deep semantic search models that projected queries and documents into one shared space.^{[1]Reference 1Learning Deep Structured Semantic Models for Web Search Using Clickthrough Data.https://www.microsoft.com/en-us/research/publication/learning-deep-structured-semantic-models-for-web-search-using-clickthrough-data/}

The separation changes serving work. Because the item tower doesn't depend on the current query, item vectors can be computed offline and loaded into an approximate nearest neighbor (ANN) index. A request runs the query tower once, retrieves a small candidate set, and reserves expensive pairwise scoring for those candidates. Exact complexity and latency depend on the index, recall target, hardware, and corpus, so ANN search shouldn't be described with one universal $O(\log N)$ bound.

The cross-feature bottleneck

An early cross-feature mixes query and item information before each side has produced its independent vector. Query-document cross-attention is one example. The resulting item representation changes with every query, so it can't be precomputed.

Using such a model as first-stage retrieval requires pairwise model work across the corpus. Using it after retrieval requires pairwise work only across the bounded candidate set. That makes cross-features useful for reranking even when they're too expensive for candidate generation.

The miniature example uses exact dot products so every step stays visible. A production ANN index changes how candidates are found, but it preserves the same separation between precomputed item vectors and live query work.

Precomputation removes item-encoder work from the request path. Candidate retrieval narrows the pairwise work by a factor of 200 in this fixture, from 1,000 query-item rows to 5. Measure ANN recall and latency on your own data because the index adds an approximation that this exact-search lab doesn't model.

A metric that values the top slots

In a ranked list, placing the best result first matters more than moving it from position 40 to position 39. Discounted Cumulative Gain (DCG) gives graded relevance near the top more weight:

$$\text{DCG@k} = \sum_{i=1}^{k} \frac{2^{rel_i} - 1}{\log_2(i+1)}$$

Here $rel_i$ is the judged relevance at position $i$. Position 1 has denominator $\log_2(2)=1$, so it keeps full weight. Lower positions receive progressively smaller weight. Normalized DCG (NDCG) divides by the score of the ideal ordering, producing a value between zero and one for a query.^{[2]Reference 2Introduction to Information Retrieval.https://nlp.stanford.edu/IR-book/}

Calculate NDCG@5 for both orders. Build the ideal top five by sorting all candidate relevances before truncating; otherwise a relevant candidate below the evaluated slate could disappear from the denominator. The guard for best == 0 handles a query with no judged relevant documents instead of dividing by zero.

This lab computes reranker NDCG over documents that survived retrieval. That boundary is deliberate: candidate recall measures misses before ranking, while reranker NDCG measures ordering quality inside the bounded candidate set. A whole-system evaluation can also compute NDCG against the eligible judged corpus so retrieval misses reduce the final score.

The scorer lifts reranker NDCG@5 from 0.910 to 1.000. That doesn't prove the model will improve a live docs portal. It only proves this frozen judged query improved after reranking.

Reranker NDCG can still look perfect when retrieval drops relevant documents. Compare it with whole-system NDCG, which builds the ideal list from the eligible judged corpus so missing relevants lower the score:

At k=3, the perfect reranker order among survivors still yields reranker NDCG 1.0, because the ideal list is built only from those three documents. System NDCG falls to about 0.673 because the ideal top three from the eligible corpus is [3, 2, 2] and two relevant documents never entered the slate. Report both when you care about end-to-end quality; keep candidate recall as the explicit retrieval diagnostic.

One query isn't an evaluation set. Average reranker NDCG over a frozen query set, report slices such as new documents and languages, and inspect failed queries. Keep candidate recall separate so a strong reranker score can't hide retrieval loss.

Why does normalized discounted cumulative gain (NDCG) penalize moving a relevant item from rank 1 to rank 10?

Answer

Discounted cumulative gain gives earlier positions more weight through a logarithmic discount. NDCG then divides by the ideal ordering, making the score comparable across queries with different relevance sets.

Turn judgments into pair preferences

Pairwise learning-to-rank methods such as RankNet train from preferences so a relevant item scores above a less relevant one.^{[3]Reference 3Learning to Rank using Gradient Descent.https://www.microsoft.com/en-us/research/publication/learning-to-rank-using-gradient-descent/} For this one query, create the preference pairs from offline judgments.

The pair ("api-guide", "blog-post") says the API guide should score higher for this query. A learned pairwise model uses features to reduce preference mistakes across many queries. The hand-written scorer stays useful as a transparent baseline.

Log exposure before reading clicks

The ranked slate is now ready to display. Persist one immutable impression row per exposed document before reading clicks. Later click and resolved events join back through request and document IDs. Without the slate, position, eligibility-policy version, candidate-set version, and ranker version, you can't reproduce what the developer saw.

The first row doesn't mean api-guide is universally best. It means this request exposed it at position 1, and one developer clicked it.

Log enough context to interpret later outcomes:

Clicks are biased labels

When a model places a document first, it receives more attention and therefore more clicks. Training directly on clicks rewards past ranking position as if it were document relevance. This is a feedback loop.

Take two documents with equal underlying appeal. Historical ranker always places same-quality-a first and same-quality-b second. Slot one is examined more often.

Naive training labels say document a is twice as attractive. The log doesn't justify that conclusion because document and position never vary independently.

A simple counterfactual correction weights each click by the inverse of its position's examination probability. This example fixes slot-one propensity at 1.0 and slot-two propensity at 0.5.

Both corrected signals become 0.40. The arithmetic exposes the idea, not a production estimator. Real propensities need careful estimation, often through randomized interventions or a validated click model. Very small propensities also create large, noisy weights, so a production estimator needs variance controls and evaluation. Joachims, Swaminathan, and Schnabel describe propensity-weighted learning-to-rank as a counterfactual correction for biased implicit feedback.

Why can't raw click rate serve as an unbiased relevance label?

Answer

Users can click only items they were shown, and higher-ranked items receive more attention. Log exposure and position, then use controlled exploration or bias-aware estimation before treating clicks as preference evidence.

Protect users and the dataset

Ranking can narrow what developers see. Add constraints for stale documents, private drafts, source policy, and overconcentration. Monitor slices such as new documents, small doc sets, languages, and stale categories rather than accepting one portal-wide NDCG.

Policy must stay outside the learned score. Reproduce a bad integration that ranks the raw corpus without applying eligibility first.

The private draft ranks first because its query match is strong. Better model training won't fix this bug. Eligibility filtering must run before retrieval and remain covered by release checks.

Gate an offline candidate

A useful release packet includes offline query judgments, candidate recall, reranker NDCG, p95 scoring latency, diversity checks, impression and outcome schemas, and an A/B stopping rule. A list that looks good offline isn't allowed to silently write its own future training labels.

Use concrete gates for this candidate:

• candidate recall at 5 must be 1.0
• reranker NDCG@5 must be at least 0.95
• p95 scorer latency must be at most 20 milliseconds
• blocked documents must stay absent
• impression rows must include fields needed for replay
• outcome events must join a logged impression

Passing offline checks earns a controlled online experiment, not an immediate global rollout. Engagement changes can reflect latency, layout, freshness, source trust, and ranking behavior. Randomized A/B assignment isolates the candidate change more reliably than comparing two time windows.^{[4]Reference 4Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/} Predeclare minimum sample size, minimum duration, and safety stops before exposure starts so the team doesn't stop when a noisy result happens to look favorable. The miniature receipt below uses illustrative commitments; real values come from traffic, power analysis, and risk policy.

Publish a receipt before exposure begins.

The receipt binds eligibility policy, candidate generator, ranker, offline evidence, release checks, assignment unit, primary metric, minimum exposure, and guardrails. Your later analysis must keep these versions pinned.

When ranking breaks