Ranking and Recommendation Systems

An incident classifier chooses one action for one alert. A search ranker makes a different prediction: among thousands of documents, which few should appear first for a query or developer?

This is a ranking problem. Retrieval finds plausible candidates quickly; a ranking model orders them using query, document, freshness, and source-trust features. A recommendation feed uses the same shape without a typed query: developer and session context replace query features. In both cases, the displayed order becomes part of the future training data.

Separate eligibility, retrieval, and ranking

Suppose a developer searches for retry idempotency key. The corpus contains one million documents. A feature-heavy model shouldn't score every document on every request.

A document absent from the candidate set can't be rescued by a perfect ranker. A forbidden document shouldn't enter the candidate set at all. Candidate recall, policy filtering, and ranking quality are separate obligations.

Build a seven-document fixture. The relevance value is an offline human judgment for one frozen query. It belongs in evaluation data, not in online model features.

The corpus intentionally includes a private draft and a stale guide. Remove both before retrieval. A learned score can't override access policy.

Measure candidate recall before ordering quality

Candidate generation runs a cheap retrieval stage. This lab uses a precomputed retrieval score so you can focus on the contract between stages. A real system might combine lexical retrieval, embeddings, popularity, and personalized retrieval.

The judged relevant eligible documents are api-guide, retry-runbook, and sdk-example. Measure how many survive candidate generation.

At k=3, retrieval keeps only one of three relevant eligible documents. No downstream ranker can recover the missing runbook or SDK example. Raising this tiny lab's candidate budget to 5 restores candidate recall before the expensive scorer runs.

Pause and predict: If the ranker becomes perfect while candidate budget stays at 3, can the runbook appear? No. It never reaches the ranker.

Rerank the surviving candidates

Keep five candidates. The first-stage retrieval order is fast but imprecise: blog-post and stale-changelog sit near the top on retrieval alone, yet their query_fit stays low for the idempotency intent.

The scorer below is hand-written so each feature remains visible. query_fit represents a richer query-document match feature. reliability represents source trust. Neither uses the offline relevance label.

A production scorer learns weights or nonlinear interactions from labeled examples. The request path remains the same: retrieve cheaply, score the bounded candidate set, then apply policy.

A metric that values the top slots

In a ranked list, placing the best result first matters more than moving it from position 40 to position 39. Discounted Cumulative Gain (DCG) gives graded relevance near the top more weight:

$$\text{DCG@k} = \sum_{i=1}^{k} \frac{2^{rel_i} - 1}{\log_2(i+1)}$$

Here $rel_i$ is the judged relevance at position $i$. Position 1 has denominator $\log_2(2)=1$, so it keeps full weight. Lower positions receive progressively smaller weight. Normalized DCG (NDCG) divides by the score of the ideal ordering, producing a value between zero and one for a query.^{[1]Reference 1Introduction to Information Retrieval.https://nlp.stanford.edu/IR-book/}

Calculate NDCG@5 for both orders. Build the ideal top five by sorting all candidate relevances before truncating; otherwise a relevant candidate below the evaluated slate could disappear from the denominator. The guard for best == 0 handles a query with no judged relevant documents instead of dividing by zero.

This lab computes reranker NDCG over documents that survived retrieval. That boundary is deliberate: candidate recall measures misses before ranking, while reranker NDCG measures ordering quality inside the bounded candidate set. A whole-system evaluation can also compute NDCG against the eligible judged corpus so retrieval misses reduce the final score.

The scorer lifts reranker NDCG@5 from 0.910 to 1.000. That doesn't prove the model will improve a live docs portal. It only proves this frozen judged query improved after reranking.

One query isn't an evaluation set. Average reranker NDCG over a frozen query set, report slices such as new documents and languages, and inspect failed queries. Keep candidate recall separate so a strong reranker score can't hide retrieval loss.

Turn judgments into pair preferences

Pairwise learning-to-rank methods such as RankNet train from preferences so a relevant item scores above a less relevant one.^{[2]Reference 2Learning to Rank using Gradient Descent.https://www.microsoft.com/en-us/research/publication/learning-to-rank-using-gradient-descent/} For this one query, create the preference pairs from offline judgments.

The pair ("api-guide", "blog-post") says the API guide should score higher for this query. A learned pairwise model uses features to reduce preference mistakes across many queries. The hand-written scorer stays useful as a transparent baseline.

Log exposure before reading clicks

The ranked slate is now ready to display. Persist one immutable impression row per exposed document before reading clicks. Later click and resolved events join back through request and document IDs. Without the slate, position, eligibility-policy version, candidate-set version, and ranker version, you can't reproduce what the developer saw.

The first row doesn't mean api-guide is universally best. It means this request exposed it at position 1, and one developer clicked it.

Log enough context to interpret later outcomes:

Clicks are biased labels

When a model places a document first, it receives more attention and therefore more clicks. Training directly on clicks rewards past ranking position as if it were document relevance. This is a feedback loop.

Take two documents with equal underlying appeal. Historical ranker always places same-quality-a first and same-quality-b second. Slot one is examined more often.

Naive training labels say document a is twice as attractive. The log doesn't justify that conclusion because document and position never vary independently.

A simple counterfactual correction weights each click by the inverse of the probability that its position was examined. The propensities below are deliberately fixed for the miniature example: slot one is examined with probability 1.0, slot two with probability 0.5.

Both corrected signals become 0.40. The arithmetic exposes the idea, not a production estimator. Real propensities need careful estimation, often through randomized interventions or a validated click model. Very small propensities also create large, noisy weights, so a production estimator needs variance controls and evaluation. Joachims, Swaminathan, and Schnabel describe propensity-weighted learning-to-rank as a counterfactual correction for biased implicit feedback.

Protect users and the dataset

Ranking can narrow what developers see. Add constraints for stale documents, private drafts, source policy, and overconcentration. Monitor slices such as new documents, small doc sets, languages, and stale categories rather than accepting one portal-wide NDCG.

Policy must stay outside the learned score. Reproduce a bad integration that ranks the raw corpus without applying eligibility first.

The private draft ranks first because its query match is strong. Better model training won't fix this bug. Eligibility filtering must run before retrieval and remain covered by release checks.

Gate an offline candidate

A useful release packet includes offline query judgments, candidate recall, reranker NDCG, p95 scoring latency, diversity checks, impression and outcome schemas, and an A/B stopping rule. A list that looks good offline isn't allowed to silently write its own future training labels.

Use concrete gates for this candidate:

• candidate recall at 5 must be 1.0
• reranker NDCG@5 must be at least 0.95
• p95 scorer latency must be at most 20 milliseconds
• blocked documents must stay absent
• impression rows must include fields needed for replay
• outcome events must join a logged impression

Passing offline checks earns a controlled online experiment, not an immediate global rollout. Engagement changes can reflect latency, layout, freshness, source trust, and ranking behavior. Randomized A/B assignment isolates the candidate change more reliably than comparing two time windows.^{[3]Reference 3Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.https://experimentguide.com/} Predeclare minimum sample size, minimum duration, and safety stops before exposure starts so the team doesn't stop when a noisy result happens to look favorable. The miniature receipt below uses illustrative commitments; real values come from traffic, power analysis, and risk policy.

Publish a receipt before exposure begins.

The receipt binds eligibility policy, candidate generator, ranker, offline evidence, release checks, assignment unit, primary metric, minimum exposure, and guardrails. Your later analysis must keep these versions pinned.

Explain the slate without looking back

Practice

1. Reduce candidate budget from 5 to 4. Predict which relevant document disappears, then rerun candidate recall and reranker NDCG. Explain why both metrics matter.
2. Change blog-post relevance from 0 to 1. Recompute reranker NDCG and explain how graded judgments change evaluation.
3. Add a timestamped save outcome event. Decide whether saves should replace resolved sessions as the primary online metric or remain a diagnostic signal.
4. Change slot-two propensity from 0.5 to 0.25. Recompute corrected signal and explain why an incorrect propensity model creates a different bias.
5. Add a diversity gate that allows at most two documents from one source in top three. Extend fixture with source_id, then reproduce one failure.

What strong answers show

When ranking breaks