Reward Modeling from Preference Data

The last chapter left you with a resumable supervised fine-tuning (SFT) job. policy-sft-v4 can continue, initialize, or export, and you already know which adaptation method to launch. Once that checkpoint answers, a new training problem shows up: two fluent replies to the same access-policy prompt. One cites policy P-7 and opens a review ticket. The other grants admin access for the night. Which one should score higher?

RLHF (Reinforcement Learning from Human Feedback) diagrams often squash that question into a box labeled reward model, then jump to Proximal Policy Optimization (PPO). The box is its own training project. If it learns the wrong shortcuts, policy optimization will amplify them.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

A scalar judge, not a generator

A reward model doesn't write the next token. It scores text that already exists.

Given a prompt x and two candidate answers:

• y+ chosen by the labeler (open a review ticket, cite P-7)
• y- rejected by the labeler (grant admin tonight)

the model should assign:

That scalar is useful in two different ways later:

1. as a reward signal for PPO-style RLHF^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}
2. as an inspectable ranking score when you want to compare many policy outputs, including best-of-N sampling

Explicit reward models still earn their keep even though Direct Preference Optimization (DPO) can skip them for offline preference optimization.^{[2]Reference 2Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} Train that scalar first, then distrust it, before you plug it into either use.

Preference pairs that can actually train

The core supervision format is a preference pair: same prompt, two different completions, one definite ranking.

Standard format

Conversational format

Hugging Face TRL's RewardTrainer accepts both standard and conversational preference formats and can apply the model's chat template during tokenization.^{[3]Reference 3Reward Modeling.https://huggingface.co/docs/trl/reward_trainer}

A pair contract before training

A row isn't ready just because it has chosen and rejected columns. For every binary pair, enforce:

• the same prompt, system message, tool context, and rendering template on both candidates
• two different candidate answers and a definite preference label
• separate handling for ties, abstentions, and low-agreement labels
• provenance such as source prompt, generator checkpoint, sampling settings, and labeling batch

That last field matters for splitting. If one prompt generated several candidates, its comparisons are near-duplicates. Put the entire prompt group in train or in evaluation, never both.

The audit below keeps only row a. Identical candidates, mismatched prompts, and ties don't enter ordinary pairwise ranking training.

A grouped split is the next check. Rows a-b and a-c share prompt support-17, so they move together.

Prompt groups also change how you train, not only how you split. InstructGPT asked labelers to rank $K=4$ to $K=9$ completions per prompt, which produces $\binom{K}{2}$ pairwise comparisons. If you shuffle those pairs into the dataset independently, each completion can contribute $K-1$ gradient updates in one epoch, and the reward model overfits. Their fix was to treat every comparison from one prompt as a single batch element and average the loss over the $\binom{K}{2}$ pairs.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

Last token, one number

Most practical reward models aren't built from scratch. You start from a pretrained or SFT checkpoint, drop the token-unembedding head, and attach a one-unit score head. InstructGPT did exactly that: replace the unembedding layer with a projection to a scalar.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} Conceptually:

For a decoder-only LM, that representation is often the final non-padding position. Reward modeling then feels like sequence classification with pairwise labels rather than generation. The output is one number per candidate response, not the next-token distribution.

The rendered sequence is part of the contract. Keep chat-template and end-of-sequence conventions consistent with the policy you'll score. Don't silently train on answers whose decisive ending was truncated: current TRL RewardConfig.max_length (default 1024) drops a pair when either candidate exceeds that limit after tokenization.^{[3]Reference 3Reward Modeling.https://huggingface.co/docs/trl/reward_trainer}

A pair that scores is only useful if the loss on those two numbers matches the label. That's the next question.

Bradley-Terry from a worked margin

Start with the access-policy pair. Suppose the ticket-and-cite answer gets reward 1.8 and the unsupported approval gets 0.7. The margin is 1.8 - 0.7 = 1.1. A positive margin means the reward model prefers the chosen answer. The Bradley-Terry model turns that margin into a preference probability: $\sigma(1.1) \approx 0.75$.

$$P(y^+ \succ y^- \mid x) = \sigma\big(r(x, y^+) - r(x, y^-)\big)$$

where $\sigma$ is the sigmoid. InstructGPT trains the negative log of that probability, averaged over the comparisons in a prompt group:^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

$$\mathcal{L}(\theta) = -\frac{1}{\binom{K}{2}} \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}}\big[\log \sigma\big(r_\theta(x, y_w) - r_\theta(x, y_l)\big)\big]$$

where $(y_w, y_l)$ is the chosen (winning) and rejected (losing) pair among the $K$ ranked completions for prompt $x$. For a single binary pair, $K=2$ and the factor is 1, so the loss is just $-\log\sigma(r^+ - r^-)$. If the chosen reward is much higher than the rejected reward, loss shrinks toward 0. If the model ranks them backwards, loss gets large.

The numerically stable form of $-\log\sigma(z)$ is $\log(1 + e^{-z})$. For negative $z$, rewrite it as $-z + \log(1 + e^{z})$ so exp doesn't overflow.

Why does a larger positive reward margin make the Bradley-Terry loss smaller?

Answer

Because the loss is -log sigma(r(chosen) - r(rejected)). As the chosen-minus-rejected margin grows, the sigmoid term moves closer to 1, so the negative log shrinks toward 0.

The batch below uses the same three margins you'll see in the quiz: 1.7, -0.2, and 0.7. Pair accuracy is 2/3 because one pair is ranked backwards.

Sweeping the margin makes the same loss visible as a curve. Zero margin is a coin flip (log 2 ≈ 0.6931). A margin of -2 is a confident wrong ranking.

That's the ranking core. Everything else in reward modeling is about whether the data and evaluation around that loss are strong enough to trust.

Multi-way rankings and a preference-strength margin

Binary chosen/rejected pairs are the default training unit. Production sets often produce $K>2$ completions per prompt. Two standard expansions:

• Pair expansion. InstructGPT's method: emit every ordered pair the ranking can score, then pack those pairs by prompt so one completion doesn't dominate the epoch.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}
• Plackett-Luce. Model a full ranking as a product of successive choices rather than independent pairs. Use it when you have true multi-way rankings and want one joint likelihood. Pair expansion remains the simpler baseline when only sparse pairwise labels exist.

Llama 2 added a discrete margin $m(y^+, y^-)$ inside the sigmoid when raters marked how much better the winner was (significantly better, slightly better, and so on):^{[4]Reference 4Llama 2: Open Foundation and Fine-Tuned Chat Models.https://arxiv.org/abs/2307.09288}

$$\mathcal{L} = -\log \sigma\big(r(x, y^+) - r(x, y^-) - m\big)$$

Positive $m$ requires the chosen score to beat the rejected score by more than $m$ before the loss saturates. Llama 3 later dropped that term after seeing diminishing returns once preference data scaled.^{[5]Reference 5The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783} Treat $m$ as a hyperparameter you validate on held-out pair accuracy and fresh-policy human agreement, not as a free constant.

A temperature $\beta$ that multiplies the reward difference, $\sigma\big(\beta(r^+ - r^-)\big)$, is multiplicative score scaling: it changes loss confidence and downstream RL strength without changing the additive invariance of pure ranking. Log any $\beta$ or $m$ next to the reward checkpoint.

What pair accuracy hides

TRL logs more than loss for a reason.^{[3]Reference 3Reward Modeling.https://huggingface.co/docs/trl/reward_trainer}

Loss can fall while the model memorizes easy stylistic cues that overfit the train pairs. InstructGPT already saw this: extra epochs quickly hurt validation loss even when the architecture was stable.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

Centering and calibration

Reward models are underdetermined up to an additive constant: adding the same number to every score leaves every margin and the Bradley-Terry loss unchanged. Scaling scores is different. It preserves a ranking but changes loss confidence and the strength of a reward signal consumed by an optimizer.

That matters operationally because:

• absolute reward level and score scale can drift over training
• PPO-style optimization is sensitive to reward scale
• long verbose answers can look better than they are if the model learned a shallow heuristic

InstructGPT fixed the additive ambiguity with a bias so labeler demonstrations scored mean zero before RL.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} TRL exposes center_rewards_coefficient (recommended 0.01) as an auxiliary term that pushes batch rewards toward zero during training. It's a centering aid, not proof that reward magnitude is calibrated for policy optimization.^{[3]Reference 3Reward Modeling.https://huggingface.co/docs/trl/reward_trainer}

Shift leaves the loss alone. Scale doesn't. If you later feed these scores into PPO, the * 3 version is a stronger (and easier to overoptimize) signal even though the ranking is the same.

Audits that pair accuracy misses

Pair accuracy can look healthy while the reward model still learns a bad shortcut. This audit ranks all three preference pairs correctly, but every chosen answer is longer than its rejected pair.

Annotator disagreement is another failure signal. A pair can be formatted correctly and still be weak supervision if raters don't agree about which completion is better. The toy gate below routes any disputed label to review. A real pipeline may adjudicate, weight, or keep disagreements on a dedicated evaluation slice.

Length-matched slices and rater agreement still evaluate the original pair distribution. The harder question is what happens when the policy starts writing answers that weren't in that distribution.

Fresh outputs are the real test

A reward model can fit the training pairs and still rank the current policy's fresh answers the wrong way. That second ranking is the one that matters before you optimize against the score.

As the policy improves, it starts producing answers unlike the ones in the original preference dataset. The reward model may then score confidently for the wrong reasons. Gao, Schulman, and Hilton measured this as overoptimization: a proxy reward you keep optimizing eventually stops tracking a gold preference signal, which is Goodhart's law with a learning curve.^{[6]Reference 6Scaling Laws for Reward Model Overoptimizationhttps://arxiv.org/abs/2210.10760} In their synthetic setup the gold score rose, then fell, while the proxy kept climbing. A KL penalty increased proxy reward at a given KL, but it didn't improve the gold-reward frontier.

Watch for that gap in production:

• the learned reward rises
• held-out human preference on fresh policy outputs stops rising
• raters see longer, repetitive, or otherwise worse answers

If you don't monitor that gap, policy optimization can amplify the shortcut. The threshold below is illustrative; set release gates from your evaluation design and risk tolerance.

KL control intuition (preview)

A common control, developed in full by the next lesson, penalizes the policy for drifting too far from its reference checkpoint. Optimization maximizes reward minus a KL-divergence term that measures policy drift.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} That discourages large departures from the reference, but Gao's measurements are the reason to keep the claim narrow: KL control isn't a certificate that the reward model is valid on new outputs.^{[6]Reference 6Scaling Laws for Reward Model Overoptimizationhttps://arxiv.org/abs/2210.10760} A reward model is a local approximation of human preference, which is exactly why it can be overoptimized.

Standardized evaluation: RewardBench

Held-out pairs you wrote yourself can share your blind spots. RewardBench is an Ai2 benchmark that scores a reward model by how often it ranks a known-better completion above a worse one. Its original sections cover chat, harder instruction-following comparisons, safety, reasoning, and prior preference test sets.^{[7]Reference 7RewardBench: Evaluating Reward Models for Language Modelinghttps://arxiv.org/abs/2403.13787}

RewardBench 2 is a harder follow-up: multi-skill, best-of-four scoring, mostly previously unused human prompts (about 70% of the set), and decontamination against twenty downstream evaluations. In their experiments, benchmark scores correlated strongly with best-of-N (Pearson 0.87 on the average they report). PPO was a weaker, saturating signal: decent-to-good reward models clustered, and transfer dropped when the reward model didn't share the policy's model lineage or prompt distribution.^{[8]Reference 8RewardBench 2: Advancing Reward Model Evaluationhttps://arxiv.org/abs/2506.01937}

One practical caveat from that work: the highest-scoring reward model on the leaderboard isn't automatically the best choice for your run. Treat absolute benchmark rank as a filter, then validate with the policy and optimization setup you'll use.^{[8]Reference 8RewardBench 2: Advancing Reward Model Evaluationhttps://arxiv.org/abs/2506.01937}

Your reward model tops the RewardBench leaderboard. Is it automatically the right choice for your PPO run?

Answer

No. A high benchmark score is a good filter, especially for best-of-N. RewardBench 2 reported better PPO transfer for same-lineage, in-distribution reward and policy models, and PPO gains saturated across many decent scores. Validate with your intended policy and optimization setup.

Other ways to build a preference signal

The scalar Bradley-Terry head is a common baseline. It isn't the only judge.

• Generative reward models. Instead of a scalar head, an LM can read candidates and emit a verdict. Mahan et al. study rationale generation and vote aggregation in that setup; treat those as design choices to evaluate, not universal guarantees.^{[9]Reference 9Generative Reward Modelshttps://arxiv.org/abs/2410.12832}
• Process reward models. Scoring only the final answer is a weak signal for multi-step reasoning. PRMs score each step. Lightman et al. compared process and outcome supervision as search over many sampled MATH solutions (best-of-N), not as RL on the generator, and the process RM beat the outcome RM.^{[10]Reference 10Let's Verify Step by Step.https://arxiv.org/abs/2305.20050}
• Verifiers and RLVR. When correctness is checkable, such as a final math answer or passing unit tests, verifiable rewards can reduce dependence on a learned preference proxy. They don't eliminate misspecified tests or gaming of the verifier.^{[11]Reference 11Tülu 3: Pushing Frontiers in Open Language Model Post-Traininghttps://arxiv.org/abs/2411.15124} Later chapters cover this family.

These don't retire the scalar reward model, and they don't all share its Bradley-Terry objective. The reusable lesson is the evaluation discipline: inspect the signal's coverage, test it on outputs produced by the system being optimized, and watch for optimization exploiting its blind spots.

When an explicit reward model is worth the cost

Use one when:

• you want PPO-style online optimization
• you want to score many candidate outputs with one scalar model
• you want to inspect and audit the preference signal directly

Start with DPO when:

• you have a clean offline preference dataset
• you want the simpler baseline first
• you don't need an explicit learned judge in the loop

That trade-off is why DPO is a strong offline-preference baseline: it removes the separate reward-model training stage.^{[2]Reference 2Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} It doesn't supply a reusable scalar judge for PPO or candidate scoring. The final toy gate combines several checks, with project-specific thresholds, to block optimization when only one static slice passes.

Your reward model has good static pair accuracy, but once PPO starts, reward climbs while human raters say answers are getting verbose and manipulative. What is the first diagnosis?

Answer

The reward model is being exploited under distribution shift. Static pair accuracy wasn't enough to prove that it would rank fresh policy outputs the way humans do.

You have a clean offline preference dataset and no need to score large candidate pools or run PPO. What simpler baseline should you evaluate first?

Answer

DPO. If you don't need an explicit learned scalar judge in the loop, it removes the separate reward-model stage.

Common pitfalls

Symptom: loss falls but held-out rankings barely improve

• Cause: chosen and rejected responses are nearly equivalent, so the pair gives little ranking signal.
• Fix: audit pair strength before training. Keep pairs where the preference is clear, policy-relevant, and tied to the same prompt.

Symptom: one labeler style dominates the reward model

• Cause: inconsistent or narrow annotator preferences become inconsistent rewards.
• Fix: measure agreement, review disagreements, and separate policy rules from personal style before training.

Symptom: PPO reward rises while human preference gets worse

• Cause: the policy has found a shortcut in the reward model under distribution shift.
• Fix: add fresh policy-output evaluations, length-matched checks, adversarial probes, and human preference gates before trusting the scalar reward.

Symptom: product dashboards treat reward as truth

• Cause: reward is being mistaken for the business or human objective itself.
• Fix: report reward beside held-out preference, refusal quality, helpfulness, safety, and downstream product metrics.