Reward Modeling from Preference Data

LoRA adapts a model's behavior cheaply. Preference alignment starts from the next training problem: once a model can answer, how do we teach it which answer people prefer?

Reinforcement Learning from Human Feedback (RLHF) diagrams often make reward modeling look trivial: collect preferences, train reward model, run Proximal Policy Optimization (PPO). In practice, the reward model is its own training project. If it learns the wrong shortcuts, policy optimization will happily amplify them.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

Start by isolating that stage. Before you think about PPO, Group Relative Policy Optimization (GRPO), or online exploration, explain what a reward model sees, what loss it optimizes, what metrics it logs, and how it fails.

What reward modeling is trying to learn

A scalar reward model doesn't generate text. It scores text.

Given a prompt x and two candidate answers:

• y+ chosen by the labeler
• y- rejected by the labeler

the reward model should assign:

That scalar score is later useful in two different ways:

1. as a scalar reward signal for PPO-style RLHF^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}
2. as an inspectable ranking signal when you want to compare policy outputs

Explicit reward models remain relevant even though Direct Preference Optimization (DPO) can skip them for offline preference optimization.^{[2]Reference 2Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290}

What the dataset looks like

The core supervision format is a preference pair.

Standard format

Conversational format

Hugging Face TRL supports both standard and conversational preference formats and can apply the model's chat template automatically during reward-model training.^{[3]Reference 3Reward Modeling.https://huggingface.co/docs/trl/reward_trainer}

A pair contract before training

A row isn't ready merely because it has chosen and rejected columns. For every binary preference pair, enforce:

• the same prompt, system message, tool context, and rendering template on both candidates
• two different candidate answers and a definite preference label
• separate handling for ties, abstentions, and ambiguous or low-agreement labels
• provenance such as source prompt, generator checkpoint, sampling settings, and labeling batch

That last field is important for splitting. If one prompt generated several candidates, its comparisons are near-duplicates. Put the entire prompt or candidate-generation group in either train or evaluation, never both.

The usual architecture

Most practical reward models aren't built from scratch. You start from a pretrained or SFT checkpoint and attach a scalar sequence-level head. Conceptually:

For a decoder-only LM, that representation is often taken from the final non-padding position, then passed through a one-unit score head. Reward modeling feels like sequence classification with pairwise labels rather than generation. The output is one number per candidate response, not the next token distribution.

The rendered sequence is part of the contract. Keep chat-template and end-of-sequence conventions consistent with the policy being evaluated. Also don't silently train on answers whose decisive ending was truncated: current TRL RewardConfig.max_length filters a pair when either candidate exceeds the configured maximum after tokenization.^{[3]Reference 3Reward Modeling.https://huggingface.co/docs/trl/reward_trainer}

Bradley-Terry loss in one page

The classic formulation says the probability that the chosen response wins depends on the reward difference:

Start with a tiny comparison. Suppose the policy-correct access answer gets reward 1.8, while the unsupported approval gets 0.7. The margin is 1.8 - 0.7 = 1.1. A positive margin means the reward model prefers the helpful answer. The Bradley-Terry model turns that margin into a preference probability: $\sigma(1.1) \approx 0.75$.

$$P(y^+ \succ y^- \mid x) = \sigma\big(r(x, y^+) - r(x, y^-)\big)$$

where σ is the sigmoid function.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

The loss maximizes the log-probability of the observed preference:

$$\mathcal{L} = -\log \sigma\big(r(x, y^+) - r(x, y^-)\big)$$

If the chosen reward is much higher than the rejected reward, loss becomes small. If the model ranks them backwards, loss becomes large.

Why does a larger positive reward margin make the Bradley-Terry loss smaller?

Answer

Because the loss is -log sigma(r(chosen) - r(rejected)). As the chosen-minus-rejected margin grows, the sigmoid term moves closer to 1, so the negative log shrinks toward 0.

That's the ranking core. Everything else in reward modeling is about making sure the data and evaluation around that loss are strong enough to be trusted.

Multi-way preferences, margin BT, and score temperature

Binary chosen/rejected pairs are the default training unit. Production preference sets often produce $K>2$ completions per prompt. Two standard expansions:

• Pair expansion (tournament / all pairs). From $K$ ranked or labeled candidates, emit every ordered pair that the rubric can score (or sample a subset). Keep the same prompt-group split so no completion from a held-out prompt leaks into train.
• Plackett-Luce. Model a full ranking as a product of successive choices rather than independent pairs. Use it when you have true multi-way rankings and want one joint likelihood; pair expansion remains the simpler baseline when only sparse pairwise labels exist.

Some trainers (including TRL-style margin variants) use a margin Bradley-Terry form:

$$\mathcal{L} = -\log \sigma\big(r(x, y^+) - r(x, y^-) - m\big)$$

with margin $m \ge 0$. Positive $m$ requires the chosen score to beat the rejected score by more than $m$ before the loss saturates, which can reduce near-ties on easy pairs. Treat $m$ as a hyperparameter validated on held-out pair accuracy and fresh-policy human agreement, not a free constant.

A temperature or inverse-temperature $\beta$ that multiplies the reward difference, $\sigma\big(\beta(r^+ - r^-)\big)$, is the same family of multiplicative score scaling already discussed under centering: it changes loss confidence and downstream RL strength without changing the additive invariance of pure ranking. Log any $\beta$ or $m$ next to the reward checkpoint.

What you should monitor during training

The TRL reward-model guide logs more than loss for a reason.^{[3]Reference 3Reward Modeling.https://huggingface.co/docs/trl/reward_trainer}

Useful metrics include:

• pair accuracy: how often chosen beats rejected
• margin: average r(chosen) - r(rejected)
• mean/min/max reward: catch drift or exploding scale
• gradient norm: catch unstable updates
• held-out preference quality: does the ranking still match human judgment off the train split?

Loss alone isn't enough. A reward model can lower training loss by overfitting to easy stylistic cues that don't hold up under real policy outputs.

Centering and calibration

Reward models are underdetermined up to an additive constant: adding the same number to every score leaves every margin and the Bradley-Terry loss unchanged. Scaling scores is different. It preserves a ranking but changes the loss confidence and the strength of a reward signal consumed by an optimizer.

Operationally, that matters because:

• absolute reward level and score scale can drift over training
• PPO-style optimization becomes sensitive to reward scale
• long verbose answers can look better than they are if the reward model learned a shallow heuristic

TRL exposes center_rewards_coefficient to encourage mean-zero rewards. It's a centering aid, not proof that reward magnitude is calibrated for policy optimization.^{[3]Reference 3Reward Modeling.https://huggingface.co/docs/trl/reward_trainer}

A tiny reward audit

Pair accuracy can look healthy while the reward model still learns a bad shortcut. This tiny audit separates pair accuracy from length bias. The model ranks all three preference pairs correctly, but its reward is suspiciously correlated with response length.

Annotator disagreement is another failure signal. A pair can be formatted correctly and still be weak supervision if raters don't agree about which completion is better. The toy gate below routes any disputed label to review; a real pipeline may adjudicate, weight, or retain disagreements for a dedicated evaluation slice.

The real evaluation question

The core question isn't whether the reward model fits the training pairs. It's whether, when the current policy produces fresh responses, the reward model still ranks them the way humans would. That's the distribution-shift problem.

As the policy improves, it starts producing answers unlike the ones in the original preference dataset. The reward model may then score confidently for the wrong reasons. This is one path to reward hacking, a concrete instance of Goodhart's law: once a proxy metric (the learned reward) becomes the optimization target, it can stop tracking the thing you cared about (real human preference).^{[4]Reference 4Scaling Laws for Reward Model Overoptimizationhttps://arxiv.org/abs/2210.10760}

• the reward rises
• held-out human preference stops rising
• human raters see longer, repetitive, or otherwise worse answers

If you don't monitor that gap, policy optimization can amplify the shortcut. The threshold below is illustrative; set release gates from your evaluation design and risk tolerance.

KL control intuition (preview)

A common control, developed in full by the next lesson, penalizes the policy for drifting too far from its reference checkpoint. Optimization maximizes reward minus a KL-divergence term measuring policy drift.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} This discourages large departures from the reference policy, but doesn't certify that the reward model is valid on new outputs. Keep the claim narrow: a reward model is a local approximation of human preference, not a global truth, which is exactly why it can be overoptimized.

Standardized evaluation: RewardBench

Held-out pairs you wrote yourself can share your blind spots. RewardBench is an Ai2 benchmark that scores a reward model by how often it ranks a known-better completion above a worse one. Its sections cover chat, harder instruction-following comparisons, safety, reasoning, and prior preference test sets.^{[5]Reference 5RewardBench: Evaluating Reward Models for Language Modelinghttps://arxiv.org/abs/2403.13787} RewardBench 2 reports a harder multi-skill, best-of-four evaluation using mostly previously unused human prompts and verifies no overlap with the downstream evaluations it compares against. In its experiments, benchmark scores correlate with best-of-N performance and provide a useful signal for PPO, rather than only measuring static pair accuracy.^{[6]Reference 6RewardBench 2: Advancing Reward Model Evaluationhttps://arxiv.org/abs/2506.01937}

One practical caveat from that work: the highest-scoring reward model on the leaderboard isn't automatically the best choice for your run. In the paper's PPO experiments, reward models based on the same model lineage as the policy transferred better than mismatched choices. Treat absolute benchmark rank as a filter, then validate with the policy and optimization setup you'll use.^{[6]Reference 6RewardBench 2: Advancing Reward Model Evaluationhttps://arxiv.org/abs/2506.01937}

Your reward model tops the RewardBench leaderboard. Is it automatically the right choice for your PPO run?

Answer

No. A high benchmark score is a good filter, but RewardBench 2 reports better PPO transfer for same-lineage reward and policy models in its experiments. Validate with your intended policy and optimization setup.

Beyond scalar reward heads

The scalar Bradley-Terry head is a common baseline, but the space has widened.

• Generative reward models (LLM-as-judge). Instead of a scalar head, an LM can read candidates and emit a verdict, optionally with a rationale. Mahan et al. report improvements from rationale generation and vote aggregation in their studied setup; these are design choices to evaluate, not universal guarantees.^{[7]Reference 7Generative Reward Modelshttps://arxiv.org/abs/2410.12832}
• Process reward models (PRMs). For multi-step reasoning, scoring only the final answer is a weak signal. PRMs score each step of a chain of thought, giving denser supervision. "Let's Verify Step by Step" showed step-level supervision beating outcome-only reward models on hard math.^{[8]Reference 8Let's Verify Step by Step.https://arxiv.org/abs/2305.20050}
• Verifiers and RLVR. When correctness is checkable, such as a final math answer or passing unit tests, verifiable rewards can reduce dependence on a learned preference proxy. They don't eliminate misspecified tests, partial specifications, or gaming of the verifier.^{[9]Reference 9Tülu 3: Pushing Frontiers in Open Language Model Post-Traininghttps://arxiv.org/abs/2411.15124} The next chapters cover this family.

These don't retire the scalar reward model, and they don't all share its Bradley-Terry objective. The reusable lesson is the evaluation discipline: inspect the signal's coverage, test it on outputs produced by the system being optimized, and watch for optimization exploiting its blind spots.

When an explicit reward model is worth the cost

Use one when:

• you want PPO-style online optimization
• you want to score many candidate outputs with one scalar model
• you want to inspect and audit the preference signal directly

Start with DPO when:

• you have a clean offline preference dataset
• you want the simpler baseline first
• you don't need an explicit learned judge in the loop

That trade-off is why DPO is a strong offline-preference baseline: it removes the separate reward-model training stage.^{[2]Reference 2Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} But it doesn't supply a reusable scalar judge for PPO or candidate scoring. This final toy gate combines several checks, with project-specific thresholds, to block optimization when only one static slice passes.

Your reward model has good static pair accuracy, but once PPO starts, reward climbs while human raters say answers are getting verbose and manipulative. What is the first diagnosis?

Answer

The reward model is being exploited under distribution shift. Static pair accuracy wasn't enough to prove that it would rank fresh policy outputs the way humans do.

You have a clean offline preference dataset and no need to score large candidate pools or run PPO. What simpler baseline should you evaluate first?

Answer

DPO. If you don't need an explicit learned scalar judge in the loop, it removes the separate reward-model stage.

Common pitfalls

Symptom: loss falls but held-out rankings barely improve

• Cause: chosen and rejected responses are nearly equivalent, so the pair gives little ranking signal.
• Fix: audit pair strength before training. Keep pairs where the preference is clear, policy-relevant, and tied to the same prompt.

Symptom: one labeler style dominates the reward model

• Cause: inconsistent or narrow annotator preferences become inconsistent rewards.
• Fix: measure agreement, review disagreements, and separate policy rules from personal style before training.

Symptom: PPO reward rises while human preference gets worse

• Cause: the policy has found a shortcut in the reward model under distribution shift.
• Fix: add fresh policy-output evaluations, length-matched checks, adversarial probes, and human preference gates before trusting the scalar reward.

Symptom: product dashboards treat reward as truth

• Cause: reward is being mistaken for the business or human objective itself.
• Fix: report reward beside held-out preference, refusal quality, helpfulness, safety, and downstream product metrics.