Skip to content
Skip to lesson
Back to Topics
LearnAdvanced Training & AdaptationReward Modeling from Preference Data
🛡️HardAlignment & Safety

Reward Modeling from Preference Data

A longer answer can still be the wrong one. Fit a scalar reward on chosen and rejected pairs, then test whether the score learned technical accuracy or a shortcut such as length.

47 min read
Learning path
Step 114 of 199 in the full curriculum

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

A developer asks a coding assistant how to prevent exponential overflow when implementing softmax(z) in Python. One reply gives the exact fix in two sentences: subtract max(z) from every logit before calling exp(z) so the largest exponent is zero. A second reply looks polished and friendly, yet advises casting the array to float64 after computing exp(z), which still overflows inside exp(z) before any cast runs. Which answer should earn the higher score?

trains a network to imitate demonstration tokens, whereas reward modeling trains an evaluative judge to separate preferred completions from rejected ones on the same prompt. A pairwise reward model assigns every completed prompt-response pair a single real number that downstream policy optimizers can rank or optimize. System diagrams for (Reinforcement Learning from Human Feedback) often compress that judge into a single box before jumping to , yet that box is simply a neural network trained on noisy human comparisons.[1] If the scorer learns to reward extra length or confident phrasing instead of technical correctness, later policy optimization will amplify the mistake rather than fixing it.[2] Throughout this lesson, we will return to this softmax stability question to test whether a trained scorer learned the underlying bug fix or a superficial shortcut.

How a scalar judge scores a completed answer

An autoregressive language model outputs a conditional probability distribution over the vocabulary VV at every token position, πθ(yt∣x,y<t)∈Δ∣V∣−1\pi_\theta(y_t \mid x, y_{<t}) \in \Delta^{|V|-1}. A instead evaluates the entire prompt and completed response (x,y)(x, y) together. It passes the combined sequence through a transformer backbone and projects the final hidden state down to a single real-valued scalar:

rθ(x,y)∈Rr_\theta(x, y) \in \mathbb{R}

For our softmax prompt xx, an annotator compares two candidate completions: a chosen completion ywy_w that subtracts max(z) before exp(z), and a rejected completion yly_l that casts exp(z) after the overflow has already happened. The training objective pushes the model to satisfy the strict ranking inequality r(x,yw)>r(x,yl)r(x, y_w) > r(x, y_l). Because pairwise comparisons record only relative order, adding a constant offset to every score preserves every ranking, leaving the raw origin arbitrary. A score of 2.0 does not mean twice as helpful as 1.0, and you should calibrate two independently trained reward models on shared data before comparing their raw outputs. At the same time, scaling all scores stretches their pairwise gaps, which directly changes Bradley-Terry win probabilities and downstream policy optimization pressure.

That scalar score supports two distinct downstream workflows in post-training pipelines, depending on whether you are generating rollouts online or filtering candidates at inference time:

  1. Terminal reward for online RL: The scorer evaluates completed policy rollouts as a sequence-level return. PPO pairs those rewards with a learned value baseline using Generalized Advantage Estimation (GAE), whereas outcome-supervised GRPO normalizes rewards relative to a group of completions sampled from the same prompt without a separate value network.[3]
  2. Best-of-N selection: During evaluation or synthetic dataset curation, a generator samples NN candidate completions for prompt xx and selects arg⁡max⁡ir(x,yi)\arg\max_i r(x, y_i). People often call this rejection sampling in post-training recipes, even though taking the argmax over NN draws does not sample directly from a target distribution.

uses the exact analytic link between a reward function and its optimal KL-regularized policy to reparameterize rewards as policy-to-reference log-ratios, turning pairwise preference loss directly into a policy objective.[4] The basic offline DPO loop updates the language model from labeled pairs without maintaining a separate scorer network. An explicit reward model remains valuable whenever you want fast reusable ranking across fresh rollouts, best-of-NN filtering, or shared feedback signals alongside generative judges and executable verifiers.[5]

Preference data anatomy and validation pipeline

Before training a scorer, inspect the exact structure and labels in your comparison dataset. Pairwise cross-entropy minimizes loss on whatever ordering the labels record, including formatting mismatches, duplicate candidates, and systematic annotator habits. Checking the raw pairs early prevents your training run from spending compute on degenerate comparisons.

Standard and conversational preference formats

In standard explicit-prompt datasets, each row holds the shared query in prompt alongside competing completions in chosen and rejected. The prompt appears once so both responses share the identical context prefix when tokenized, preventing minor formatting discrepancies in the prompt from leaking into score margins. Keeping the prompt separated also makes group-by-prompt dataset splitting straightforward during validation:

preference_pair.json
{
  "prompt": "How do you prevent exponential overflow when implementing softmax in Python?",
  "chosen": "Subtract max(z) from each logit before calling exp(z) so the largest exponent is zero.",
  "rejected": "Cast the output array to float64 after calling exp(z) so large values fit safely."
}

Multi-turn dialogues store full role-tagged message lists for both chosen and rejected. Both transcripts must match token for token across the system message and prior user turns, diverging only at the final assistant message under comparison. If an earlier system directive or intermediate tool call differs between the two transcripts, the reward head can pick up context noise instead of scoring assistant quality:

chat_preference_pair.json
{
  "chosen": [
    {"role": "system", "content": "Give concise, numerically accurate Python advice."},
    {"role": "user", "content": "How do you prevent exponential overflow when implementing softmax in Python?"},
    {"role": "assistant", "content": "Subtract max(z) from each logit before calling exp(z) so the largest exponent is zero."}
  ],
  "rejected": [
    {"role": "system", "content": "Give concise, numerically accurate Python advice."},
    {"role": "user", "content": "How do you prevent exponential overflow when implementing softmax in Python?"},
    {"role": "assistant", "content": "Cast the output array to float64 after calling exp(z) so large values fit safely."}
  ]
}

Hugging Face TRL's RewardTrainer accepts both plain strings and conversational message lists, whether the prompt is implicit or explicit. Conversational examples are formatted through the tokenizer's , whereas raw strings are concatenated and tokenized directly without applying conversation delimiters. These behaviors match TRL 1.17.1 as verified on November 4, 2026.[6]

Pair validation before training

Having chosen and rejected columns in a table does not make a dataset ready for gradient descent. Malformed rows such as mismatched prompts, duplicate completions, or non-binary tie labels distort loss magnitudes or silently zero out gradients during minibatch updates. Before tokenizing pairs into minibatches, verify four structural invariants across every record so training gradients reflect actual response quality rather than preprocessing bugs:

  • Identical context prefix: Both completions must receive the exact same user prompt, system instructions, tool declarations, and chat template formatting.
  • Distinct scored inputs: If normalization, truncation, or templating collapses chosen and rejected into identical token IDs and attention masks, a shared deterministic scorer outputs identical values, giving zero margin and zero parameter gradient. Verify uniqueness both before and after tokenization.
  • Labels aligned with the loss: Winner-only Bradley-Terry loss expects strict preference outcomes, while ties and ambiguous ratings need separate handling or soft targets. Low annotator agreement can signal an unclear rubric, an error, or genuine stylistic disagreement, so inspect low-agreement clusters instead of dropping them blindly.
  • Candidate lineage metadata: Record the generator checkpoint, decoding temperature, and reviewer cohort for each completion so downstream audits can isolate source artifacts.

The validation check below keeps only strict chosen-over-rejected rows whose prompts match and whose completions differ cleanly. Text checks catch surface defects first, while a full pipeline repeats the uniqueness check after running the tokenizer:

preference_pair_checks.py
from collections import Counter

pairs = [
    {"id": "a", "prompt_left": "softmax?", "prompt_right": "softmax?", "chosen": "Subtract max(z).", "rejected": "Cast after exp(z).", "label": "chosen"},
    {"id": "b", "prompt_left": "softmax?", "prompt_right": "softmax?", "chosen": "Subtract max(z).", "rejected": "Subtract max(z).", "label": "chosen"},
    {"id": "c", "prompt_left": "softmax?", "prompt_right": "attention?", "chosen": "Subtract max(z).", "rejected": "Scale by sqrt(d).", "label": "chosen"},
    {"id": "d", "prompt_left": "attention?", "prompt_right": "attention?", "chosen": "Scale by sqrt(d).", "rejected": "Omit scaling.", "label": "tie"},
]

def rejection_reason(pair):
    fields = ("prompt_left", "prompt_right", "chosen", "rejected")
    if any(not isinstance(pair.get(field), str) or not pair[field].strip() for field in fields):
        return "missing_or_empty_text"
    if pair.get("label") != "chosen":
        return "tie_or_abstention"
    if pair["prompt_left"] != pair["prompt_right"]:
        return "context_mismatch"
    if pair["chosen"].strip() == pair["rejected"].strip():
        return "identical_candidates"
    return None

reasons = Counter(reason for pair in pairs if (reason := rejection_reason(pair)))
kept = [pair["id"] for pair in pairs if rejection_reason(pair) is None]

print(f"kept={kept}")
print(f"rejected={dict(sorted(reasons.items()))}")
assert kept == ["a"]
assert rejection_reason({**pairs[0], "chosen": "  "}) == "missing_or_empty_text"
assert rejection_reason({}) == "missing_or_empty_text"
Pair validation output
kept=['a']
rejected={'context_mismatch': 1, 'identical_candidates': 1, 'tie_or_abstention': 1}

Choose an evaluation split that matches generalization

A single prompt with four candidate answers yields six unordered pairs. If you shuffle and split those pairs row by row, the exact same prompt and even the same candidate completion appear in both training and validation splits, turning validation accuracy into a test of memorization rather than unseen-prompt generalization. To measure generalization to new prompts, group comparisons by prompt_id, cluster near-duplicate prompts, and assign every pair from a prompt to the same split. If your actual goal is ranking fresh answers to a fixed prompt set, use a response-held-out split instead and state that choice explicitly so evaluation metrics are never misread.

grouped_preference_split.py
pairs = [
    {"pair_id": "a-b", "prompt_id": "softmax-17"},
    {"pair_id": "a-c", "prompt_id": "softmax-17"},
    {"pair_id": "d-e", "prompt_id": "attention-04"},
    {"pair_id": "f-g", "prompt_id": "pooling-09"},
]
eval_prompt_ids = {"softmax-17"}

train = [pair for pair in pairs if pair["prompt_id"] not in eval_prompt_ids]
evaluation = [pair for pair in pairs if pair["prompt_id"] in eval_prompt_ids]
overlap = {pair["prompt_id"] for pair in train} & {pair["prompt_id"] for pair in evaluation}

assert not overlap
print(f"train_pairs={len(train)} eval_pairs={len(evaluation)}")
print(f"prompt_overlap={sorted(overlap)}")
Grouped split output
train_pairs=2 eval_pairs=2
prompt_overlap=[]

InstructGPT collected rankings over K=4K=4 to K=9K=9 completions per prompt, yielding (K2)\binom{K}{2} correlated comparisons per prompt.[1] Shuffling those comparisons into independent training rows led to rapid overfitting, so the authors placed all (K2)\binom{K}{2} pairs from one prompt into a single batch item. Doing so lets the forward pass score each of the KK responses once and average the pairwise losses within the prompt group. Without within-prompt averaging, a prompt with K=9K=9 responses receives 36 times as much loss weight as a prompt with K=2K=2 responses, biasing updates toward prompts that happened to have larger candidate pools.

Turning a language model into a scalar scorer

A generative transformer already reads prompts and candidate answers, so turning it into a reward model requires minimal architectural changes. InstructGPT initialized its reward models from an SFT checkpoint in the main text, while noting in its appendix that the final model used a checkpoint fine-tuned on public NLP datasets with similar outcomes.[1] Llama 3 instead initialized its reward models directly from a pretrained checkpoint.[7] Pick a backbone with suitable reasoning capacity, context window, and template compatibility, then replace its vocabulary projection head with a linear score projection:

  1. Remove the unembedding projection layer (Wlm∈Rd×∣V∣W_{\text{lm}} \in \mathbb{R}^{d \times |V|}).
  2. Attach a scalar linear projection (Wr∈Rd×1W_r \in \mathbb{R}^{d \times 1}); any additive scalar bias cancels out of score differences and receives zero gradient from pairwise loss alone.
  3. Train all weights, train LoRA adapters alongside the scalar head, or train the scalar head over a frozen backbone. When saving LoRA runs, ensure the scalar head parameters are marked trainable and exported together with the adapters.[8]
text
Input Tokens -> Transformer Hidden States (d) -> Last-Token Representation -> Linear Head -> Scalar Reward r(x, y)

Final real-token pooling and sequence padding

In standard causal decoder attention, token position tt attends only to positions j≤tj \le t, including itself on the diagonal. Consequently, any token before the end of the answer lacks hidden-state information from the remaining suffix. Reading from the final non-padding token, typically the end-of-turn delimiter at the close of the response, ensures the representation summarizes the entire prompt and response. Other architectures sometimes append a dedicated scoring token or pool multiple positions, but in every design the readout must observe the complete text it judges.

Contiguous batch padding moves that final real token index across examples in a batch. Under left padding, the final non-padding token sits at index -1 for every nonempty sequence. Under contiguous right padding, it sits at mask.sum(dim=-1) - 1, assuming real tokens begin at index zero without internal gaps. Never assume a transformer leaves harmless zeros at masked padding slots, because attention quirks or positional encodings can produce large activations there; always gather the last unmasked index explicitly.

Filtering long sequences vs. truncating suffixes

Sequence-length limits pose a trade-off between memory efficiency and label fidelity. If a human annotator rejected a response because of a subtle overflow bug in the final code block, right-truncating the response removes the exact reason for the preference label. Filtering oversized pairs preserves label meaning at the cost of altering the prompt-length distribution, so log which pairs and task types are excluded. In TRL 1.17, RewardConfig.max_length defaults to 1024 tokens covering the full templated prompt plus completion, dropping pairs that exceed the limit rather than slicing their endings unless set to None.[6][9] The short length check below demonstrates that boundary rule by keeping every pair up to max_length inclusive while excluding oversized candidates whole:

sequence_length_filter.py
max_length = 1024
pairs = [
    {"id": "fits", "chosen_tokens": 412, "rejected_tokens": 390},
    {"id": "at_limit", "chosen_tokens": 1024, "rejected_tokens": 1024},
    {"id": "chosen_too_long", "chosen_tokens": 1088, "rejected_tokens": 401},
    {"id": "rejected_too_long", "chosen_tokens": 288, "rejected_tokens": 1030},
]

kept = [p["id"] for p in pairs if max(p["chosen_tokens"], p["rejected_tokens"]) <= max_length]
dropped = [p["id"] for p in pairs if p["id"] not in kept]

print(f"kept={kept}")
print(f"dropped_instead_of_truncated={dropped}")
assert kept == ["fits", "at_limit"]
Sequence-length filter output
kept=['fits', 'at_limit']
dropped_instead_of_truncated=['chosen_too_long', 'rejected_too_long']

The pooling check next verifies with NumPy that masking selects the exact final real hidden vector [2.0, 3.0] regardless of whether padding tokens ([99.0, 99.0]) sit on the right or on the left:

last_nonpadding_reward.py
import numpy as np

def last_nonpadding(hidden, mask):
    hidden, mask = np.asarray(hidden), np.asarray(mask)
    if hidden.ndim != 3 or mask.shape != hidden.shape[:2]:
        raise ValueError("expected hidden [batch, time, width] and matching mask")
    if not np.isin(mask, [0, 1]).all() or not mask.any(axis=1).all():
        raise ValueError("each sequence needs at least one unmasked position")
    positions = np.arange(mask.shape[1])
    last = np.where(mask == 1, positions, -1).max(axis=1)
    return hidden[np.arange(len(hidden)), last]

states = np.array([[[1., 0.], [2., 3.], [99., 99.]],
                   [[99., 99.], [1., 0.], [2., 3.]]])
pooled = last_nonpadding(states, [[1, 1, 0], [0, 1, 1]])
scores = pooled @ np.array([0.5, 1.0])
assert np.allclose(scores, [4.0, 4.0])
try:
    last_nonpadding(states[:1], [[0, 0, 0]])
except ValueError:
    pass
else:
    raise AssertionError("empty sequence accepted")
print("right/left-padded scores:", scores.tolist())
Reward pooling index output
right/left-padded scores: [4.0, 4.0]
Right-padded tokens have mask [1, 1, 0], and left-padded tokens have mask [0, 1, 1]. Both select the final real hidden state [2, 3] and ignore the padding hidden state [99, 99]. The shared reward head [0.5, 1.0] gives score 4.0 in either layout.
Mask-indexed readout selects the final unmasked token regardless of padding side, producing score 4.0 for both sequences while ignoring padding values.

Bradley-Terry formulation, BCE loss, and gradient dynamics

Suppose our trained head scores the two softmax completions as r(x,yw)=1.80r(x, y_w) = 1.80 for subtracting max(z) first and r(x,yl)=0.70r(x, y_l) = 0.70 for casting after exp(z). Their score difference is Δ=1.80−0.70=+1.10\Delta = 1.80 - 0.70 = +1.10. The Bradley-Terry (1952) model converts that difference into an implied preference probability via normalized exponentials:

P(yw≻yl∣x)=exp⁡(r(x,yw))exp⁡(r(x,yw))+exp⁡(r(x,yl))=11+exp⁡(−(r(x,yw)−r(x,yl)))=σ(r(x,yw)−r(x,yl))P(y_w \succ y_l \mid x) = \frac{\exp(r(x, y_w))}{\exp(r(x, y_w)) + \exp(r(x, y_l))} = \frac{1}{1 + \exp\big(-(r(x, y_w) - r(x, y_l))\big)} = \sigma\big(r(x, y_w) - r(x, y_l)\big)

Substituting Δ=1.10\Delta = 1.10 gives P(yw≻yl∣x)=σ(1.10)≈0.7503P(y_w \succ y_l \mid x) = \sigma(1.10) \approx 0.7503. Any positive margin yields an implied win probability above 50%, while a margin of zero yields 0.50. Keep in mind that σ(Δ)\sigma(\Delta) is the model's parameterized probability rather than a calibrated frequency across human raters, nor can a 0.50 score distinguish genuine annotator uncertainty from an untried pair.

The same scalar reward head scores a chosen completion at 1.80 and a rejected completion at 0.70. Their 1.10 margin lands at 0.7503 on the sigmoid curve, giving pair loss 0.2873.
Bradley-Terry maps the score margin 1.80 − 0.70 = 1.10 through the logistic sigmoid to model-implied win probability 0.7503 and negative log-likelihood 0.2873.

Direct equivalence to binary cross-entropy with logits

Fitting the reward parameters maximizes the log-likelihood of observed preferences over dataset D\mathcal{D}, which minimizes the expected negative log-likelihood:

LBT(θ)=−E(x,yw,yl)∼D[log⁡σ(rθ(x,yw)−rθ(x,yl))]\mathcal{L}_{\text{BT}}(\theta) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \Big[ \log \sigma\big(r_\theta(x, y_w) - r_\theta(x, y_l)\big) \Big]

This objective is identical to binary cross-entropy (BCE) with target z=1z = 1 evaluated on the scalar difference Δ\Delta: LBCE(Δ,z=1)=−1⋅log⁡σ(Δ)−(1−1)⋅log⁡(1−σ(Δ))=−log⁡σ(Δ)\mathcal{L}_{\text{BCE}}(\Delta, z=1) = -1\cdot\log\sigma(\Delta) - (1-1)\cdot\log(1-\sigma(\Delta)) = -\log\sigma(\Delta). Passing the logit margin directly into PyTorch's binary_cross_entropy_with_logits (or softplus(-logits_diff)) avoids underflow when σ(Δ)\sigma(\Delta) rounds to zero. In the PyTorch check below, we also add a shared bias = 10.0 to both candidates to verify that a constant score shift receives exact zero gradient from the pairwise objective:

bce_loss_equivalence.py
import torch

# One row per pair, columns chosen/rejected.
rewards = torch.tensor([[1.8, 0.7]], dtype=torch.float64, requires_grad=True)
bias = torch.tensor(10.0, dtype=torch.float64, requires_grad=True)
scores = rewards + bias
logits_diff = scores[:, 0] - scores[:, 1]
loss = torch.nn.functional.binary_cross_entropy_with_logits(
    logits_diff, torch.ones_like(logits_diff)
)
assert torch.allclose(loss, torch.nn.functional.softplus(-logits_diff).mean())
loss.backward()
expected = torch.sigmoid(logits_diff.detach()) - 1
assert torch.allclose(rewards.grad[:, 0], expected)
assert torch.allclose(rewards.grad[:, 1], -expected)
assert bias.grad.item() == 0.0
print(f"loss={loss.item():.4f}")
print(f"chosen_gradient={rewards.grad[0, 0].item():+.4f} rejected_gradient={rewards.grad[0, 1].item():+.4f}")
print(f"shared_bias_gradient={bias.grad.item():.1f}")
BCE and score-gradient output
loss=0.2873
chosen_gradient=-0.2497 rejected_gradient=+0.2497
shared_bias_gradient=0.0

Gradient dynamics and shared-weight movement

Differentiating LBT=−log⁡σ(Δ)\mathcal{L}_{\text{BT}} = -\log\sigma(\Delta) with respect to Δ=rw−rl\Delta = r_w - r_l yields:

∂L∂Δ=−1σ(Δ)⋅σ(Δ)(1−σ(Δ))=σ(Δ)−1\frac{\partial \mathcal{L}}{\partial \Delta} = -\frac{1}{\sigma(\Delta)} \cdot \sigma(\Delta)(1 - \sigma(\Delta)) = \sigma(\Delta) - 1

Applying the chain rule gives equal-magnitude, opposite-sign derivatives with respect to the individual candidate rewards:

∂L∂rw=σ(Δ)−1≤0,∂L∂rl=1−σ(Δ)≥0\frac{\partial \mathcal{L}}{\partial r_w} = \sigma(\Delta) - 1 \le 0, \qquad \frac{\partial \mathcal{L}}{\partial r_l} = 1 - \sigma(\Delta) \ge 0

These score derivatives show where training signal concentrates. When the margin is already strongly positive (Δ≫0\Delta \gg 0), σ(Δ)≈1\sigma(\Delta) \approx 1 and the pair derivatives shrink toward zero; when Δ=0\Delta = 0, the derivatives are −0.5-0.5 and +0.5+0.5; and when the model ranks the wrong answer ahead (Δ≪0\Delta \ll 0), the derivatives saturate near −1.0-1.0 and +1.0+1.0. Crucially, opposite signs on ∂L/∂rw\partial\mathcal{L}/\partial r_w and ∂L/∂rl\partial\mathcal{L}/\partial r_l do not force the actual scores rwr_w and rlr_l to move in opposite directions during optimization, because both scores share parameters θ\theta. Suppose a single scalar head r=whr = wh starts at w=−1w = -1 on frozen features hw=1h_w = 1 and hl=2h_l = 2, giving initial scores −1-1 and −2-2. One SGD step makes ww more negative, so both scores drop while the rejected score drops twice as fast, widening the margin. Monitoring individual rewards without logging their margin can therefore create false alarms when valid margins are actually growing:

shared_reward_head_update.py
import torch

weight = torch.nn.Parameter(torch.tensor(-1.0, dtype=torch.float64))
features = torch.tensor([1.0, 2.0], dtype=torch.float64)
optimizer = torch.optim.SGD([weight], lr=0.1)
before = (weight * features).detach().clone()
scores = weight * features
loss = torch.nn.functional.softplus(-(scores[0] - scores[1]))
loss.backward()
assert torch.allclose(weight.grad, torch.sigmoid(torch.tensor(-1.0, dtype=torch.float64)))
optimizer.step()
after = (weight * features).detach()
assert torch.all(after < before)
assert (after[0] - after[1]) > (before[0] - before[1])
print("before:", [round(value, 4) for value in before.tolist()])
print("after: ", [round(value, 4) for value in after.tolist()])
print(f"margin: {before[0] - before[1]:.4f} -> {after[0] - after[1]:.4f}")
Shared-head score movement
before: [-1.0, -2.0]
after:  [-1.0269, -2.0538]
margin: 1.0000 -> 1.0269
One shared scalar weight starts at minus 1.0000 and an SGD step with learning rate 0.1 moves it to minus 1.0269. With chosen feature 1 and rejected feature 2, both scores fall: minus 1.0000 to minus 1.0269 and minus 2.0000 to minus 2.0538. The rejected score moves twice as far, widening the margin from 1.0000 to 1.0269.
A shared-parameter PyTorch gradient step lowers both scores simultaneously while moving the rejected score twice as far, widening the margin from 1.0000 to 1.0269.

Translation invariance vs. scale sensitivity

Adding any prompt-dependent scalar offset c(x)c(x) to both completions leaves the pairwise difference Δ′=(rw+c(x))−(rl+c(x))=rw−rl\Delta' = (r_w + c(x)) - (r_l + c(x)) = r_w - r_l strictly unchanged, so scores (11.80, 10.70) yield the exact same loss, probability, and gradients as (1.80, 0.70). Multiplying both scores by 3, however, triples the margin from 1.101.10 to 3.303.30, driving the sigmoid probability to 0.96440.9644 and shrinking the loss. Because separable preference batches allow the network to decrease loss indefinitely simply by scaling weights upward, regularization and held-out probability evaluation remain important. Below, we verify this contrast directly with NumPy across shifted and scaled margin vectors:

reward_model_loss.py
import numpy as np

chosen_rewards = np.array([2.1, 0.8, 1.9])
rejected_rewards = np.array([0.4, 1.0, 1.2])
margins = chosen_rewards - rejected_rewards

def pairwise_bt_loss(delta):
    return np.logaddexp(0.0, -delta)

losses = pairwise_bt_loss(margins)
print("margins:", np.round(margins, 3).tolist())
print("reward_loss:", round(float(losses.mean()), 4))
print("pair_accuracy:", round(float((margins > 0).mean()), 4))
for margin in [-2.0, 0.0, 2.0]:
    print(f"margin={margin:+.1f} loss={float(pairwise_bt_loss(margin)):.4f}")
assert np.isclose(pairwise_bt_loss(0.0), np.log(2.0))
assert np.isclose(pairwise_bt_loss(-1000.0), 1000.0)
assert pairwise_bt_loss(1000.0) == 0.0
Reward loss and margin invariance
margins: [1.7, -0.2, 0.7]
reward_loss: 0.4564
pair_accuracy: 0.6667
margin=-2.0 loss=2.1269
margin=+0.0 loss=0.6931
margin=+2.0 loss=0.1269

Decreasing loss tells you the margin grew larger on the training pairs, not which feature drove that separation. Next, we fit a minimal two-feature reward head to expose how correlated training pairs let a length shortcut masquerade as correctness.

Training a small reward head and isolating shortcut features

A linear reward head r=wThr = w^T h fitted on just two input features (technical correctness and scaled response length) is enough to expose how training correlations create hidden shortcuts. In a confounded training set, every correct answer is also one length unit longer than the buggy alternative, so correctness and length always increase together. In a balanced training set, a concise fix beats a verbose buggy answer, breaking that spurious correlation.

Both heads easily achieve 100% accuracy on their training pairs, yet when evaluated on counterexamples where a long padded response recommends casting after exp(z), the confounded head drops to 33% accuracy. The NumPy experiment below computes analytical logistic-loss gradients, verifies them via central finite differences, checks input shape validity, saves the fitted weights alongside feature names, and reloads them for inference:

train_tiny_reward_head.py
import json
import tempfile
from pathlib import Path
import numpy as np

def objective_and_gradient(weights, pairs, l2=0.02):
    pairs = np.asarray(pairs, dtype=float)
    if pairs.ndim != 3 or pairs.shape[1:] != (2, 2) or len(pairs) == 0:
        raise ValueError("expected nonempty [pairs, chosen/rejected, two features]")
    if not np.isfinite(pairs).all():
        raise ValueError("features must be finite")
    differences = pairs[:, 0] - pairs[:, 1]
    margins = differences @ weights
    loss = np.logaddexp(0.0, -margins).mean() + 0.5 * l2 * (weights @ weights)
    sigmoid_negative = np.exp(-np.logaddexp(0.0, margins))
    gradient = -(differences.T @ sigmoid_negative) / len(pairs) + l2 * weights
    return float(loss), gradient

def fit(pairs):
    weights = np.zeros(2)
    initial, _ = objective_and_gradient(weights, pairs)
    for _ in range(300):
        _, gradient = objective_and_gradient(weights, pairs)
        weights -= 0.1 * gradient
    final, _ = objective_and_gradient(weights, pairs)
    assert final < initial
    return weights

def accuracy(weights, pairs):
    return float(np.mean((pairs[:, 0] - pairs[:, 1]) @ weights > 0))

# Each row is [chosen representation, rejected representation] with features [correctness, scaled_length].
confounded = np.array([[[1., 1.], [0., 0.]], [[1., 2.], [0., 1.]]])
balanced = np.array([[[1., 1.], [0., 0.]], [[1., 0.], [0., 1.]]])
counterexamples = np.array([[[1., 0.], [0., 2.]],
                            [[1., 0.], [0., 3.]],
                            [[1., 1.], [0., 1.]]])

# Check analytic gradients against central finite differences.
probe = np.array([0.2, -0.3])
_, analytic = objective_and_gradient(probe, confounded)
for invalid in [np.empty((0, 2, 2)), np.zeros((2, 2)), np.full((1, 2, 2), np.nan)]:
    try:
        objective_and_gradient(probe, invalid)
        raise AssertionError("invalid pair tensor accepted")
    except ValueError:
        pass
for coordinate in range(2):
    delta = np.zeros(2)
    delta[coordinate] = 1e-6
    numerical = (objective_and_gradient(probe + delta, confounded)[0]
                 - objective_and_gradient(probe - delta, confounded)[0]) / 2e-6
    assert abs(numerical - analytic[coordinate]) < 1e-7

for name, training in [("confounded", confounded), ("balanced", balanced)]:
    weights = fit(training)
    print(f"{name}: weights={weights.round(3).tolist()}, train={accuracy(weights, training):.2f}, counterexamples={accuracy(weights, counterexamples):.2f}")
    if name == "confounded":
        assert accuracy(weights, counterexamples) == 1 / 3
    else:
        assert accuracy(weights, counterexamples) == 1.0

with tempfile.TemporaryDirectory() as directory:
    path = Path(directory) / "reward_head.json"
    path.write_text(json.dumps({"features": ["correctness", "scaled_length"], "weights": weights.tolist()}))
    artifact = json.loads(path.read_text())
    assert artifact["features"] == ["correctness", "scaled_length"]
    assert np.allclose(counterexamples @ np.array(artifact["weights"]), counterexamples @ weights)
print("gradient check and head reload: passed")
Learned reward-head shortcut
confounded: weights=[1.629, 1.629], train=1.00, counterexamples=0.33
balanced: weights=[2.667, 0.0], train=1.00, counterexamples=1.00
gradient check and head reload: passed
The first counterexample, chosen feature vector (1, 0) and verbose-error vector (0, 2), is projected by two heads fitted for 300 updates. On shared equal-unit axes, the confounded weight direction (1.629, 1.629) scores them 1.629 and 3.259; the balanced direction (2.667, 0) scores them 2.667 and 0. Both heads fit 2 of 2 training pairs, while their measured accuracy on the three controlled counterexamples is 1 of 3 and 3 of 3.
Both heads reach 2 of 2 on their respective training pairs, yet only the head trained on length-balanced pairs keeps the concise fix ahead of the verbose error (3 of 3 vs. 1 of 3).

After 300 gradient steps, the confounded head settles at w≈[1.629,1.629]w \approx [1.629, 1.629] because both features change identically across its training pairs, splitting optimization pressure evenly between correctness and length. On the first counterexample, a verbose wrong answer at (0,2)(0, 2) scores 2×1.629=3.2582 \times 1.629 = 3.258, comfortably defeating the concise fix (1,0)(1, 0) at 1.6291.629. By contrast, the balanced set has difference vectors [1,1][1, 1] and [1,−1][1, -1], so starting at zero length weight causes length gradients from the two pairs to cancel identically. Notice that L2 regularization shrinks weight magnitude rather than identifying which feature is spurious; clean contrast pairs are what eliminate the length weight in this controlled CPU experiment.

Multi-response rankings and margin-augmented losses

When labelers rank groups of K>2K > 2 candidate completions for a prompt, teams commonly convert rankings into training losses in one of two ways: expanding all (K2)\binom{K}{2} pairs and averaging within the prompt group,[1] or modeling the full permutation under a Plackett-Luce multinomial distribution. Averaging over (K2)\binom{K}{2} within each prompt is critical so prompts sampled with larger candidate pools do not dominate batch gradients. Some pipelines also capture preference strength labels such as "much better" versus "slightly better" and incorporate a rating-dependent margin m(x,yw,yl)≥0m(x, y_w, y_l) \ge 0 inside the sigmoid, setting m=0m = 0 for the weakest preference category:[10]

LMargin=−log⁡σ(r(x,yw)−r(x,yl)−m(x,yw,yl))\mathcal{L}_{\text{Margin}} = -\log \sigma\big(r(x, y_w) - r(x, y_l) - m(x, y_w, y_l)\big)

Subtracting a positive margin shifts the loss curve horizontally rather than saturating it. At rw−rl=mr_w - r_l = m, the shifted logit is zero, the loss remains log⁡2\log 2, and the gradient with respect to the margin is still −0.5-0.5. Reaching an effective win probability ρ\rho demands rw−rl≥m+log⁡(ρ/(1−ρ))r_w - r_l \ge m + \log(\rho / (1 - \rho)); for m=1.0m = 1.0 and ρ=0.9\rho = 0.9, the score difference must reach 3.1973.197. Llama 3 later removed the margin term after observing diminishing gains as dataset size scaled.[7]

preference_margin_threshold.py
import numpy as np

m = 1.0
required_difference = m + np.log(0.9 / 0.1)
differences = np.array([m, required_difference])
logits = differences - m
probabilities = 1.0 / (1.0 + np.exp(-logits))
losses = np.logaddexp(0.0, -logits)
for diff, prob, loss in zip(differences, probabilities, losses):
    print(f"score_difference={diff:.3f} effective_p={prob:.2f} loss={loss:.4f} derivative={prob - 1:+.2f}")
assert np.isclose(probabilities[1], 0.9)
Preference-margin threshold output
score_difference=1.000 effective_p=0.50 loss=0.6931 derivative=-0.50
score_difference=3.197 effective_p=0.90 loss=0.1054 derivative=-0.10

Notice that a larger margin alters loss slopes and probability separation without necessarily changing binary pair accuracy, which tracks only whether the sign of Δ\Delta is positive.

Metrics, centering invariance, and calibration

Pair accuracy reduces each comparison to a binary indicator, I(rw>rl)\mathbb{I}(r_w > r_l), concealing shrinking score margins, drifting score origins, and heavy reliance on spurious features. A model can sustain 95% accuracy on training splits even while raw margins explode or depend almost entirely on response length. Monitoring the distributions below alongside pair accuracy reveals trouble early:

Logged MetricWhat It MeasuresWhat It Conceals
Pair accuracyBinary win frequency (rw>rlr_w > r_l)Margin collapse or shortcut reliance
Mean margin E[rw−rl]\mathbb{E}[r_w - r_l]Average separation distanceProbability calibration and extreme outliers
Mean reward E[r]\mathbb{E}[r]Overall score driftOrdering quality across prompts
Gradient normOptimization step magnitudeReliance on spurious features such as length
Fresh-policy pair accuracyAgreement on newly sampled model textUnsampled failures, calibration error, and human disagreement

Centering invariance and PPO scale sensitivity

In PPO-based RLHF, policy optimization balances expected scalar reward against a KL-divergence penalty relative to a reference policy πref\pi_{\text{ref}}:

max⁡πEx∼D[Ey∼π(⋅∣x)[rθ(x,y)]−βKLDKL(π(⋅∣x)∥πref(⋅∣x))]\max_\pi \mathbb{E}_{x \sim \mathcal{D}}\left[\mathbb{E}_{y \sim \pi(\cdot\mid x)}[r_\theta(x,y)]-\beta_{\text{KL}}D_{\text{KL}}\big(\pi(\cdot\mid x)\parallel\pi_{\text{ref}}(\cdot\mid x)\big)\right]

Because shifting every score by a prompt-dependent constant adds a constant to the expected reward, the maximizing policy stays invariant to score translation. Scaling all rewards by 3 while leaving βKL\beta_{\text{KL}} unchanged, however, yields the same ideal objective maximizer as dividing βKL\beta_{\text{KL}} by 3, effectively weakening the KL constraint. Real policy drift also reflects advantage normalization, ratio clipping, and optimization steps.

To keep score origins anchored before RL, InstructGPT subtracted the mean reward over labeler demonstrations prior to optimization.[1] TRL 1.17 exposes center_rewards_coefficient, recommending 0.01 as a starting point (default None), which adds the auxiliary term popularized by Eisenstein et al.:[6][9][11]

Lcenter=γ Epairs[(rw+rl)2]\mathcal{L}_{\text{center}}=\gamma\,\mathbb{E}_{\text{pairs}}\left[(r_w+r_l)^2\right]

In terms of the pair midpoint u=(rw+rl)/2u = (r_w + r_l) / 2, this penalty equals 4γu24\gamma u^2. Note what it omits: pairs with scores +100+100 and −100-100 have zero midpoint and pay zero penalty, so centering pulls pair averages near zero without bounding individual reward magnitudes or calibrating win probabilities. In the NumPy check below, adding 10 leaves pairwise loss unchanged while multiplying scores by 3 shifts the loss downward, and the +100/−100+100/-100 pair confirms zero midpoint penalty:

reward_centering_invariance.py
import numpy as np

chosen = np.array([1.2, 0.7])
rejected = np.array([0.2, 0.4])

def pair_loss(left, right):
    return float(np.logaddexp(0.0, -(left - right)).mean())

shifted = (chosen + 10.0, rejected + 10.0)
scaled = (chosen * 3.0, rejected * 3.0)

print(f"base_loss={pair_loss(chosen, rejected):.4f}")
print(f"shifted_loss={pair_loss(*shifted):.4f}")
print(f"scaled_loss={pair_loss(*scaled):.4f}")
gamma = 0.01
sum_penalty = gamma * (100 + (-100)) ** 2
individual_penalty = gamma * (100 ** 2 + (-100) ** 2) / 2
assert sum_penalty == 0.0 and individual_penalty == 100.0
print(f"scores=+100,-100 pair_sum_penalty={sum_penalty:.1f} individual_square_penalty={individual_penalty:.1f}")
Centering invariance output
base_loss=0.4338
shifted_loss=0.4338
scaled_loss=0.1949
scores=+100,-100 pair_sum_penalty=0.0 individual_square_penalty=100.0
Three pairs share one score axis. The original scores 1.80 and 0.70 have margin 1.10 and sigmoid probability 0.7503. Adding 10 to both scores gives 11.80 and 10.70 with the same margin and probability. Multiplying both by 3 gives 5.40 and 2.10, margin 3.30, and probability 0.9644 under the same fixed sigmoid link. Translation preserves the pair loss; scaling changes it.
Adding +10 preserves the score margin and sigmoid win probability, whereas multiplying by 3 stretches the margin to 3.30 and pushes the implied probability to 0.9644.

Probabilistic calibration and Brier scores

To measure whether Bradley-Terry probabilities match actual annotator frequencies, fix candidate identifiers A and B before collecting votes rather than sorting every row so the winner sits in slot A. Let zi=1z_i = 1 if A wins vote ii and zi=0z_i = 0 if B wins, and evaluate pi=σ(rA−rB)p_i = \sigma(r_A - r_B) using two proper metrics:

BS=1N∑i=1N(pi−zi)2,ECE=∑m:∣Bm∣>0∣Bm∣N∣1∣Bm∣∑i∈Bmpi−1∣Bm∣∑i∈Bmzi∣\text{BS}=\frac{1}{N}\sum_{i=1}^N(p_i-z_i)^2, \qquad \text{ECE}=\sum_{m:|B_m|>0}\frac{|B_m|}{N}\left|\frac{1}{|B_m|}\sum_{i\in B_m}p_i-\frac{1}{|B_m|}\sum_{i\in B_m}z_i\right|

The Brier score (BS) averages squared probability errors across individual votes, while Expected Calibration Error (ECE) groups predictions into bins BmB_m and compares average predicted win probability with observed win fraction in each bin. When comparing calibration across experiments, state whether bins track fixed A-versus-B win probabilities or top-class confidence, along with bin edges and sample counts. Suppose four independent reviewers compare two softmax answers and split 3 votes for A to 1 vote for B. Both p(A)=0.75p(A) = 0.75 and overconfident p(A)=0.99p(A) = 0.99 pick A and achieve the same 75% binary accuracy, yet Brier score and one-bin ECE sharply separate them:

reward_calibration.py
import numpy as np

# Fixed A/B ordering evaluated across four independent votes.
votes = np.array([1.0, 1.0, 1.0, 0.0])
for probability in [0.75, 0.99]:
    accuracy = float(np.mean((probability > 0.5) == (votes == 1.0)))
    brier = float(np.mean((probability - votes) ** 2))
    one_bin_ece = float(abs(probability - votes.mean()))
    assert accuracy == 0.75
    print(f"p_A={probability:.2f} accuracy={accuracy:.2f} Brier={brier:.4f} one_bin_ECE={one_bin_ece:.2f}")
Probability calibration output
p_A=0.75 accuracy=0.75 Brier=0.1875 one_bin_ECE=0.00
p_A=0.99 accuracy=0.75 Brier=0.2451 one_bin_ECE=0.24

Forecasting 0.750.75 matches this four-vote empirical rate, though four votes are far too small to prove population calibration across prompt categories. When training on repeated votes per pair, using the empirical vote share q∈(0,1)q \in (0, 1) as a soft BCE target −qlog⁡p−(1−q)log⁡(1−p)-q\log p - (1 - q)\log(1 - p) has unique minimum p=qp = q. Keep in mind that a 50-50 reviewer split and an explicit "equally good" tie annotation record different phenomena; use a tie-aware objective such as Davidson or Rao-Kupper when modeling explicit ties directly.

Four fixed A/B votes are A, A, A, B, so the observed sample share for A is 0.75. Predictions 0.75 and 0.99 both choose A for 3-of-4 accuracy. The per-vote squared errors average to Brier scores 0.1875 and 0.2451; one-bin ECE is 0 and 0.24.
Predictions 0.75 and 0.99 achieve identical 3-of-4 rank accuracy, yet per-vote Brier squared error and one-bin ECE favor the calibrated 0.75 probability.

Reward hacking, Goodhart's Law, and policy overoptimization

High accuracy on a held-out preference split never guarantees safe downstream policy optimization. Held-out accuracy only inspects past outputs drawn from static reference policies, whereas RL actively explores new prompt-completion trajectories where learned correlations break down. Goodhart's Law warns that optimizing an imperfect proxy score relentlessly will eventually expose and amplify discrepancies between the proxy and true quality, even when the proxy correlates well initially.

The Goodhart mechanism and Gao et al. scaling laws

Gao et al. (2023) studied reward overoptimization in controlled synthetic setups by using a much larger gold reward model both to generate preference labels and to evaluate PPO and best-of-NN policies trained against a smaller proxy scorer.[2] Under sustained optimization, proxy reward continued rising while gold reward reached a peak and then declined. Their fitted scaling relationships show how evaluator errors rare on static splits become dominant modes once RL searches into uncharted regions of completion space, though exact peak heights depend on model family, data budget, and the proxy-vs-gold gap rather than forming universal constants.

When an RL optimizer explores candidate completions, four common failure modes repeatedly break naive scorers:

  1. Verbosity bias: Longer replies often correlate with thoroughness in preference data, tempting the scorer to equate extra tokens with higher quality even when the extra text is redundant filler. Always test length-controlled slices where factual content is fixed while token count varies.
  2. Sycophancy: Preference models and human raters can reward completions that agree with a user's incorrect assumption rather than gently correcting it.[12] Distinguishing helpful tone from factual deference requires counterexamples where polite disagreement wins.
  3. Formatting shortcuts: Markdown structure aids readability, yet optimization can exaggerate surface layout. In Eisenstein et al.'s helpfulness experiments, list-formatted generations rose to roughly half of policy outputs compared with about 8% in the underlying preference dataset.[11]
  4. Omitted criteria: A reward head trained solely on helpfulness comparisons omits truthfulness, safety, or privacy signals unless those dimensions are explicitly represented in both training pairs and held-out audits.

The audit snippet below checks whether winning candidates are systematically longer across held-out pairs; seeing 100% accuracy alongside 100% longer winners signals the need for length-matched counterexamples before declaring success:

reward_audit.py
import numpy as np

chosen_rewards = np.array([1.8, 2.4, 1.6])
rejected_rewards = np.array([0.7, 1.1, 0.3])
chosen_tokens = np.array([36, 58, 33])
rejected_tokens = np.array([19, 22, 14])

margins = chosen_rewards - rejected_rewards
accuracy = float(np.mean(margins > 0))
length_gaps = chosen_tokens - rejected_tokens

print(f"pair_accuracy={accuracy:.2f}")
print(f"mean_margin={float(margins.mean()):.2f}")
print(f"chosen_answers_longer={bool(np.all(length_gaps > 0))}")
print("next_check=build length-matched eval pairs")
Reward audit output
pair_accuracy=1.00
mean_margin=1.23
chosen_answers_longer=True
next_check=build length-matched eval pairs

Annotator disagreement audits complement length audits by exposing ambiguous slices where strict binary labels are noisy. In practice, unanimous pairs train cleanly under strict Bradley-Terry loss, while split votes warrant inspection, additional annotation, or soft-target modeling rather than automatic deletion. Separating confident agreement from contested comparisons prevents label noise from masking real learning failures on unambiguous bugs:

agreement_audit.py
votes = {
    "clear_bugfix": ["chosen", "chosen", "chosen"],
    "style_only": ["chosen", "rejected", "chosen"],
    "ambiguous_format": ["chosen", "rejected", "tie"],
}

for pair_id, labels in votes.items():
    chosen_share = labels.count("chosen") / len(labels)
    status = "train" if chosen_share == 1.0 else "review_or_hold_out"
    print(f"{pair_id}: chosen_share={chosen_share:.2f} status={status}")
Agreement audit output
clear_bugfix: chosen_share=1.00 status=train
style_only: chosen_share=0.67 status=review_or_hold_out
ambiguous_format: chosen_share=0.33 status=review_or_hold_out

Testing mitigations before trusting them

1. Why a naive length margin fails to block length shortcuts

One tempting heuristic subtracts a length margin m(yw,yl)=α⋅max⁡(0,len(yw)−len(yl))m(y_w, y_l) = \alpha \cdot \max(0, \text{len}(y_w) - \text{len}(y_l)) inside the sigmoid:

L=−log⁡σ(r(x,yw)−r(x,yl)−α⋅max⁡(0,len(yw)−len(yl)))\mathcal{L} = -\log \sigma\Big(r(x, y_w) - r(x, y_l) - \alpha \cdot \max\big(0, \text{len}(y_w) - \text{len}(y_l)\big)\Big)

Subtracting that offset requires a longer winner to beat the loser by a larger margin during training, yet it does not prevent a reward head from relying solely on token length. For a pure length scorer r(y)=b⋅len(y)r(y) = b \cdot \text{len}(y) evaluated on a pair with positive length gap Δlen>0\Delta_{\text{len}} > 0, the shifted logit equals (b−α)Δlen(b - \alpha)\Delta_{\text{len}}. As the NumPy check below shows, raising the length weight bb from 0 to 6 drives training loss down to 0.0067 while failing every length-reversed counterexample where the correct fix is short and the wrong answer is long:

length_margin_shortcut.py
import numpy as np

alpha = 1.0
chosen_length, rejected_length = 2.0, 1.0
gap = chosen_length - rejected_length
for length_weight in np.array([0.0, 3.0, 6.0]):
    effective_logit = (length_weight - alpha) * gap
    loss = float(np.logaddexp(0.0, -effective_logit))
    short_good_score = length_weight * 1.0
    long_wrong_score = length_weight * 2.0
    passes_reversal = bool(short_good_score > long_wrong_score)
    print(f"length_weight={length_weight:.0f} training_loss={loss:.4f} passes_length_reversal={passes_reversal}")
Length-margin counterexample
length_weight=0 training_loss=1.3133 passes_length_reversal=False
length_weight=3 training_loss=0.1269 passes_length_reversal=False
length_weight=6 training_loss=0.0067 passes_length_reversal=False

Rather than adding fragile length penalties that can punish genuinely thorough answers, construct balanced training and evaluation data containing concise correct answers, detailed explanations that earn their length, and verbose errors. Adding targeted counterexamples forces gradient updates to align with semantic quality across every length band instead of shifting scores uniformly by character count. Always verify shortcut resistance directly on length-matched and length-reversed slices.

2. Ensemble disagreement penalties (μ−λσ\mu - \lambda \sigma)

An ensemble of K≥2K \ge 2 reward models trained with varied seeds, data splits, or architectures can expose unstable regions during optimization. Before aggregating scores across members, align their origins and scales on shared calibration data so arbitrary model offsets do not masquerade as disagreement.[11] Computing the sample mean and sample standard deviation yields a conservative Lower Confidence Bound (LCB) score:

μ(x,y)=1K∑k=1Krk(x,y),σ(x,y)=1K−1∑k=1K(rk(x,y)−μ(x,y))2,rconservative(x,y)=μ(x,y)−λ⋅σ(x,y)\mu(x, y) = \frac{1}{K}\sum_{k=1}^K r_k(x, y), \qquad \sigma(x, y) = \sqrt{\frac{1}{K-1}\sum_{k=1}^K \big(r_k(x, y) - \mu(x, y)\big)^2}, \qquad r_{\text{conservative}}(x, y) = \mu(x, y) - \lambda \cdot \sigma(x, y)

Despite its name, this LCB penalty is an optimization heuristic rather than a calibrated statistical confidence bound. While Eisenstein et al. showed that reward ensembles reduce reward hacking, ensembles cannot stop it completely because correlated errors across members produce high mean reward with zero disagreement.[11]

Auditing fresh policy rollouts before optimization

Before committing compute to a full RL run, generate fresh completions from your policy model, form new preference pairs, and obtain human reviews covering factuality, refusal quality, and shortcut slices. Static held-out accuracy only inspects past outputs, whereas fresh-rollout accuracy measures whether the reward model ranks what the policy currently emits. The screening check below applies an 80% fresh-pair accuracy threshold; passing a screen supports proceeding to a small monitored RL trial rather than certifying production release:

fresh_policy_gate.py
evaluation = {
    "static_held_out_pairs": {"accuracy": 0.92, "human_reviewed": False},
    "fresh_policy_pairs": {"accuracy": 0.64, "human_reviewed": True},
}
minimum_fresh_accuracy = 0.80
def passes_fresh_gate(result, minimum):
    return result["human_reviewed"] and result["accuracy"] >= minimum

assert not passes_fresh_gate({"accuracy": 0.99, "human_reviewed": False}, 0.80)
assert passes_fresh_gate({"accuracy": 0.80, "human_reviewed": True}, 0.80)
passes_pair_screen = passes_fresh_gate(evaluation["fresh_policy_pairs"], minimum_fresh_accuracy)

print(f"static_accuracy={evaluation['static_held_out_pairs']['accuracy']:.2f}")
print(f"fresh_accuracy={evaluation['fresh_policy_pairs']['accuracy']:.2f}")
print(f"passes_pair_screen={passes_pair_screen}")
Fresh-policy gate output
static_accuracy=0.92
fresh_accuracy=0.64
passes_pair_screen=False

Standardized evaluation: RewardBench and RewardBench 2

Internal validation pairs often share the blind spots of your own annotation pipeline, so standardized benchmarks provide a broader external check. RewardBench (Lambert et al., 2024) scores preference models across four pairwise categories: Chat (general dialogue and helpfulness), Chat Hard (challenging instruction edge cases), Safety (harm refusal and jailbreak resistance), and Reasoning (mathematics and code correctness).[13] RewardBench 2 (Malik et al., 2025; revised April 2026) expands evaluation to Factuality, Precise Instruction Following, Math, Safety, Focus, and Ties, moving ordinary categories to one-among-four selection (25% random baseline) with a separate Ties metric so scores are not directly comparable to RewardBench V1.[14]

In Tulu 3 8B experiments reported by the RewardBench 2 authors, benchmark scores correlated strongly with best-of-16 sampling performance (Pearson 0.87), whereas PPO outcomes proved far less sensitive among capable reward models and degraded sharply when reward-model initialization or prompt distributions differed from training.[14] Log the exact benchmark version, model checkpoint, tokenizer, and chat template with every run so results stay comparable.

Architectural spectrum: ORMs, PRMs, verifiers, and generative judges

A scalar Bradley-Terry Outcome Reward Model (ORM) is only one way to score completions. Each supervision paradigm trades off labeling cost, inference speed, interpretability, and vulnerability to reward hacking in a different way. Choosing among supervision designs depends on whether correctness can be checked deterministically, whether step-level attribution is required, and how quickly the scorer must run:

ApproachFeedback or objectivePrimary strengthsPrimary vulnerabilities
Scalar Bradley-Terry ORMSingle scalar r(x,y)∈Rr(x, y) \in \mathbb{R}Fast inference, standard PPO/GRPO terminal reward, inspectable marginsOveroptimization, length bias, Goodhart drift
Direct Preference Optimization (DPO)Implicit policy/reference log-ratioOptimizes policy directly from pairs without a separate scorerRelies on offline pair coverage, reference anchor, and regularization
Process Reward Model (PRM)Step-level scalars r(x,y1:t)r(x, y_{1:t})Locates errors across multi-step reasoning tracesHigh step-annotation cost, still susceptible to gaming[15]
Generative RM / LLM-as-judgeVerdict text or verdict token probabilitiesFlexible rubrics and synthetic label scalingInference latency, position bias, rationales that may not reflect true reasons[16]
Verifiable reward / RLVRProgrammatic execution or constraint resultDeterministic validation of specified checksIncomplete unit tests, parser loopholes, gap between check and intent[5]

Process Reward Models (PRMs) assign step-by-step scores across mathematical or coding derivations. Lightman et al. showed that step-level supervision improved best-of-N MATH selection on a fixed generator compared with outcome supervision, though they did not train the generator itself via RL and process benefits vary across domains.[15] In domains with deterministic checkers, Tulu 3 replaced learned reward models with executable RLVR verifiers for math and instruction constraints.[5] Executable checks remove learned-head vulnerabilities, though reward hacking still persists whenever tests are incomplete or a parser credits a right final token paired with broken reasoning.

Put the pieces together with TRL

Our two-feature NumPy head isolated shortcut dynamics on CPU. At full transformer scale, TRL 1.13.0 wires together sequence classification, pair collation, and stable BCE optimization via RewardTrainer. The configuration fragment below assumes train_pairs and eval_pairs are Hugging Face Dataset objects already grouped by prompt ID, because RewardTrainer expects your split to prevent leakage beforehand.

transformer_reward_training.py
from trl import RewardConfig, RewardTrainer

# Configured for trl==1.13.0 with pre-split train_pairs and eval_pairs Datasets.
trainer = RewardTrainer(
    model="Qwen/Qwen3-0.6B",
    args=RewardConfig(
        output_dir="reward-head",
        max_length=2048,  # Set to cover full templated sequences without truncating endings.
        center_rewards_coefficient=0.01,  # Starting offset penalty; verify against held-out margins.
    ),
    train_dataset=train_pairs,
    eval_dataset=eval_pairs,
)
trainer.train()
metrics = trainer.evaluate()
trainer.save_model("reward-head/final")
trainer.processing_class.save_pretrained("reward-head/final")

Passing a causal model ID instructs TRL to initialize a one-label sequence-classification head automatically. Always save and reload both the trained scalar head and the exact tokenizer and chat template, verifying held-out pair scores after reloading so an uninitialized or dropped linear head is caught immediately.[8] Before scaling into policy optimization, review calibration error, fresh-rollout agreement, and shortcut counterexamples alongside clear stop criteria.

Production diagnostic reference

When a reward-modeling run or downstream RL trial diverges, match the symptom against these diagnostic checks:

1. Training loss drops while held-out accuracy stalls

Stalled validation accuracy paired with shrinking training loss can indicate overfitting, noisy preference labels, widening margins on already easy pairs, or prompt mismatch between splits. Because Bradley-Terry loss keeps pushing already-correct margins further positive, aggregate loss can drop long after ranking decisions stop improving on boundary cases. Check prompt-group overlap, audit label consistency, and inspect held-out loss and counterexample accuracy before changing learning rates.

2. Scorer prefers wordier, padded answers

When verbose errors beat concise correct solutions, training correlations likely linked length with reward or evaluators systematically favored detail over accuracy. Standard held-out evaluation can hide this failure if the validation split obeys the same length skew as training. Build length-matched and length-reversed evaluation slices with technical content held constant, add balanced counterexamples to training, and verify accuracy on those slices rather than relying on length penalties alone.

3. Proxy reward climbs during PPO while human ratings drop

A rising internal score alongside falling user satisfaction signals proxy exploitation or distributional drift into regions where the reward model errs. Pause the optimization run, save policy and scorer checkpoints, and evaluate fresh generations with human judges or verifiable checks. Tightening KL regularization, lowering the optimization budget, diversifying training pairs, or switching to a calibrated ensemble can reduce hacking without removing every evaluator blind spot.

4. Raw scores drift rapidly across steps

Wide shifts in score magnitude typically stem from unanchored score origins, runaway margins on separable pairs, or mismatched pooling and template masks between training and evaluation. Unbounded drift distorts effective KL balance during PPO even when validation pair accuracy looks stable. Log pair midpoints separately from pair margins, verify padding masks and end-of-turn token alignment, and combine centering penalties with L2 regularization and held-out calibration audits.

Complete the lesson

Mastery Check

Answer every question, then check your score. Score 75% or higher to mark this lesson complete.

1.A team wants an offline preference update for a writer that summarizes technical reports. For one report, the reviewer prefers summary S, but the fixed reference writer assigns summary T a higher probability. Which update learns the reviewer’s pairwise preference relative to that reference?
2.A right-padded sequence has mask [1,1,1,0] and hidden vectors [1,2], [3,1], [2,4], [99,99]. Its scalar head is [0.25,0.5]. Under the final-real-token readout, which index and score are correct?
3.For a strict chosen-over-rejected label, scores are 1.4 and 2.1. Use the Bradley-Terry link and natural-log loss. Which approximate probability and loss correspond to the observed chosen outcome?
4.Two different raw completions become identical token IDs and masks after preprocessing. A deterministic shared scorer is trained only with their strict-winner pair loss. What does this pair contribute?
5.Prompt P has five ranked completions and prompt Q has two. Expand all unordered pairs. Compare averaging every pair equally with averaging within prompt and then equally across prompts. What are the relative P:Q loss weights?
6.A scorer gives 4 points per unit of length. In a training pair, the chosen answer is 2 units longer. The loss subtracts 1.5 times that length gap from the score difference. What logit enters the loss, and how does the same length-only scorer rank a short correct answer against a longer wrong answer on a held-out pair?
7.Answer identities A and B are fixed before ten independent recorded votes: six choose A and four choose B. Compare constant forecasts p(A)=0.6 and p(A)=0.9. Which Brier-score pair is correct?
8.The ideal objective is expected reward minus beta×KL from a fixed reference, with beta=0.12. All rewards are multiplied by four while beta stays fixed. Which KL coefficient, with the original rewards, gives the same ideal maximizer?
9.A live LoRA reward scorer and its reloaded copy score the same two held-out replies to one prompt. For each reply, their pooled vectors match exactly. Each scorer applies a linear head directly to its pooled vector. In reply order, the live scores are [6, 2] and the reloaded scores are [2, 4]. The live weights are available. Which repair and check would recover and verify both raw scores?
10.A validation screen requires at least 85% agreement on 28 freshly reviewed pairs; 23 agree. Three calibrated scorers also each assign score 7 to an answer whose citation fails an independent source check. Which outcome follows?

10 questions remaining.

References

Training Language Models to Follow Instructions with Human Feedback (InstructGPT).

Ouyang, L., et al. · 2022 · NeurIPS 2022

https://arxiv.org/abs/2203.02155

Scaling Laws for Reward Model Overoptimization

Gao, L., Schulman, J., & Hilton, J. · 2023 · ICML 2023

https://proceedings.mlr.press/v202/gao23h.html

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Shao, Z., et al. · 2024

https://arxiv.org/abs/2402.03300

Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

Rafailov, R., et al. · 2023

https://arxiv.org/abs/2305.18290

Tülu 3: Pushing Frontiers in Open Language Model Post-Training

Lambert, N., et al. · 2024 · arXiv preprint

https://arxiv.org/abs/2411.15124

TRL 1.17.1: RewardTrainer Implementation

Hugging Face TRL Contributors · 2026

https://github.com/huggingface/trl/blob/v1.17.1/trl/trainer/reward_trainer.py

The Llama 3 Herd of Models.

Dubey, A., et al. · 2024 · arXiv preprint

https://arxiv.org/abs/2407.21783

TRL 1.13.0 RewardTrainer implementation

Hugging Face · 2026

https://github.com/huggingface/trl/blob/3d9261f1fec9f9a8140099c78a65c7da73dce79c/trl/trainer/reward_trainer.py

TRL 1.17.1: RewardConfig

Hugging Face TRL Contributors · 2026

https://github.com/huggingface/trl/blob/v1.17.1/trl/trainer/reward_config.py

Llama 2: Open Foundation and Fine-Tuned Chat Models.

Touvron, H., et al. · 2023 · arXiv preprint

https://arxiv.org/abs/2307.09288

Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

Eisenstein, J., et al. · 2023 · CoLM 2024

https://arxiv.org/abs/2312.09244v3

Towards Understanding Sycophancy in Language Models

Sharma, M., et al. · 2023

https://arxiv.org/abs/2310.13548v4

RewardBench: Evaluating Reward Models for Language Modeling

Lambert, N., et al. · 2024

https://arxiv.org/abs/2403.13787

RewardBench 2: Advancing Reward Model Evaluation

Malik, S., et al. · 2025

https://arxiv.org/abs/2506.01937

Let's Verify Step by Step.

Lightman, H., et al. · 2024 · ICLR 2024

https://arxiv.org/abs/2305.20050

Generative Reward Models

Mahan, D., et al. · 2024

https://arxiv.org/abs/2410.12832

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.