Personalize this lesson
Adapt explanations and teaching visuals to your background and preferred voice.
A developer asks a coding assistant how to prevent exponential overflow when implementing softmax(z) in Python. One reply gives the exact fix in two sentences: subtract max(z) from every logit before calling exp(z) so the largest exponent is zero. A second reply looks polished and friendly, yet advises casting the array to float64 after computing exp(z), which still overflows inside exp(z) before any cast runs. Which answer should earn the higher score?
Supervised fine-tuning (SFT) trains a network to imitate demonstration tokens, whereas reward modeling trains an evaluative judge to separate preferred completions from rejected ones on the same prompt. A pairwise reward model assigns every completed prompt-response pair a single real number that downstream policy optimizers can rank or optimize. System diagrams for RLHF (Reinforcement Learning from Human Feedback) often compress that judge into a single box before jumping to Proximal Policy Optimization (PPO), yet that box is simply a neural network trained on noisy human comparisons.[1] If the scorer learns to reward extra length or confident phrasing instead of technical correctness, later policy optimization will amplify the mistake rather than fixing it.[2] Throughout this lesson, we will return to this softmax stability question to test whether a trained scorer learned the underlying bug fix or a superficial shortcut.
How a scalar judge scores a completed answer
An autoregressive language model outputs a conditional probability distribution over the vocabulary at every token position, . A reward model instead evaluates the entire prompt and completed response together. It passes the combined sequence through a transformer backbone and projects the final hidden state down to a single real-valued scalar:
For our softmax prompt , an annotator compares two candidate completions: a chosen completion that subtracts max(z) before exp(z), and a rejected completion that casts exp(z) after the overflow has already happened. The training objective pushes the model to satisfy the strict ranking inequality . Because pairwise comparisons record only relative order, adding a constant offset to every score preserves every ranking, leaving the raw origin arbitrary. A score of 2.0 does not mean twice as helpful as 1.0, and you should calibrate two independently trained reward models on shared data before comparing their raw outputs. At the same time, scaling all scores stretches their pairwise gaps, which directly changes Bradley-Terry win probabilities and downstream policy optimization pressure.
That scalar score supports two distinct downstream workflows in post-training pipelines, depending on whether you are generating rollouts online or filtering candidates at inference time:
- Terminal reward for online RL: The scorer evaluates completed policy rollouts as a sequence-level return. PPO pairs those rewards with a learned value baseline using Generalized Advantage Estimation (GAE), whereas outcome-supervised GRPO normalizes rewards relative to a group of completions sampled from the same prompt without a separate value network.[3]
- Best-of-N selection: During evaluation or synthetic dataset curation, a generator samples candidate completions for prompt and selects . People often call this rejection sampling in post-training recipes, even though taking the argmax over draws does not sample directly from a target distribution.
Direct Preference Optimization (DPO) uses the exact analytic link between a reward function and its optimal KL-regularized policy to reparameterize rewards as policy-to-reference log-ratios, turning pairwise preference loss directly into a policy objective.[4] The basic offline DPO loop updates the language model from labeled pairs without maintaining a separate scorer network. An explicit reward model remains valuable whenever you want fast reusable ranking across fresh rollouts, best-of- filtering, or shared feedback signals alongside generative judges and executable verifiers.[5]
Preference data anatomy and validation pipeline
Before training a scorer, inspect the exact structure and labels in your comparison dataset. Pairwise cross-entropy minimizes loss on whatever ordering the labels record, including formatting mismatches, duplicate candidates, and systematic annotator habits. Checking the raw pairs early prevents your training run from spending compute on degenerate comparisons.
Standard and conversational preference formats
In standard explicit-prompt datasets, each row holds the shared query in prompt alongside competing completions in chosen and rejected. The prompt appears once so both responses share the identical context prefix when tokenized, preventing minor formatting discrepancies in the prompt from leaking into score margins. Keeping the prompt separated also makes group-by-prompt dataset splitting straightforward during validation:
{
"prompt": "How do you prevent exponential overflow when implementing softmax in Python?",
"chosen": "Subtract max(z) from each logit before calling exp(z) so the largest exponent is zero.",
"rejected": "Cast the output array to float64 after calling exp(z) so large values fit safely."
}Multi-turn dialogues store full role-tagged message lists for both chosen and rejected. Both transcripts must match token for token across the system message and prior user turns, diverging only at the final assistant message under comparison. If an earlier system directive or intermediate tool call differs between the two transcripts, the reward head can pick up context noise instead of scoring assistant quality:
{
"chosen": [
{"role": "system", "content": "Give concise, numerically accurate Python advice."},
{"role": "user", "content": "How do you prevent exponential overflow when implementing softmax in Python?"},
{"role": "assistant", "content": "Subtract max(z) from each logit before calling exp(z) so the largest exponent is zero."}
],
"rejected": [
{"role": "system", "content": "Give concise, numerically accurate Python advice."},
{"role": "user", "content": "How do you prevent exponential overflow when implementing softmax in Python?"},
{"role": "assistant", "content": "Cast the output array to float64 after calling exp(z) so large values fit safely."}
]
}Hugging Face TRL's RewardTrainer accepts both plain strings and conversational message lists, whether the prompt is implicit or explicit. Conversational examples are formatted through the tokenizer's chat template, whereas raw strings are concatenated and tokenized directly without applying conversation delimiters. These behaviors match TRL 1.17.1 as verified on November 4, 2026.[6]
Pair validation before training
Having chosen and rejected columns in a table does not make a dataset ready for gradient descent. Malformed rows such as mismatched prompts, duplicate completions, or non-binary tie labels distort loss magnitudes or silently zero out gradients during minibatch updates. Before tokenizing pairs into minibatches, verify four structural invariants across every record so training gradients reflect actual response quality rather than preprocessing bugs:
- Identical context prefix: Both completions must receive the exact same user prompt, system instructions, tool declarations, and chat template formatting.
- Distinct scored inputs: If normalization, truncation, or templating collapses
chosenandrejectedinto identical token IDs and attention masks, a shared deterministic scorer outputs identical values, giving zero margin and zero parameter gradient. Verify uniqueness both before and after tokenization. - Labels aligned with the loss: Winner-only Bradley-Terry loss expects strict preference outcomes, while ties and ambiguous ratings need separate handling or soft targets. Low annotator agreement can signal an unclear rubric, an error, or genuine stylistic disagreement, so inspect low-agreement clusters instead of dropping them blindly.
- Candidate lineage metadata: Record the generator checkpoint, decoding temperature, and reviewer cohort for each completion so downstream audits can isolate source artifacts.
The validation check below keeps only strict chosen-over-rejected rows whose prompts match and whose completions differ cleanly. Text checks catch surface defects first, while a full pipeline repeats the uniqueness check after running the tokenizer:
from collections import Counter
pairs = [
{"id": "a", "prompt_left": "softmax?", "prompt_right": "softmax?", "chosen": "Subtract max(z).", "rejected": "Cast after exp(z).", "label": "chosen"},
{"id": "b", "prompt_left": "softmax?", "prompt_right": "softmax?", "chosen": "Subtract max(z).", "rejected": "Subtract max(z).", "label": "chosen"},
{"id": "c", "prompt_left": "softmax?", "prompt_right": "attention?", "chosen": "Subtract max(z).", "rejected": "Scale by sqrt(d).", "label": "chosen"},
{"id": "d", "prompt_left": "attention?", "prompt_right": "attention?", "chosen": "Scale by sqrt(d).", "rejected": "Omit scaling.", "label": "tie"},
]
def rejection_reason(pair):
fields = ("prompt_left", "prompt_right", "chosen", "rejected")
if any(not isinstance(pair.get(field), str) or not pair[field].strip() for field in fields):
return "missing_or_empty_text"
if pair.get("label") != "chosen":
return "tie_or_abstention"
if pair["prompt_left"] != pair["prompt_right"]:
return "context_mismatch"
if pair["chosen"].strip() == pair["rejected"].strip():
return "identical_candidates"
return None
reasons = Counter(reason for pair in pairs if (reason := rejection_reason(pair)))
kept = [pair["id"] for pair in pairs if rejection_reason(pair) is None]
print(f"kept={kept}")
print(f"rejected={dict(sorted(reasons.items()))}")
assert kept == ["a"]
assert rejection_reason({**pairs[0], "chosen": " "}) == "missing_or_empty_text"
assert rejection_reason({}) == "missing_or_empty_text"kept=['a']
rejected={'context_mismatch': 1, 'identical_candidates': 1, 'tie_or_abstention': 1}Choose an evaluation split that matches generalization
A single prompt with four candidate answers yields six unordered pairs. If you shuffle and split those pairs row by row, the exact same prompt and even the same candidate completion appear in both training and validation splits, turning validation accuracy into a test of memorization rather than unseen-prompt generalization. To measure generalization to new prompts, group comparisons by prompt_id, cluster near-duplicate prompts, and assign every pair from a prompt to the same split. If your actual goal is ranking fresh answers to a fixed prompt set, use a response-held-out split instead and state that choice explicitly so evaluation metrics are never misread.
pairs = [
{"pair_id": "a-b", "prompt_id": "softmax-17"},
{"pair_id": "a-c", "prompt_id": "softmax-17"},
{"pair_id": "d-e", "prompt_id": "attention-04"},
{"pair_id": "f-g", "prompt_id": "pooling-09"},
]
eval_prompt_ids = {"softmax-17"}
train = [pair for pair in pairs if pair["prompt_id"] not in eval_prompt_ids]
evaluation = [pair for pair in pairs if pair["prompt_id"] in eval_prompt_ids]
overlap = {pair["prompt_id"] for pair in train} & {pair["prompt_id"] for pair in evaluation}
assert not overlap
print(f"train_pairs={len(train)} eval_pairs={len(evaluation)}")
print(f"prompt_overlap={sorted(overlap)}")train_pairs=2 eval_pairs=2
prompt_overlap=[]InstructGPT collected rankings over to completions per prompt, yielding correlated comparisons per prompt.[1] Shuffling those comparisons into independent training rows led to rapid overfitting, so the authors placed all pairs from one prompt into a single batch item. Doing so lets the forward pass score each of the responses once and average the pairwise losses within the prompt group. Without within-prompt averaging, a prompt with responses receives 36 times as much loss weight as a prompt with responses, biasing updates toward prompts that happened to have larger candidate pools.
Turning a language model into a scalar scorer
A generative transformer already reads prompts and candidate answers, so turning it into a reward model requires minimal architectural changes. InstructGPT initialized its reward models from an SFT checkpoint in the main text, while noting in its appendix that the final model used a checkpoint fine-tuned on public NLP datasets with similar outcomes.[1] Llama 3 instead initialized its reward models directly from a pretrained checkpoint.[7] Pick a backbone with suitable reasoning capacity, context window, and template compatibility, then replace its vocabulary projection head with a linear score projection:
- Remove the unembedding projection layer ().
- Attach a scalar linear projection (); any additive scalar bias cancels out of score differences and receives zero gradient from pairwise loss alone.
- Train all weights, train LoRA adapters alongside the scalar head, or train the scalar head over a frozen backbone. When saving LoRA runs, ensure the scalar head parameters are marked trainable and exported together with the adapters.[8]
Input Tokens -> Transformer Hidden States (d) -> Last-Token Representation -> Linear Head -> Scalar Reward r(x, y)Final real-token pooling and sequence padding
In standard causal decoder attention, token position attends only to positions , including itself on the diagonal. Consequently, any token before the end of the answer lacks hidden-state information from the remaining suffix. Reading from the final non-padding token, typically the end-of-turn delimiter at the close of the response, ensures the representation summarizes the entire prompt and response. Other architectures sometimes append a dedicated scoring token or pool multiple positions, but in every design the readout must observe the complete text it judges.
Contiguous batch padding moves that final real token index across examples in a batch. Under left padding, the final non-padding token sits at index -1 for every nonempty sequence. Under contiguous right padding, it sits at mask.sum(dim=-1) - 1, assuming real tokens begin at index zero without internal gaps. Never assume a transformer leaves harmless zeros at masked padding slots, because attention quirks or positional encodings can produce large activations there; always gather the last unmasked index explicitly.
Filtering long sequences vs. truncating suffixes
Sequence-length limits pose a trade-off between memory efficiency and label fidelity. If a human annotator rejected a response because of a subtle overflow bug in the final code block, right-truncating the response removes the exact reason for the preference label. Filtering oversized pairs preserves label meaning at the cost of altering the prompt-length distribution, so log which pairs and task types are excluded. In TRL 1.17, RewardConfig.max_length defaults to 1024 tokens covering the full templated prompt plus completion, dropping pairs that exceed the limit rather than slicing their endings unless set to None.[6][9] The short length check below demonstrates that boundary rule by keeping every pair up to max_length inclusive while excluding oversized candidates whole:
max_length = 1024
pairs = [
{"id": "fits", "chosen_tokens": 412, "rejected_tokens": 390},
{"id": "at_limit", "chosen_tokens": 1024, "rejected_tokens": 1024},
{"id": "chosen_too_long", "chosen_tokens": 1088, "rejected_tokens": 401},
{"id": "rejected_too_long", "chosen_tokens": 288, "rejected_tokens": 1030},
]
kept = [p["id"] for p in pairs if max(p["chosen_tokens"], p["rejected_tokens"]) <= max_length]
dropped = [p["id"] for p in pairs if p["id"] not in kept]
print(f"kept={kept}")
print(f"dropped_instead_of_truncated={dropped}")
assert kept == ["fits", "at_limit"]kept=['fits', 'at_limit']
dropped_instead_of_truncated=['chosen_too_long', 'rejected_too_long']The pooling check next verifies with NumPy that masking selects the exact final real hidden vector [2.0, 3.0] regardless of whether padding tokens ([99.0, 99.0]) sit on the right or on the left:
import numpy as np
def last_nonpadding(hidden, mask):
hidden, mask = np.asarray(hidden), np.asarray(mask)
if hidden.ndim != 3 or mask.shape != hidden.shape[:2]:
raise ValueError("expected hidden [batch, time, width] and matching mask")
if not np.isin(mask, [0, 1]).all() or not mask.any(axis=1).all():
raise ValueError("each sequence needs at least one unmasked position")
positions = np.arange(mask.shape[1])
last = np.where(mask == 1, positions, -1).max(axis=1)
return hidden[np.arange(len(hidden)), last]
states = np.array([[[1., 0.], [2., 3.], [99., 99.]],
[[99., 99.], [1., 0.], [2., 3.]]])
pooled = last_nonpadding(states, [[1, 1, 0], [0, 1, 1]])
scores = pooled @ np.array([0.5, 1.0])
assert np.allclose(scores, [4.0, 4.0])
try:
last_nonpadding(states[:1], [[0, 0, 0]])
except ValueError:
pass
else:
raise AssertionError("empty sequence accepted")
print("right/left-padded scores:", scores.tolist())right/left-padded scores: [4.0, 4.0]![Right-padded tokens have mask [1, 1, 0], and left-padded tokens have mask [0, 1, 1]. Both select the final real hidden state [2, 3] and ignore the padding hidden state [99, 99]. The shared reward head [0.5, 1.0] gives score 4.0 in either layout.](/cdn/content-image/training/reward-modeling-from-preference-data/illustrations/_generated/last_real_token_readout_dark.png?v=c4ab27a4917a)
Bradley-Terry formulation, BCE loss, and gradient dynamics
Suppose our trained head scores the two softmax completions as for subtracting max(z) first and for casting after exp(z). Their score difference is . The Bradley-Terry (1952) model converts that difference into an implied preference probability via normalized exponentials:
Substituting gives . Any positive margin yields an implied win probability above 50%, while a margin of zero yields 0.50. Keep in mind that is the model's parameterized probability rather than a calibrated frequency across human raters, nor can a 0.50 score distinguish genuine annotator uncertainty from an untried pair.

Direct equivalence to binary cross-entropy with logits
Fitting the reward parameters maximizes the log-likelihood of observed preferences over dataset , which minimizes the expected negative log-likelihood:
This objective is identical to binary cross-entropy (BCE) with target evaluated on the scalar difference : . Passing the logit margin directly into PyTorch's binary_cross_entropy_with_logits (or softplus(-logits_diff)) avoids underflow when rounds to zero. In the PyTorch check below, we also add a shared bias = 10.0 to both candidates to verify that a constant score shift receives exact zero gradient from the pairwise objective:
import torch
# One row per pair, columns chosen/rejected.
rewards = torch.tensor([[1.8, 0.7]], dtype=torch.float64, requires_grad=True)
bias = torch.tensor(10.0, dtype=torch.float64, requires_grad=True)
scores = rewards + bias
logits_diff = scores[:, 0] - scores[:, 1]
loss = torch.nn.functional.binary_cross_entropy_with_logits(
logits_diff, torch.ones_like(logits_diff)
)
assert torch.allclose(loss, torch.nn.functional.softplus(-logits_diff).mean())
loss.backward()
expected = torch.sigmoid(logits_diff.detach()) - 1
assert torch.allclose(rewards.grad[:, 0], expected)
assert torch.allclose(rewards.grad[:, 1], -expected)
assert bias.grad.item() == 0.0
print(f"loss={loss.item():.4f}")
print(f"chosen_gradient={rewards.grad[0, 0].item():+.4f} rejected_gradient={rewards.grad[0, 1].item():+.4f}")
print(f"shared_bias_gradient={bias.grad.item():.1f}")loss=0.2873
chosen_gradient=-0.2497 rejected_gradient=+0.2497
shared_bias_gradient=0.0Gradient dynamics and shared-weight movement
Differentiating with respect to yields:
Applying the chain rule gives equal-magnitude, opposite-sign derivatives with respect to the individual candidate rewards:
These score derivatives show where training signal concentrates. When the margin is already strongly positive (), and the pair derivatives shrink toward zero; when , the derivatives are and ; and when the model ranks the wrong answer ahead (), the derivatives saturate near and . Crucially, opposite signs on and do not force the actual scores and to move in opposite directions during optimization, because both scores share parameters . Suppose a single scalar head starts at on frozen features and , giving initial scores and . One SGD step makes more negative, so both scores drop while the rejected score drops twice as fast, widening the margin. Monitoring individual rewards without logging their margin can therefore create false alarms when valid margins are actually growing:
import torch
weight = torch.nn.Parameter(torch.tensor(-1.0, dtype=torch.float64))
features = torch.tensor([1.0, 2.0], dtype=torch.float64)
optimizer = torch.optim.SGD([weight], lr=0.1)
before = (weight * features).detach().clone()
scores = weight * features
loss = torch.nn.functional.softplus(-(scores[0] - scores[1]))
loss.backward()
assert torch.allclose(weight.grad, torch.sigmoid(torch.tensor(-1.0, dtype=torch.float64)))
optimizer.step()
after = (weight * features).detach()
assert torch.all(after < before)
assert (after[0] - after[1]) > (before[0] - before[1])
print("before:", [round(value, 4) for value in before.tolist()])
print("after: ", [round(value, 4) for value in after.tolist()])
print(f"margin: {before[0] - before[1]:.4f} -> {after[0] - after[1]:.4f}")before: [-1.0, -2.0]
after: [-1.0269, -2.0538]
margin: 1.0000 -> 1.0269
Translation invariance vs. scale sensitivity
Adding any prompt-dependent scalar offset to both completions leaves the pairwise difference strictly unchanged, so scores (11.80, 10.70) yield the exact same loss, probability, and gradients as (1.80, 0.70). Multiplying both scores by 3, however, triples the margin from to , driving the sigmoid probability to and shrinking the loss. Because separable preference batches allow the network to decrease loss indefinitely simply by scaling weights upward, regularization and held-out probability evaluation remain important. Below, we verify this contrast directly with NumPy across shifted and scaled margin vectors:
import numpy as np
chosen_rewards = np.array([2.1, 0.8, 1.9])
rejected_rewards = np.array([0.4, 1.0, 1.2])
margins = chosen_rewards - rejected_rewards
def pairwise_bt_loss(delta):
return np.logaddexp(0.0, -delta)
losses = pairwise_bt_loss(margins)
print("margins:", np.round(margins, 3).tolist())
print("reward_loss:", round(float(losses.mean()), 4))
print("pair_accuracy:", round(float((margins > 0).mean()), 4))
for margin in [-2.0, 0.0, 2.0]:
print(f"margin={margin:+.1f} loss={float(pairwise_bt_loss(margin)):.4f}")
assert np.isclose(pairwise_bt_loss(0.0), np.log(2.0))
assert np.isclose(pairwise_bt_loss(-1000.0), 1000.0)
assert pairwise_bt_loss(1000.0) == 0.0margins: [1.7, -0.2, 0.7]
reward_loss: 0.4564
pair_accuracy: 0.6667
margin=-2.0 loss=2.1269
margin=+0.0 loss=0.6931
margin=+2.0 loss=0.1269Decreasing loss tells you the margin grew larger on the training pairs, not which feature drove that separation. Next, we fit a minimal two-feature reward head to expose how correlated training pairs let a length shortcut masquerade as correctness.
Training a small reward head and isolating shortcut features
A linear reward head fitted on just two input features (technical correctness and scaled response length) is enough to expose how training correlations create hidden shortcuts. In a confounded training set, every correct answer is also one length unit longer than the buggy alternative, so correctness and length always increase together. In a balanced training set, a concise fix beats a verbose buggy answer, breaking that spurious correlation.
Both heads easily achieve 100% accuracy on their training pairs, yet when evaluated on counterexamples where a long padded response recommends casting after exp(z), the confounded head drops to 33% accuracy. The NumPy experiment below computes analytical logistic-loss gradients, verifies them via central finite differences, checks input shape validity, saves the fitted weights alongside feature names, and reloads them for inference:
import json
import tempfile
from pathlib import Path
import numpy as np
def objective_and_gradient(weights, pairs, l2=0.02):
pairs = np.asarray(pairs, dtype=float)
if pairs.ndim != 3 or pairs.shape[1:] != (2, 2) or len(pairs) == 0:
raise ValueError("expected nonempty [pairs, chosen/rejected, two features]")
if not np.isfinite(pairs).all():
raise ValueError("features must be finite")
differences = pairs[:, 0] - pairs[:, 1]
margins = differences @ weights
loss = np.logaddexp(0.0, -margins).mean() + 0.5 * l2 * (weights @ weights)
sigmoid_negative = np.exp(-np.logaddexp(0.0, margins))
gradient = -(differences.T @ sigmoid_negative) / len(pairs) + l2 * weights
return float(loss), gradient
def fit(pairs):
weights = np.zeros(2)
initial, _ = objective_and_gradient(weights, pairs)
for _ in range(300):
_, gradient = objective_and_gradient(weights, pairs)
weights -= 0.1 * gradient
final, _ = objective_and_gradient(weights, pairs)
assert final < initial
return weights
def accuracy(weights, pairs):
return float(np.mean((pairs[:, 0] - pairs[:, 1]) @ weights > 0))
# Each row is [chosen representation, rejected representation] with features [correctness, scaled_length].
confounded = np.array([[[1., 1.], [0., 0.]], [[1., 2.], [0., 1.]]])
balanced = np.array([[[1., 1.], [0., 0.]], [[1., 0.], [0., 1.]]])
counterexamples = np.array([[[1., 0.], [0., 2.]],
[[1., 0.], [0., 3.]],
[[1., 1.], [0., 1.]]])
# Check analytic gradients against central finite differences.
probe = np.array([0.2, -0.3])
_, analytic = objective_and_gradient(probe, confounded)
for invalid in [np.empty((0, 2, 2)), np.zeros((2, 2)), np.full((1, 2, 2), np.nan)]:
try:
objective_and_gradient(probe, invalid)
raise AssertionError("invalid pair tensor accepted")
except ValueError:
pass
for coordinate in range(2):
delta = np.zeros(2)
delta[coordinate] = 1e-6
numerical = (objective_and_gradient(probe + delta, confounded)[0]
- objective_and_gradient(probe - delta, confounded)[0]) / 2e-6
assert abs(numerical - analytic[coordinate]) < 1e-7
for name, training in [("confounded", confounded), ("balanced", balanced)]:
weights = fit(training)
print(f"{name}: weights={weights.round(3).tolist()}, train={accuracy(weights, training):.2f}, counterexamples={accuracy(weights, counterexamples):.2f}")
if name == "confounded":
assert accuracy(weights, counterexamples) == 1 / 3
else:
assert accuracy(weights, counterexamples) == 1.0
with tempfile.TemporaryDirectory() as directory:
path = Path(directory) / "reward_head.json"
path.write_text(json.dumps({"features": ["correctness", "scaled_length"], "weights": weights.tolist()}))
artifact = json.loads(path.read_text())
assert artifact["features"] == ["correctness", "scaled_length"]
assert np.allclose(counterexamples @ np.array(artifact["weights"]), counterexamples @ weights)
print("gradient check and head reload: passed")confounded: weights=[1.629, 1.629], train=1.00, counterexamples=0.33
balanced: weights=[2.667, 0.0], train=1.00, counterexamples=1.00
gradient check and head reload: passed
After 300 gradient steps, the confounded head settles at because both features change identically across its training pairs, splitting optimization pressure evenly between correctness and length. On the first counterexample, a verbose wrong answer at scores , comfortably defeating the concise fix at . By contrast, the balanced set has difference vectors and , so starting at zero length weight causes length gradients from the two pairs to cancel identically. Notice that L2 regularization shrinks weight magnitude rather than identifying which feature is spurious; clean contrast pairs are what eliminate the length weight in this controlled CPU experiment.
Suppose you remove L2 regularization while keeping the balanced training pairs and zero initial weights. Does the length weight grow during training?
Answer
No. Because the two training pairs have equal correctness differences and opposite length differences, their gradients on the length coordinate still cancel exactly at zero length weight. Data symmetry drives that cancellation, whereas L2 regularization only pulls weights toward zero.
Multi-response rankings and margin-augmented losses
When labelers rank groups of candidate completions for a prompt, teams commonly convert rankings into training losses in one of two ways: expanding all pairs and averaging within the prompt group,[1] or modeling the full permutation under a Plackett-Luce multinomial distribution. Averaging over within each prompt is critical so prompts sampled with larger candidate pools do not dominate batch gradients. Some pipelines also capture preference strength labels such as "much better" versus "slightly better" and incorporate a rating-dependent margin inside the sigmoid, setting for the weakest preference category:[10]
Subtracting a positive margin shifts the loss curve horizontally rather than saturating it. At , the shifted logit is zero, the loss remains , and the gradient with respect to the margin is still . Reaching an effective win probability demands ; for and , the score difference must reach . Llama 3 later removed the margin term after observing diminishing gains as dataset size scaled.[7]
import numpy as np
m = 1.0
required_difference = m + np.log(0.9 / 0.1)
differences = np.array([m, required_difference])
logits = differences - m
probabilities = 1.0 / (1.0 + np.exp(-logits))
losses = np.logaddexp(0.0, -logits)
for diff, prob, loss in zip(differences, probabilities, losses):
print(f"score_difference={diff:.3f} effective_p={prob:.2f} loss={loss:.4f} derivative={prob - 1:+.2f}")
assert np.isclose(probabilities[1], 0.9)score_difference=1.000 effective_p=0.50 loss=0.6931 derivative=-0.50
score_difference=3.197 effective_p=0.90 loss=0.1054 derivative=-0.10Notice that a larger margin alters loss slopes and probability separation without necessarily changing binary pair accuracy, which tracks only whether the sign of is positive.
Metrics, centering invariance, and calibration
Pair accuracy reduces each comparison to a binary indicator, , concealing shrinking score margins, drifting score origins, and heavy reliance on spurious features. A model can sustain 95% accuracy on training splits even while raw margins explode or depend almost entirely on response length. Monitoring the distributions below alongside pair accuracy reveals trouble early:
| Logged Metric | What It Measures | What It Conceals |
|---|---|---|
| Pair accuracy | Binary win frequency () | Margin collapse or shortcut reliance |
| Mean margin | Average separation distance | Probability calibration and extreme outliers |
| Mean reward | Overall score drift | Ordering quality across prompts |
| Gradient norm | Optimization step magnitude | Reliance on spurious features such as length |
| Fresh-policy pair accuracy | Agreement on newly sampled model text | Unsampled failures, calibration error, and human disagreement |
Centering invariance and PPO scale sensitivity
In PPO-based RLHF, policy optimization balances expected scalar reward against a KL-divergence penalty relative to a reference policy :
Because shifting every score by a prompt-dependent constant adds a constant to the expected reward, the maximizing policy stays invariant to score translation. Scaling all rewards by 3 while leaving unchanged, however, yields the same ideal objective maximizer as dividing by 3, effectively weakening the KL constraint. Real policy drift also reflects advantage normalization, ratio clipping, and optimization steps.
To keep score origins anchored before RL, InstructGPT subtracted the mean reward over labeler demonstrations prior to optimization.[1] TRL 1.17 exposes center_rewards_coefficient, recommending 0.01 as a starting point (default None), which adds the auxiliary term popularized by Eisenstein et al.:[6][9][11]
In terms of the pair midpoint , this penalty equals . Note what it omits: pairs with scores and have zero midpoint and pay zero penalty, so centering pulls pair averages near zero without bounding individual reward magnitudes or calibrating win probabilities. In the NumPy check below, adding 10 leaves pairwise loss unchanged while multiplying scores by 3 shifts the loss downward, and the pair confirms zero midpoint penalty:
import numpy as np
chosen = np.array([1.2, 0.7])
rejected = np.array([0.2, 0.4])
def pair_loss(left, right):
return float(np.logaddexp(0.0, -(left - right)).mean())
shifted = (chosen + 10.0, rejected + 10.0)
scaled = (chosen * 3.0, rejected * 3.0)
print(f"base_loss={pair_loss(chosen, rejected):.4f}")
print(f"shifted_loss={pair_loss(*shifted):.4f}")
print(f"scaled_loss={pair_loss(*scaled):.4f}")
gamma = 0.01
sum_penalty = gamma * (100 + (-100)) ** 2
individual_penalty = gamma * (100 ** 2 + (-100) ** 2) / 2
assert sum_penalty == 0.0 and individual_penalty == 100.0
print(f"scores=+100,-100 pair_sum_penalty={sum_penalty:.1f} individual_square_penalty={individual_penalty:.1f}")base_loss=0.4338
shifted_loss=0.4338
scaled_loss=0.1949
scores=+100,-100 pair_sum_penalty=0.0 individual_square_penalty=100.0
Probabilistic calibration and Brier scores
To measure whether Bradley-Terry probabilities match actual annotator frequencies, fix candidate identifiers A and B before collecting votes rather than sorting every row so the winner sits in slot A. Let if A wins vote and if B wins, and evaluate using two proper metrics:
The Brier score (BS) averages squared probability errors across individual votes, while Expected Calibration Error (ECE) groups predictions into bins and compares average predicted win probability with observed win fraction in each bin. When comparing calibration across experiments, state whether bins track fixed A-versus-B win probabilities or top-class confidence, along with bin edges and sample counts. Suppose four independent reviewers compare two softmax answers and split 3 votes for A to 1 vote for B. Both and overconfident pick A and achieve the same 75% binary accuracy, yet Brier score and one-bin ECE sharply separate them:
import numpy as np
# Fixed A/B ordering evaluated across four independent votes.
votes = np.array([1.0, 1.0, 1.0, 0.0])
for probability in [0.75, 0.99]:
accuracy = float(np.mean((probability > 0.5) == (votes == 1.0)))
brier = float(np.mean((probability - votes) ** 2))
one_bin_ece = float(abs(probability - votes.mean()))
assert accuracy == 0.75
print(f"p_A={probability:.2f} accuracy={accuracy:.2f} Brier={brier:.4f} one_bin_ECE={one_bin_ece:.2f}")p_A=0.75 accuracy=0.75 Brier=0.1875 one_bin_ECE=0.00
p_A=0.99 accuracy=0.75 Brier=0.2451 one_bin_ECE=0.24Forecasting matches this four-vote empirical rate, though four votes are far too small to prove population calibration across prompt categories. When training on repeated votes per pair, using the empirical vote share as a soft BCE target has unique minimum . Keep in mind that a 50-50 reviewer split and an explicit "equally good" tie annotation record different phenomena; use a tie-aware objective such as Davidson or Rao-Kupper when modeling explicit ties directly.

Reward hacking, Goodhart's Law, and policy overoptimization
High accuracy on a held-out preference split never guarantees safe downstream policy optimization. Held-out accuracy only inspects past outputs drawn from static reference policies, whereas RL actively explores new prompt-completion trajectories where learned correlations break down. Goodhart's Law warns that optimizing an imperfect proxy score relentlessly will eventually expose and amplify discrepancies between the proxy and true quality, even when the proxy correlates well initially.
The Goodhart mechanism and Gao et al. scaling laws
Gao et al. (2023) studied reward overoptimization in controlled synthetic setups by using a much larger gold reward model both to generate preference labels and to evaluate PPO and best-of- policies trained against a smaller proxy scorer.[2] Under sustained optimization, proxy reward continued rising while gold reward reached a peak and then declined. Their fitted scaling relationships show how evaluator errors rare on static splits become dominant modes once RL searches into uncharted regions of completion space, though exact peak heights depend on model family, data budget, and the proxy-vs-gold gap rather than forming universal constants.
When an RL optimizer explores candidate completions, four common failure modes repeatedly break naive scorers:
- Verbosity bias: Longer replies often correlate with thoroughness in preference data, tempting the scorer to equate extra tokens with higher quality even when the extra text is redundant filler. Always test length-controlled slices where factual content is fixed while token count varies.
- Sycophancy: Preference models and human raters can reward completions that agree with a user's incorrect assumption rather than gently correcting it.[12] Distinguishing helpful tone from factual deference requires counterexamples where polite disagreement wins.
- Formatting shortcuts: Markdown structure aids readability, yet optimization can exaggerate surface layout. In Eisenstein et al.'s helpfulness experiments, list-formatted generations rose to roughly half of policy outputs compared with about 8% in the underlying preference dataset.[11]
- Omitted criteria: A reward head trained solely on helpfulness comparisons omits truthfulness, safety, or privacy signals unless those dimensions are explicitly represented in both training pairs and held-out audits.
The audit snippet below checks whether winning candidates are systematically longer across held-out pairs; seeing 100% accuracy alongside 100% longer winners signals the need for length-matched counterexamples before declaring success:
import numpy as np
chosen_rewards = np.array([1.8, 2.4, 1.6])
rejected_rewards = np.array([0.7, 1.1, 0.3])
chosen_tokens = np.array([36, 58, 33])
rejected_tokens = np.array([19, 22, 14])
margins = chosen_rewards - rejected_rewards
accuracy = float(np.mean(margins > 0))
length_gaps = chosen_tokens - rejected_tokens
print(f"pair_accuracy={accuracy:.2f}")
print(f"mean_margin={float(margins.mean()):.2f}")
print(f"chosen_answers_longer={bool(np.all(length_gaps > 0))}")
print("next_check=build length-matched eval pairs")pair_accuracy=1.00
mean_margin=1.23
chosen_answers_longer=True
next_check=build length-matched eval pairsAnnotator disagreement audits complement length audits by exposing ambiguous slices where strict binary labels are noisy. In practice, unanimous pairs train cleanly under strict Bradley-Terry loss, while split votes warrant inspection, additional annotation, or soft-target modeling rather than automatic deletion. Separating confident agreement from contested comparisons prevents label noise from masking real learning failures on unambiguous bugs:
votes = {
"clear_bugfix": ["chosen", "chosen", "chosen"],
"style_only": ["chosen", "rejected", "chosen"],
"ambiguous_format": ["chosen", "rejected", "tie"],
}
for pair_id, labels in votes.items():
chosen_share = labels.count("chosen") / len(labels)
status = "train" if chosen_share == 1.0 else "review_or_hold_out"
print(f"{pair_id}: chosen_share={chosen_share:.2f} status={status}")clear_bugfix: chosen_share=1.00 status=train
style_only: chosen_share=0.67 status=review_or_hold_out
ambiguous_format: chosen_share=0.33 status=review_or_hold_outTesting mitigations before trusting them
1. Why a naive length margin fails to block length shortcuts
One tempting heuristic subtracts a length margin inside the sigmoid:
Subtracting that offset requires a longer winner to beat the loser by a larger margin during training, yet it does not prevent a reward head from relying solely on token length. For a pure length scorer evaluated on a pair with positive length gap , the shifted logit equals . As the NumPy check below shows, raising the length weight from 0 to 6 drives training loss down to 0.0067 while failing every length-reversed counterexample where the correct fix is short and the wrong answer is long:
import numpy as np
alpha = 1.0
chosen_length, rejected_length = 2.0, 1.0
gap = chosen_length - rejected_length
for length_weight in np.array([0.0, 3.0, 6.0]):
effective_logit = (length_weight - alpha) * gap
loss = float(np.logaddexp(0.0, -effective_logit))
short_good_score = length_weight * 1.0
long_wrong_score = length_weight * 2.0
passes_reversal = bool(short_good_score > long_wrong_score)
print(f"length_weight={length_weight:.0f} training_loss={loss:.4f} passes_length_reversal={passes_reversal}")length_weight=0 training_loss=1.3133 passes_length_reversal=False
length_weight=3 training_loss=0.1269 passes_length_reversal=False
length_weight=6 training_loss=0.0067 passes_length_reversal=FalseRather than adding fragile length penalties that can punish genuinely thorough answers, construct balanced training and evaluation data containing concise correct answers, detailed explanations that earn their length, and verbose errors. Adding targeted counterexamples forces gradient updates to align with semantic quality across every length band instead of shifting scores uniformly by character count. Always verify shortcut resistance directly on length-matched and length-reversed slices.
2. Ensemble disagreement penalties ()
An ensemble of reward models trained with varied seeds, data splits, or architectures can expose unstable regions during optimization. Before aggregating scores across members, align their origins and scales on shared calibration data so arbitrary model offsets do not masquerade as disagreement.[11] Computing the sample mean and sample standard deviation yields a conservative Lower Confidence Bound (LCB) score:
Despite its name, this LCB penalty is an optimization heuristic rather than a calibrated statistical confidence bound. While Eisenstein et al. showed that reward ensembles reduce reward hacking, ensembles cannot stop it completely because correlated errors across members produce high mean reward with zero disagreement.[11]
Three aligned reward models all assign score 5 to an answer that cites a nonexistent API flag. With , what does the conservative ensemble rule return, and does it flag the hallucination?
Answer
The ensemble mean is 5 and the standard deviation is 0, so the penalized score stays 5. Variance across models cannot detect a mistake that all three models share; catching fabricated citations requires external verification or an independently grounded evaluator.
Auditing fresh policy rollouts before optimization
Before committing compute to a full RL run, generate fresh completions from your policy model, form new preference pairs, and obtain human reviews covering factuality, refusal quality, and shortcut slices. Static held-out accuracy only inspects past outputs, whereas fresh-rollout accuracy measures whether the reward model ranks what the policy currently emits. The screening check below applies an 80% fresh-pair accuracy threshold; passing a screen supports proceeding to a small monitored RL trial rather than certifying production release:
evaluation = {
"static_held_out_pairs": {"accuracy": 0.92, "human_reviewed": False},
"fresh_policy_pairs": {"accuracy": 0.64, "human_reviewed": True},
}
minimum_fresh_accuracy = 0.80
def passes_fresh_gate(result, minimum):
return result["human_reviewed"] and result["accuracy"] >= minimum
assert not passes_fresh_gate({"accuracy": 0.99, "human_reviewed": False}, 0.80)
assert passes_fresh_gate({"accuracy": 0.80, "human_reviewed": True}, 0.80)
passes_pair_screen = passes_fresh_gate(evaluation["fresh_policy_pairs"], minimum_fresh_accuracy)
print(f"static_accuracy={evaluation['static_held_out_pairs']['accuracy']:.2f}")
print(f"fresh_accuracy={evaluation['fresh_policy_pairs']['accuracy']:.2f}")
print(f"passes_pair_screen={passes_pair_screen}")static_accuracy=0.92
fresh_accuracy=0.64
passes_pair_screen=FalseStandardized evaluation: RewardBench and RewardBench 2
Internal validation pairs often share the blind spots of your own annotation pipeline, so standardized benchmarks provide a broader external check. RewardBench (Lambert et al., 2024) scores preference models across four pairwise categories: Chat (general dialogue and helpfulness), Chat Hard (challenging instruction edge cases), Safety (harm refusal and jailbreak resistance), and Reasoning (mathematics and code correctness).[13] RewardBench 2 (Malik et al., 2025; revised April 2026) expands evaluation to Factuality, Precise Instruction Following, Math, Safety, Focus, and Ties, moving ordinary categories to one-among-four selection (25% random baseline) with a separate Ties metric so scores are not directly comparable to RewardBench V1.[14]
In Tulu 3 8B experiments reported by the RewardBench 2 authors, benchmark scores correlated strongly with best-of-16 sampling performance (Pearson 0.87), whereas PPO outcomes proved far less sensitive among capable reward models and degraded sharply when reward-model initialization or prompt distributions differed from training.[14] Log the exact benchmark version, model checkpoint, tokenizer, and chat template with every run so results stay comparable.
A candidate reward model leads the public RewardBench leaderboard. Can you skip fresh rollout checks and run full PPO optimization directly?
Answer
No. Leaderboard rank in one benchmark setup did not reliably separate PPO outcomes in controlled experiments, especially across distribution shifts. Validate the scorer on fresh outputs from your target policy and prompt mix first, then run a short monitored RL trial.
Architectural spectrum: ORMs, PRMs, verifiers, and generative judges
A scalar Bradley-Terry Outcome Reward Model (ORM) is only one way to score completions. Each supervision paradigm trades off labeling cost, inference speed, interpretability, and vulnerability to reward hacking in a different way. Choosing among supervision designs depends on whether correctness can be checked deterministically, whether step-level attribution is required, and how quickly the scorer must run:
| Approach | Feedback or objective | Primary strengths | Primary vulnerabilities |
|---|---|---|---|
| Scalar Bradley-Terry ORM | Single scalar | Fast inference, standard PPO/GRPO terminal reward, inspectable margins | Overoptimization, length bias, Goodhart drift |
| Direct Preference Optimization (DPO) | Implicit policy/reference log-ratio | Optimizes policy directly from pairs without a separate scorer | Relies on offline pair coverage, reference anchor, and regularization |
| Process Reward Model (PRM) | Step-level scalars | Locates errors across multi-step reasoning traces | High step-annotation cost, still susceptible to gaming[15] |
| Generative RM / LLM-as-judge | Verdict text or verdict token probabilities | Flexible rubrics and synthetic label scaling | Inference latency, position bias, rationales that may not reflect true reasons[16] |
| Verifiable reward / RLVR | Programmatic execution or constraint result | Deterministic validation of specified checks | Incomplete unit tests, parser loopholes, gap between check and intent[5] |
Process Reward Models (PRMs) assign step-by-step scores across mathematical or coding derivations. Lightman et al. showed that step-level supervision improved best-of-N MATH selection on a fixed generator compared with outcome supervision, though they did not train the generator itself via RL and process benefits vary across domains.[15] In domains with deterministic checkers, Tulu 3 replaced learned reward models with executable RLVR verifiers for math and instruction constraints.[5] Executable checks remove learned-head vulnerabilities, though reward hacking still persists whenever tests are incomplete or a parser credits a right final token paired with broken reasoning.
Put the pieces together with TRL
Our two-feature NumPy head isolated shortcut dynamics on CPU. At full transformer scale, TRL 1.13.0 wires together sequence classification, pair collation, and stable BCE optimization via RewardTrainer. The configuration fragment below assumes train_pairs and eval_pairs are Hugging Face Dataset objects already grouped by prompt ID, because RewardTrainer expects your split to prevent leakage beforehand.
from trl import RewardConfig, RewardTrainer
# Configured for trl==1.13.0 with pre-split train_pairs and eval_pairs Datasets.
trainer = RewardTrainer(
model="Qwen/Qwen3-0.6B",
args=RewardConfig(
output_dir="reward-head",
max_length=2048, # Set to cover full templated sequences without truncating endings.
center_rewards_coefficient=0.01, # Starting offset penalty; verify against held-out margins.
),
train_dataset=train_pairs,
eval_dataset=eval_pairs,
)
trainer.train()
metrics = trainer.evaluate()
trainer.save_model("reward-head/final")
trainer.processing_class.save_pretrained("reward-head/final")Passing a causal model ID instructs TRL to initialize a one-label sequence-classification head automatically. Always save and reload both the trained scalar head and the exact tokenizer and chat template, verifying held-out pair scores after reloading so an uninitialized or dropped linear head is caught immediately.[8] Before scaling into policy optimization, review calibration error, fresh-rollout agreement, and shortcut counterexamples alongside clear stop criteria.
Production diagnostic reference
When a reward-modeling run or downstream RL trial diverges, match the symptom against these diagnostic checks:
1. Training loss drops while held-out accuracy stalls
Stalled validation accuracy paired with shrinking training loss can indicate overfitting, noisy preference labels, widening margins on already easy pairs, or prompt mismatch between splits. Because Bradley-Terry loss keeps pushing already-correct margins further positive, aggregate loss can drop long after ranking decisions stop improving on boundary cases. Check prompt-group overlap, audit label consistency, and inspect held-out loss and counterexample accuracy before changing learning rates.
2. Scorer prefers wordier, padded answers
When verbose errors beat concise correct solutions, training correlations likely linked length with reward or evaluators systematically favored detail over accuracy. Standard held-out evaluation can hide this failure if the validation split obeys the same length skew as training. Build length-matched and length-reversed evaluation slices with technical content held constant, add balanced counterexamples to training, and verify accuracy on those slices rather than relying on length penalties alone.
3. Proxy reward climbs during PPO while human ratings drop
A rising internal score alongside falling user satisfaction signals proxy exploitation or distributional drift into regions where the reward model errs. Pause the optimization run, save policy and scorer checkpoints, and evaluate fresh generations with human judges or verifiable checks. Tightening KL regularization, lowering the optimization budget, diversifying training pairs, or switching to a calibrated ensemble can reduce hacking without removing every evaluator blind spot.
4. Raw scores drift rapidly across steps
Wide shifts in score magnitude typically stem from unanchored score origins, runaway margins on separable pairs, or mismatched pooling and template masks between training and evaluation. Unbounded drift distorts effective KL balance during PPO even when validation pair accuracy looks stable. Log pair midpoints separately from pair margins, verify padding masks and end-of-turn token alignment, and combine centering penalties with L2 regularization and held-out calibration audits.
For one preference pair, the centering penalty is with . Compare score pairs [8, −8] and [2, 2]. Which penalty pair is correct?
- 1.28 and 0.08.
- 2.56 and 0.32.
- 0.00 and 0.32.
- 0.00 and 0.08.
Answer
0.00 and 0.32.
The sums are 0 and 4, giving penalties and . Squaring each score individually or forgetting the factor of four alters the formula, and note that zero centering penalty still leaves the margin [8, −8] unconstrained.
A rating-margin loss uses effective logit . With , what score difference is required to reach an effective chosen probability of 0.8?
- About 1.9863.
- About 1.3863.
- About 0.7863.
- Exactly 0.6000.
Answer
About 1.9863.
Solving gives . Dropping or subtracting shifts the required score gap incorrectly, while yields probability 0.50.
Three score-aligned ensemble members return [1, 3, 5]. Using sample standard deviation with denominator and conservative aggregation , which value and interpretation hold?
- Aggregate 3; it applies no disagreement penalty and yields an uncalibrated score.
- Aggregate about 2.184; it applies sample deviation and yields an uncalibrated score.
- Aggregate 2; it applies sample deviation and yields an uncalibrated score.
- Aggregate 1; it applies half the variance and yields an uncalibrated score.
Answer
Aggregate 2; it applies sample deviation and yields an uncalibrated score.
The mean is 3 and the sample variance is , so sample standard deviation is 2 and . Remember that this penalty is a conservative heuristic rather than a calibrated confidence interval and cannot fix errors shared across all ensemble members.
A benchmark leaderboard score rises, yet human reviewers find more unsupported technical claims in fresh outputs sampled during a trial run. The trial's predeclared stop rule triggers on unsupported-claim regressions. What action should you take?
- Continue the trial and reuse the static benchmark rank as evidence of quality.
- Continue the trial and drop the new failures from the previous benchmark split.
- Pause fresh output reviews while increasing the proxy-reward optimization budget.
- Stop the trial, save the generated outputs, and investigate the separate support failure.
Answer
Stop the trial, save the generated outputs, and investigate the separate support failure.
Once the independent regression rule triggers, preserve the checkpoint and failing outputs to diagnose why fresh generations regressed rather than overriding the finding with static leaderboard scores or changing the evaluation set.