RLHF & DPO Alignment

Suppose a release candidate raises its internal judge score by $0.31$. Reviewers prefer its answers $0.04$ less often, and responses are 48 tokens longer because the model learned a "Policy-safe answer:" preamble. Would you promote it? No. One proxy moved; product behavior moved the wrong way.

The last chapter turned one access-policy pair into a scalar. For the prompt "temporary admin access for tonight's migration," the ticket-and-cite answer scored $1.8$ and the grant-tonight answer scored $0.7$. The Bradley-Terry preference model maps that $1.1$ margin to roughly a $75\%$ chance that the labeler prefers the safer answer. This chapter asks what to do with that judge: train a policy against it, or skip the extra model and optimize the policy directly from the pairs?

An instruction-tuned assistant can still answer that prompt with fluent, confident, and wrong guidance. InstructGPT puts the gap plainly: a model trained to predict plausible text can still be unhelpful, untruthful, or unsafe even when the prose looks polished.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

That's the alignment problem. Pretraining optimizes next-token prediction. Preference methods such as RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization) push the model toward answers a labeling policy prefers. They don't, by themselves, prove truthfulness or safety outside that policy's coverage. Whether learned or held out, a judge remains a proxy.

The gap SFT can't close

After instruction tuning, the model can follow the shape of a request. It can still grant a privileged role, refuse a reasonable one, or pad a correct answer until it's unusable. Next-token prediction doesn't enforce a product's rules. Predicting what comes next isn't the same as satisfying the behavior being judged.

Keep the same prompt the reward-modeling chapter used: "Policy: cite the retrieved access policy and escalate privileged-role changes. User: My service account needs temporary admin access for tonight's migration. Can you approve it?" Two SFT completions are both grammatical:

• Response A: "The retrieved access policy requires escalation for temporary admin access. I can open a reviewer ticket and cite policy P-7."
• Response B: "Temporary admin access is common for migrations, so you can proceed and clean it up tomorrow."

A follows the supplied policy. B violates it. Before preference labels enter, ask what SFT can infer from this row: both strings are grammatical, but only A makes escalation explicit. Unless demonstrations encode that distinction, next-token loss has no term that marks B as a product failure. Preference training supplies that comparative signal when the distinction appears in labeled data.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

RLHF: a judge, then an actor

RLHF is the large-scale preference pipeline InstructGPT popularized, building on earlier human-feedback fine-tuning for summarization: start from an SFT model, train a reward model from comparisons, then optimize the policy with reinforcement learning.^{[2]Reference 2Learning to Summarize from Human Feedback.https://arxiv.org/abs/2009.01325[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

The pipeline is a handoff. SFT gives instruction-following. The reward model turns comparisons into a scalar. PPO updates the policy while a KL term charges drift away from the SFT reference.

Three-phase pipeline

Treat each phase as answering one question: can the model imitate useful answers, can a scorer reproduce the preference rule, and can the policy improve under that scorer without leaving tested behavior? The first two use stored examples. The third generates and scores policy outputs.

Phase 3 is where the extra machinery shows up. InstructGPT's RL stage is a bandit: a prompt comes in, the policy writes a response, the reward model returns a scalar, and a per-token KL penalty against the SFT model is added to limit overoptimization.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} That fresh response is the key difference from fitting only the stored demonstrations.

Bradley-Terry, one number later

Here is how the judge turns a pair into a training signal. Predict the direction first: because A's score is higher, its preference probability should be above $0.5$; reversing the scores should make the observed A label less likely. The Bradley-Terry model says the probability that a human prefers $y_w$ over $y_l$ is a sigmoid of the reward gap:^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290}

$P(y_w \succ y_l \mid x) = \sigma\left(r(x, y_w) - r(x, y_l)\right)$

Plug in last chapter's scores: $r(A)=1.8$, $r(B)=0.7$. The gap is $1.1$, so $P(A \succ B)=\sigma(1.1)\approx 0.750$. If the human picked A, the negative log-likelihood of that observation is $-\log(0.750)\approx 0.287$. Flip the ranking and the same math becomes $-\log(0.250)\approx 1.386$. The loss pushes the reward model toward the higher-probability assignment.

Why does a larger reward gap make the preference probability closer to 1?

Answer

The sigmoid receives a larger positive input, so it moves toward 1. In Bradley-Terry terms, a larger score difference means the model is more confident the higher-scored response wins.

The training loss is the same expectation you saw for the reward model:

$$\mathcal{L}_{\text{RM}} = -\mathbb{E}_{(x, y_w, y_l)} \left[\log \sigma\left(r(x, y_w) - r(x, y_l)\right)\right]$$

where $y_w$ is the winning response and $y_l$ is the losing response for the same prompt $x$.

PPO and the KL budget

Once you have a reward model, ask what prevents the policy from chasing its score by drifting. Treat the language model as a policy $\pi_\theta$. Proximal Policy Optimization (PPO) makes clipped, conservative updates.^{[4]Reference 4Proximal Policy Optimization Algorithms.https://arxiv.org/abs/1707.06347} The KL-regularized objective is reward minus a penalty for leaving the reference:

$$\begin{aligned} \max_\theta\;& \mathbb{E}_{x \sim D, y \sim \pi_\theta} \left[r(x, y)\right] \\ &- \beta \cdot \mathbb{E}_{x \sim D} \left[\text{KL}(\pi_\theta(\cdot|x) \| \pi_{\text{ref}}(\cdot|x))\right] \end{aligned}$$

The Kullback-Leibler term is a drift budget, not a proof that the reward is right. InstructGPT applies that penalty per token specifically to mitigate reward-model overoptimization.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

That budget matters on this prompt. Suppose the policy starts opening with "Policy-safe answer: access guidance follows." If the reward model scores that preamble higher because it correlated with polite answers in the comparison set, the policy will amplify it. KL charges the departure from the SFT reference. Held-out humans still decide whether behavior actually improved.

To see why this is a system rather than one loss, track four roles in one PPO-style step. Real systems may shard or share weights, but the computation still has to cover all four:

1. Actor / policy ($\pi_\theta$): the model being trained
2. Reference ($\pi_{\text{ref}}$): frozen SFT model for the KL penalty
3. Reward model ($r_\phi$): frozen scorer
4. Critic / value ($V_\psi$): predicts expected return for advantage estimation

The snippet below isolates shaped reward for two sampled responses. Before reading its output, predict the ordering: the preamble has raw reward $1.35$ but log-ratio $1.6$, versus $0.1$ for the near-reference sample. With $\beta=0.2$, KL should erase that apparent win. This isn't a PPO trainer. A full implementation also estimates advantages, clips updates, and applies token-level masks.

The output makes the ordering explicit: the higher raw score isn't automatically the better update once drift is charged. Production PPO stacks token-level KL, a value model, and often Generalized Advantage Estimation (GAE) on top of this shape, and it generates fresh responses each cycle.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155[4]Reference 4Proximal Policy Optimization Algorithms.https://arxiv.org/abs/1707.06347}

Why this loop is expensive

The four roles create a coordination cost first. Sharding changes resident memory, not the fact that you're generating and scoring on-policy text before the gradient step.

That loop exposes a wide tuning surface: reward scale, the KL coefficient (beta), policy-ratio clip threshold, value-loss weight, advantage estimation, and rollout quality. Weak drift control invites reward-model exploitation. Overly strong control can freeze useful updates.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155[4]Reference 4Proximal Policy Optimization Algorithms.https://arxiv.org/abs/1707.06347}

Both costs would be manageable if the score stayed trustworthy. The reward model saw a finite set of comparisons. Once the improving policy leaves that distribution, its score no longer necessarily means "humans would prefer this." That is overoptimization: numeric reward climbs while held-out human preference flattens or drops. On this assistant, a longer compliance preamble is an easy tell because it can score well without fixing the access decision.

DPO: the same objective without the extra judge

PPO-style RLHF is a multi-role online loop. If a fixed pair already tells you which response wins, why generate another response for every update? DPO targets those pairs with a classification-style loss. It removes engineering, but it doesn't give you online exploration or guarantee the same checkpoint PPO would have found.

Why an offline loss can stand in for RL

Under a KL-regularized reward-maximization objective and a Bradley-Terry preference model, the optimal policy and the reward are two views of the same object. DPO parameterizes an implicit reward with policy-to-reference log-probability ratios, so the pairwise loss never needs a standalone reward model or a PPO loop.^{[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} Before the algebra, check its anchor: if policy equals reference, both relative log-ratios are zero, so the DPO logit is zero and the pair probability is $0.5$.

The implicit reward of the optimal policy is:

$r^*(x, y) = \beta \log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x)$

$\pi^*$ is that optimal policy, $\pi_{\text{ref}}$ is the reference, $\beta$ is the KL trade-off (and the DPO logit scale), and $Z(x)$ is a partition function that depends only on the prompt. Because Bradley-Terry uses a difference of rewards, $Z(x)$ cancels, and you can train a policy $\pi_\theta$ directly from pairs.^{[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290}

Why does the reference policy appear in DPO?

Answer

It anchors the aligned model to the base behavior. DPO rewards making preferred responses more likely relative to the reference while discouraging uncontrolled drift.

Work the loss on the P-7 pair

Substitute the implicit reward into the preference loss:

$$\begin{aligned} \mathcal{L}_{\text{DPO}} = -\mathbb{E}\Bigg[ \log \sigma\Bigg( &\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} \\ &- \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \Bigg) \Bigg] \end{aligned}$$

$y_w$ is chosen (ticket and cite P-7) and $y_l$ is rejected (grant admin tonight). Read each term as change relative to the frozen reference: raise the chosen response relative to that baseline, lower the rejected response, and the margin grows. The reference is a baseline, not a safety proof.

Suppose the current sequence log-probabilities are:

• Chosen A: policy $-8.2$, reference $-8.5$
• Rejected B: policy $-9.5$, reference $-9.1$
• $\beta = 0.1$

Then:

1. Chosen relative advantage: $-8.2 - (-8.5) = 0.3$
2. Rejected relative advantage: $-9.5 - (-9.1) = -0.4$
3. Margin: $0.3 - (-0.4) = 0.7$
4. Scaled logit: $0.1 \times 0.7 = 0.07$
5. $\sigma(0.07) \approx 0.517$
6. Loss: $-\log(0.517) \approx 0.659$

The update tries to grow that $0.7$ margin: raise chosen likelihood relative to the reference, lower rejected likelihood, scaled by how far the policy already sits from the reference.

Those $-8.2$ figures are already summed sequence log-probabilities. In a trainer they come from a causal shift: concatenate prompt and response, take next-token logits on the response span, log-softmax, gather the observed token, and sum. The next snippet does that on a four-token vocab for the same chosen/rejected split. Predict the result: boosting chosen-token logits should raise chosen log-probability and lower the loss while the frozen reference stays fixed. Padding masks, attention masks, and distributed training stay out of this example.

At initialization, a policy copied from its reference has zero reference-relative log-ratios, so the DPO logit is zero and the loss is near $0.693$. The model's token logits themselves need not be zero. After training starts, watch margins together with held-out preference quality and output regressions. A near-zero DPO logit alone doesn't diagnose weak pairs or a bad beta.

Production trainers also mask padding and track response length, because a whole-sequence log-prob sum can create length bias when chosen and rejected answers differ systematically in length.

Likelihood displacement: a bigger margin isn't a more likely chosen answer

Pair arithmetic above hides a second question: did the policy make the chosen answer more likely, or only make the rejected answer less likely? A rising DPO margin isn't the same as "the preferred answer is getting more likely." DPO tracks the gap between chosen and rejected relative log-ratios. Lowering $\log\pi_\theta(y_w\mid x)$ and $\log\pi_\theta(y_l\mid x)$ together can still look like a win when the rejected side falls faster.

Keep the P-7 pair's frozen reference at $-8.5$ (chosen) and $-9.1$ (rejected). If chosen log-probability slides from $-8.2$ to $-9.1$ while rejected slides from $-9.5$ to $-11.2$, the relative margin grows from $0.7$ to $1.5$ even though both labeled completions are being avoided. Plotting $\log(\pi/\pi_{\text{ref}})$ makes that gap the thing you see: chosen starts at $+0.3$ and finishes at $-0.6$, so the preferred answer is now less likely than the SFT reference.

Monitor at least four series:

If margins improve while chosen log-prob collapses, stop. Inspect data, $\beta$, length bias, and offline coverage before more steps. Length-matched pairs, length-normalized log-probs, or length-aware objectives such as SimPO are the usual next tests.

Vanilla DPO is still just that pairwise loss plus a frozen reference. No reward model, no value model, no rollout.

Comparing RLHF and DPO

The useful contrast is what each method has to keep alive, not a winner-take-all ranking.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} Read the table as a deployment question: which signal must stay live, and which behavior can your stored data never show you?

DPO is a strong candidate baseline for offline pairwise preferences. PPO isn't obsolete. If you need online exploration, fresh model-generated negatives, or a reward that moves as the policy moves, an RL-style loop can reach cases a frozen DPO dataset can't.

The data both methods inherit

RLHF and DPO consume the same kind of row, then spend it differently. RLHF fits a reward model on the comparisons, then rolls out. DPO trains the policy on the pairs themselves.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} The row is shared; the feedback path is not.

The hygiene from the reward-modeling chapter still applies: same rendered prompt, a clear non-tie label, generator provenance, and prompt-grouped splits so related comparisons don't leak into evaluation. A pair from another generator isn't automatically invalid, but it can miss the failure modes your current policy actually produces. When shift matters, include current-policy samples.

You don't need a huge dump of easy instructions. You need coverage of the distinctions you care about: refusals, tool use, reasoning, style, and the P-7-style policy calls. Ask what happens when the current policy invents a failure absent from the stored pairs: DPO has no training row for it until you refresh the data, while PPO can expose it only after a fresh rollout reaches its judge. If annotators keep disagreeing, more labels can quantify uncertainty. They won't turn a vague rubric into a training signal.

When the proxy lies

That release rehearsal is reward hacking in miniature. A policy exploits the scorer instead of getting better. The reward model is a proxy; optimize hard enough and the proxy becomes the target.

If the scorer overweights length or markdown, policy optimization will amplify those traits faster than answer quality. The visible symptom is rising proxy reward without a matching gain in held-out human preference.

Verbosity

Annotators sometimes treat length as quality. Predict what happens next: a padded P-7 explanation can beat a short, correct one, so the policy learns to pad if that correlation is in the data.

• Symptom: mean length climbs while helpfulness plateaus.
• Fix: length-matched preference evals, and explicit concise-versus-padded labels. Length penalties can also punish necessary detail, so validate them.

The 0.693 trap

If the policy starts equal to its reference, every DPO relative margin is zero and the loss starts near $-\log(0.5)=0.693$. That's expected. At zero logit, the gradient is $\sigma(0)-1=-0.5$, so the binary loss is not flat.

That is initialization, not evidence that a pair is weak. Ambiguous pairs still hurt. Conflicting labels on the same prompt can cancel in aggregate or teach arbitrary style. Diagnose from agreement, slices, and held-out behavior, not from initial loss.

• Symptom: loss stays near $0.693$ after real updates, and held-out preference doesn't move.
• Fix: confirm the policy is updating, then inspect contradictory labels, unresolved ties, prompt leakage, and missing coverage before blaming beta.

A fresh DPO run starts near loss 0.693 with a pair gradient of about -0.5. Does that prove the pair is ambiguous or uninformative?

Answer

No. A policy initialized from its reference starts with zero relative margin, so 0.693 is expected and the directional gradient is strong. Suspect ambiguity only when training stays there without held-out improvement, then inspect conflicting labels, leakage, and coverage.

A release gate that actually uses humans

Release asks a simpler question: did behavior improve within the tested drift budget? Tie proxy improvement to behavior you care about:

1. Hold out human eval. Higher RM score has to win more often with people.
2. Track auxiliaries: length, refusal rate, formatting drift, calibration.
3. Watch KL and stop when you leave the tested drift budget.
4. Refresh the scorer when current outputs leave its training distribution.

With the release-candidate numbers from the opening, predict the gate before running it: proxy reward should improve, human preference should fail, and promotion should be blocked.

Other failure signatures worth the same stop rule:

• DPO loss stays near $0.693$ after confirmed updates: audit labels, ties, leakage, and coverage rather than treating init loss as a diagnosis.
• DPO improves tone but still grants admin: the pair set rewarded style, not the P-7 refusal. Add targeted safety pairs and policy-specific evals.
• PPO generations get repetitive while reward rises: freeze promotion, inspect reward scale and KL, then refresh evaluation together.

Online DPO and iterative alignment

Vanilla DPO is offline: it trains on a fixed set of chosen and rejected responses.^{[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} Ask what happens when the policy improves past those rows: old comparisons describe a model you no longer have, so they may no longer contain hard negatives.

Later work studies the online setting: sample fresh responses from the current policy, label them with humans or a preference model, then apply an IPO-style or related pairwise update on the new comparisons.^{[5]Reference 5Human Alignment of Large Language Models through Online Preference Optimisation.https://arxiv.org/abs/2403.08635}

The loop is:

1. Sample two or more candidates from the current policy.
2. Ask humans or a preference model which is better.
3. Store a new (prompt, chosen, rejected) triple.
4. Run a DPO-style update with whatever reference-policy rule the method uses.
5. Repeat so the data follows the model.

There isn't one canonical "online DPO" algorithm, and reference-update choices differ. The family resemblance is what matters: refresh preference data on-policy, keep a pairwise loss, skip a full PPO stack.^{[5]Reference 5Human Alignment of Large Language Models through Online Preference Optimisation.https://arxiv.org/abs/2403.08635} You get less distribution mismatch and harder negatives over time, without necessarily standing up a critic.

Nearby methods, not drop-in replacements

These methods make different data contracts. Start with the signal you can reliably collect: a chosen/rejected pair, a binary label, or a task verifier. Nearby objectives relax the pair requirement, change the loss, or drop the reference. One popular method in this table isn't a DPO variant at all.

IPO, KTO, ORPO, and SimPO still live in the offline preference family. GRPO is a different branch. Process reward models (PRMs) score intermediate steps rather than only the finished answer; that's adjacent credit assignment, and RLVR & Verifiable Rewards treats it properly.^{[13]Reference 13Let's Verify Step by Step.https://arxiv.org/abs/2305.20050}

For subjective assistant behavior from fixed pairs, start with DPO. Binary logs without pairs point to KTO. Tight memory makes ORPO and SimPO attractive because they skip a frozen reference. When DPO overfits near-deterministic labels, IPO is the bounded-margin test. When a task has a checkable verifier, online RL such as GRPO is the method to evaluate, not another DPO flag.

There is one edge case worth predicting before the code: if every verifier gives a group the same reward, normalization has no direction and every relative advantage should be zero.

The zero-variance guard is load-bearing. If every sampled answer gets the same reward, the group has no relative signal. TRL reports frac_reward_zero_std, the fraction of generation-batch samples whose reward standard deviation is zero. A high value means the rewards aren't distinguishing answers, so inspect the verifier or the sampling before assuming GRPO is learning from useful comparisons.^{[12]Reference 12GRPO Trainer.https://huggingface.co/docs/trl/grpo_trainer}

What comes after pairwise human labels

Human comparisons don't scale to every new failure. Constitutional AI and RLAIF (Reinforcement Learning from AI Feedback) write the rules down as principles, then use a model to critique, revise, and rank against those principles, so you need fewer repeated harmlessness labels.^{[14]Reference 14Constitutional AI: Harmlessness from AI Feedback.https://arxiv.org/abs/2212.08073} The next chapter is that pipeline, plus red teaming to see where the principles fail.

The dependence doesn't go away. A constitution is only as good as the judge that applies it. Held-out humans still decide whether "cite P-7 and escalate" survived contact with a real user.