RLHF & DPO Alignment

Reward modeling showed how to turn chosen/rejected responses into a scalar preference signal. RLHF and DPO ask what to do with that signal: train a separate reward-driven policy, or optimize the policy directly from preference pairs.

Imagine asking a customer-support AI whether a delayed order should be refunded, and it returns something fluent, confident, and wrong. Or asking about a return-policy exception and getting advice that sounds authoritative but violates company rules. InstructGPT frames this gap clearly: pretrained language models can be untruthful, toxic, or unhelpful even when their text is polished.^[1]

That is the alignment problem. Pretraining optimizes next-token prediction, not direct human preference. A language model optimized to predict plausible text can still produce responses that sound useful whether or not they are helpful, truthful, or safe. RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization) are post-training methods that push a model toward responses preferred under a labeling policy. They don't, by themselves, prove truthfulness or safety outside that policy's coverage.

The gap SFT can't close

After instruction tuning, a model can follow instructions. But it may still generate harmful content, be dishonest, refuse reasonable requests, or produce verbose and unhelpful answers. This misalignment happens because next-token prediction doesn't inherently enforce a product's safety rules or a labeler's preference rubric. Predicting what comes next isn't the same as satisfying the behavior being evaluated.

Alignment optimizes for human preferences beyond task completion. It bridges the gap between next-token prediction (what's probable?) and helpful interaction (what's desirable?).

To see why SFT alone isn't enough, consider a customer-support bot given this context: "Policy: offer a refund after seven calendar days without delivery. Customer: My package hasn't arrived after 10 days. Can I get a refund?" The SFT model might generate two grammatically correct responses:

• Response A: "I can help with that. Since your package is 10 days late, you're eligible for a refund under our 7-day late-delivery policy. Would you like me to process that now?"
• Response B: "Packages sometimes take longer than expected. Please wait another week and contact us if it still hasn't arrived."

Response A follows the supplied policy. Response B is unhelpful and violates that stated policy. Both can look fluent. SFT teaches the model to imitate demonstrations; preference training supplies comparative signal when those distinctions are represented in the labeled data.^[1]

That preference signal is what alignment adds.

RLHF: building a judge and an actor

RLHF (Reinforcement Learning from Human Feedback) is the classic large-scale preference-alignment pipeline popularized by InstructGPT: start from an SFT model, train a reward model from human comparisons, then optimize the policy with reinforcement learning.^[1]

The key distinction is separation of roles. SFT gives the model baseline instruction-following behavior, the reward model approximates labeled preference, and PPO updates the policy while a KL term discourages large drift from the SFT reference.

Three-phase pipeline

RLHF runs in three distinct steps. First, collect demonstrations and perform Supervised Fine-Tuning (SFT) to establish baseline instruction-following. Then gather comparison data to train a proxy judge (the reward model). Finally, use reinforcement learning to optimize the language model against that judge.

Reward models and the Bradley-Terry trick

Think of the reward model like a support-policy auditor comparing two draft replies. The auditor doesn't need to generate the reply; it needs to score the policy-correct, helpful answer above the vague or unsafe one.

The mathematical foundation is the Bradley-Terry model, which assumes the probability that a human prefers response $y_w$ over $y_l$ depends on their underlying "reward" scores:^[1][2]

$P(y_w \succ y_l \mid x) = \sigma\left(r(x, y_w) - r(x, y_l)\right)$

This model captures the intuition that the larger the gap in quality between two responses, the more confident we are about which one a human would prefer.

Let's walk through a concrete example. Suppose the reward model scores Response A at 2.0 and Response B at 0.5 for the same refund prompt. The probability that a human prefers A over B is:

$P(A \succ B) = \sigma(2.0 - 0.5) = \sigma(1.5) \approx 0.818$

That means the model predicts an 81.8% chance the human picks A. If the human picked A, the log-likelihood of that observation is $\log(0.818) \approx -0.201$. If the model had instead predicted B > A, the log-likelihood would be $\log(1 - 0.818) = \log(0.182) \approx -1.704$. The loss pushes the reward model toward the higher probability assignment.

The reward model is trained to predict which of two responses a human would prefer. The loss function maximizes the log-likelihood of the observed preferences:

$$\mathcal{L}_{\text{RM}} = -\mathbb{E}_{(x, y_w, y_l)} \left[\log \sigma\left(r(x, y_w) - r(x, y_l)\right)\right]$$

where $y_w$ is the preferred (winning) response and $y_l$ is the rejected (losing) response for the same prompt $x$.

PPO and KL drift control

Once we have a reward model, we treat the language model as a policy $\pi_\theta$. We use Proximal Policy Optimization (PPO), a reinforcement learning algorithm designed to make clipped, conservative policy updates^[3], to maximize the expected reward. The objective maximizes reward while penalizing deviation from the reference policy:

$$\begin{aligned} \max_\theta\;& \mathbb{E}_{x \sim D, y \sim \pi_\theta} \left[r(x, y)\right] \\ &- \beta \cdot \mathbb{E}_{x \sim D} \left[\text{KL}(\pi_\theta(\cdot|x) \| \pi_{\text{ref}}(\cdot|x))\right] \end{aligned}$$

Think of the Kullback-Leibler (KL) term as a drift budget. The policy may seek higher learned reward, while deviations from the reference cost reward. This can reduce exposure to regions where the reward model is unreliable; it can't prove the reward is correct or prevent every exploitable shortcut. InstructGPT applies a per-token KL penalty to mitigate reward-model overoptimization.^[1]

To see why drift control matters, imagine the policy starts generating refund responses that begin with "I am an AI assistant designed to help with refunds." If the reward model accidentally scores that opening higher because it correlates with politeness in training data, the policy can amplify it. A KL term charges departures from the reference, while held-out human checks still determine whether behavior improved.

In the common conceptual PPO-style RLHF setup, one update coordinates four model roles. Implementations can shard, offload, or share parts of these roles, so this is a compute-and-state picture rather than a universal memory layout:

1. Actor/Policy ($\pi_\theta$): The model being trained
2. Reference ($\pi_{\text{ref}}$): Frozen SFT model for KL penalty
3. Reward Model ($r_\phi$): Frozen model that scores outputs
4. Critic/Value ($V_\psi$): Predicts expected returns for advantage estimation

The calculation below isolates the shaped reward for sampled responses. It isn't a PPO trainer: a full implementation also estimates advantages, clips updates, and applies token-level masks.

The higher raw score isn't automatically the preferred optimization target once policy drift is charged. In production, PPO implementations compute token-level KL penalties, estimate returns with a value model and often GAE, and separate rollout collection from policy updates. The point here is the system shape: PPO-style RLHF generates fresh responses and coordinates policy, reference, reward, and value roles.^[1][3]

Why RLHF is hard to run

Despite its power, the RLHF pipeline introduces practical challenges that make it expensive to run and hard to debug at scale.

The first problem is systems overhead. Standard PPO-style RLHF needs actor, reference, reward, and critic/value roles during a training cycle. Sharding and offload change resident memory, but not the computation and coordination burden. Each rollout batch also begins with current-policy generation before gradient updates, so throughput becomes partly an inference problem.

Reinforcement learning also adds more knobs than supervised fine-tuning. PPO is sensitive to reward scale, the KL coefficient (beta), clipping threshold (epsilon), value-loss weighting, advantage estimation, and rollout quality. Weak drift control increases exposure to reward-model exploitation; overly strong control can suppress useful updates.^[1][3]

The reward model itself is a silent failure point. It was trained on a finite collection of human comparisons. When the improving policy starts generating responses outside that distribution, the proxy scores become unreliable. That is when overoptimization appears: the numeric reward keeps climbing while real human preference scores plateau or drop. In a customer-support setting, you might see the model start adding long disclaimers or repetitive apologies because those patterns happened to score high in the limited preference data.

DPO: the mathematical shortcut

PPO-style RLHF has a multi-role online loop. DPO instead targets fixed preference pairs with a simpler objective. It removes engineering components; it doesn't supply online exploration or guarantee the same outcome as every RLHF run.

The key insight

At first glance, DPO looks like a completely different alignment recipe. The key mathematical result in the DPO paper is narrower: under a KL-regularized reward-maximization objective and a Bradley-Terry-style preference model, an implicit reward can be parameterized through policy-to-reference log-probability ratios.^[2] That relationship yields an offline pairwise loss without training a standalone reward model or running PPO during DPO training.

Here's the "aha" moment in plain English. Under those modeling assumptions, an implicit reward is encoded by how policy log-probabilities change relative to a reference model. DPO can therefore train directly on labeled pairs. It learns only from the coverage and quality of those comparisons unless you build a separate data-refresh loop.

The implicit reward is:

$r^*(x, y) = \beta \log \frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta \log Z(x)$

Where $\pi^*$ is the optimal policy for the stated objective, $\pi_{\text{ref}}$ is the reference model, $\beta$ is tied to the KL-regularization trade-off and scales the DPO logit, and $Z(x)$ is a normalization term that depends only on prompt $x$. DPO then trains a policy $\pi_\theta$ from pairwise preferences under this parameterization.^[2]

DPO loss with a worked example

By substituting this relationship back into the preference loss, DPO derives a loss function that depends only on the policy and reference model:

$$\begin{aligned} \mathcal{L}_{\text{DPO}} = -\mathbb{E}\Bigg[ \log \sigma\Bigg( &\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} \\ &- \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \Bigg) \Bigg] \end{aligned}$$

Where $y_w$ is the preferred response and $y_l$ is the rejected response for the same prompt $x$, and $\pi_{\text{ref}}$ is the reference model that provides the baseline distribution. The reference model defines the relative log-probability baseline and shapes the divergence trade-off; it isn't proof that outputs stay fluent, calibrated, or safe.

Let's trace the loss with actual numbers. Suppose for our refund prompt we have:

• Chosen response (A): policy log-prob = -8.2, reference log-prob = -8.5
• Rejected response (B): policy log-prob = -9.5, reference log-prob = -9.1
• DPO scale / regularization parameter: $\beta = 0.1$

Step by step:

1. Policy advantage for chosen: $-8.2 - (-8.5) = 0.3$
2. Policy advantage for rejected: $-9.5 - (-9.1) = -0.4$
3. Margin inside the sigmoid: $0.3 - (-0.4) = 0.7$
4. Scaled by beta: $0.1 \times 0.7 = 0.07$
5. Sigmoid probability: $\sigma(0.07) \approx 0.517$
6. Loss: $-\log(0.517) \approx 0.659$

The loss pushes the policy to increase that 0.7 margin. If the policy becomes more confident on the chosen response (say, log-prob rises to -7.5 while the rejected stays similar), the margin grows, the sigmoid probability approaches 1.0, and the loss drops toward 0. In gradient terms, the update increases the likelihood of the chosen answer relative to the reference while decreasing the likelihood of the rejected answer, all scaled by how far the current policy already deviates from the reference.

The implementation of the DPO loss is straightforward. The runnable example below uses a tiny causal language model so the tensor shapes are real without downloading an external checkpoint. A production implementation would add padding masks, attention masks, distributed training, and tokenizer-specific details.

Visible feedback: At initialization, a policy copied from its reference naturally produces logits near zero and loss near 0.693. After training begins, monitor margins together with held-out preference quality and output regressions. Near-zero logits alone don't diagnose weak pairs or a bad beta.

In production, DPO implementations also mask padding tokens and explicitly track the chosen-vs-rejected log-probability margin during training. Teams usually monitor response length too, because whole-sequence log-probability sums can create length bias when chosen and rejected responses have systematically different lengths.

DPO training flow

Vanilla DPO training is operationally simpler than PPO-style RLHF. It applies an offline pairwise loss instead of coordinating rollouts, a learned reward model, and a value model. A reference model provides baseline log-probabilities while the active policy is updated from preference data.

DPO turns preference alignment into a supervised pairwise classification loss. You still need a frozen reference model and high-quality chosen/rejected pairs, but you no longer train a separate reward model or value function.

Comparing RLHF and DPO

To better understand the trade-offs between the traditional RLHF pipeline and DPO, compare what each method optimizes and what it has to keep alive during training.^[1][2]

DPO lowers system complexity and is a practical candidate baseline for offline pairwise preferences. That doesn't mean PPO is obsolete. If a team needs online exploration, fresh model-generated negatives, or a reward signal that changes as the model improves, RL-style alignment has capabilities fixed-dataset DPO doesn't.

Preference data: the foundation

Whether a team uses RLHF or DPO, preference-signal quality sets a major performance ceiling. In practice, collecting and cleaning that signal is often the slowest and most expensive part of the pipeline. Both methods start from pairwise preference data, but they use it differently: RLHF trains a reward model on those comparisons, while DPO optimizes the policy directly on the same comparisons.^[1][2]

Data requirements

Regardless of the specific algorithm chosen, the foundation of preference optimization relies on high-quality preference data. A usable row needs more than chosen and rejected: it needs the same rendered prompt context, a clear non-tie label, and split provenance that prevents related comparisons leaking into evaluation.

Annotation quality guidelines

Poorly annotated data can lead to undesirable behavior or failure to generalize the intended policy. That is why preference annotation needs a clear rubric, disagreement handling, and held-out checks.

Preference data from another generator isn't automatically invalid. The risk is coverage mismatch: labels may compare styles or capabilities the target policy rarely produces, while ignoring its actual failure modes. Record each generator, evaluate on fresh target-policy outputs, and include current-policy samples when distribution shift matters.

While data quality matters more than raw count, dataset size still changes what the model can learn.

What scale changes

• A narrow domain can improve from a relatively small, carefully curated preference set.
• A general assistant needs much broader coverage across refusals, tool use, reasoning, style, and failure cases.
• If annotators keep disagreeing, more labels can quantify uncertainty, but they won't turn an ambiguous rubric into clear training direction.

When alignment backfires

The most subtle failure mode in alignment is reward hacking: the model learns to exploit reward model weaknesses rather than genuinely improving. This happens because the reward model is merely a proxy for human preference, not a perfect representation of it.

When the language model is optimized against this proxy, it can discover edge cases where the proxy assigns high scores to degenerate outputs. The proxy becomes the target, and the target can stop representing true preference.

For example, if the reward model overweights length or surface formatting, policy optimization can amplify those traits faster than actual answer quality. The visible symptom is rising proxy reward without a matching improvement in held-out human preference.

The verbosity bias

Aligned models can become wordy when human annotators equate length with quality. A long, rambling refund explanation might score higher than a concise, accurate one because it looks more thorough. If your reward data has this bias, the policy learns to pad responses.

Symptom: Average response length grows steadily during training while helpfulness scores plateau. Fix: Add length-matched preference evaluations and annotate concise versus padded answers explicitly. Length penalties or normalization are interventions to validate, because they can also punish necessary detail.

Ambiguous labels and the 0.693 trap

If the policy begins equal to its reference, every DPO relative margin starts at zero and the loss starts near -log(0.5) = 0.693. That is expected, not evidence that an individual pair provides no gradient. At zero logit, the binary loss has its largest directional gradient magnitude.

Ambiguous pairs still hurt: if different annotators or duplicate prompts point in conflicting directions, their updates can cancel in aggregate or teach arbitrary style preferences. Diagnose data quality from agreement, slices, and held-out behavior, not from initial loss alone.

Symptom: Loss remains near 0.693 after meaningful training and held-out preference metrics don't improve. Fix: Check that the policy is updating, then inspect contradictory labels, unresolved ambiguity, prompt leakage, and missing task coverage before changing beta.

Mitigation strategies

To combat reward hacking and ensure the model genuinely improves, alignment engineers employ several defensive techniques:

1. Hold out human eval data: Check whether higher proxy reward also improves real human preference wins
2. Track auxiliary metrics: Monitor length, refusal rate, formatting drift, and calibration rather than trusting one scalar score
3. KL budgets: Cap total KL divergence from the reference model during training
4. Refresh the scorer: Re-label or retrain the reward signal when current policy outputs drift too far from the scorer's training data

Online DPO and iterative alignment

Vanilla DPO is an offline algorithm: it trains on a fixed dataset of chosen and rejected responses.^[2] As the policy improves, those old comparisons become less representative of what the current model produces.

Several later papers study this online setting: sample fresh responses from the current policy, label those pairs with humans or a learned preference model, then apply an IPO-style update, or a closely related pairwise preference loss, on the new comparisons.^[4]

Conceptually, an online DPO loop has five steps:

1. Sample two or more candidate responses from the current policy for each prompt.
2. Ask humans or a preference model to choose the better candidate.
3. Convert the result into a fresh (prompt, chosen, rejected) triple.
4. Run a DPO-style update using the chosen reference-policy strategy.
5. Repeat with new policy outputs so the data distribution follows the model as it changes.

There isn't one canonical "online DPO" algorithm, and reference-policy update choices differ across methods. The family resemblance is the important part: refresh preference data on-policy while keeping a pairwise preference loss instead of a full PPO loop.^[4]

Online DPO advantages

• Reduced distribution mismatch: The model learns from its own current outputs, not only from stale comparisons.
• Harder negatives over time: As the policy improves, the rejected samples become stronger and more informative.
• Keeps the DPO loss: You retain a simpler pairwise objective instead of moving all the way to PPO.

Beyond vanilla DPO

While DPO is the standard offline baseline, it still requires high-quality pairwise preference data $(y_w, y_l)$. That data is expensive to collect and often noisy, especially when annotators disagree or the difference between two good answers is subtle. Newer methods attempt to relax this requirement, improve the optimization objective, or combine training stages to reduce pipeline complexity.

From that broader design space, several practical variants and adjacent post-training methods have become common discussion points:

IPO, KTO, ORPO, and SimPO all still live in the offline preference-optimization family. GRPO sits on a different branch: it is online reinforcement learning, not a DPO variant. Programmatic correctness checks are one high-value reward source, not a requirement of the algorithm.

For teams optimizing subjective assistant behavior from fixed pairs, DPO is a useful baseline candidate. If you only have binary feedback rather than strict pairs, KTO is designed for that shape. If memory is tight, ORPO and SimPO avoid a separate frozen reference model. If DPO overfits near-deterministic labels, IPO tests a bounded-margin alternative. If the task has strong programmatic verifiers, online RL methods such as GRPO become worth evaluating.

Modern post-training family map

These names get mixed together too easily. Some are optimizers, some are reward-model families, and some are full pipeline patterns.

Two clarifications keep taxonomy clean:

1. PRM and ORM aren't alternatives to DPO in the same sense as ORPO or KTO. They are scorers that can feed RLHF or reasoning-time search loops.
2. GRPO isn't a DPO variant. It's online reinforcement learning that uses group-relative advantages instead of a learned critic.^[9]

Use this shortcut:

• offline subjective alignment: evaluate DPO as a baseline
• offline binary feedback: consider KTO
• single-stage preference plus SFT: consider ORPO
• reference-free, length-bias-sensitive: consider SimPO
• near-deterministic labels, DPO overfits: consider IPO
• online verifiable reasoning: consider GRPO with programmatic verifier rewards

That map is also why curriculum splits reward modeling, RLHF/DPO, and RLVR into separate lessons. Names overlap in conversation, but actual training loops differ.

Constitutional AI and RLAIF

A parallel evolution in alignment reduces the amount of human labeling by replacing most pairwise annotations with AI feedback. Constitutional AI (CAI), introduced by Bai et al. (2022)^[12], defines a written set of principles (a "constitution") and uses the model itself to:

1. Critique its own outputs against each principle
2. Revise the output to comply with the violated principle
3. Rank candidate outputs under those principles to generate preference data

In the original Constitutional AI paper, those AI critiques and revisions are used in two stages.^[12] First, revised responses from self-critique supervise an SFT-style phase. Then the model samples alternative answers, an AI judge ranks them under the constitution, a preference model is trained on those rankings, and an RL stage optimizes against that model.^[12] The core advantage is scale: you can synthesize far more feedback from a constitution than by hiring humans to label every harmful example. Bai et al. show that this can train a more harmless, less evasive assistant with far fewer human labels, though the result still depends heavily on the quality of the constitution and the judge model.^[12]

Constitutional AI is especially valuable when alignment requirements can be written as explicit rules, such as "cite retrieved sources when making factual claims" or "escalate high-risk refund and safety issues instead of sounding certain." A written constitution makes the feedback policy easier to audit and update than ad hoc human-labeling instructions.

Common pitfalls

• Reward rises while human preference gets worse. Symptom: offline reward looks better, but held-out human wins flatten or drop. Fix: keep human eval held out, watch length and formatting drift, cap KL movement, and refresh the reward signal when policy outputs shift.
• DPO loss stays near 0.693 after updates. Symptom: held-out pair quality also fails to improve. Fix: first verify gradients and optimizer steps, then audit contradictory labels, ties, leakage, and coverage rather than treating initialization loss as a diagnosis.
• DPO improves style but misses safety boundaries. Symptom: outputs sound polished yet still cross policy lines. Fix: add targeted safety pairs, refusal-quality checks, and policy-specific evals instead of assuming generic preference data covers safety.
• PPO updates make the model incoherent. Symptom: reward rises, but generations get unstable or repetitive. Fix: stop promotion, examine reward scale and drift, then tune KL/update size and refresh evaluation together rather than trusting proxy reward.

Practice checkpoints

Evaluation rubric

• Foundational: Outline the RLHF pipeline from SFT to reward modeling to PPO.
• Foundational: Explain why PPO needs a reference policy, reward model, value model, and KL constraint.
• Intermediate: Compute a Bradley-Terry preference probability and a DPO loss from small log-probability numbers.
• Intermediate: Compare RLHF's online rollout loop with DPO's offline pairwise objective.
• Advanced: Diagnose reward hacking, length bias, weak preference pairs, and overtraining symptoms.
• Advanced: Explain when KTO, ORPO, GRPO, ORM, and PRM belong in the post-training map.
• Advanced: Explain how Constitutional AI and RLAIF reduce human labeling pressure while introducing judge-quality dependence.

What to remember

1. Why alignment: Instruction following doesn't equate to helpful, safe, or honest behavior.
2. RLHF pipeline: SFT to reward model to PPO. A common PPO setup coordinates four model roles during training.
3. Reward model: Trained on pairwise preference comparisons to predict human judgments.
4. PPO: Maximizes reward while keeping the model close to the reference using a KL constraint.
5. DPO: Under its KL-regularized preference-model assumptions, optimizes an implicit reward without a separate reward model or PPO loop.
6. Tradeoffs: RLHF supports online exploration but has more moving parts; DPO has a simpler offline loop but still depends on data and objective choices.
7. Modern variants: KTO and ORPO change offline preference tuning requirements, while GRPO is often used for online training with verifiable rewards.
8. Constitutional AI: RLAIF scales alignment by replacing much of the human labeling with principled AI feedback.
9. Production reality: Evaluate DPO as an offline baseline when pair data fits the task. Evaluate online methods when fresh sampling or verifier feedback is central.

Follow-up questions

Try this quick exercise before moving on. Suppose you have a DPO training batch with the following log probabilities (all base-e logs):

With $\beta = 0.1$, compute the DPO loss by hand. Then answer: is the margin positive (the policy already prefers chosen over rejected relative to the reference) or negative?