Skip to content
Skip to lesson
Back to Topics
LearnAdvanced Training & AdaptationRLHF & DPO Alignment
๐Ÿ›ก๏ธHardAlignment & Safety

RLHF & DPO Alignment

An internal judge score can rise while blinded comparisons stay flat. Turn preference labels into a PPO update or a DPO loss, then test the score on held-out behavior.

59 min read
Learning path
Step 115 of 199 in the full curriculum

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

Suppose a release candidate raises its internal judge score by 0.310.31, but wins only 184 of 400 blinded comparisons against the old assistant. Its answers are also 48 tokens longer on average, often starting with "Policy-safe answer:". Would you promote it? These illustrative results don't establish a human-preference improvement, even though the training score rose. Keep that disagreement in view as we turn preference labels into policy updates.

The last chapter turned one access-policy pair into a scalar. For the prompt "temporary admin access for tonight's migration," the ticket-and-cite answer scored 1.81.8 and the grant-tonight answer scored 0.70.7. The maps that 1.11.1 margin to roughly a 75%75\% chance that the labeler prefers the safer answer. This chapter asks what to do with that judge: train a policy against it, or skip the extra model and optimize the policy directly from the pairs?

An instruction-tuned assistant can still answer that prompt with fluent, confident, and wrong guidance. InstructGPT puts the gap plainly: a model trained to predict plausible text can still be unhelpful, untruthful, or unsafe even when the prose looks polished.[1]

That's the alignment problem. optimizes next- prediction. Preference methods such as (Reinforcement Learning from Human Feedback) and (Direct Preference Optimization) push the model toward answers a labeling policy prefers. They don't, by themselves, prove truthfulness or safety outside that policy's coverage. Whether learned or held out, a judge remains a proxy.

What preference pairs add to SFT

After instruction tuning, the model can follow the shape of a request. It can still grant a privileged role, refuse a reasonable one, or pad a correct answer until it's unusable. Next-token prediction doesn't enforce a product's rules. Predicting what comes next isn't the same as satisfying the behavior being judged.

Keep the same prompt the reward-modeling chapter used: "Policy: cite the retrieved access policy and escalate privileged-role changes. User: My service account needs temporary admin access for tonight's migration. Can you approve it?" Two completions are both grammatical:

  • Response A: "The retrieved access policy requires escalation for temporary admin access. I can open a reviewer ticket and cite policy P-7."
  • Response B: "Temporary admin access is common for migrations, so you can proceed and clean it up tomorrow."

A follows the supplied policy. B violates it. SFT on good demonstrations can teach escalation too; preference training isn't the only route to that behavior. A pair adds a different signal: under the same prompt, prefer A over B. That contrast can be easier to label than writing a new ideal answer, but it only teaches the distinctions the labels actually contain.[1]

The same P-7 request branches into two short actions: open a reviewer ticket and cite the policy, or grant temporary admin access. An illustrative reward scorer assigns 1.80 and 0.70, a 1.10 gap that maps through Bradleyโ€“Terry to about 75 percent for the first answer. The scores are example values, not calibrated human probabilities.
An illustrative 1.10 reward gap gives a Bradleyโ€“Terry prediction near 75%. Calibrate the scorer against held-out judgments.

RLHF: a judge, then an actor

RLHF is the large-scale preference pipeline InstructGPT popularized, building on earlier human-feedback fine-tuning for summarization: start from an SFT model, train a from comparisons, then optimize the policy with reinforcement learning.[2][1]

The pipeline is a handoff. SFT gives instruction-following. The reward model turns comparisons into a scalar. PPO updates the policy while a KL term charges drift away from the SFT reference.

Three-phase pipeline

Treat each phase as answering one question: can the model imitate useful answers, can a scorer reproduce the preference rule, and can the policy improve under that scorer without leaving tested behavior? The first two use stored examples. The third generates and scores policy outputs.

Phase 3 is where the extra machinery shows up. InstructGPT's RL stage is a bandit: a prompt comes in, the policy writes a response, the reward model returns a scalar, and a per-token KL penalty against the SFT model is added to limit overoptimization.[1] That fresh response is the key difference from fitting only the stored demonstrations.

Bradley-Terry, one number later

Here is how the judge turns a pair into a training signal. Predict the direction first: because A's score is higher, its preference probability should be above 0.50.5; reversing the scores should make the observed A label less likely. The Bradley-Terry model says the probability that a human prefers ywy_w over yly_l is a sigmoid of the reward gap:[1][3]

P(ywโ‰ปylโˆฃx)=ฯƒ(r(x,yw)โˆ’r(x,yl))P(y_w \succ y_l \mid x) = \sigma\left(r(x, y_w) - r(x, y_l)\right)

Plug in last chapter's scores: r(A)=1.8r(A)=1.8, r(B)=0.7r(B)=0.7. The gap is 1.11.1, so P(Aโ‰ปB)=ฯƒ(1.1)โ‰ˆ0.750P(A \succ B)=\sigma(1.1)\approx 0.750. If the human picked A, the negative log-likelihood of that observation is about 0.2870.287. Flip the scores and the loss becomes about 1.3871.387. These are the scorer's preference probabilities, not independently calibrated guarantees about people. The loss pushes the reward model toward the observed label.

bradley_terry_p7_pair.py
import math

def sigmoid(value: float) -> float:
    return 1 / (1 + math.exp(-value))

reward_a = 1.8
reward_b = 0.7
preference_probability = sigmoid(reward_a - reward_b)
negative_log_likelihood = -math.log(preference_probability)

print(f"P(A preferred over B) = {preference_probability:.1%}")
print(f"NLL = {negative_log_likelihood:.3f}")
Bradley-Terry output
P(A preferred over B) = 75.0%
NLL = 0.287

The training loss is the same expectation you saw for the reward model:

LRM=โˆ’E(x,yw,yl)[logโกฯƒ(r(x,yw)โˆ’r(x,yl))]\mathcal{L}_{\text{RM}} = -\mathbb{E}_{(x, y_w, y_l)} \left[\log \sigma\left(r(x, y_w) - r(x, y_l)\right)\right]

where ywy_w is the winning response and yly_l is the losing response for the same prompt xx.

PPO and the cost of leaving the reference

Once you have a reward model, ask what prevents the policy from chasing its score by drifting into gibberish. Treat the language model as a policy ฯ€ฮธ\pi_\theta: a conditional probability distribution over responses given a prompt. clips the objective's incentive for large probability ratios on sampled actions; policy changes elsewhere still depend on the data and update process.[4] We start by formalizing what we want to optimize: expected reward minus a penalty for leaving the reference model, scaled by coefficient ฮฒ>0\beta > 0:

maxโกฮธExโˆผD,yโˆผฯ€ฮธ(โ‹…โˆฃx)[r(x,y)]โˆ’ฮฒExโˆผD[DKL(ฯ€ฮธ(โ‹…โˆฃx)โˆฅฯ€ref(โ‹…โˆฃx))]\max_\theta \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot \mid x)} \left[ r(x, y) \right] - \beta \mathbb{E}_{x \sim \mathcal{D}} \left[ \mathbb{D}_{\text{KL}}\left(\pi_\theta(\cdot \mid x) \parallel \pi_{\text{ref}}(\cdot \mid x)\right) \right]

The is an expectation over complete responses: DKL(ฯ€ฮธโˆฅฯ€ref)=Eyโˆผฯ€ฮธ[logโกฯ€ฮธ(yโˆฃx)โˆ’logโกฯ€ref(yโˆฃx)]\mathbb{D}_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) = \mathbb{E}_{y \sim \pi_\theta} [\log \pi_\theta(y \mid x) - \log \pi_{\text{ref}}(y \mid x)]. It's non-negative, with zero for identical distributions, even though an individual sampled log-ratio can be negative. A plain sample mean estimates this KL when rollouts come directly from the same ฯ€ฮธ\pi_\theta whose log-probabilities you score. Greedy search, temperature changes, nucleus filtering, and top-kk truncation change the sampling distribution; scores from an older policy do too. Correct for that mismatch or label the metric accordingly.

Before inspecting the PPO approximation, look at what the unconstrained mathematical optimum of this objective actually is. For any fixed prompt xx, write the objective directly:

J(ฯ€)=โˆ‘yฯ€(yโˆฃx)r(x,y)โˆ’ฮฒโˆ‘yฯ€(yโˆฃx)logโกฯ€(yโˆฃx)ฯ€ref(yโˆฃx)\mathcal{J}(\pi) = \sum_y \pi(y \mid x) r(x, y) - \beta \sum_y \pi(y \mid x) \log \frac{\pi(y \mid x)}{\pi_{\text{ref}}(y \mid x)}

Pull out โˆ’ฮฒ-\beta:

J(ฯ€)=โˆ’ฮฒโˆ‘yฯ€(yโˆฃx)[logโกฯ€(yโˆฃx)ฯ€ref(yโˆฃx)โˆ’1ฮฒr(x,y)]=โˆ’ฮฒโˆ‘yฯ€(yโˆฃx)logโก(ฯ€(yโˆฃx)ฯ€ref(yโˆฃx)expโก(r(x,y)/ฮฒ))\mathcal{J}(\pi) = -\beta \sum_y \pi(y \mid x) \left[ \log \frac{\pi(y \mid x)}{\pi_{\text{ref}}(y \mid x)} - \frac{1}{\beta} r(x, y) \right] = -\beta \sum_y \pi(y \mid x) \log \left( \frac{\pi(y \mid x)}{\pi_{\text{ref}}(y \mid x) \exp(r(x, y)/\beta)} \right)

Define the prompt partition function (normalizer) Z(x)=โˆ‘yฯ€ref(yโˆฃx)expโก(r(x,y)/ฮฒ)Z(x) = \sum_y \pi_{\text{ref}}(y \mid x) \exp(r(x, y)/\beta), which is independent of the response yy. Multiplying and dividing inside the logarithm by Z(x)Z(x) yields:

J(ฯ€)=โˆ’ฮฒโˆ‘yฯ€(yโˆฃx)logโก(ฯ€(yโˆฃx)1Z(x)ฯ€ref(yโˆฃx)expโก(r(x,y)/ฮฒ)โ‹…1Z(x))\mathcal{J}(\pi) = -\beta \sum_y \pi(y \mid x) \log \left( \frac{\pi(y \mid x)}{\frac{1}{Z(x)} \pi_{\text{ref}}(y \mid x) \exp(r(x, y)/\beta)} \cdot \frac{1}{Z(x)} \right)

Splitting the log product separates the terms:

J(ฯ€)=โˆ’ฮฒDKL(ฯ€(โ‹…โˆฃx)โˆฅฯ€โˆ—(โ‹…โˆฃx))+ฮฒlogโกZ(x)\mathcal{J}(\pi) = -\beta \mathbb{D}_{\text{KL}}\left( \pi(\cdot \mid x) \parallel \pi^*(\cdot \mid x) \right) + \beta \log Z(x)

where the optimal distribution is defined analytically as:

ฯ€โˆ—(yโˆฃx)=1Z(x)ฯ€ref(yโˆฃx)expโก(r(x,y)ฮฒ)\pi^*(y \mid x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y \mid x) \exp\left( \frac{r(x, y)}{\beta} \right)

Assume ฮฒ>0\beta>0, a finite normalizer Z(x)Z(x), and restrict policies to responses supported by the reference. Then KL is non-negative, with equality at ฯ€=ฯ€โˆ—\pi=\pi^*, so this is the unique optimal distribution for the fixed reward. A parameterized language model might not represent or reach it. This result is an exponentially reweighted reference policy, not a convergence guarantee for an optimizer.

Computing the normalizer over language sequences is generally intractable. PPO instead makes approximate policy updates from sampled responses; finite data, clipping, and critic error can keep the learned policy away from this optimum.[3]

Token-level reward shaping and the four model roles

The PPO-style RLHF recipe here coordinates four logical model roles:

  1. Actor / policy (ฯ€ฮธ\pi_\theta): the generative language model being trained.
  2. Reference (ฯ€ref\pi_{\text{ref}}): frozen SFT model providing baseline log-probabilities for the KL penalty.
  3. Reward model (rฯ•r_\phi): frozen scorer that evaluates sequence quality.
  4. Critic / value (VฯˆV_\psi): predicts expected future return from each prefix to help estimate advantages.

These roles don't require four independent, simultaneously resident GPU backbones. The critic can be smaller or share policy parameters; frozen roles can use separate devices or offloading. InstructGPT, for example, used a 6B reward model even for its 175B policy and initialized its value function from that reward model.[1][4]

In this sequence-score recipe, the reward model scores the completed response y=(y1,โ€ฆ,yT)y=(y_1,\dots,y_T). A terminal reward can train earlier tokens: their returns include that future outcome. Assigning it to each immediate reward would instead count it repeatedly. KL shaping adds a different signal, charging each generated token for its sampled policy/reference log-ratio. When collecting a rollout, evaluate that charge under the rollout policy ฯ€old\pi_{\mathrm{old}}:

Rt={โˆ’ฮฒ(logโกฯ€old(ytโˆฃx,y<t)โˆ’logโกฯ€ref(ytโˆฃx,y<t)),1โ‰คt<Trฯ•(x,y)โˆ’ฮฒ(logโกฯ€old(yTโˆฃx,y<T)โˆ’logโกฯ€ref(yTโˆฃx,y<T)),t=TR_t = \begin{cases} -\beta \left( \log \pi_{\mathrm{old}}(y_t \mid x, y_{<t}) - \log \pi_{\text{ref}}(y_t \mid x, y_{<t}) \right), & 1 \le t < T \\ r_\phi(x, y) - \beta \left( \log \pi_{\mathrm{old}}(y_T \mid x, y_{<T}) - \log \pi_{\text{ref}}(y_T \mid x, y_{<T}) \right), & t = T \end{cases}

For an undiscounted episode, adding these rewards uses the autoregressive log-probability factorization:

โˆ‘t=1TRt=rฯ•(x,y)โˆ’ฮฒโˆ‘t=1Tlogโกฯ€old(ytโˆฃx,y<t)ฯ€ref(ytโˆฃx,y<t)=rฯ•(x,y)โˆ’ฮฒlogโกฯ€old(yโˆฃx)ฯ€ref(yโˆฃx)\sum_{t=1}^T R_t = r_\phi(x, y) - \beta \sum_{t=1}^T \log \frac{\pi_{\mathrm{old}}(y_t \mid x, y_{<t})}{\pi_{\text{ref}}(y_t \mid x, y_{<t})} = r_\phi(x, y) - \beta \log \frac{\pi_{\mathrm{old}}(y \mid x)}{\pi_{\text{ref}}(y \mid x)}

Its expectation under ฯ€old\pi_{\mathrm{old}} is the reward-minus-KL objective for that rollout policy. It isn't a fresh evaluation of the actor after every PPO update. Store rollout rewards, old log-probabilities, and detached advantage targets for the batch.

Generalized Advantage Estimation (GAE) uses residuals ฮดt=Rt+ฮณV(st+1)โˆ’V(st)\delta_t=R_t+\gamma V(s_{t+1})-V(s_t) and, for an episode ending at TT, A^t=โˆ‘l=0Tโˆ’t(ฮณฮป)lฮดt+l\hat A_t=\sum_{l=0}^{T-t}(\gamma\lambda)^l\delta_{t+l}. Here ฮณ\gamma discounts future rewards, while ฮป\lambda trades off reliance on the critic against longer sampled returns. Use V(sT+1)=0V(s_{T+1})=0 for a true terminal state; a deliberately truncated continuing episode needs an appropriate bootstrap.[5]

Try terminal-only rewards with no KL and a zero baseline. Does the first token have zero advantage?

terminal_reward_reaches_earlier_tokens.py
def gae(rewards: list[float], values: list[float], lam: float) -> list[float]:
    if len(values) != len(rewards) + 1:
        raise ValueError("need one baseline per step plus the terminal bootstrap")
    advantages = [0.0] * len(rewards)
    running = 0.0
    for t in reversed(range(len(rewards))):
        delta = rewards[t] + values[t + 1] - values[t]  # gamma = 1
        running = delta + lam * running
        advantages[t] = running
    return advantages

rewards = [0.0, 0.0, 1.0]
values = [0.0, 0.0, 0.0, 0.0]
for lam in [1.0, 0.5]:
    advantages = gae(rewards, values, lam)
    print(f"lambda={lam:.1f}: advantages={advantages}")
    assert all(value > 0 for value in advantages)
Terminal credit output
lambda=1.0: advantages=[1.0, 1.0, 1.0]
lambda=0.5: advantages=[0.25, 0.5, 1.0]

With ฮณ=ฮป=1\gamma=\lambda=1, the terminal score reaches every earlier decision through its return. With this inaccurate zero critic and a smaller ฮป\lambda, the signal fades backward. The scalar reward propagates through the sequence; it does not identify which token caused the outcome.

Three token transitions have rewards 0, 0, and 1. The critic predicts zero at all four states, including terminal V3, so the TD residuals are 0, 0, and 1. Backward GAE accumulation gives advantages [1, 1, 1] at lambda 1 and [0.25, 0.5, 1] at lambda 0.5; arrows point from the terminal decision toward earlier tokens.
Backward GAE carries the terminal reward to earlier tokens. A smaller ฮป\lambda discounts more distant credit when this zero critic supplies the baseline.
One sampled response produces reward 1.35 and a sampled reference charge of 0.32, giving return 1.03. The critic value 0.80 is subtracted to produce advantage 0.23. With policy ratio 1.20, the clipped surrogate is 0.276. Reward and reference are frozen roles; critic and actor are trainable roles. This example does not imply four simultaneously resident model copies.
One illustrative decision connects the reward, reference charge, critic baseline, advantage, and actor ratio. Real token trajectories use prefix values and GAE.

The snippet below isolates shaped reward for two sampled responses. Before reading its output, predict the ordering: the preamble has raw reward 1.351.35 but log-ratio 1.61.6, versus 0.10.1 for the near-reference sample. With ฮฒ=0.2\beta=0.2, KL erases that apparent win.

kl_regularized_reward.py
samples = [
    {"name": "near_reference", "reward": 1.20, "policy_logp": -2.1, "ref_logp": -2.2},
    {"name": "preamble_hack", "reward": 1.35, "policy_logp": -1.2, "ref_logp": -2.8},
]
beta = 0.20

for sample in samples:
    sampled_log_ratio = sample["policy_logp"] - sample["ref_logp"]
    shaped_reward = sample["reward"] - beta * sampled_log_ratio
    print(f"{sample['name']}: raw={sample['reward']:.2f} shaped={shaped_reward:.2f}")
KL-shaped reward output
near_reference: raw=1.20 shaped=1.18
preamble_hack: raw=1.35 shaped=1.03

The higher raw score isn't automatically the larger shaped return. These two rows illustrate arithmetic, not an estimate of the full KL. On a complete two-response distribution, one log-ratio can be negative while its policy-weighted average stays positive:

sampled_ratio_vs_kl.py
import math

policy = [0.8, 0.2]
reference = [0.5, 0.5]
ratios = [math.log(p / q) for p, q in zip(policy, reference, strict=True)]
kl = sum(p * ratio for p, ratio in zip(policy, ratios, strict=True))
print("sample log-ratios:", [round(value, 3) for value in ratios])
print(f"expected KL={kl:.3f}")
assert ratios[1] < 0 < kl
Output
sample log-ratios: [0.47, -0.916]
expected KL=0.193

What PPO actually clips

The reference anchors the overall training run. A second policy snapshot, ฯ€old\pi_{\mathrm{old}}, generated the current rollout batch. PPO compares the changing actor to that snapshot, not to the SFT reference. At token tt, let sts_t be the prompt plus generated prefix and ata_t the sampled next token:

ฯt(ฮธ)=ฯ€ฮธ(atโˆฃst)ฯ€old(atโˆฃst),Lt=minโก(ฯtA^t,clipโก(ฯt,1โˆ’ฯต,1+ฯต)A^t).\rho_t(\theta)=\frac{\pi_\theta(a_t \mid s_t)}{\pi_{\mathrm{old}}(a_t \mid s_t)},\qquad L_t=\min\left(\rho_t \hat{A}_t,\operatorname{clip}(\rho_t,1-\epsilon,1+\epsilon)\hat{A}_t\right).

We maximize the batch average of LtL_t. A positive advantage A^t\hat{A}_t says the outcome beat the critic's expectation; a negative one says it fell short.

Clipping behaves asymmetrically depending on the sign of the advantage:

  • When A^t>0\hat{A}_t>0 and ฯต=0.2\epsilon=0.2, this sample's surrogate stops improving once the ratio exceeds 1.21.2. It still penalizes a ratio that moves downward in the wrong direction.
  • When A^t<0\hat{A}_t<0, reducing the ratio improves the surrogate only down to 0.80.8. Below that, its value is constant. Increasing the ratio remains harmful and isn't clipped away.

Check both signs across varying ratios:

ppo_clipped_surrogate.py
def surrogate(ratio: float, advantage: float, epsilon: float = 0.2) -> float:
    clipped = min(max(ratio, 1 - epsilon), 1 + epsilon)
    return min(ratio * advantage, clipped * advantage)

for advantage in [1.0, -1.0]:
    values = [surrogate(ratio, advantage) for ratio in [0.5, 1.0, 1.5]]
    print(f"A={advantage:+.0f}: {values}")
assert surrogate(1.5, 1.0) == 1.2
assert surrogate(0.5, -1.0) == -0.8
assert surrogate(1.5, -1.0) == -1.5
Output
A=+1: [0.5, 1.0, 1.2]
A=-1: [-0.8, -1.0, -1.5]
Two PPO clipped-surrogate curves share a labeled probability-ratio axis from 0.5 to 1.5 and a signed objective axis. With epsilon 0.2, positive advantage flattens above 1.2 and negative advantage flattens below 0.8. The clipped objective removes added surrogate gain while the policy ratio can still cross either boundary.
Solid curves show the clipped objective; dashed curves show the raw surrogate. The flat segment depends on the advantage sign, and neither boundary forces the ratio to stop.

Clipping removes an incentive in this surrogate. It doesn't forcibly clamp every actual token probability or guarantee a hard KL boundary. PPO can take several gradient passes over one batch, retaining its old-policy probabilities and advantage targets, before collecting fresh rollouts.

The systems cost of online RL

Estimate the allocations before choosing a layout. Suppose all four roles have independent 70B backbones, both trainable roles use BF16 weights and gradients, and Adam stores two FP32 moments plus an FP32 master copy. That's an assumed 16 bytes per trainable parameter and 2 per frozen parameter. The master copy and gradient precision are implementation choices; adapters or a smaller critic change this budget.

four_role_state_budget.py
parameters = 70_000_000_000
trainable_bytes = 2 + 2 + 4 + 4 + 4  # weights, gradients, moments, master
frozen_bytes = 2
actor_gb = parameters * trainable_bytes / 1e9
critic_gb = actor_gb
frozen_gb = parameters * frozen_bytes / 1e9
ppo_gb = actor_gb + critic_gb + 2 * frozen_gb
without_critic_gb = actor_gb + 2 * frozen_gb

print(f"each trainable backbone={actor_gb:.0f} GB")
print(f"each frozen backbone={frozen_gb:.0f} GB")
print(f"four-role state total={ppo_gb:.0f} GB")
print(f"remove critic: {without_critic_gb:.0f} GB, saved={critic_gb / ppo_gb:.1%}")
Alignment state budget
each trainable backbone=1120 GB
each frozen backbone=140 GB
four-role state total=2520 GB
remove critic: 1400 GB, saved=44.4%

These are decimal GB of aggregate parameter-related state, excluding activations, generation KV caches, temporary buffers, replication, and communication. They aren't measured peak GPU allocations or a universal PPO requirement. Sharding and offloading distribute or relocate state; they don't make the underlying bytes vanish. Whether the run needs multiple nodes depends on the recipe and available hardware.

Rollouts add work that offline DPO avoids. Prompt prefill and token-by-token decoding have different compute and memory demands; decoding can be bandwidth-bound at small batches, while larger rollout batches improve throughput until another limit takes over. A training run usually values throughput more than an interactive assistant's response latency.

Both colocated and separate generation/training layouts are valid. OpenRLHF documents a colocated hybrid engine that alternates generation and training, and an asynchronous mode that can overlap them. Overlap brings weight synchronization, possible stale-policy samples, and extra resource costs. Compare measured utilization and policy lag rather than assuming decoupling always wins.[6]

Cost or failureEvidence to inspectOptions to test
Parameter statePer-role sizes, optimizer/gradient dtypes, shared weightsSharding, offload, adapters, a smaller critic
Rollout timePrefill/decode times, batch utilization, policy lagBatch scheduling, KV-cache budget, colocated or asynchronous engines
Unstable updatesValue loss, advantages, ratio distribution, reference driftLearning rate, clipping, GAE, reward scaling; verify each change
Scorer exploitationControlled rewrites and independent judgmentsBetter labels/scorer coverage, shorter optimization, evaluated KL budgets

DPO: turn reward fitting into policy fitting

PPO-style RLHF is a multi-role online loop. If a fixed pair already tells you which response wins, why generate another response for every update? DPO targets those pairs with a classification-style loss. It removes engineering overhead, but it doesn't give you online exploration or guarantee the same checkpoint PPO would have found.

Why an offline loss can stand in for RL

Under a KL-regularized reward-maximization objective and a Bradley-Terry preference model, the optimal policy and the reward function are two mathematical views of the same object. DPO parameterizes an implicit reward with policy-to-reference log-probability ratios, so the pairwise loss never needs a standalone reward model or a PPO loop.[3]

Recall the closed-form analytical optimum derived earlier for a fixed prompt xx and frozen reference policy ฯ€ref\pi_{\text{ref}}:

ฯ€โˆ—(yโˆฃx)=1Z(x)ฯ€ref(yโˆฃx)expโก(r(x,y)ฮฒ)\pi^*(y \mid x) = \frac{1}{Z(x)} \pi_{\text{ref}}(y \mid x) \exp\left( \frac{r(x, y)}{\beta} \right)

Taking the natural logarithm of both sides isolates the reward:

logโกฯ€โˆ—(yโˆฃx)=logโกฯ€ref(yโˆฃx)+r(x,y)ฮฒโˆ’logโกZ(x)\log \pi^*(y \mid x) = \log \pi_{\text{ref}}(y \mid x) + \frac{r(x, y)}{\beta} - \log Z(x)

Rearranging expresses the fixed reward r(x,y)r(x,y) in terms of its optimal policy, the reference, and the partition function. The algebra also works for a proxy reward; evaluate that reward against ground-truth labels:

r(x,y)=ฮฒlogโกฯ€โˆ—(yโˆฃx)ฯ€ref(yโˆฃx)+ฮฒlogโกZ(x)r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)

Now, substitute this expression into the Bradley-Terry preference model. The probability that a human prefers ywy_w over yly_l depends strictly on the difference between their rewards:

P(ywโ‰ปylโˆฃx)=ฯƒ(r(x,yw)โˆ’r(x,yl))P(y_w \succ y_l \mid x) = \sigma\left( r(x, y_w) - r(x, y_l) \right)

Watch what happens when you subtract the two implicit reward terms:

r(x,yw)โˆ’r(x,yl)=[ฮฒlogโกฯ€โˆ—(ywโˆฃx)ฯ€ref(ywโˆฃx)+ฮฒlogโกZ(x)]โˆ’[ฮฒlogโกฯ€โˆ—(ylโˆฃx)ฯ€ref(ylโˆฃx)+ฮฒlogโกZ(x)]r(x, y_w) - r(x, y_l) = \left[ \beta \log \frac{\pi^*(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} + \beta \log Z(x) \right] - \left[ \beta \log \frac{\pi^*(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} + \beta \log Z(x) \right]

The partition function Z(x)=โˆ‘yฯ€ref(yโˆฃx)expโก(r(x,y)/ฮฒ)Z(x) = \sum_y \pi_{\text{ref}}(y \mid x) \exp(r(x, y)/\beta) depends solely on the prompt xx, not the response. It cancels out completely:

r(x,yw)โˆ’r(x,yl)=ฮฒlogโกฯ€โˆ—(ywโˆฃx)ฯ€ref(ywโˆฃx)โˆ’ฮฒlogโกฯ€โˆ—(ylโˆฃx)ฯ€ref(ylโˆฃx)r(x, y_w) - r(x, y_l) = \beta \log \frac{\pi^*(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi^*(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}

This cancellation is the mathematical foundation of DPO.[3] By substituting the parameterized policy ฯ€ฮธ\pi_\theta for the optimal policy ฯ€โˆ—\pi^*, we define the implicit reward:

r^ฮธ(x,y)=ฮฒlogโกฯ€ฮธ(yโˆฃx)ฯ€ref(yโˆฃx)\hat{r}_\theta(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}

We can now train the language model directly on preference pairs by minimizing the negative log-likelihood of the observed preference labels. This isn't the SFT loss โˆ’logโกฯ€ฮธ(ywโˆฃx)-\log\pi_\theta(y_w|x): the sigmoid models which answer wins, rather than the probability of generating the chosen text.

LDPO(ฮธ)=โˆ’E(x,yw,yl)โˆผD[logโกฯƒ(ฮฒlogโกฯ€ฮธ(ywโˆฃx)ฯ€ref(ywโˆฃx)โˆ’ฮฒlogโกฯ€ฮธ(ylโˆฃx)ฯ€ref(ylโˆฃx))]\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]

What exactly is equivalent to RLHF here? The change of variables connects a reward function to its exact KL-regularized optimal distribution. Recovering the same optimum from preference learning additionally needs a suitable preference model, enough comparisons to identify relevant reward differences, a policy class that can represent the solution, and successful optimization. Under the Bradley-Terry population assumptions, the objectives can share an optimum.[3][7] A finite DPO run and a clipped PPO run needn't produce the same policy. Neither the algebra nor a low training loss determines behavior on answers absent from the comparisons.

Gradient dynamics and implicit sample weighting

Taking the gradient of LDPO\mathcal{L}_{\text{DPO}} with respect to the policy parameters ฮธ\theta reveals how DPO steers the model:

โˆ‡ฮธLDPO(ฮธ)=โˆ’ฮฒE(x,yw,yl)[ฯƒ(r^ฮธ(x,yl)โˆ’r^ฮธ(x,yw))โŸDynamicย Sampleย Weightโ‹…(โˆ‡ฮธlogโกฯ€ฮธ(ywโˆฃx)โˆ’โˆ‡ฮธlogโกฯ€ฮธ(ylโˆฃx))โŸPush-Pullย Direction]\nabla_\theta \mathcal{L}_{\text{DPO}}(\theta) = -\beta \mathbb{E}_{(x, y_w, y_l)} \left[ \underbrace{\sigma\left(\hat{r}_\theta(x, y_l) - \hat{r}_\theta(x, y_w)\right)}_{\text{Dynamic Sample Weight}} \cdot \underbrace{\left( \nabla_\theta \log \pi_\theta(y_w \mid x) - \nabla_\theta \log \pi_\theta(y_l \mid x) \right)}_{\text{Push-Pull Direction}} \right]

This gradient contains two intuitive mechanisms:

  1. Contrast direction: Gradient descent follows โˆ‡ฮธlogโกฯ€ฮธ(ywโˆฃx)โˆ’โˆ‡ฮธlogโกฯ€ฮธ(ylโˆฃx)\nabla_\theta\log\pi_\theta(y_w|x)-\nabla_\theta\log\pi_\theta(y_l|x). The two terms favor the chosen and oppose the rejected completion, but shared parameters can make both absolute probabilities fall. Each token's probability change depends on its shared parameter gradients and the optimizer update.
  2. Pair weight: ฯƒ(r^lโˆ’r^w)\sigma(\hat r_l-\hat r_w) lies above 0.50.5 when the implicit reward ordering is wrong and approaches 1 for a large negative chosen margin. It approaches zero for a large positive margin. This scales the pair's gradient; it isn't the optimizer's learning rate or a guarantee against overfitting.

The role of ฮฒ\beta deserves care. For a fixed reward, a larger ฮฒ\beta makes the analytical optimum stay closer to the reference. In the empirical DPO loss, it also scales the logit and the gradient. At initialization, โˆ‚L/โˆ‚m=โˆ’ฮฒ/2\partial L/\partial m=-\beta/2 for unscaled margin mm. A larger ฮฒ\beta therefore produces a larger initial margin gradient, even though it represents stronger KL regularization in the reward-to-policy derivation. Learning rate, number of updates, and saturation affect the actual drift.

For a complete two-answer policy with a uniform reference, suppose the true probability of preferring A is q=0.75q=0.75. Matching that probability requires ฮฒm=logโก(q/(1โˆ’q))=logโก3\beta m=\log(q/(1-q))=\log3, so ฯ€โˆ—(A)=ฯƒ(logโก3/ฮฒ)\pi^*(A)=\sigma(\log3/\beta). These are population optima, not measurements from training:

ฮฒ\betaInitial โˆ‚L/โˆ‚m\partial L/\partial mOptimal margin mmOptimal ฯ€(A)\pi(A)
0.1โˆ’0.05-0.0510.98699.998%
0.5โˆ’0.25-0.252.19790.0%
1.0โˆ’0.50-0.501.09975.0%

Larger ฮฒ\beta moves the ideal policy closer to the 50% reference here. It doesn't turn empirical DPO into an explicit hard KL constraint.[3][7]

Work the loss on the P-7 pair

Apply the loss to the access-policy prompt. ywy_w is chosen (ticket and cite P-7) and yly_l is rejected (grant admin tonight). Read each term as change relative to the frozen reference: raise the chosen response relative to that baseline, lower the rejected response, and the margin grows.

Suppose the current sequence log-probabilities are:

  • Chosen A: policy โˆ’8.2-8.2, reference โˆ’8.5-8.5
  • Rejected B: policy โˆ’9.5-9.5, reference โˆ’9.1-9.1
  • ฮฒ=0.1\beta = 0.1

Then:

  1. Chosen relative advantage: โˆ’8.2โˆ’(โˆ’8.5)=+0.3-8.2 - (-8.5) = +0.3
  2. Rejected relative advantage: โˆ’9.5โˆ’(โˆ’9.1)=โˆ’0.4-9.5 - (-9.1) = -0.4
  3. Margin: 0.3โˆ’(โˆ’0.4)=+0.70.3 - (-0.4) = +0.7
  4. Scaled : 0.1ร—0.7=0.070.1 \times 0.7 = 0.07
  5. ฯƒ(0.07)โ‰ˆ0.517\sigma(0.07) \approx 0.517
  6. Loss: โˆ’logโก(0.517)โ‰ˆ0.659-\log(0.517) \approx 0.659

The loss rewards growing that 0.70.7 margin. Its weight depends on the pair's reference-relative ordering, not on a single global distance between models. Shared model parameters couple the chosen and rejected probabilities, so their absolute changes need separate monitoring.

dpo_margin_from_logprobs.py
from math import exp, log

chosen_policy, chosen_reference = -8.2, -8.5
rejected_policy, rejected_reference = -9.5, -9.1
beta = 0.1
margin = (chosen_policy - chosen_reference) - (rejected_policy - rejected_reference)
logit = beta * margin
loss = -log(1 / (1 + exp(-logit)))

print(f"relative_margin={margin:.1f}")
print(f"scaled_logit={logit:.2f}")
print(f"dpo_loss={loss:.3f}")
DPO margin output
relative_margin=0.7
scaled_logit=0.07
dpo_loss=0.659

Those โˆ’8.2-8.2 figures are already summed response log-probabilities. For response tokens y1,โ€ฆ,yTy_1,\ldots,y_T, compute logโกฯ€(yโˆฃx)=โˆ‘tlogโกฯ€(ytโˆฃx,y<t)\log\pi(y|x)=\sum_t\log\pi(y_t|x,y_{<t}). Teacher forcing supplies each observed prefix; it doesn't sample new text. The log- must include the whole vocabulary even though we gather only the observed token.

The toy below makes the causal shift and masks explicit. Token IDs 0, 2 form the prompt, the next two IDs form the response, and 4 is end-of-sequence. A final 0 is padding; another 0 is real response content on the rejected branch, so masking by token ID alone would be wrong. Logit row t-1 predicts input token t. The arbitrary scores don't come from a trained model.

dpo_loss_from_token_logits.py
import math

CHOSEN_IDS = [0, 2, 1, 2, 4, 0]
REJECTED_IDS = [0, 2, 3, 0, 4, 0]
ATTENTION_MASK = [1, 1, 1, 1, 1, 0]
PROMPT_LENGTH = 2
BETA = 0.1

chosen_policy = [
    [0.0] * 5,  # predicts a prompt token: excluded from the loss
    [0.0, 2.4, 0.1, -0.8, -1.0],
    [0.0, 0.2, 2.1, -0.6, -1.0],
    [0.0, 0.0, 0.0, 0.0, 2.0],  # predicts EOS: included
    [0.0] * 5,  # predicts padding: excluded
    [0.0] * 5,
]
chosen_ref = [
    [0.0] * 5,
    [0.0, 0.8, 0.2, 0.0, -1.0],
    [0.0, 0.1, 0.9, 0.0, -1.0],
    [0.0, 0.0, 0.0, 0.0, 2.0],
    [0.0] * 5,
    [0.0] * 5,
]
rejected_policy = [
    [0.0] * 5,
    [0.2, -0.4, -0.2, 1.1, -1.0],
    [1.0, -0.3, -0.4, 0.2, -1.0],
    [0.0, 0.0, 0.0, 0.0, 2.0],
    [0.0] * 5,
    [0.0] * 5,
]
rejected_ref = [
    [0.0] * 5,
    [0.1, 0.0, 0.0, 0.6, -1.0],
    [0.7, 0.0, 0.0, 0.2, -1.0],
    [0.0, 0.0, 0.0, 0.0, 2.0],
    [0.0] * 5,
    [0.0] * 5,
]

def log_softmax(logits: list[float]) -> list[float]:
    peak = max(logits)
    log_normalizer = peak + math.log(sum(math.exp(value - peak) for value in logits))
    return [value - log_normalizer for value in logits]

def sequence_logprob(steps: list[list[float]], token_ids: list[int]) -> float:
    if len(steps) != len(token_ids) or len(token_ids) != len(ATTENTION_MASK):
        raise ValueError("logits, tokens, and mask must have matching lengths")
    return sum(log_softmax(steps[t - 1])[token_ids[t]]
               for t in range(PROMPT_LENGTH, len(token_ids)) if ATTENTION_MASK[t])

def dpo_loss(
    chosen_pi: list[list[float]],
    rejected_pi: list[list[float]],
) -> float:
    margin = (
        sequence_logprob(chosen_pi, CHOSEN_IDS)
        - sequence_logprob(chosen_ref, CHOSEN_IDS)
        - (
            sequence_logprob(rejected_pi, REJECTED_IDS)
            - sequence_logprob(rejected_ref, REJECTED_IDS)
        )
    )
    logit = BETA * margin
    return max(0.0, -logit) + math.log1p(math.exp(-abs(logit)))

loss = dpo_loss(chosen_policy, rejected_policy)
boosted_chosen = [row[:] for row in chosen_policy]
boosted_chosen[1][1] += 1.0
boosted_chosen[2][2] += 1.0
boosted_loss = dpo_loss(boosted_chosen, rejected_policy)
masked_changes = [row[:] for row in chosen_policy]
masked_changes[0][2] = 1000.0
masked_changes[4][0] = 1000.0

print(f"chosen_logp={sequence_logprob(chosen_policy, CHOSEN_IDS):.3f}")
print(f"rejected_logp={sequence_logprob(rejected_policy, REJECTED_IDS):.3f}")
print(f"dpo_loss={loss:.3f}")
print(f"after_boost_loss={boosted_loss:.3f}")
assert math.isfinite(loss) and boosted_loss < loss
assert sequence_logprob(masked_changes, CHOSEN_IDS) == sequence_logprob(chosen_policy, CHOSEN_IDS)
assert math.isfinite(log_softmax([1000.0, -1000.0])[1])
print("prompt and padding predictions excluded:", True)
DPO token logprob output
chosen_logp=-0.993
rejected_logp=-1.882
dpo_loss=0.662
after_boost_loss=0.646
prompt and padding predictions excluded: True

At initialization, identical policy and reference probabilities give a DPO logit of zero and loss logโก2โ‰ˆ0.693\log 2\approx0.693. Token logits themselves need not be zero. Dropout, mismatched tokenization, or different masks can break the identity before the first update. Keep the reference frozen, compute its scores without gradients, and use the same tokenizer, chat template, response boundaries, and EOS convention on both branches.

Whole-response sums can create length effects when chosen and rejected answers differ systematically. Dividing by token count changes the objective; it isn't a harmless numerical optimization. To see why, consider two illustrative completions whose counted lengths include the end token:

CompletionCounted tokensPolicy log-probabilityReference log-probabilitySummed log-ratioAverage log-ratio
Chosen A4โˆ’3.60-3.60โˆ’4.00-4.00+0.40+0.40+0.10+0.10
Rejected B2โˆ’1.70-1.70โˆ’2.00-2.00+0.30+0.30+0.15+0.15

Standard DPO sees margin 0.40โˆ’0.30=+0.100.40-0.30=+0.10. Averaging each completion first gives 0.10โˆ’0.15=โˆ’0.050.10-0.15=-0.05, reversing the implicit preference ordering. Averaging the final loss across dataset rows is a separate operation and doesn't change each row's sequence score. Track completion lengths and verify truncation hasn't removed the behavior that explains the label.[3][8]

A larger DPO margin can coexist with lower chosen probability

Look closely at the DPO gradient direction: it optimizes the difference between chosen and rejected log-probability changes, not their absolute values. That leads directly to a deceptive failure mode. Did the policy make the chosen answer more likely, or did it merely crush the rejected answer?

Coupled parameters and the model's embedding geometry can make rejected-answer suppression reduce chosen-answer probability too. If rejected log-probability drops by 1.71.7 and chosen by 0.90.9, the margin grows by 0.80.8 despite both completions losing probability. Whether chosen probability falls below the SFT reference depends on where it started.

Pal et al. proposed DPO-Positive (DPOP), adding a penalty when chosen likelihood falls below the reference.[9] Razin et al. analyzed likelihood displacement and showed that it can move mass to either acceptable alternatives or worse responses, including unsafe compliance in their experiments.[10] The pair's margin alone doesn't reveal which happened; repetitive text or hallucinations aren't automatic consequences of the arithmetic.

Keep the P-7 pair's frozen reference at โˆ’8.5-8.5 (chosen) and โˆ’9.1-9.1 (rejected). In an illustrative before/after comparison, chosen log-probability falls from โˆ’8.2-8.2 to โˆ’9.1-9.1 while rejected falls from โˆ’9.5-9.5 to โˆ’11.2-11.2. The relative margin grows from 0.70.7 to 1.51.5 even though both labeled completions lose probability. Mass moves to other responses, whose quality the pair doesn't tell us. Plotting logโก(ฯ€/ฯ€ref)\log(\pi/\pi_{\text{ref}}) exposes the change: chosen starts at +0.3+0.3 and finishes at โˆ’0.6-0.6.

Four authored policy states share one log-probability-ratio axis. The chosen response moves from 0 to 0.30, โˆ’0.15, and โˆ’0.60; the rejected response moves from 0 to โˆ’0.40, โˆ’1.20, and โˆ’2.10. Their pairwise margin grows from 0 to 0.70, 1.05, and 1.50 even as the chosen response falls below its reference likelihood. These are illustrative states, not a measured training trajectory.
The chosen log-ratio falls below the reference while the pair margin grows. These authored intermediate states are not a measured training trajectory.
dpo_likelihood_displacement.py
ref_w, ref_l = -8.5, -9.1
steps = [
    ("before", -8.2, -9.5),
    ("after", -9.1, -11.2),
]

for name, pi_w, pi_l in steps:
    margin = (pi_w - ref_w) - (pi_l - ref_l)
    print(f"{name}: chosen={pi_w:.1f} rejected={pi_l:.1f} margin={margin:.1f}")
Displacement margin output
before: chosen=-8.2 rejected=-9.5 margin=0.7
after: chosen=-9.1 rejected=-11.2 margin=1.5

Monitor at least four series:

SignalWhat it catches
logโกฯ€ฮธ(ywโˆฃx)\log \pi_\theta(y_w \mid x)Whether preferred answers stay likely
logโกฯ€ฮธ(ylโˆฃx)\log \pi_\theta(y_l \mid x)Whether rejected answers are suppressed
Relative margin mm and logit ฮฒm\beta mWhether pairwise ranking is improving
On-policy KL estimate plus labeled-pair log-ratiosDistribution drift versus movement on these particular rows

If margins improve while chosen log-prob collapses, inspect data, ฮฒ\beta, length effects, and held-out generations before extending the run. A decrease in one exact answer's probability can reflect probability mass shifting to other good answers. Compare length-matched slices. Evaluate a length-aware objective such as SimPO on held-out generations to see whether it addresses the length effects.[11]

Vanilla DPO needs policy and reference scores, not a separate reward or value model and not fresh rollouts during optimization. A frozen reference's scores can be precomputed for fixed token sequences, so two logical roles don't require two resident GPU models. TRL 1.17.1, rechecked November 4, 2026, exposes precompute_ref_log_probs and rejects combining it with sync_ref_model=True. Its IPO loss also uses per-completion average reference-relative scores, whereas standard sigmoid DPO sums token log-probabilities. Check the exact loss and token masks before comparing margins or reusing caches.[12][13]

A reference cache belongs to an exact scoring recipe: reference weights, tokenizer and template, token IDs, completion masks, truncation, and sum/mean convention. Recompute affected scores when any of those change. Validate a cache against the full scoring recipe, not only the dataset row's text.

To connect that arithmetic to the library, prepare two map-style Hugging Face datasets, train_pairs and eval_pairs, with prompt, chosen, and rejected fields. For conversational rows, use role/content messages and the SFT tokenizer's chat template. Set sft_checkpoint to a full model checkpoint directory, rather than an adapter-only artifact. This source-checked TRL 1.13.0 fragment is a starting configuration; it hasn't been run as transformer training here:

trl_dpo_setup_fragment.py
from transformers import AutoTokenizer
from trl import DPOConfig, DPOTrainer

tokenizer = AutoTokenizer.from_pretrained(sft_checkpoint)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

trainer = DPOTrainer(
    model=sft_checkpoint,
    processing_class=tokenizer,
    args=DPOConfig(
        output_dir="runs/p7-dpo",
        beta=0.1,
        precompute_ref_log_probs=True,
        max_length=2048,
        per_device_train_batch_size=1,
        learning_rate=5e-6,
        max_steps=10,
    ),
    train_dataset=train_pairs,
    eval_dataset=eval_pairs,
)
trainer.train()
pair_metrics = trainer.evaluate()
trainer.save_model("artifacts/p7-dpo")
tokenizer.save_pretrained("artifacts/p7-dpo")

Ten updates are a wiring exercise, not a quality target. Choose precision and batch settings for the model and device. Count rows before and after preparation, and inspect collated pairs before training. As of TRL 1.17, the default keep_start truncation cuts each sequence at max_length, which can erase the chosen/rejected distinction, and the trainer drops any row whose prompt alone fills max_length before collation, without an error. On a path that skips that step, such as a custom collator, require at least one scored next-token target per branch after the causal shift, and check that the label-defining difference survives. A nonempty input tensor can still contain zero scored completion targets. The trainer's reference cache is constructed before updates. Evaluate fresh generations separately; trainer.evaluate() reports losses and metrics on stored pairs, not a human win rate.[13][14]

For this full-model setup with precomputation, supply every planned pair-evaluation dataset at initialization. Without a separate ref_model, the trainer doesn't retain the frozen reference after updates. Passing a new uncached dataset later can raise Cannot compute reference log-probs for a dataset passed to `evaluate()` after training has started because its reference log-probabilities were not cached.[8]

Comparing RLHF and DPO

The useful contrast is what each method has to keep alive, not a winner-take-all ranking.[1][3] Read the table as a deployment question: which signal must stay live, and which behavior can your stored data never show you?

PPO draws a fresh response from the current actor, scores it with reward, reference, and critic roles, then updates the actor for a later rollout. Vanilla DPO keeps one stored chosen/rejected pair, scores the same responses under the updated policy and a frozen reference cache, computes pairwise loss, and updates the policy to rescore that stored pair. The DPO path does not sample new responses.
Follow the return paths: PPO updates the next rollout, while vanilla DPO rescales the same stored pair. Cached reference scores stay fixed.
FeatureRLHF (PPO)DPO
Reward modelTrain a separate reward modelNo separate RM; reward is implicit
Training loopOn-policy RL with rolloutsOffline pairwise loss on fixed comparisons
Logical rolesPolicy + ref + reward + criticPolicy + reference
Main cost driversSampling, scoring, and policy/value updatesForward/backward on stored pairs
StabilitySensitive to reward scale, KL, and PPO settingsFewer loops, still sensitive to data and beta
Online explorationYesNot in vanilla DPO
Data dependenceCan collect fresh comparisons during trainingLimited by current pair coverage unless you refresh
MonitoringReward drift, KL, value loss, rollout qualityPreference loss, chosen log-prob, margin, length

DPO is a strong candidate baseline for offline pairwise preferences. PPO isn't obsolete. If you need online exploration, fresh model-generated negatives, or a reward that moves as the policy moves, an RL-style loop can reach cases a frozen DPO dataset can't.

The data both methods inherit

RLHF and DPO consume the same kind of row, then spend it differently. RLHF fits a reward model on the comparisons, then rolls out. DPO trains the policy on the pairs themselves.[1][3] The row is shared; the feedback path is not.

For the strict chosen/rejected loss here, use the same rendered prompt, record generator , and handle ties or disagreement deliberately. Standard DPO treats the selected ordering as a binary observation; repeated opposite votes can represent uncertainty rather than mislabeled data. Group prompts when measuring generalization to unseen prompts. A known-prompt/new-response evaluation asks a different question and still must keep evaluated responses and labels out of training. Samples from another generator aren't automatically invalid, but can miss current-policy failures.

You don't need a huge dump of easy instructions. You need coverage of the distinctions you care about: refusals, tool use, reasoning, style, and the P-7-style policy calls. Ask what happens when the current policy invents a failure absent from the stored pairs: DPO has no training row for it until you refresh the data, while PPO can expose it only after a fresh rollout reaches its judge. If annotators keep disagreeing, more labels can quantify uncertainty. They won't turn a vague rubric into a training signal.

Here is a precise coverage failure. Restrict the P-7 assistant to three possible responses: A escalates, B grants admin directly, and an unseen C grants admin while citing a nonexistent exception. Suppose the dataset compares only A against B. Two policies can assign the same A-to-B odds while allocating very different mass to C:

dpo_unseen_response_mass.py
from math import log, log1p, exp

reference = {"A": 1 / 3, "B": 1 / 3, "C": 1 / 3}
policies = [
    {"A": 0.40, "B": 0.10, "C": 0.50},
    {"A": 0.08, "B": 0.02, "C": 0.90},
]
losses = []
for policy in policies:
    assert abs(sum(policy.values()) - 1) < 1e-12
    margin = log(policy["A"] / reference["A"]) - log(policy["B"] / reference["B"])
    logit = 0.1 * margin
    loss = max(0.0, -logit) + log1p(exp(-abs(logit)))
    losses.append(loss)
    print(f"A/B odds={policy['A'] / policy['B']:.1f} DPO loss={loss:.3f} unseen C mass={policy['C']:.0%}")
assert abs(losses[0] - losses[1]) < 1e-12
Unseen response coverage
A/B odds=4.0 DPO loss=0.626 unseen C mass=50%
A/B odds=4.0 DPO loss=0.626 unseen C mass=90%

Both margins are logโก4\log4, so the observed pair loss is identical even when the unseen violation receives 90% probability. This comparison isolates what the observed objective leaves unidentified. Fresh rollout evaluation can discover C, and new A-versus-C labels can constrain its ranking. An online reward model also needs coverage of that fabricated exception before its score is useful.[3][7]

A probability simplex contains a uniform reference and two policies. Policy 1 assigns probabilities 0.40, 0.10, and 0.50 to A, B, and unseen C; policy 2 assigns 0.08, 0.02, and 0.90. Both keep A-to-B odds at 4 to 1, so the DPO pair loss remains 0.62623 at beta 0.1 while C mass rises from 50 to 90 percent. The normalized response probabilities are an authored analytical example.
Both policies keep the observed A/B odds at 4:1. The pair loss follows that comparison while Cโ€™s probability rises from 50% to 90%.

When the proxy lies

That release rehearsal raises a reward-hacking suspicion, not a diagnosis from three numbers. Reward hacking means the policy exploits the scorer instead of improving the intended behavior. Inspect the longer answers and score controlled rewrites: does adding "Policy-safe answer:" raise the reward without fixing a policy error? That comparison can expose what the scorer rewards. The reward model is a proxy, so optimizing its score still needs independent checks.

If the scorer overweights length or markdown, policy optimization can amplify those traits without improving answer quality. Rising proxy reward without matching independent preference gains is a warning; controlled comparisons help distinguish a shortcut from useful extra detail.

HackSymptomDetection
Length gamingAnswers get steadily longerTrack mean response length
SycophancyModel agrees with a wrong userTest with false user claims
Format exploitationEverything wrapped in listsCompare formatted vs plain scores
Confidence mimicrySounds sure on hard items checks on known-hard questions

Verbosity

Annotators sometimes treat length as quality. Predict what happens next: a padded P-7 explanation can beat a short, correct one, so the policy learns to pad if that correlation is in the data.

  • Symptom: mean length climbs while helpfulness plateaus.
  • Fix: length-matched preference evals, and explicit concise-versus-padded labels. Length penalties can also punish necessary detail, so validate them.

The 0.693 trap

If the policy starts equal to its reference, every DPO relative margin is zero and the loss starts at โˆ’logโก(0.5)โ‰ˆ0.693-\log(0.5)\approx0.693. That's expected. Let mm be the unscaled reference-relative margin and z=ฮฒmz=\beta m. For one pair,

โˆ‚Lโˆ‚z=ฯƒ(z)โˆ’1,โˆ‚Lโˆ‚m=ฮฒ(ฯƒ(z)โˆ’1).\frac{\partial L}{\partial z}=\sigma(z)-1,\qquad\frac{\partial L}{\partial m}=\beta(\sigma(z)-1).

At zero logit the first derivative is โˆ’0.5-0.5, but at ฮฒ=0.1\beta=0.1 the second is โˆ’0.05-0.05. The model-parameter gradient is that second factor times โˆ‡ฮธlogโกฯ€ฮธ(ywโˆฃx)โˆ’โˆ‡ฮธlogโกฯ€ฮธ(ylโˆฃx)\nabla_\theta\log\pi_\theta(y_w|x)-\nabla_\theta\log\pi_\theta(y_l|x). Its size depends on the model too. The loss isn't flat at initialization, but -0.5 isn't the gradient of every parameter.[3]

Initial loss alone says little about pair strength. Ambiguous pairs still hurt. Conflicting labels on the same prompt can cancel in aggregate or teach arbitrary style. Diagnose with agreement, slices, and held-out behavior.

dpo_zero_margin_is_not_zero_gradient.py
from math import exp, log1p

def sigmoid(value: float) -> float:
    return 1 / (1 + exp(-value))

def loss_at_margin(margin: float, beta: float) -> float:
    logit = beta * margin
    return max(0.0, -logit) + log1p(exp(-abs(logit)))

beta = 0.1
for margin in [0.0, 20.0]:
    logit = beta * margin
    d_logit = sigmoid(logit) - 1
    d_margin = beta * d_logit
    h = 1e-5
    finite_difference = (loss_at_margin(margin + h, beta) - loss_at_margin(margin - h, beta)) / (2 * h)
    assert abs(finite_difference - d_margin) < 1e-9
    print(f"logit={logit:.1f} loss={loss_at_margin(margin, beta):.3f} dL/dz={d_logit:.3f} dL/dm={d_margin:.3f}")

# A two-response softmax policy with logits [theta, 0] and uniform reference.
# Its reference-relative log-odds margin is exactly theta.
theta = 0.0
for _ in range(20):
    theta -= beta * (sigmoid(beta * theta) - 1)  # learning rate 1
print(f"toy P(chosen)={sigmoid(theta):.3f} loss={loss_at_margin(theta, beta):.3f}")
assert sigmoid(theta) > 0.5 and loss_at_margin(theta, beta) < loss_at_margin(0.0, beta)
DPO zero-margin output
logit=0.0 loss=0.693 dL/dz=-0.500 dL/dm=-0.050
logit=2.0 loss=0.127 dL/dz=-0.119 dL/dm=-0.012
toy P(chosen)=0.726 loss=0.646
  • Symptom: loss stays near 0.6930.693 after real updates, and held-out preference doesn't move.
  • Fix: confirm the policy is updating, then inspect contradictory labels, unresolved ties, prompt , and missing coverage before blaming beta.

A release gate that actually uses humans

The scalar toy took real gradient steps, but it didn't train a language model. Full-model validation must also exercise tokenization, reference freezing, optimizer state, precision, memory, and held-out generation. GPU throughput and PPO rollout stability aren't established by these CPU examples.

Release asks another question: did behavior improve within the tested drift budget? Tie proxy improvement to behavior you care about:

  1. Hold out human evaluation, blind response order, and specify the rubric and decision rule before looking at results.
  2. Track auxiliaries: length, refusal rate, formatting drift, calibration.
  3. Watch KL and stop when you leave the tested drift budget.
  4. Refresh the scorer when current outputs leave its training distribution.

Return to the illustrative 184 wins in 400 comparisons. Assume independent held-out prompts, one judgment per prompt, no ties, and one planned evaluation. A 95% Wilson interval for the candidate's win probability accounts for sampling uncertainty.[15] Repeated checkpoints, shared users, or ties require a matching analysis rather than pretending every judgment is an independent Bernoulli trial.

For this toy gate, require the interval's lower bound to exceed 50% and mean length growth to stay below 20 tokens. That length threshold is an invented product constraint, not an alignment standard. Predict the result: higher proxy reward won't authorize promotion.

alignment_release_gate.py
from math import sqrt

def wilson_interval(wins: int, total: int, z: float = 1.96) -> tuple[float, float]:
    if total <= 0 or not 0 <= wins <= total:
        raise ValueError("need positive total and 0 <= wins <= total")
    p = wins / total
    denominator = 1 + z * z / total
    center = (p + z * z / (2 * total)) / denominator
    radius = z * sqrt(p * (1 - p) / total + z * z / (4 * total * total)) / denominator
    return center - radius, center + radius

wins, total = 184, 400
lower, upper = wilson_interval(wins, total)
proxy_reward_delta = 0.31
mean_length_delta_tokens = 48
ready = lower > 0.50 and mean_length_delta_tokens < 20

print(f"proxy_improved={proxy_reward_delta > 0}")
print(f"candidate_win_rate={wins / total:.1%}")
print(f"95% Wilson interval=[{lower:.1%}, {upper:.1%}]")
print(f"evidence_of_improvement={lower > 0.50}")
print(f"release_ready={ready}")
Alignment release gate output
proxy_improved=True
candidate_win_rate=46.0%
95% Wilson interval=[41.2%, 50.9%]
evidence_of_improvement=False
release_ready=False
A release example records 184 wins in 400 blinded comparisons, a 46.0 percent rate with a 95 percent Wilson interval from 41.2 to 50.9 percent that crosses the 50 percent baseline. Proxy reward rises by 0.31, while mean response length grows 48 tokens against a 20-token limit. The lower-confidence win gate and length gate both fail, so promotion is held; the interval establishes neither improvement nor degradation.
The Wilson interval crosses 50%, and the mean-length limit also fails. Hold promotion while the evidence establishes neither a gain nor a clear drop.

Other failure signatures worth the same stop rule:

  • DPO loss stays near 0.6930.693 after confirmed updates: audit labels, ties, leakage, and coverage rather than treating init loss as a diagnosis.
  • DPO improves tone but still grants admin: inspect whether labels cover the P-7 rule, then check the deployed prompt/template and fresh generations. Add missing policy contrasts when the data audit supports them.
  • PPO generations get repetitive while reward rises: freeze promotion, inspect reward scale and KL, then refresh evaluation together.

Online DPO and iterative alignment

Vanilla DPO is offline: it trains on a fixed set of chosen and rejected responses.[3] Ask what happens when the policy improves past those rows: old comparisons describe a model you no longer have, so they may no longer contain hard negatives.

Later work studies the online setting: sample fresh responses from the current policy, label them with humans or a preference model, then apply an IPO-style or related pairwise update on the new comparisons.[16]

The loop is:

  1. Sample two or more candidates from the current policy.
  2. Ask humans or a preference model which is better.
  3. Store a new (prompt, chosen, rejected) triple.
  4. Run a DPO-style update with whatever reference-policy rule the method uses.
  5. Repeat so the data follows the model.

There isn't one canonical "online DPO" algorithm, and sampling/reference rules differ. Calandriello et al. study current-policy online IPO and IPO-MD, which samples from a mixture with the reference, alongside online DPO variants.[16] Refreshing samples can reduce mismatch and expose harder negatives without a critic. A stale or biased judge can still mislabel the new rows.

These methods make different data contracts. Start with the signal you can reliably collect: a chosen/rejected pair, an unpaired binary label, or an automated task verifier.

MethodObjective & MechanismKey Systems Trade-Off
IPO (Identity Preference Optimization)Derived from identity-transformed preferences. Squared loss targets reference-relative margin 1/(2ฯ„)1/(2\tau) for regularization parameter ฯ„>0\tau>0.[7]Penalizes overshooting that finite target; no universal stability or safety guarantee.
KTO (Kahneman-Tversky Optimization)Uses individual desirable/undesirable labels, with opposite sigmoid directions around a reference point.[17]No paired alternatives required; feedback semantics, selection bias, and class weighting still need review.
ORPO (Odds Ratio Preference Optimization)Combines SFT cross-entropy with an odds-ratio preference penalty into a single training stage, dropping the reference model entirely.[18]Saves reference model memory, but couples instruction-following and preference updates.
SimPO (Simple Preference Optimization)Uses scaled average token log-probability as reward, with a target preference margin and no reference.[11]Changes length dependence; evaluate concision and correctness on target prompts.
GRPO (Group Relative Policy Optimization)Online group-relative RL without a learned critic. Outcome supervision centers/scales each completion's reward within its prompt group.[19][20]Removes critic state and updates; actual memory savings depend on role sizes and the execution layout.

Three short calculations show how these data contracts alter the update.

IPO targets a finite margin

Let m=logโก(ฯ€ฮธ(ywโˆฃx)/ฯ€ref(ywโˆฃx))โˆ’logโก(ฯ€ฮธ(ylโˆฃx)/ฯ€ref(ylโˆฃx))m=\log(\pi_\theta(y_w|x)/\pi_{\mathrm{ref}}(y_w|x))-\log(\pi_\theta(y_l|x)/\pi_{\mathrm{ref}}(y_l|x)). Vanilla DPO's loss โˆ’logโกฯƒ(ฮฒm)-\log\sigma(\beta m) keeps decreasing as mm grows. On separable comparisons it can favor ever larger margins. That can concentrate the policy on a narrow set of answers; it does not imply that forward KL must diverge.

IPO instead derives an identity-preference objective whose sampled squared loss is:[7]

LIPO=ED[(mโˆ’12ฯ„)2],ฯ„>0.\mathcal L_{\mathrm{IPO}}=\mathbb E_{\mathcal D} \left[\left(m-\frac{1}{2\tau}\right)^2\right],\qquad \tau>0.

The target is 1/(2ฯ„)1/(2\tau), not ฯ„/2\tau/2. At ฯ„=0.1\tau=0.1, it is 5. The derivative 2(mโˆ’5)2(m-5) favors increasing a smaller margin and decreasing an overshoot. It doesn't clip margins, bound all model parameters, or guarantee stable neural-network optimization. TRL's beta represents ฯ„\tau for its IPO loss, with the averaging convention described above.[8]

Compare both losses on a complete two-response policy. With logits [ฮธ,0][\theta,0] and a uniform reference, m=ฮธm=\theta. Forward KL is at most logโก2\log 2, even as the margin tends to infinity.

dpo_ipo_margin_and_kl.py
from math import exp, log, log1p

beta = tau = 0.1
target = 1 / (2 * tau)
print(f"IPO target={target:.1f}")
for margin in [0.0, 5.0, 10.0]:
    p = 1 / (1 + exp(-margin))
    kl = p * log(2 * p) + (1 - p) * log(2 * (1 - p))
    dpo = log1p(exp(-beta * margin))
    ipo = (margin - target) ** 2
    print(f"m={margin:.0f}: DPO={dpo:.3f} IPO={ipo:.1f} KL={kl:.3f}")
    assert -1e-12 <= kl <= log(2) + 1e-12
assert (10 - target) ** 2 > (5 - target) ** 2
Finite margin output
IPO target=5.0
m=0: DPO=0.693 IPO=25.0 KL=0.000
m=5: DPO=0.474 IPO=0.0 KL=0.653
m=10: DPO=0.313 IPO=25.0 KL=0.693

KTO: prospect theory on unpaired telemetry

An application may collect one answer and a thumbs-up/down label, without another answer to the same prompt. KTO consumes desirable/undesirable labels directly. These labels must have defined semantics: a click can reflect user satisfaction, formatting, or bias rather than factual correctness.

KTO uses a prospect-theory-inspired value function as a modeling choice for human feedback. Define rฮธ(x,y)=logโก(ฯ€ฮธ(yโˆฃx)/ฯ€ref(yโˆฃx))r_\theta(x,y)=\log(\pi_\theta(y|x)/\pi_{\mathrm{ref}}(y|x)) and a detached reference point z0z_0. Its loss contribution has opposite directions for the two labels:[17]

โ„“KTO={ฮปD[1โˆ’ฯƒ(ฮฒ(rฮธโˆ’z0))],desirable,ฮปU[1โˆ’ฯƒ(ฮฒ(z0โˆ’rฮธ))],undesirable.\ell_{\mathrm{KTO}}= \begin{cases} \lambda_D[1-\sigma(\beta(r_\theta-z_0))], & \text{desirable}, \\ \lambda_U[1-\sigma(\beta(z_0-r_\theta))], & \text{undesirable}. \end{cases}

The paper defines z0z_0 using policy/reference KL and implements a biased, non-negative batch estimate from mismatched prompt/response rows. It doesn't backpropagate through that estimate. Both class weights default to 1 in the paper; class balance can justify raising either weight. A larger undesirable weight isn't part of the required definition.

Hold the reference point fixed. If increasing an undesirable answer's log-ratio lowers your implemented loss, you've lost the negative-label branch.

kto_desirable_vs_undesirable.py
from math import exp

def kto_loss(log_ratio: float, desirable: bool) -> float:
    beta, reference_point = 0.5, 0.2
    direction = 1 if desirable else -1
    logit = direction * beta * (log_ratio - reference_point)
    return 1 - 1 / (1 + exp(-logit))  # both class weights = 1

for label, desirable in [("desirable", True), ("undesirable", False)]:
    before = kto_loss(1.0, desirable)
    after = kto_loss(2.0, desirable)
    print(f"{label}: loss {before:.3f} -> {after:.3f}")
    assert (after < before) == desirable
KTO label directions
desirable: loss 0.401 -> 0.289
undesirable: loss 0.599 -> 0.711

This snippet isolates the scalar loss. Audit telemetry selection and label quality, then evaluate model behavior before interpreting a falling KTO loss as an improvement.

GRPO: group-relative advantage without a critic

The PPO recipe above learns a critic for GAE. The clipped surrogate itself can use other advantage estimators; it doesn't mathematically require a separate 70B value backbone. GRPO, introduced in DeepSeekMath and used in DeepSeek-R1, replaces the learned critic with group statistics.[19][20] For outcome supervision on prompt xx:

  1. The policy samples a group of GG candidate completions: {y1,y2,โ€ฆ,yG}โˆผฯ€old(โ‹…โˆฃx)\{y_1, y_2, \dots, y_G\} \sim \pi_{\text{old}}(\cdot \mid x).
  2. A verifier or reward model produces scalar scores {r1,โ€ฆ,rG}\{r_1,\dots,r_G\}. Run tests to check selected behaviors after compilation.
  3. The original outcome formulation centers and scales scores within the prompt group:
A^i=riโˆ’meanโก({r1,โ€ฆ,rG})stdโก({r1,โ€ฆ,rG})+ฯต\hat{A}_i = \frac{r_i - \operatorname{mean}\left(\{r_1, \dots, r_G\}\right)}{\operatorname{std}\left(\{r_1, \dots, r_G\}\right) + \epsilon}
  1. Assign that outcome advantage to the generated tokens of completion ii, then use token ratios ฯi,t=ฯ€ฮธ(yi,tโˆฃsi,t)/ฯ€old(yi,tโˆฃsi,t)\rho_{i,t}=\pi_\theta(y_{i,t}|s_{i,t})/\pi_{\mathrm{old}}(y_{i,t}|s_{i,t}). A sampled-batch loss for the original token-averaged formulation is:
LGRPO=โˆ’1Gโˆ‘i=1G1Tiโˆ‘t=1Ti[minโก(ฯi,tA^i,clipโก(ฯi,t,1โˆ’ฯต,1+ฯต)A^i)โˆ’ฮฒki,t].\mathcal L_{\mathrm{GRPO}}= -\frac{1}{G}\sum_{i=1}^G\frac{1}{T_i}\sum_{t=1}^{T_i} \left[\min\left(\rho_{i,t}\hat A_i, \operatorname{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\hat A_i\right) -\beta k_{i,t}\right].

Here TiT_i counts generated tokens. DeepSeekMath uses the non-negative sample term ki,t=expโก(โ„“i,t)โˆ’โ„“i,tโˆ’1k_{i,t}=\exp(\ell_{i,t})-\ell_{i,t}-1, with โ„“i,t=logโก(ฯ€ref/ฯ€ฮธ)\ell_{i,t}=\log(\pi_{\mathrm{ref}}/\pi_\theta) at that observed token. Its expectation estimates conditional forward KL when the token is sampled from the scored current policy; old-policy samples during repeated updates need the corresponding correction. KL is added to the loss, not mixed into these group reward advantages.[19]

Replacing token ratios with the product across the entire response changes clipping. Even two ratios of 1.1 multiply to 1.21 and trigger a sequence clip when neither token is clipped.

grpo_token_ratio_is_not_sequence_ratio.py
from math import prod

ratios = [1.1, 1.1]
epsilon, advantage = 0.2, 1.0

def clipped_surrogate(ratio: float) -> float:
    clipped = min(max(ratio, 1 - epsilon), 1 + epsilon)
    return min(ratio * advantage, clipped * advantage)

token_average = sum(clipped_surrogate(ratio) for ratio in ratios) / len(ratios)
sequence_value = clipped_surrogate(prod(ratios))
print(f"token-average surrogate={token_average:.3f}")
print(f"whole-response surrogate={sequence_value:.3f}")
assert abs(token_average - 1.1) < 1e-12
assert abs(sequence_value - 1.2) < 1e-12
Token versus sequence clipping
token-average surrogate=1.100
whole-response surrogate=1.200

This calculation concerns DeepSeekMath's token-ratio formulation. The R1 report presents response-level notation, and current trainers also offer other importance ratios and normalizers; identify the equation and implementation you're using.[20][21]

GRPO removes critic state, computation, and value-fitting error. It still needs rollouts, a useful scoring signal, and stable policy updates. The earlier equal-backbone state example saves 44.4% by removing its critic; that isn't a measured peak-VRAM reduction for another recipe.

There is one edge case worth predicting before inspecting the code: if every response in a group earns the exact same reward, the standard deviation is zero, and every relative advantage collapses to zero.

grpo_group_relative_advantage.py
from statistics import mean, pstdev

def group_advantages(rewards: list[float]) -> tuple[float, float, list[float]]:
    center = mean(rewards)
    scale = pstdev(rewards)
    if scale == 0:
        return center, scale, [0.0 for _ in rewards]
    return center, scale, [(reward - center) / scale for reward in rewards]

reward_groups = {
    "mixed": [1.0, 0.0, 0.5, 1.0],
    "all_equal": [1.0, 1.0, 1.0, 1.0],
}

for name, rewards in reward_groups.items():
    center, scale, advantages = group_advantages(rewards)
    print(
        f"{name}: mean={center:.3f} std={scale:.3f} "
        f"advantages={[round(value, 3) for value in advantages]}"
    )
Group-relative advantage output
mixed: mean=0.625 std=0.415 advantages=[0.905, -1.508, -0.302, 0.905]
all_equal: mean=1.000 std=0.000 advantages=[0.0, 0.0, 0.0, 0.0]

The helper uses population standard deviation for transparent arithmetic; trainer conventions and reward-scaling options can differ. If every sampled answer gets the same reward, the centered reward advantages are zero. That removes this group's reward-driven update, not necessarily every gradient: an enabled KL penalty or another auxiliary loss can still contribute.

Four completion rewards [1, 0, 0.5, 1] have population mean 0.625 and standard deviation about 0.4146. Subtracting the mean and dividing by the standard deviation yields advantages about [0.90, โˆ’1.51, โˆ’0.30, 0.90]. A flat group [1, 1, 1, 1] has zero standard deviation and the implementation assigns all four advantages zero. A KL term remains separate.
The mixed group centers and scales each completion score. With identical scores, this helper returns zero reward advantages; KL remains a separate loss term.

TRL 1.13.0 logs frac_reward_zero_std from its near-zero standard-deviation check. Under group scaling, a high value can mean all samples pass, all fail, or the scorer can't distinguish them. Under batch scaling, that diagnostic follows the batch scale instead. Inspect the configuration and tasks. This release also offers different token-loss normalizers, and defaults to beta=0.0, so it doesn't load a reference model for KL by default. These choices differ from the original formulation above.[22][21]

What comes after pairwise human labels

Human comparisons don't scale to every new failure. Constitutional AI uses written principles for model critique and revision, supervised learning on revised answers, and AI preference labels for harmlessness. Its RL stage fits a preference model using those AI labels alongside human helpfulness labels, then uses that scorer as reward. This is one form of (Reinforcement Learning from AI Feedback). RLAIF more broadly doesn't require constitutional critique and revision.[23] Writing the rules down still doesn't show whether they hold once someone tries to break them.

The dependence doesn't go away. A constitution is only as good as the judge that applies it. Held-out humans still decide whether "cite P-7 and escalate" survived contact with a real user.

Complete the lesson

Mastery Check

Answer every question, then check your score. Score 75% or higher to mark this lesson complete.

1.Use the four separate logical backbones taught for PPO-style RLHF, each with 3 billion parameters. Trainable state uses 16 bytes per parameter and frozen state 2. Which aggregate budget and frozen-role pairing follow, before activation or rollout storage?
2.Three generated tokens have old-policy minus reference log ratios [0.2,โˆ’0.1,0.4]. Charge beta 0.5 per token and add sequence reward 2 only at the last token. With gamma 1, lambda 0.5, four zero critic values and a true terminal boundary, what are the three GAE advantages?
3.For one stored PPO batch, advantage is โˆ’2 and epsilon 0.2. Compare current-to-old token ratios 0.6 and 1.4. Which surrogate values and denominator follow from the taught minimum rule?
4.Chosen policy/reference summed response log scores are โˆ’4.2/โˆ’5.0; rejected scores are โˆ’6.3/โˆ’5.9. With beta 0.4, which unscaled margin, scaled logit and observed-label DPO loss follow?
5.A six-position response-scoring input is [2,0,1,0,4,0], with two prompt positions, attention validity [1,1,1,1,1,0] and ending ID 4. In a separate length example, chosen summed log ratio 1.2 has 3 targets and rejected sum 0.9 has 2. Which zero-indexed scored target positions, sum-margin and per-token-average margin are correct?
6.A complete two-answer population has a uniform reference and true preference probability 0.8 for A. At its ideal fitted comparison probability, compare beta 0.5 with beta 1. Which policy-A probabilities and initial unscaled-margin derivatives follow?
7.A run retains its original explicit frozen reference R0 after actor updates. New evaluation dataset E2 arrives. Cached row C4 contains chosen/rejected reference sums โˆ’6/โˆ’5 over 3/2 valid targets; the new scoring request requires per-completion means with the same tokens and masks. Which eligibility and C4 scores follow?
8.With a uniform three-answer reference and beta 1, policy U is [A=.45,B=.15,C=.40] and policy V is [A=.15,B=.05,C=.80]. Only A-over-B is labeled. Which pair loss and unseen-C conclusion follow?
9.Check three separate scalar implementations: IPO uses tau 0.2 at margin 4; KTO raises an undesirable rowโ€™s log ratio while its reference point stays fixed; a GRPO group has equal rewards and a separately enabled KL term. Which tuple is consistent with the taught objectives?
10.Two predeclared candidate evaluations supply valid 95% win-probability intervals for their stated comparison plan. Candidate A has [0.54,0.63] and mean length growth 23 tokens; B has [0.49,0.58] and growth 12. The product gates require lower bound>0.50 and growth<20. Which gate outcomes and eligibility follow?

10 questions remaining.

References

Training Language Models to Follow Instructions with Human Feedback (InstructGPT).

Ouyang, L., et al. ยท 2022 ยท NeurIPS 2022

https://arxiv.org/abs/2203.02155

Learning to Summarize from Human Feedback.

Stiennon et al. ยท 2020

https://arxiv.org/abs/2009.01325

Direct Preference Optimization: Your Language Model is Secretly a Reward Model.

Rafailov, R., et al. ยท 2023

https://arxiv.org/abs/2305.18290

Proximal Policy Optimization Algorithms.

Schulman, J., et al. ยท 2017

https://arxiv.org/abs/1707.06347

High-Dimensional Continuous Control Using Generalized Advantage Estimation

Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. ยท 2015 ยท ICLR 2016

https://arxiv.org/abs/1506.02438

OpenRLHF: Execution Modes and Performance Tuning

OpenRLHF Contributors ยท 2026

https://github.com/OpenRLHF/OpenRLHF

A General Theoretical Paradigm to Understand Learning from Human Preferences

Azar, M. G., Rowland, M., et al. ยท 2023

https://arxiv.org/abs/2310.12036

TRL 1.13.0: DPOTrainer Implementation

Hugging Face TRL Contributors ยท 2026

https://github.com/huggingface/trl/blob/3d9261f1fec9f9a8140099c78a65c7da73dce79c/trl/trainer/dpo_trainer.py

Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., & White, C. ยท 2024

https://arxiv.org/abs/2402.13228

Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization

Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., & Hanin, B. ยท 2025 ยท ICLR 2025

https://arxiv.org/abs/2410.08847

SimPO: Simple Preference Optimization with a Reference-Free Reward

Meng, Y., et al. ยท 2024

https://arxiv.org/abs/2405.14734

DPO Trainer

Hugging Face TRL Contributors ยท 2026

https://huggingface.co/docs/trl/dpo_trainer

TRL 1.17.1: DPOTrainer Implementation

Hugging Face TRL Contributors ยท 2026

https://github.com/huggingface/trl/blob/v1.17.1/trl/trainer/dpo_trainer.py

TRL 1.17.1: DPOConfig

Hugging Face TRL Contributors ยท 2026

https://github.com/huggingface/trl/blob/v1.17.1/trl/trainer/dpo_config.py

NIST/SEMATECH e-Handbook: Confidence Intervals for Proportions

NIST/SEMATECH ยท 2026

https://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm

Human Alignment of Large Language Models through Online Preference Optimisation.

Calandriello, D., et al. ยท 2024

https://arxiv.org/abs/2403.08635

KTO: Model Alignment as Prospect Theoretic Optimization.

Ethayarajh, K., et al. ยท 2024 ยท ICML 2024

https://arxiv.org/abs/2402.01306

ORPO: Monolithic Preference Optimization without Reference Model.

Hong, J., Lee, N., & Thorne, J. ยท 2024

https://arxiv.org/abs/2403.07691

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Shao, Z., et al. ยท 2024

https://arxiv.org/abs/2402.03300

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI ยท 2025

https://arxiv.org/abs/2501.12948

TRL 1.13.0: GRPOTrainer Implementation

Hugging Face TRL Contributors ยท 2026

https://github.com/huggingface/trl/blob/3d9261f1fec9f9a8140099c78a65c7da73dce79c/trl/trainer/grpo_trainer.py

GRPO Trainer.

Hugging Face ยท 2026

https://huggingface.co/docs/trl/grpo_trainer

Constitutional AI: Harmlessness from AI Feedback.

Bai, Y., et al. ยท 2022 ยท arXiv preprint

https://arxiv.org/abs/2212.08073

Discussion

Questions and insights from fellow learners.

Discussion loads when you reach this section.