RLHF & DPO Alignment

Suppose a release candidate raises its internal judge score by $0.31$, but wins only 184 of 400 blinded comparisons against the old assistant. Its answers are also 48 tokens longer on average, often starting with "Policy-safe answer:". Would you promote it? These illustrative results don't establish a human-preference improvement, even though the training score rose. Keep that disagreement in view as we turn preference labels into policy updates.

The last chapter turned one access-policy pair into a scalar. For the prompt "temporary admin access for tonight's migration," the ticket-and-cite answer scored $1.8$ and the grant-tonight answer scored $0.7$. The Bradley-Terry preference model maps that $1.1$ margin to roughly a $75\%$ chance that the labeler prefers the safer answer. This chapter asks what to do with that judge: train a policy against it, or skip the extra model and optimize the policy directly from the pairs?

An instruction-tuned assistant can still answer that prompt with fluent, confident, and wrong guidance. InstructGPT puts the gap plainly: a model trained to predict plausible text can still be unhelpful, untruthful, or unsafe even when the prose looks polished.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

That's the alignment problem. Pretraining optimizes next-token prediction. Preference methods such as RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization) push the model toward answers a labeling policy prefers. They don't, by themselves, prove truthfulness or safety outside that policy's coverage. Whether learned or held out, a judge remains a proxy.

What preference pairs add to SFT

After instruction tuning, the model can follow the shape of a request. It can still grant a privileged role, refuse a reasonable one, or pad a correct answer until it's unusable. Next-token prediction doesn't enforce a product's rules. Predicting what comes next isn't the same as satisfying the behavior being judged.

Keep the same prompt the reward-modeling chapter used: "Policy: cite the retrieved access policy and escalate privileged-role changes. User: My service account needs temporary admin access for tonight's migration. Can you approve it?" Two SFT completions are both grammatical:

• Response A: "The retrieved access policy requires escalation for temporary admin access. I can open a reviewer ticket and cite policy P-7."
• Response B: "Temporary admin access is common for migrations, so you can proceed and clean it up tomorrow."

A follows the supplied policy. B violates it. SFT on good demonstrations can teach escalation too; preference training isn't the only route to that behavior. A pair adds a different signal: under the same prompt, prefer A over B. That contrast can be easier to label than writing a new ideal answer, but it only teaches the distinctions the labels actually contain.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

RLHF: a judge, then an actor

RLHF is the large-scale preference pipeline InstructGPT popularized, building on earlier human-feedback fine-tuning for summarization: start from an SFT model, train a reward model from comparisons, then optimize the policy with reinforcement learning.^{[2]Reference 2Learning to Summarize from Human Feedback.https://arxiv.org/abs/2009.01325[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155}

The pipeline is a handoff. SFT gives instruction-following. The reward model turns comparisons into a scalar. PPO updates the policy while a KL term charges drift away from the SFT reference.

Three-phase pipeline

Treat each phase as answering one question: can the model imitate useful answers, can a scorer reproduce the preference rule, and can the policy improve under that scorer without leaving tested behavior? The first two use stored examples. The third generates and scores policy outputs.

Phase 3 is where the extra machinery shows up. InstructGPT's RL stage is a bandit: a prompt comes in, the policy writes a response, the reward model returns a scalar, and a per-token KL penalty against the SFT model is added to limit overoptimization.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} That fresh response is the key difference from fitting only the stored demonstrations.

Bradley-Terry, one number later

Here is how the judge turns a pair into a training signal. Predict the direction first: because A's score is higher, its preference probability should be above $0.5$; reversing the scores should make the observed A label less likely. The Bradley-Terry model says the probability that a human prefers $y_w$ over $y_l$ is a sigmoid of the reward gap:^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290}

$P(y_w \succ y_l \mid x) = \sigma\left(r(x, y_w) - r(x, y_l)\right)$

Plug in last chapter's scores: $r(A)=1.8$, $r(B)=0.7$. The gap is $1.1$, so $P(A \succ B)=\sigma(1.1)\approx 0.750$. If the human picked A, the negative log-likelihood of that observation is about $0.287$. Flip the scores and the loss becomes about $1.387$. These are the scorer's preference probabilities, not independently calibrated guarantees about people. The loss pushes the reward model toward the observed label.

import math

def sigmoid(value: float) -> float:
    return 1 / (1 + math.exp(-value))

reward_a = 1.8
reward_b = 0.7
preference_probability = sigmoid(reward_a - reward_b)
negative_log_likelihood = -math.log(preference_probability)

print(f"P(A preferred over B) = {preference_probability:.1%}")
print(f"NLL = {negative_log_likelihood:.3f}")

P(A preferred over B) = 75.0%
NLL = 0.287

The training loss is the same expectation you saw for the reward model:

where $y_w$ is the winning response and $y_l$ is the losing response for the same prompt $x$.

PPO and the cost of leaving the reference

Once you have a reward model, ask what prevents the policy from chasing its score by drifting into gibberish. Treat the language model as a policy $\pi_\theta$: a conditional probability distribution over responses given a prompt. Proximal Policy Optimization (PPO) uses a clipped update objective; it doesn't guarantee small changes everywhere.^{[4]Reference 4Proximal Policy Optimization Algorithms.https://arxiv.org/abs/1707.06347} We start by formalizing what we want to optimize: expected reward minus a penalty for leaving the reference model, scaled by coefficient $\beta > 0$:

The Kullback-Leibler divergence is an expectation over complete responses: $\mathbb{D}_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) = \mathbb{E}_{y \sim \pi_\theta} [\log \pi_\theta(y \mid x) - \log \pi_{\text{ref}}(y \mid x)]$. It's non-negative, with zero for identical distributions, even though an individual sampled log-ratio can be negative. A plain sample mean estimates this KL when rollouts come directly from the same $\pi_\theta$ whose log-probabilities you score. Greedy search, temperature changes, nucleus filtering, and top-$k$ truncation change the sampling distribution; scores from an older policy do too. Correct for that mismatch or label the metric accordingly.

Before inspecting the PPO approximation, look at what the unconstrained mathematical optimum of this objective actually is. For any fixed prompt $x$, write the objective directly:

Pull out $-\beta$:

Define the prompt partition function (normalizer) $Z(x) = \sum_y \pi_{\text{ref}}(y \mid x) \exp(r(x, y)/\beta)$, which is independent of the response $y$. Multiplying and dividing inside the logarithm by $Z(x)$ yields:

Splitting the log product separates the terms:

where the optimal distribution is defined analytically as:

Assume $\beta>0$, a finite normalizer $Z(x)$, and restrict policies to responses supported by the reference. Then KL is non-negative, with equality at $\pi=\pi^*$, so this is the unique optimal distribution for the fixed reward. A parameterized language model might not represent or reach it. This result is an exponentially reweighted reference policy, not a convergence guarantee for an optimizer.

Computing the normalizer over language sequences is generally intractable. PPO instead makes approximate policy updates from sampled responses; finite data, clipping, and critic error can keep the learned policy away from this optimum.^{[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290}

Token-level reward shaping and the four model roles

The PPO-style RLHF recipe here coordinates four logical model roles:

1. Actor / policy ($\pi_\theta$): the generative language model being trained.
2. Reference ($\pi_{\text{ref}}$): frozen SFT model providing baseline log-probabilities for the KL penalty.
3. Reward model ($r_\phi$): frozen scorer that evaluates sequence quality.
4. Critic / value ($V_\psi$): predicts expected future return from each prefix to help estimate advantages.

These roles don't require four independent, simultaneously resident GPU backbones. The critic can be smaller or share policy parameters; frozen roles can use separate devices or offloading. InstructGPT, for example, used a 6B reward model even for its 175B policy and initialized its value function from that reward model.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155[4]Reference 4Proximal Policy Optimization Algorithms.https://arxiv.org/abs/1707.06347}

In this sequence-score recipe, the reward model scores the completed response $y=(y_1,\dots,y_T)$. A terminal reward can train earlier tokens: their returns include that future outcome. Assigning it to each immediate reward would instead count it repeatedly. KL shaping adds a different signal, charging each generated token for its sampled policy/reference log-ratio. When collecting a rollout, evaluate that charge under the rollout policy $\pi_{\mathrm{old}}$:

For an undiscounted episode, adding these rewards uses the autoregressive log-probability factorization:

Its expectation under $\pi_{\mathrm{old}}$ is the reward-minus-KL objective for that rollout policy. It isn't a fresh evaluation of the actor after every PPO update. Store rollout rewards, old log-probabilities, and detached advantage targets for the batch.

Generalized Advantage Estimation (GAE) uses residuals $\delta_t=R_t+\gamma V(s_{t+1})-V(s_t)$ and, for an episode ending at $T$, $\hat A_t=\sum_{l=0}^{T-t}(\gamma\lambda)^l\delta_{t+l}$. Here $\gamma$ discounts future rewards, while $\lambda$ trades off reliance on the critic against longer sampled returns. Use $V(s_{T+1})=0$ for a true terminal state; a deliberately truncated continuing episode needs an appropriate bootstrap.^{[5]Reference 5High-Dimensional Continuous Control Using Generalized Advantage Estimationhttps://arxiv.org/abs/1506.02438}

Try terminal-only rewards with no KL and a zero baseline. Does the first token have zero advantage?

def gae(rewards: list[float], values: list[float], lam: float) -> list[float]:
    if len(values) != len(rewards) + 1:
        raise ValueError("need one baseline per step plus the terminal bootstrap")
    advantages = [0.0] * len(rewards)
    running = 0.0
    for t in reversed(range(len(rewards))):
        delta = rewards[t] + values[t + 1] - values[t]  # gamma = 1
        running = delta + lam * running
        advantages[t] = running
    return advantages

rewards = [0.0, 0.0, 1.0]
values = [0.0, 0.0, 0.0, 0.0]
for lam in [1.0, 0.5]:
    advantages = gae(rewards, values, lam)
    print(f"lambda={lam:.1f}: advantages={advantages}")
    assert all(value > 0 for value in advantages)

lambda=1.0: advantages=[1.0, 1.0, 1.0]
lambda=0.5: advantages=[0.25, 0.5, 1.0]

With $\gamma=\lambda=1$, the terminal score reaches every earlier decision through its return. With this inaccurate zero critic and a smaller $\lambda$, the signal fades backward. This is credit assignment, not evidence that every token individually deserved the reward.

The snippet below isolates shaped reward for two sampled responses. Before reading its output, predict the ordering: the preamble has raw reward $1.35$ but log-ratio $1.6$, versus $0.1$ for the near-reference sample. With $\beta=0.2$, KL erases that apparent win.

samples = [
    {"name": "near_reference", "reward": 1.20, "policy_logp": -2.1, "ref_logp": -2.2},
    {"name": "preamble_hack", "reward": 1.35, "policy_logp": -1.2, "ref_logp": -2.8},
]
beta = 0.20

for sample in samples:
    sampled_log_ratio = sample["policy_logp"] - sample["ref_logp"]
    shaped_reward = sample["reward"] - beta * sampled_log_ratio
    print(f"{sample['name']}: raw={sample['reward']:.2f} shaped={shaped_reward:.2f}")

near_reference: raw=1.20 shaped=1.18
preamble_hack: raw=1.35 shaped=1.03

The higher raw score isn't automatically the larger shaped return. These two rows illustrate arithmetic, not an estimate of the full KL. On a complete two-response distribution, one log-ratio can be negative while its policy-weighted average stays positive:

import math

policy = [0.8, 0.2]
reference = [0.5, 0.5]
ratios = [math.log(p / q) for p, q in zip(policy, reference, strict=True)]
kl = sum(p * ratio for p, ratio in zip(policy, ratios, strict=True))
print("sample log-ratios:", [round(value, 3) for value in ratios])
print(f"expected KL={kl:.3f}")
assert ratios[1] < 0 < kl

sample log-ratios: [0.47, -0.916]
expected KL=0.193

What PPO actually clips

The reference anchors the overall training run. A second policy snapshot, $\pi_{\mathrm{old}}$, generated the current rollout batch. PPO compares the changing actor to that snapshot, not to the SFT reference. At token $t$, let $s_t$ be the prompt plus generated prefix and $a_t$ the sampled next token:

We maximize the batch average of $L_t$. A positive advantage $\hat{A}_t$ says the outcome beat the critic's expectation; a negative one says it fell short.

Clipping behaves asymmetrically depending on the sign of the advantage:

• When $\hat{A}_t>0$ and $\epsilon=0.2$, this sample's surrogate stops improving once the ratio exceeds $1.2$. It still penalizes a ratio that moves downward in the wrong direction.
• When $\hat{A}_t<0$, reducing the ratio improves the surrogate only down to $0.8$. Below that, its value is constant. Increasing the ratio remains harmful and isn't clipped away.

Check both signs across varying ratios:

def surrogate(ratio: float, advantage: float, epsilon: float = 0.2) -> float:
    clipped = min(max(ratio, 1 - epsilon), 1 + epsilon)
    return min(ratio * advantage, clipped * advantage)

for advantage in [1.0, -1.0]:
    values = [surrogate(ratio, advantage) for ratio in [0.5, 1.0, 1.5]]
    print(f"A={advantage:+.0f}: {values}")
assert surrogate(1.5, 1.0) == 1.2
assert surrogate(0.5, -1.0) == -0.8
assert surrogate(1.5, -1.0) == -1.5

A=+1: [0.5, 1.0, 1.2]
A=-1: [-0.8, -1.0, -1.5]

Clipping removes an incentive in this surrogate. It doesn't forcibly clamp every actual token probability or guarantee a hard KL boundary. PPO can take several gradient passes over one batch, retaining its old-policy probabilities and advantage targets, before collecting fresh rollouts.

The systems cost of online RL

Estimate the allocations before choosing a layout. Suppose all four roles have independent 70B backbones, both trainable roles use BF16 weights and gradients, and Adam stores two FP32 moments plus an FP32 master copy. That's an assumed 16 bytes per trainable parameter and 2 per frozen parameter. The master copy and gradient precision are implementation choices; adapters or a smaller critic change this budget.

parameters = 70_000_000_000
trainable_bytes = 2 + 2 + 4 + 4 + 4  # weights, gradients, moments, master
frozen_bytes = 2
actor_gb = parameters * trainable_bytes / 1e9
critic_gb = actor_gb
frozen_gb = parameters * frozen_bytes / 1e9
ppo_gb = actor_gb + critic_gb + 2 * frozen_gb
without_critic_gb = actor_gb + 2 * frozen_gb

print(f"each trainable backbone={actor_gb:.0f} GB")
print(f"each frozen backbone={frozen_gb:.0f} GB")
print(f"four-role state total={ppo_gb:.0f} GB")
print(f"remove critic: {without_critic_gb:.0f} GB, saved={critic_gb / ppo_gb:.1%}")

each trainable backbone=1120 GB
each frozen backbone=140 GB
four-role state total=2520 GB
remove critic: 1400 GB, saved=44.4%

These are decimal GB of aggregate parameter-related state, excluding activations, generation KV caches, temporary buffers, replication, and communication. They aren't measured peak GPU allocations or a universal PPO requirement. Sharding and offloading distribute or relocate state; they don't make the underlying bytes vanish. Whether the run needs multiple nodes depends on the recipe and available hardware.

Rollouts add work that offline DPO avoids. Prompt prefill and token-by-token decoding have different compute and memory demands; decoding can be bandwidth-bound at small batches, while larger rollout batches improve throughput until another limit takes over. A training run usually values throughput more than an interactive assistant's response latency.

Both colocated and separate generation/training layouts are valid. OpenRLHF documents a colocated hybrid engine that alternates generation and training, and an asynchronous mode that can overlap them. Overlap brings weight synchronization, possible stale-policy samples, and extra resource costs. Compare measured utilization and policy lag rather than assuming decoupling always wins.^{[6]Reference 6OpenRLHF: Execution Modes and Performance Tuninghttps://github.com/OpenRLHF/OpenRLHF}

DPO: turn reward fitting into policy fitting

PPO-style RLHF is a multi-role online loop. If a fixed pair already tells you which response wins, why generate another response for every update? DPO targets those pairs with a classification-style loss. It removes engineering overhead, but it doesn't give you online exploration or guarantee the same checkpoint PPO would have found.

Why an offline loss can stand in for RL

Under a KL-regularized reward-maximization objective and a Bradley-Terry preference model, the optimal policy and the reward function are two mathematical views of the same object. DPO parameterizes an implicit reward with policy-to-reference log-probability ratios, so the pairwise loss never needs a standalone reward model or a PPO loop.^{[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290}

Recall the closed-form analytical optimum derived earlier for a fixed prompt $x$ and frozen reference policy $\pi_{\text{ref}}$:

Taking the natural logarithm of both sides isolates the reward:

Rearranging expresses the fixed reward $r(x,y)$ in terms of its optimal policy, the reference, and the partition function. The algebra works for a proxy reward too; it doesn't establish ground truth:

Now, substitute this expression into the Bradley-Terry preference model. The probability that a human prefers $y_w$ over $y_l$ depends strictly on the difference between their rewards:

Watch what happens when you subtract the two implicit reward terms:

The partition function $Z(x) = \sum_y \pi_{\text{ref}}(y \mid x) \exp(r(x, y)/\beta)$ depends solely on the prompt $x$, not the response. It cancels out completely:

This cancellation is the mathematical foundation of DPO.^{[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} By substituting the parameterized policy $\pi_\theta$ for the optimal policy $\pi^*$, we define the implicit reward:

We can now train the language model directly on preference pairs by minimizing the negative log-likelihood of the observed preference labels. This isn't the SFT loss $-\log\pi_\theta(y_w|x)$: the sigmoid models which answer wins, rather than the probability of generating the chosen text.

What exactly is equivalent to RLHF here? The change of variables connects a reward function to its exact KL-regularized optimal distribution. Recovering the same optimum from preference learning additionally needs a suitable preference model, enough comparisons to identify relevant reward differences, a policy class that can represent the solution, and successful optimization. Under the Bradley-Terry population assumptions, the objectives can share an optimum.^{[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290[7]Reference 7A General Theoretical Paradigm to Understand Learning from Human Preferenceshttps://arxiv.org/abs/2310.12036} A finite DPO run and a clipped PPO run needn't produce the same policy. Neither the algebra nor a low training loss determines behavior on answers absent from the comparisons.

Gradient dynamics and implicit sample weighting

Taking the gradient of $\mathcal{L}_{\text{DPO}}$ with respect to the policy parameters $\theta$ reveals how DPO steers the model:

This gradient contains two intuitive mechanisms:

1. Contrast direction: Gradient descent follows $\nabla_\theta\log\pi_\theta(y_w|x)-\nabla_\theta\log\pi_\theta(y_l|x)$. The two terms favor the chosen and oppose the rejected completion, but shared parameters can make both absolute probabilities fall. It isn't a promise about each token's update.
2. Pair weight: $\sigma(\hat r_l-\hat r_w)$ lies above $0.5$ when the implicit reward ordering is wrong and approaches 1 for a large negative chosen margin. It approaches zero for a large positive margin. This scales the pair's gradient; it isn't the optimizer's learning rate or a guarantee against overfitting.

The role of $\beta$ deserves care. For a fixed reward, a larger $\beta$ makes the analytical optimum stay closer to the reference. In the empirical DPO loss, it also scales the logit and the gradient. At initialization, $\partial L/\partial m=-\beta/2$ for unscaled margin $m$. A larger $\beta$ therefore produces a larger initial margin gradient, even though it represents stronger KL regularization in the reward-to-policy derivation. Learning rate, number of updates, and saturation affect the actual drift.

For a complete two-answer policy with a uniform reference, suppose the true probability of preferring A is $q=0.75$. Matching that probability requires $\beta m=\log(q/(1-q))=\log3$, so $\pi^*(A)=\sigma(\log3/\beta)$. These are population optima, not measurements from training:

Larger $\beta$ moves the ideal policy closer to the 50% reference here. It doesn't turn empirical DPO into an explicit hard KL constraint.^{[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290[7]Reference 7A General Theoretical Paradigm to Understand Learning from Human Preferenceshttps://arxiv.org/abs/2310.12036}

Work the loss on the P-7 pair

Let's test this formulation on the access-policy prompt. $y_w$ is chosen (ticket and cite P-7) and $y_l$ is rejected (grant admin tonight). Read each term as change relative to the frozen reference: raise the chosen response relative to that baseline, lower the rejected response, and the margin grows.

Suppose the current sequence log-probabilities are:

• Chosen A: policy $-8.2$, reference $-8.5$
• Rejected B: policy $-9.5$, reference $-9.1$
• $\beta = 0.1$

Then:

1. Chosen relative advantage: $-8.2 - (-8.5) = +0.3$
2. Rejected relative advantage: $-9.5 - (-9.1) = -0.4$
3. Margin: $0.3 - (-0.4) = +0.7$
4. Scaled logit: $0.1 \times 0.7 = 0.07$
5. $\sigma(0.07) \approx 0.517$
6. Loss: $-\log(0.517) \approx 0.659$

The loss rewards growing that $0.7$ margin. Its weight depends on the pair's reference-relative ordering, not on a single global distance between models. Shared model parameters couple the chosen and rejected probabilities, so their absolute changes need separate monitoring.

from math import exp, log

chosen_policy, chosen_reference = -8.2, -8.5
rejected_policy, rejected_reference = -9.5, -9.1
beta = 0.1
margin = (chosen_policy - chosen_reference) - (rejected_policy - rejected_reference)
logit = beta * margin
loss = -log(1 / (1 + exp(-logit)))

print(f"relative_margin={margin:.1f}")
print(f"scaled_logit={logit:.2f}")
print(f"dpo_loss={loss:.3f}")

relative_margin=0.7
scaled_logit=0.07
dpo_loss=0.659

Those $-8.2$ figures are already summed response log-probabilities. For response tokens $y_1,\ldots,y_T$, compute $\log\pi(y|x)=\sum_t\log\pi(y_t|x,y_{<t})$. Teacher forcing supplies each observed prefix; it doesn't sample new text. The log-softmax must include the whole vocabulary even though we gather only the observed token.

The toy below makes the causal shift and masks explicit. Token IDs 0, 2 form the prompt, the next two IDs form the response, and 4 is end-of-sequence. A final 0 is padding; another 0 is real response content on the rejected branch, so masking by token ID alone would be wrong. Logit row t-1 predicts input token t. The arbitrary scores don't come from a trained model.

import math

CHOSEN_IDS = [0, 2, 1, 2, 4, 0]
REJECTED_IDS = [0, 2, 3, 0, 4, 0]
ATTENTION_MASK = [1, 1, 1, 1, 1, 0]
PROMPT_LENGTH = 2
BETA = 0.1

chosen_policy = [
    [0.0] * 5,  # predicts a prompt token: excluded from the loss
    [0.0, 2.4, 0.1, -0.8, -1.0],
    [0.0, 0.2, 2.1, -0.6, -1.0],
    [0.0, 0.0, 0.0, 0.0, 2.0],  # predicts EOS: included
    [0.0] * 5,  # predicts padding: excluded
    [0.0] * 5,
]
chosen_ref = [
    [0.0] * 5,
    [0.0, 0.8, 0.2, 0.0, -1.0],
    [0.0, 0.1, 0.9, 0.0, -1.0],
    [0.0, 0.0, 0.0, 0.0, 2.0],
    [0.0] * 5,
    [0.0] * 5,
]
rejected_policy = [
    [0.0] * 5,
    [0.2, -0.4, -0.2, 1.1, -1.0],
    [1.0, -0.3, -0.4, 0.2, -1.0],
    [0.0, 0.0, 0.0, 0.0, 2.0],
    [0.0] * 5,
    [0.0] * 5,
]
rejected_ref = [
    [0.0] * 5,
    [0.1, 0.0, 0.0, 0.6, -1.0],
    [0.7, 0.0, 0.0, 0.2, -1.0],
    [0.0, 0.0, 0.0, 0.0, 2.0],
    [0.0] * 5,
    [0.0] * 5,
]

def log_softmax(logits: list[float]) -> list[float]:
    peak = max(logits)
    log_normalizer = peak + math.log(sum(math.exp(value - peak) for value in logits))
    return [value - log_normalizer for value in logits]

def sequence_logprob(steps: list[list[float]], token_ids: list[int]) -> float:
    if len(steps) != len(token_ids) or len(token_ids) != len(ATTENTION_MASK):
        raise ValueError("logits, tokens, and mask must have matching lengths")
    return sum(log_softmax(steps[t - 1])[token_ids[t]]
               for t in range(PROMPT_LENGTH, len(token_ids)) if ATTENTION_MASK[t])

def dpo_loss(
    chosen_pi: list[list[float]],
    rejected_pi: list[list[float]],
) -> float:
    margin = (
        sequence_logprob(chosen_pi, CHOSEN_IDS)
        - sequence_logprob(chosen_ref, CHOSEN_IDS)
        - (
            sequence_logprob(rejected_pi, REJECTED_IDS)
            - sequence_logprob(rejected_ref, REJECTED_IDS)
        )
    )
    logit = BETA * margin
    return max(0.0, -logit) + math.log1p(math.exp(-abs(logit)))

loss = dpo_loss(chosen_policy, rejected_policy)
boosted_chosen = [row[:] for row in chosen_policy]
boosted_chosen[1][1] += 1.0
boosted_chosen[2][2] += 1.0
boosted_loss = dpo_loss(boosted_chosen, rejected_policy)
masked_changes = [row[:] for row in chosen_policy]
masked_changes[0][2] = 1000.0
masked_changes[4][0] = 1000.0

print(f"chosen_logp={sequence_logprob(chosen_policy, CHOSEN_IDS):.3f}")
print(f"rejected_logp={sequence_logprob(rejected_policy, REJECTED_IDS):.3f}")
print(f"dpo_loss={loss:.3f}")
print(f"after_boost_loss={boosted_loss:.3f}")
assert math.isfinite(loss) and boosted_loss < loss
assert sequence_logprob(masked_changes, CHOSEN_IDS) == sequence_logprob(chosen_policy, CHOSEN_IDS)
assert math.isfinite(log_softmax([1000.0, -1000.0])[1])
print("prompt and padding predictions excluded:", True)

chosen_logp=-0.993
rejected_logp=-1.882
dpo_loss=0.662
after_boost_loss=0.646
prompt and padding predictions excluded: True

At initialization, identical policy and reference probabilities give a DPO logit of zero and loss $\log 2\approx0.693$. Token logits themselves need not be zero. Dropout, mismatched tokenization, or different masks can break the identity before the first update. Keep the reference frozen, compute its scores without gradients, and use the same tokenizer, chat template, response boundaries, and EOS convention on both branches.

Whole-response sums can create length effects when chosen and rejected answers differ systematically. Dividing by token count changes the objective; it isn't a harmless numerical optimization. To see why, consider two illustrative completions whose counted lengths include the end token:

Standard DPO sees margin $0.40-0.30=+0.10$. Averaging each completion first gives $0.10-0.15=-0.05$, reversing the implicit preference ordering. Averaging the final loss across dataset rows is a separate operation and doesn't change each row's sequence score. Track completion lengths and verify truncation hasn't removed the behavior that explains the label.^{[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290[8]Reference 8TRL 1.13.0: DPOTrainer Implementationhttps://github.com/huggingface/trl/blob/3d9261f1fec9f9a8140099c78a65c7da73dce79c/trl/trainer/dpo_trainer.py}

Likelihood displacement: a bigger margin isn't a more likely chosen answer

Look closely at the DPO gradient direction: it optimizes the difference between chosen and rejected log-probability changes, not their absolute values. That leads directly to a deceptive failure mode. Did the policy make the chosen answer more likely, or did it merely crush the rejected answer?

Coupled parameters and the model's embedding geometry can make rejected-answer suppression reduce chosen-answer probability too. If rejected log-probability drops by $1.7$ and chosen by $0.9$, the margin grows by $0.8$ despite both completions losing probability. Whether chosen probability falls below the SFT reference depends on where it started.

Pal et al. proposed DPO-Positive (DPOP), adding a penalty when chosen likelihood falls below the reference.^{[9]Reference 9Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivehttps://arxiv.org/abs/2402.13228} Razin et al. analyzed likelihood displacement and showed that it can move mass to either acceptable alternatives or worse responses, including unsafe compliance in their experiments.^{[10]Reference 10Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationhttps://arxiv.org/abs/2410.08847} The pair's margin alone doesn't reveal which happened; repetitive text or hallucinations aren't automatic consequences of the arithmetic.

Keep the P-7 pair's frozen reference at $-8.5$ (chosen) and $-9.1$ (rejected). In an illustrative before/after comparison, chosen log-probability falls from $-8.2$ to $-9.1$ while rejected falls from $-9.5$ to $-11.2$. The relative margin grows from $0.7$ to $1.5$ even though both labeled completions lose probability. Mass moves to other responses, whose quality the pair doesn't tell us. Plotting $\log(\pi/\pi_{\text{ref}})$ exposes the change: chosen starts at $+0.3$ and finishes at $-0.6$.

ref_w, ref_l = -8.5, -9.1
steps = [
    ("before", -8.2, -9.5),
    ("after", -9.1, -11.2),
]

for name, pi_w, pi_l in steps:
    margin = (pi_w - ref_w) - (pi_l - ref_l)
    print(f"{name}: chosen={pi_w:.1f} rejected={pi_l:.1f} margin={margin:.1f}")

before: chosen=-8.2 rejected=-9.5 margin=0.7
after: chosen=-9.1 rejected=-11.2 margin=1.5

Monitor at least four series:

If margins improve while chosen log-prob collapses, inspect data, $\beta$, length effects, and held-out generations before extending the run. A decrease in one exact answer's probability alone doesn't prove quality fell: other good answers may gain mass. Compare length-matched slices; treat a length-aware objective such as SimPO as a separate experiment, not a guaranteed cure.^{[11]Reference 11SimPO: Simple Preference Optimization with a Reference-Free Rewardhttps://arxiv.org/abs/2405.14734}

Vanilla DPO needs policy and reference scores, not a separate reward or value model and not fresh rollouts during optimization. A frozen reference's scores can be precomputed for fixed token sequences, so two logical roles don't require two resident GPU models. TRL 1.13.0, rechecked October 10, 2026, exposes precompute_ref_log_probs and rejects combining it with sync_ref_model=True. Its IPO loss also uses per-completion average reference-relative scores, whereas standard sigmoid DPO sums token log-probabilities. Check the exact loss and token masks before comparing margins or reusing caches.^{[12]Reference 12DPO Trainerhttps://huggingface.co/docs/trl/dpo_trainer[8]Reference 8TRL 1.13.0: DPOTrainer Implementationhttps://github.com/huggingface/trl/blob/3d9261f1fec9f9a8140099c78a65c7da73dce79c/trl/trainer/dpo_trainer.py}

A reference cache belongs to an exact scoring recipe: reference weights, tokenizer and template, token IDs, completion masks, truncation, and sum/mean convention. Recompute affected scores when any of those change. Matching the dataset row's text alone doesn't establish that a cached number still represents its current scored tokens.

To connect that arithmetic to the library, prepare two map-style Hugging Face datasets, train_pairs and eval_pairs, with prompt, chosen, and rejected fields. For conversational rows, use role/content messages and the SFT tokenizer's chat template. Set sft_checkpoint to a full model checkpoint directory, rather than an adapter-only artifact. This source-checked TRL 1.13.0 fragment is a starting configuration; it hasn't been run as transformer training here:

from transformers import AutoTokenizer
from trl import DPOConfig, DPOTrainer

tokenizer = AutoTokenizer.from_pretrained(sft_checkpoint)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

trainer = DPOTrainer(
    model=sft_checkpoint,
    processing_class=tokenizer,
    args=DPOConfig(
        output_dir="runs/p7-dpo",
        beta=0.1,
        precompute_ref_log_probs=True,
        max_length=2048,
        per_device_train_batch_size=1,
        learning_rate=5e-6,
        max_steps=10,
    ),
    train_dataset=train_pairs,
    eval_dataset=eval_pairs,
)
trainer.train()
pair_metrics = trainer.evaluate()
trainer.save_model("artifacts/p7-dpo")
tokenizer.save_pretrained("artifacts/p7-dpo")

Ten updates are a wiring exercise, not a quality target. Choose precision and batch settings for the model and device. Inspect collated pairs before training: this release's default keep_start truncation can erase the chosen/rejected distinction or all completion tokens. Require at least one scored next-token target per branch after the causal shift, and check that the label-defining difference survives. A nonempty input tensor can still contain zero scored completion targets. Its reference cache is constructed before updates. Evaluate fresh generations separately; trainer.evaluate() reports losses and metrics on stored pairs, not a human win rate.^{[8]Reference 8TRL 1.13.0: DPOTrainer Implementationhttps://github.com/huggingface/trl/blob/3d9261f1fec9f9a8140099c78a65c7da73dce79c/trl/trainer/dpo_trainer.py}

For this full-model setup with precomputation, supply every planned pair-evaluation dataset at initialization. Without a separate ref_model, the trainer doesn't retain the frozen reference after updates. Passing a new uncached dataset later can raise Cannot compute reference log-probs for a dataset passed to `evaluate()` after training has started. This is a cache-lifecycle failure, not evidence that the new evaluation data is malformed.^{[8]Reference 8TRL 1.13.0: DPOTrainer Implementationhttps://github.com/huggingface/trl/blob/3d9261f1fec9f9a8140099c78a65c7da73dce79c/trl/trainer/dpo_trainer.py}

Comparing RLHF and DPO

The useful contrast is what each method has to keep alive, not a winner-take-all ranking.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} Read the table as a deployment question: which signal must stay live, and which behavior can your stored data never show you?

DPO is a strong candidate baseline for offline pairwise preferences. PPO isn't obsolete. If you need online exploration, fresh model-generated negatives, or a reward that moves as the policy moves, an RL-style loop can reach cases a frozen DPO dataset can't.

The data both methods inherit

RLHF and DPO consume the same kind of row, then spend it differently. RLHF fits a reward model on the comparisons, then rolls out. DPO trains the policy on the pairs themselves.^{[1]Reference 1Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} The row is shared; the feedback path is not.

For the strict chosen/rejected loss here, use the same rendered prompt, record generator provenance, and handle ties or disagreement deliberately. Standard DPO treats the selected ordering as a binary observation; repeated opposite votes can represent uncertainty rather than mislabeled data. Group prompts when measuring generalization to unseen prompts. A known-prompt/new-response evaluation asks a different question and still must keep evaluated responses and labels out of training. Samples from another generator aren't automatically invalid, but can miss current-policy failures.

You don't need a huge dump of easy instructions. You need coverage of the distinctions you care about: refusals, tool use, reasoning, style, and the P-7-style policy calls. Ask what happens when the current policy invents a failure absent from the stored pairs: DPO has no training row for it until you refresh the data, while PPO can expose it only after a fresh rollout reaches its judge. If annotators keep disagreeing, more labels can quantify uncertainty. They won't turn a vague rubric into a training signal.

Here is a precise coverage failure. Restrict the P-7 assistant to three possible responses: A escalates, B grants admin directly, and an unseen C grants admin while citing a nonexistent exception. Suppose the dataset compares only A against B. Two policies can assign the same A-to-B odds while allocating very different mass to C:

from math import log, log1p, exp

reference = {"A": 1 / 3, "B": 1 / 3, "C": 1 / 3}
policies = [
    {"A": 0.40, "B": 0.10, "C": 0.50},
    {"A": 0.08, "B": 0.02, "C": 0.90},
]
losses = []
for policy in policies:
    assert abs(sum(policy.values()) - 1) < 1e-12
    margin = log(policy["A"] / reference["A"]) - log(policy["B"] / reference["B"])
    logit = 0.1 * margin
    loss = max(0.0, -logit) + log1p(exp(-abs(logit)))
    losses.append(loss)
    print(f"A/B odds={policy['A'] / policy['B']:.1f} DPO loss={loss:.3f} unseen C mass={policy['C']:.0%}")
assert abs(losses[0] - losses[1]) < 1e-12

A/B odds=4.0 DPO loss=0.626 unseen C mass=50%
A/B odds=4.0 DPO loss=0.626 unseen C mass=90%

Both margins are $\log4$, so the observed pair loss is identical even when the unseen violation receives 90% probability. These are hand-constructed distributions, not a claim that one optimizer step moves between them. They demonstrate what the observed objective leaves unidentified. Fresh rollout evaluation can discover C, and new A-versus-C labels can constrain its ranking. An online reward model also needs coverage of that fabricated exception before its score is useful.^{[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290[7]Reference 7A General Theoretical Paradigm to Understand Learning from Human Preferenceshttps://arxiv.org/abs/2310.12036}

When the proxy lies

That release rehearsal raises a reward-hacking suspicion, not a diagnosis from three numbers. Reward hacking means the policy exploits the scorer instead of improving the intended behavior. Inspect the longer answers and score controlled rewrites: does adding "Policy-safe answer:" raise the reward without fixing a policy error? That comparison can expose what the scorer rewards. The reward model is a proxy, so optimizing its score still needs independent checks.

If the scorer overweights length or markdown, policy optimization can amplify those traits without improving answer quality. Rising proxy reward without matching independent preference gains is a warning; controlled comparisons help distinguish a shortcut from useful extra detail.

Verbosity

Annotators sometimes treat length as quality. Predict what happens next: a padded P-7 explanation can beat a short, correct one, so the policy learns to pad if that correlation is in the data.

• Symptom: mean length climbs while helpfulness plateaus.
• Fix: length-matched preference evals, and explicit concise-versus-padded labels. Length penalties can also punish necessary detail, so validate them.

The 0.693 trap

If the policy starts equal to its reference, every DPO relative margin is zero and the loss starts at $-\log(0.5)\approx0.693$. That's expected. Let $m$ be the unscaled reference-relative margin and $z=\beta m$. For one pair,

$\frac{\partial L}{\partial z}=\sigma(z)-1,\qquad\frac{\partial L}{\partial m}=\beta(\sigma(z)-1).$

At zero logit the first derivative is $-0.5$, but at $\beta=0.1$ the second is $-0.05$. The model-parameter gradient is that second factor times $\nabla_\theta\log\pi_\theta(y_w|x)-\nabla_\theta\log\pi_\theta(y_l|x)$. Its size depends on the model too. The loss isn't flat at initialization, but -0.5 isn't the gradient of every parameter.^{[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290}

That is initialization, not evidence that a pair is weak. Ambiguous pairs still hurt. Conflicting labels on the same prompt can cancel in aggregate or teach arbitrary style. Diagnose from agreement, slices, and held-out behavior, not from initial loss.

from math import exp, log1p

def sigmoid(value: float) -> float:
    return 1 / (1 + exp(-value))

def loss_at_margin(margin: float, beta: float) -> float:
    logit = beta * margin
    return max(0.0, -logit) + log1p(exp(-abs(logit)))

beta = 0.1
for margin in [0.0, 20.0]:
    logit = beta * margin
    d_logit = sigmoid(logit) - 1
    d_margin = beta * d_logit
    h = 1e-5
    finite_difference = (loss_at_margin(margin + h, beta) - loss_at_margin(margin - h, beta)) / (2 * h)
    assert abs(finite_difference - d_margin) < 1e-9
    print(f"logit={logit:.1f} loss={loss_at_margin(margin, beta):.3f} dL/dz={d_logit:.3f} dL/dm={d_margin:.3f}")

# A two-response softmax policy with logits [theta, 0] and uniform reference.
# Its reference-relative log-odds margin is exactly theta.
theta = 0.0
for _ in range(20):
    theta -= beta * (sigmoid(beta * theta) - 1)  # learning rate 1
print(f"toy P(chosen)={sigmoid(theta):.3f} loss={loss_at_margin(theta, beta):.3f}")
assert sigmoid(theta) > 0.5 and loss_at_margin(theta, beta) < loss_at_margin(0.0, beta)

logit=0.0 loss=0.693 dL/dz=-0.500 dL/dm=-0.050
logit=2.0 loss=0.127 dL/dz=-0.119 dL/dm=-0.012
toy P(chosen)=0.726 loss=0.646

• Symptom: loss stays near $0.693$ after real updates, and held-out preference doesn't move.
• Fix: confirm the policy is updating, then inspect contradictory labels, unresolved ties, prompt leakage, and missing coverage before blaming beta.

A release gate that actually uses humans

The scalar toy took real gradient steps, but it didn't train a language model. Full-model validation must also exercise tokenization, reference freezing, optimizer state, precision, memory, and held-out generation. GPU throughput and PPO rollout stability aren't established by these CPU examples.

Release asks another question: did behavior improve within the tested drift budget? Tie proxy improvement to behavior you care about:

1. Hold out human evaluation, blind response order, and specify the rubric and decision rule before looking at results.
2. Track auxiliaries: length, refusal rate, formatting drift, calibration.
3. Watch KL and stop when you leave the tested drift budget.
4. Refresh the scorer when current outputs leave its training distribution.

Return to the illustrative 184 wins in 400 comparisons. Assume independent held-out prompts, one judgment per prompt, no ties, and one planned evaluation. A 95% Wilson interval for the candidate's win probability accounts for sampling uncertainty.^{[13]Reference 13NIST/SEMATECH e-Handbook: Confidence Intervals for Proportionshttps://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm} Repeated checkpoints, shared users, or ties require a matching analysis rather than pretending every judgment is an independent Bernoulli trial.

For this toy gate, require the interval's lower bound to exceed 50% and mean length growth to stay below 20 tokens. That length threshold is an invented product constraint, not an alignment standard. Predict the result: higher proxy reward won't authorize promotion.

from math import sqrt

def wilson_interval(wins: int, total: int, z: float = 1.96) -> tuple[float, float]:
    if total <= 0 or not 0 <= wins <= total:
        raise ValueError("need positive total and 0 <= wins <= total")
    p = wins / total
    denominator = 1 + z * z / total
    center = (p + z * z / (2 * total)) / denominator
    radius = z * sqrt(p * (1 - p) / total + z * z / (4 * total * total)) / denominator
    return center - radius, center + radius

wins, total = 184, 400
lower, upper = wilson_interval(wins, total)
proxy_reward_delta = 0.31
mean_length_delta_tokens = 48
ready = lower > 0.50 and mean_length_delta_tokens < 20

print(f"proxy_improved={proxy_reward_delta > 0}")
print(f"candidate_win_rate={wins / total:.1%}")
print(f"95% Wilson interval=[{lower:.1%}, {upper:.1%}]")
print(f"evidence_of_improvement={lower > 0.50}")
print(f"release_ready={ready}")

proxy_improved=True
candidate_win_rate=46.0%
95% Wilson interval=[41.2%, 50.9%]
evidence_of_improvement=False
release_ready=False

Other failure signatures worth the same stop rule:

• DPO loss stays near $0.693$ after confirmed updates: audit labels, ties, leakage, and coverage rather than treating init loss as a diagnosis.
• DPO improves tone but still grants admin: inspect whether labels cover the P-7 rule, then check the deployed prompt/template and fresh generations. Add missing policy contrasts when the data audit supports them.
• PPO generations get repetitive while reward rises: freeze promotion, inspect reward scale and KL, then refresh evaluation together.

Online DPO and iterative alignment

Vanilla DPO is offline: it trains on a fixed set of chosen and rejected responses.^{[3]Reference 3Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} Ask what happens when the policy improves past those rows: old comparisons describe a model you no longer have, so they may no longer contain hard negatives.

Later work studies the online setting: sample fresh responses from the current policy, label them with humans or a preference model, then apply an IPO-style or related pairwise update on the new comparisons.^{[14]Reference 14Human Alignment of Large Language Models through Online Preference Optimisation.https://arxiv.org/abs/2403.08635}

The loop is:

1. Sample two or more candidates from the current policy.
2. Ask humans or a preference model which is better.
3. Store a new (prompt, chosen, rejected) triple.
4. Run a DPO-style update with whatever reference-policy rule the method uses.
5. Repeat so the data follows the model.

There isn't one canonical "online DPO" algorithm, and sampling/reference rules differ. Calandriello et al. study current-policy online IPO and IPO-MD, which samples from a mixture with the reference, alongside online DPO variants.^{[14]Reference 14Human Alignment of Large Language Models through Online Preference Optimisation.https://arxiv.org/abs/2403.08635} Refreshing samples can reduce mismatch and expose harder negatives without a critic. A stale or biased judge can still mislabel the new rows.

Nearby methods, not drop-in replacements

These methods make different data contracts. Start with the signal you can reliably collect: a chosen/rejected pair, an unpaired binary label, or an automated task verifier.

Three short calculations show how these data contracts alter the update.

IPO: a finite target, not a hard boundary

Let $m=\log(\pi_\theta(y_w|x)/\pi_{\mathrm{ref}}(y_w|x))-\log(\pi_\theta(y_l|x)/\pi_{\mathrm{ref}}(y_l|x))$. Vanilla DPO's loss $-\log\sigma(\beta m)$ keeps decreasing as $m$ grows. On separable comparisons it can favor ever larger margins. That can concentrate the policy on a narrow set of answers; it does not imply that forward KL must diverge.

IPO instead derives an identity-preference objective whose sampled squared loss is:^{[7]Reference 7A General Theoretical Paradigm to Understand Learning from Human Preferenceshttps://arxiv.org/abs/2310.12036}

The target is $1/(2\tau)$, not $\tau/2$. At $\tau=0.1$, it is 5. The derivative $2(m-5)$ favors increasing a smaller margin and decreasing an overshoot. It doesn't clip margins, bound all model parameters, or guarantee stable neural-network optimization. TRL's beta represents $\tau$ for its IPO loss, with the averaging convention described above.^{[8]Reference 8TRL 1.13.0: DPOTrainer Implementationhttps://github.com/huggingface/trl/blob/3d9261f1fec9f9a8140099c78a65c7da73dce79c/trl/trainer/dpo_trainer.py}

Compare both losses on a complete two-response policy. With logits $[\theta,0]$ and a uniform reference, $m=\theta$. Forward KL is at most $\log 2$, even as the margin tends to infinity.

from math import exp, log, log1p

beta = tau = 0.1
target = 1 / (2 * tau)
print(f"IPO target={target:.1f}")
for margin in [0.0, 5.0, 10.0]:
    p = 1 / (1 + exp(-margin))
    kl = p * log(2 * p) + (1 - p) * log(2 * (1 - p))
    dpo = log1p(exp(-beta * margin))
    ipo = (margin - target) ** 2
    print(f"m={margin:.0f}: DPO={dpo:.3f} IPO={ipo:.1f} KL={kl:.3f}")
    assert -1e-12 <= kl <= log(2) + 1e-12
assert (10 - target) ** 2 > (5 - target) ** 2

IPO target=5.0
m=0: DPO=0.693 IPO=25.0 KL=0.000
m=5: DPO=0.474 IPO=0.0 KL=0.653
m=10: DPO=0.313 IPO=25.0 KL=0.693

KTO: prospect theory on unpaired telemetry

An application may collect one answer and a thumbs-up/down label, without another answer to the same prompt. KTO consumes desirable/undesirable labels directly. These labels must have defined semantics: a click can reflect user satisfaction, formatting, or bias rather than factual correctness.

KTO uses a prospect-theory-inspired value function, not a universal measured law of how people value text. Define $r_\theta(x,y)=\log(\pi_\theta(y|x)/\pi_{\mathrm{ref}}(y|x))$ and a detached reference point $z_0$. Its loss contribution has opposite directions for the two labels:^{[15]Reference 15KTO: Model Alignment as Prospect Theoretic Optimization.https://arxiv.org/abs/2402.01306}

The paper defines $z_0$ using policy/reference KL and implements a biased, non-negative batch estimate from mismatched prompt/response rows. It doesn't backpropagate through that estimate. Both class weights default to 1 in the paper; class balance can justify raising either weight. A larger undesirable weight isn't part of the required definition.

Hold the reference point fixed. If increasing an undesirable answer's log-ratio lowers your implemented loss, you've lost the negative-label branch.

from math import exp

def kto_loss(log_ratio: float, desirable: bool) -> float:
    beta, reference_point = 0.5, 0.2
    direction = 1 if desirable else -1
    logit = direction * beta * (log_ratio - reference_point)
    return 1 - 1 / (1 + exp(-logit))  # both class weights = 1

for label, desirable in [("desirable", True), ("undesirable", False)]:
    before = kto_loss(1.0, desirable)
    after = kto_loss(2.0, desirable)
    print(f"{label}: loss {before:.3f} -> {after:.3f}")
    assert (after < before) == desirable

desirable: loss 0.401 -> 0.289
undesirable: loss 0.599 -> 0.711

This isolates the scalar loss, not a full-model training run. Audit telemetry selection and label quality before interpreting a falling KTO loss as better behavior.

GRPO: group-relative advantage without a critic

The PPO recipe above learns a critic for GAE. The clipped surrogate itself can use other advantage estimators; it doesn't mathematically require a separate 70B value backbone. GRPO, introduced in DeepSeekMath and used in DeepSeek-R1, replaces the learned critic with group statistics.^{[17]Reference 17DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelshttps://arxiv.org/abs/2402.03300[18]Reference 18DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} For outcome supervision on prompt $x$:

1. The policy samples a group of $G$ candidate completions: $\{y_1, y_2, \dots, y_G\} \sim \pi_{\text{old}}(\cdot \mid x)$.
2. A verifier or reward model produces scalar scores $\{r_1,\dots,r_G\}$. Code execution with tests can check selected behaviors; merely compiling doesn't establish correctness.
3. The original outcome formulation centers and scales scores within the prompt group:

1. Assign that outcome advantage to the generated tokens of completion $i$, then use token ratios $\rho_{i,t}=\pi_\theta(y_{i,t}|s_{i,t})/\pi_{\mathrm{old}}(y_{i,t}|s_{i,t})$. A sampled-batch loss for the original token-averaged formulation is:

Here $T_i$ counts generated tokens. DeepSeekMath uses the non-negative sample term $k_{i,t}=\exp(\ell_{i,t})-\ell_{i,t}-1$, with $\ell_{i,t}=\log(\pi_{\mathrm{ref}}/\pi_\theta)$ at that observed token. Its expectation estimates conditional forward KL when the token is sampled from the scored current policy; old-policy samples during repeated updates need the corresponding correction. KL is added to the loss, not mixed into these group reward advantages.^{[17]Reference 17DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelshttps://arxiv.org/abs/2402.03300}

Replacing token ratios with the product across the entire response changes clipping. Even two ratios of 1.1 multiply to 1.21 and trigger a sequence clip when neither token is clipped.

from math import prod

ratios = [1.1, 1.1]
epsilon, advantage = 0.2, 1.0

def clipped_surrogate(ratio: float) -> float:
    clipped = min(max(ratio, 1 - epsilon), 1 + epsilon)
    return min(ratio * advantage, clipped * advantage)

token_average = sum(clipped_surrogate(ratio) for ratio in ratios) / len(ratios)
sequence_value = clipped_surrogate(prod(ratios))
print(f"token-average surrogate={token_average:.3f}")
print(f"whole-response surrogate={sequence_value:.3f}")
assert abs(token_average - 1.1) < 1e-12
assert abs(sequence_value - 1.2) < 1e-12

token-average surrogate=1.100
whole-response surrogate=1.200

This calculation concerns DeepSeekMath's token-ratio formulation. The R1 report presents response-level notation, and current trainers also offer other importance ratios and normalizers; identify the equation and implementation you're using.^{[18]Reference 18DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948[19]Reference 19TRL 1.13.0: GRPOTrainer Implementationhttps://github.com/huggingface/trl/blob/3d9261f1fec9f9a8140099c78a65c7da73dce79c/trl/trainer/grpo_trainer.py}

GRPO removes critic state, computation, and value-fitting error. It still needs rollouts, a useful scoring signal, and stable policy updates. The earlier equal-backbone state example saves 44.4% by removing its critic; that isn't a measured peak-VRAM reduction for another recipe.

There is one edge case worth predicting before inspecting the code: if every response in a group earns the exact same reward, the standard deviation is zero, and every relative advantage collapses to zero.

from statistics import mean, pstdev

def group_advantages(rewards: list[float]) -> tuple[float, float, list[float]]:
    center = mean(rewards)
    scale = pstdev(rewards)
    if scale == 0:
        return center, scale, [0.0 for _ in rewards]
    return center, scale, [(reward - center) / scale for reward in rewards]

reward_groups = {
    "mixed": [1.0, 0.0, 0.5, 1.0],
    "all_equal": [1.0, 1.0, 1.0, 1.0],
}

for name, rewards in reward_groups.items():
    center, scale, advantages = group_advantages(rewards)
    print(
        f"{name}: mean={center:.3f} std={scale:.3f} "
        f"advantages={[round(value, 3) for value in advantages]}"
    )

mixed: mean=0.625 std=0.415 advantages=[0.905, -1.508, -0.302, 0.905]
all_equal: mean=1.000 std=0.000 advantages=[0.0, 0.0, 0.0, 0.0]

The helper uses population standard deviation for transparent arithmetic; trainer conventions and reward-scaling options can differ. If every sampled answer gets the same reward, the centered reward advantages are zero. That removes this group's reward-driven update, not necessarily every gradient: an enabled KL penalty or another auxiliary loss can still contribute.

TRL 1.13.0 logs frac_reward_zero_std from its near-zero standard-deviation check. Under group scaling, a high value can mean all samples pass, all fail, or the scorer can't distinguish them. Under batch scaling, that diagnostic follows the batch scale instead. Inspect the configuration and tasks. This release also offers different token-loss normalizers, and defaults to beta=0.0, so it doesn't load a reference model for KL by default. These choices differ from the original formulation above.^{[20]Reference 20GRPO Trainer.https://huggingface.co/docs/trl/grpo_trainer[19]Reference 19TRL 1.13.0: GRPOTrainer Implementationhttps://github.com/huggingface/trl/blob/3d9261f1fec9f9a8140099c78a65c7da73dce79c/trl/trainer/grpo_trainer.py}

What comes after pairwise human labels

Human comparisons don't scale to every new failure. Constitutional AI uses written principles for model critique/revision, supervised learning on revised answers, and AI preference labels for harmlessness. Its RL stage fits a preference model using those AI labels alongside human helpfulness labels, then uses that scorer as reward. This is one form of RLAIF (Reinforcement Learning from AI Feedback); RLAIF more broadly doesn't require constitutional critique and revision.^{[21]Reference 21Constitutional AI: Harmlessness from AI Feedback.https://arxiv.org/abs/2212.08073} The next chapter examines this pipeline and red teaming.

The dependence doesn't go away. A constitution is only as good as the judge that applies it. Held-out humans still decide whether "cite P-7 and escalate" survived contact with a real user.