RLVR & Verifiable Rewards

The previous chapter trained models against written constitutional principles. An automated critic or human reviewer ranks two responses using those guidelines, but two judges can still disagree, and a written constitution doesn't execute in a sandbox.

Consider a nominal capacity calculation: 14 workers, 12 available slots each, one slot per job, and no other resource constraint. The answer is 168 concurrent jobs. Under that declared arithmetic contract, 168 passes and 148 fails. Predicting real cluster throughput would need additional measurements; this exercise checks multiplication.

Reinforcement Learning with Verifiable Rewards (RLVR) uses executable checks as a policy-training signal.^{[1]Reference 1Tülu 3: Pushing Frontiers in Open Language Model Post-Traininghttps://arxiv.org/abs/2411.15124} A check can establish a declared property of an answer, proof, program, or schema. Deciding whether that property captures the task is the first engineering decision.

In the reward-model/RL form of RLHF, a learned scorer evaluates generated responses. Preferences can cover factual correctness as well as tone and helpfulness. Optimizing an imperfect scorer can amplify shortcuts, but it doesn't inevitably produce sycophancy or jargon. The previous lessons examined that gap between a proxy reward and the behavior we wanted.

An executable checker replaces that scorer, not the need to question the proxy. Passing tests establishes their assertions under their execution conditions; it isn't an exact oracle for all program behavior. Tülu 3 introduced the RLVR label for its post-training phase, building on earlier execution-feedback work. DeepSeek-R1 used rule-based rewards at scale within a multi-stage pipeline.^{[1]Reference 1Tülu 3: Pushing Frontiers in Open Language Model Post-Traininghttps://arxiv.org/abs/2411.15124[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

Keep our 14×12 prompt nearby. A candidate that boxes 168 earns reward; a candidate that boxes 148 gets zero. Whether the written chain of thought was logically sound is a distinct question, and that gap resurfaces whenever the verifier inspects only the final boxed token. The CPU exercises below test verifier contracts and normalized policy updates. They clarify mechanics without demanding a multi-GPU training cluster.

A weak contract might reward seeing \boxed{168} anywhere, even beside a contradictory final answer. That gives optimization a shortcut. Whether a particular run exploits it requires measurement; the loophole itself is already enough reason to fix the checker.

The alignment space

RLVR can follow or accompany SFT and preference optimization. The central question is which supervision your target domain supports, then which learning loop uses it well.

Supervised Fine-Tuning (SFT) fits demonstration targets through cross-entropy loss. For our capacity prompt, a target might supply a worked 14×12 solution. This fixed-target objective doesn't itself sample and verify new answers, though learned behavior can generalize beyond the demonstrations.

Reinforcement Learning from Human Feedback (RLHF)^{[3]Reference 3Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} trains a parameterized reward model on preference rankings. That fits subjective dimensions like tone, conciseness, or helpfulness, but its reliability depends heavily on labeler agreement and reward-model out-of-distribution stability.

Direct Preference Optimization (DPO)^{[4]Reference 4Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} fits chosen/rejected pairs without a separately trained reward model. A verifier can supply their ordering, provided sampling finds distinguishable outcomes. Ordinary offline DPO updates from a fixed dataset; online variants can refresh comparisons. The difference from verifier-reward RL is the objective and update loop, not an exclusive right to use executable supervision.

Useful checkable properties include math answers, tests, JSON schemas, and constraints such as "return exactly three bullet points." These checks remove per-response preference labels for the checked property. People still specify tasks, audit references, and assess properties the checks omit. A sandbox is especially important when the candidate contains executable code.

Ground your pipeline in what the verifier establishes. Tests check selected behaviors; separate checks can measure resource use, formatting, or other requirements. Passing them doesn't establish maintainability. Training stages can combine signals, but SFT isn't limited to formatting, RLVR doesn't guarantee deep reasoning, and DPO isn't limited to etiquette.

Audit task generators and reference targets independently. A wrong gold answer can reject a valid solution and reward the wrong one. In our binary-reward GRPO setup, a sampled all-pass or all-fail group has no comparative reward signal. That can reflect easy/hard tasks, correlated samples, parsing problems, or a broken checker; it doesn't establish that the prompt is impossible.

Defining verifiable rewards

In its minimal outcome-only configuration, a verifiable reward function $r(x, y)$ takes problem $x$ and generated candidate $y$, returning a binary indicator:

This is one outcome-only design. Tülu 3 used success reward $\alpha=10$, failure reward 0, and a separate $-10$ missing-EOS penalty in its PPO setup.^{[1]Reference 1Tülu 3: Pushing Frontiers in Open Language Model Post-Traininghttps://arxiv.org/abs/2411.15124} Other designs use partial test credit, negative rewards, or auxiliary format signals. Specify the composition: positive total reward needn't imply semantic correctness when an auxiliary term can pay on its own.

A tiny arithmetic prompt highlights the blind spot of outcome-only grading: "Is 97 a prime number? Format your answer as $\boxed{Yes}$ or $\boxed{No}$."

• Candidate 1: "It ends in 7, so $\boxed{Yes}$." Our final-field checker awards 1.0, despite the invalid rule.
• Candidate 2: "97 is divisible by 3, so $\boxed{No}$." It awards 0.0 for the incorrect verdict.
• Candidate 3: "None of the primes up to $\sqrt{97}$ divide it: 2, 3, 5, 7. $\boxed{Yes}$." It awards 1.0. We can inspect this argument; the final-field checker doesn't inspect its arithmetic.

Candidates 1 and 3 collect identical credit. An outcome verifier doesn't tell the optimizer whether intermediate steps were valid. A diverse problem set can make brittle shortcuts less useful, but validity of an individual explanation remains a separate property.

Our check validates only the boxed field, treating an unsupported guess and a structured divisibility proof identically:

import re

def verdict_reward(response: str, expected: str) -> float:
    if response.count(r"\boxed") != 1:
        return 0.0
    matches = re.findall(r"\\boxed\{(Yes|No)\}", response)
    return 1.0 if matches == [expected] else 0.0

rollouts = {
    "lucky_reason": r"It ends in 7, so it must be prime. \boxed{Yes}",
    "valid_check": r"Check divisors at most sqrt(97): 2, 3, 5, 7 fail. \boxed{Yes}",
    "wrong_answer": r"97 is divisible by 3. \boxed{No}",
}

for name, response in rollouts.items():
    print(f"{name}: reward={verdict_reward(response, 'Yes')}")

lucky_reason: reward=1.0
valid_check: reward=1.0
wrong_answer: reward=0.0

Many RLVR pipelines employ an all-or-nothing outcome signal because deterministic checkers compute it rapidly. Others introduce auxiliary rule-based terms to reward structure without sacrificing determinism. Sparse credit assignment is the price you pay: if a candidate makes a trivial calculation slip at line 40 of a proof, an outcome checker returns zero despite excellent preceding reasoning.

Checkable isn't complete: Passing a finite test suite establishes its assertions under the tested conditions. Even thorough tests don't prove correctness on every untested input. A formal proof establishes a formal statement under audited assumptions, which may still differ from the user's intended task.

DeepSeek-R1-Zero used accuracy and format rewards to encourage <think>...</think> structure.^{[5]Reference 5DeepSeek-R1: Original January 2025 Report (v1)https://arxiv.org/html/2501.12948v1} A reward encourages behavior; it doesn't guarantee compliance. Its format term also illustrates why not every RLVR design gates all auxiliary credit on a correct answer.

Track extraction, checking, and execution statuses separately. Healthy infrastructure can establish that a candidate exceeded its declared time/memory budget: that is a failed candidate. An unrelated host crash, unavailable prover service, or invalid reference leaves no trustworthy measurement. Quarantine those faults with sample IDs and bounded retries; don't insert them as reward-zero peers. A surge in zeros could mean harder tasks, a parser regression, or changed policy behavior. Zero reward alone also doesn't establish a negative advantage: an all-zero group has none under guarded normalization.

Outcome vs. process supervision

Outcome supervision evaluates the completed artifact: the final boxed integer, the compiled binary, or the terminal test pass rate. RLVR executes this check with deterministic code. Related efforts train learned outcome verifiers to score final answers.^{[6]Reference 6Training Verifiers to Solve Math Word Problems (GSM8K).https://arxiv.org/abs/2110.14168} Credit assignment remains sparse.

Process supervision evaluates intermediate steps. Lightman et al. studied learned process reward models trained on human step labels, evaluating selection among a fixed generator's MATH solutions.^{[7]Reference 7Let's Verify Step by Step.https://arxiv.org/abs/2305.20050} More step labels can support more localized feedback; they don't guarantee dense, correct policy gradients. In formal systems, tactics construct proof terms or new goals. Completing a tactic successfully isn't the same as finishing a valid proof of the intended theorem.

Caption: Outcome supervision scores the final answer. Process supervision supplies step-level feedback, either from labels or from a checker when one exists, at higher construction cost.

Concrete verifier implementations

Mathematics and symbolic algebra

Evaluating mathematical expressions requires normalizing representations across equivalent representations. The verifier below enforces a strict single-marker contract and verifies numeric values using fractions.Fraction, allowing $1/2$ and $0.5$ to match while rejecting missing or duplicated boxes.

Symbolic comparison also needs domains, assumptions, and time limits. Simplification to zero can establish equivalence under those assumptions; failure to simplify isn't automatically proof of inequality. Hugging Face's Math-Verify separates extraction, SymPy representation, and comparison, with configurable tolerance and extraction priorities.^{[8]Reference 8Math-Verify: Extraction, Parsing, and Comparisonhttps://github.com/huggingface/Math-Verify} Its flexible extraction policy differs from our deliberately narrow single-box contract. Neither policy suits every task. If intermediate working legitimately contains several boxes, define a dedicated final-answer field instead of rejecting useful work accidentally.

from fractions import Fraction
import re

def extract_boxed_exprs(text: str) -> list[str]:
    if len(text) > 4096 or text.count(r"\boxed") != 1:
        return []
    marker = "\\boxed{"
    expressions: list[str] = []
    offset = 0
    while (start := text.find(marker, offset)) != -1:
        depth = 0
        expr_chars: list[str] = []
        for ch in text[start + len(marker):]:
            if ch == "{":
                depth += 1
                expr_chars.append(ch)
            elif ch == "}":
                if depth == 0:
                    expressions.append("".join(expr_chars).strip())
                    offset = start + len(marker) + len(expr_chars) + 1
                    break
                depth -= 1
                expr_chars.append(ch)
            else:
                expr_chars.append(ch)
        else:
            return []
    return expressions

def math_verifier(problem: str, answer: str, ground_truth: str) -> float:
    expected_fields = extract_boxed_exprs(ground_truth)
    numeric = r"[+-]?[0-9]+(?:/[0-9]+|\.[0-9]+)?"
    def parse_field(field: str) -> Fraction:
        if len(field) > 64 or re.fullmatch(numeric, field) is None:
            raise ValueError("Unsupported numeric field")
        return Fraction(field)
    try:
        if len(expected_fields) != 1:
            raise ValueError("Reference needs one field")
        expected = parse_field(expected_fields[0])
    except (ValueError, ZeroDivisionError) as error:
        raise ValueError("Invalid reference; quarantine the task") from error
    predicted_fields = extract_boxed_exprs(answer)
    if len(predicted_fields) != 1:
        return 0.0
    try:
        predicted = parse_field(predicted_fields[0])
    except (ValueError, ZeroDivisionError):
        return 0.0
    return 1.0 if predicted == expected else 0.0

equivalent_score = math_verifier("Compute half of one.", r"The answer is \boxed{0.5}.", r"\boxed{1/2}")
wrong_score = math_verifier("Compute 14 times 12.", r"\boxed{148}", r"\boxed{168}")
ambiguous_score = math_verifier("Compute 14 times 12.", r"Maybe \boxed{168} or \boxed{148}.", r"\boxed{168}")
print(equivalent_score)
print(wrong_score)
print(ambiguous_score)
print(f"equivalent_passes={equivalent_score == 1.0}")
print(f"wrong_answer_rejected={wrong_score == 0.0}")
print(f"ambiguous_answer_rejected={ambiguous_score == 0.0}")
assert math_verifier("half", r"\boxed{1/0}", r"\boxed{1/2}") == 0.0
try:
    math_verifier("half", r"\boxed{1/2}", r"\boxed{1/0}")
except ValueError as error:
    assert "quarantine" in str(error)
else:
    raise AssertionError("A broken reference must not become candidate reward zero")

1.0
0.0
0.0
equivalent_passes=True
wrong_answer_rejected=True
ambiguous_answer_rejected=True

Code generation and microVM sandboxing

Testing model-generated software demands strict execution containment. Running untrusted Python code directly on host machines invites system tampering, fork bombs, and network exfiltration of hidden evaluation cases. Interpreter isolation leaves filesystem and kernel access available.

MicroVMs such as Firecracker and application kernels such as gVisor supply stronger execution boundaries.^{[9]Reference 9Firecracker: Production Host Setup Recommendationshttps://github.com/firecracker-microvm/firecracker/blob/main/docs/prod-host-setup.md[10]Reference 10gVisor Security Modelhttps://gvisor.dev/docs/architecture_guide/security/} Their production security depends on configuration and maintained hosts. Configure least privilege, filesystem mounts, network access, resource budgets, and output limits for your chosen backend. A Docker network flag isn't a universal Firecracker setting, and choosing a sandbox doesn't automatically make every mount read-only or hide every test.

Enforce elapsed-time limits as well as CPU budgets. A sleeping process uses little CPU, so a CPU limit alone doesn't bound elapsed time. Production cancellation must terminate the candidate's complete sandbox or process tree, and bound captured output. The exercise below runs only the fixed, trusted strings shown here in local subprocesses. Python -I isolates interpreter environment/import settings; this fixture isn't an execution sandbox for arbitrary model code.

import json
import subprocess
import sys
from dataclasses import dataclass

@dataclass
class TestCase:
    inputs: tuple[int, ...]
    expected: int

def code_verifier(problem: str, code: str, test_cases: list[TestCase]) -> float:
    if not test_cases:
        raise ValueError("A verifier needs test cases")
    for case in test_cases:
        # Candidate process receives inputs, never gold answers or grading authority.
        runner = code + "\nimport json\n" + f"print(json.dumps(solve(*{case.inputs!r})))\n"
        try:
            completed = subprocess.run(
                [sys.executable, "-I", "-c", runner],
                capture_output=True, text=True, timeout=1.0, check=False,
            )
        except subprocess.TimeoutExpired:
            return 0.0
        if completed.returncode != 0:
            return 0.0
        try:
            value = json.loads(completed.stdout)
        except json.JSONDecodeError:
            return 0.0
        # bool is a subclass of int in Python; require an actual integer.
        if type(value) is not int or value != case.expected:
            return 0.0
    return 1.0

cases = [TestCase((14, 12), 168), TestCase((8, 12), 96)]
good = "def solve(a, b):\n    return a * b\n"
bad = "def solve(a, b):\n    return a + b\n"
self_report = 'print(\'{"passed": true}\')\nraise SystemExit(0)\n'
print(f"good_score={code_verifier('multiply', good, cases)}")
print(f"bad_score={code_verifier('multiply', bad, cases)}")
print(f"self_report_score={code_verifier('multiply', self_report, cases)}")

good_score=1.0
bad_score=0.0
self_report_score=0.0

Keep tests used for optimization as training or regression evidence. Use separately held-out tasks/cases to estimate generalization. Hiding test text from the candidate is useful, but repeated scalar reward feedback can still cause adaptive overfitting. Also split related prompts and references, audit pretraining contamination where feasible, and limit final-test reuse. This authored shortcut happens to fail every held-out case below; a small hidden suite doesn't guarantee it catches every shortcut.

from collections.abc import Callable

def shortcut_multiplier(a: int, b: int) -> int:
    known = {(14, 12): 168, (8, 12): 96}
    return known.get((a, b), 0)

def pass_rate(fn: Callable[[int, int], int], cases: list[tuple[int, int, int]]) -> float:
    passed = sum(fn(a, b) == expected for a, b, expected in cases)
    if not cases:
        raise ValueError("An empty suite supplies no correctness evidence")
    return passed / len(cases)

visible = [(14, 12, 168), (8, 12, 96)]
held_out = [(7, 9, 63), (11, 13, 143)]

print(f"visible_pass_rate={pass_rate(shortcut_multiplier, visible):.0%}")
print(f"held_out_pass_rate={pass_rate(shortcut_multiplier, held_out):.0%}")
print(f"ship={pass_rate(shortcut_multiplier, held_out) == 1.0}")

visible_pass_rate=100%
held_out_pass_rate=0%
ship=False

Instruction constraints

Verifiable rewards extend beyond math and code. Tülu 3 used IFEval-derived instruction constraints.^{[1]Reference 1Tülu 3: Pushing Frontiers in Open Language Model Post-Traininghttps://arxiv.org/abs/2411.15124} The grading contract must say what counts as a bullet. Here is a deliberately weak check that counts only lines starting with - :

def weak_constraint_reward(response: str) -> float:
    bullets = [line for line in response.splitlines() if line.startswith("- ")]
    return 1.0 if len(bullets) == 3 else 0.0

def constraint_reward(response: str) -> float:
    lines = response.splitlines()
    return float(len(lines) == 3 and all(line.startswith("- ") and line[2:].strip() for line in lines))

valid = "- name the GPU\n- name the slot count\n- name the job total"
too_short = "- name the GPU\n- name the slot count"
extra_prose = valid + "\nThis extra sentence violates the declared three-line contract."
print(constraint_reward(valid))
print(constraint_reward(too_short))
print(f"three_bullets_pass={constraint_reward(valid) == 1.0}")
print(f"two_bullets_rejected={constraint_reward(too_short) == 0.0}")
print(f"extra_prose: weak={weak_constraint_reward(extra_prose):.0f}; strict={constraint_reward(extra_prose):.0f}")
assert constraint_reward("- \n- \n- ") == 0.0

1.0
0.0
three_bullets_pass=True
two_bullets_rejected=True
extra_prose: weak=1; strict=0

Formal logic and SQL solvers

Interactive theorem provers such as Lean 4, Isabelle, and Coq supply formal checking. Freeze the intended theorem and trusted imports outside candidate control. Audit the completed artifact and assumptions, rather than trusting compilation or a candidate's self-reported success.

Lean 4's #print axioms reports transitive dependencies. sorry uses sorryAx, which can inhabit arbitrary types; a finished proof mustn't rely on it.^{[11]Reference 11Axiomshttps://lean-lang.org/doc/reference/latest/Axioms/} Standard mathematical axioms can be legitimate under your contract. Native evaluation also adds compiler-dependent axioms, so "no literal sorry in the response" and "the file compiled" are insufficient audits.

For SQL, define duplicate, NULL, type, and ordering semantics before comparing results. If order is irrelevant, compare row multisets without dropping duplicates. If order matters, require a specified ordering with deterministic tie handling. SQLite leaves row order undefined without ORDER BY, and equal sort keys don't fix tie order.^{[12]Reference 12SQLite SELECT: Ordering Semanticshttps://sqlite.org/lang_select.html} One matching database snapshot doesn't establish equivalence on every database.

This fixture consumes authored audit records; it doesn't run a prover or inspect real proof terms:

from dataclasses import dataclass

@dataclass(frozen=True)
class ProverResult:
    accepted: bool
    stderr: str
    theorem_matches: bool = True
    axioms: frozenset[str] = frozenset()

def proof_reward(result: ProverResult) -> float:
    allowed = {"Classical.choice", "propext", "Quot.sound"}
    return float(result.accepted and result.theorem_matches and result.axioms <= allowed)

accepted = proof_reward(ProverResult(accepted=True, stderr=""))
rejected = proof_reward(ProverResult(accepted=False, stderr="unknown identifier"))
print(accepted)
print(rejected)
print(f"accepted_maps_to_one={accepted == 1.0}")
print(f"rejected_maps_to_zero={rejected == 0.0}")
stub = ProverResult(True, "", axioms=frozenset({"sorryAx"}))
changed = ProverResult(True, "", theorem_matches=False)
print(f"compiled_sorry_score={proof_reward(stub):.0f}")
print(f"changed_theorem_score={proof_reward(changed):.0f}")
assert proof_reward(ProverResult(True, "", axioms=frozenset({"Classical.choice"}))) == 1.0

1.0
0.0
accepted_maps_to_one=True
rejected_maps_to_zero=True
compiled_sorry_score=0
changed_theorem_score=0

What can't be verified?

An executable check needs a specification. Some properties of an open-ended task are checkable while its overall quality still needs judgment:

• Creative prose: "Draft three catchy newsletter hooks for our product release" has no objective truth.
• Complex business tradeoffs: "Assess whether migrating to Kubernetes reduces total cost of ownership" admits multiple sound arguments.
• Safety and conversational boundaries: authorization decisions and exact synthetic-secret disclosures can be tested programmatically. Broader toxicity, contextual privacy, and nuanced refusals need policy judgments and complementary evaluation. Passing one check doesn't establish the whole boundary.

Group relative policy optimization (GRPO)

DeepSeek-R1 popularized Group Relative Policy Optimization (GRPO), an algorithm developed in DeepSeekMath that eliminates the learned value model by calculating advantages across a group of rollouts generated for the identical prompt.^{[13]Reference 13DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelshttps://arxiv.org/abs/2402.03300} DeepSeek-R1 relied on GRPO throughout its reasoning stages.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} GRPO is an optimization algorithm, not a synonym for RLVR. Tülu 3 conducted its RLVR experiments using standard PPO.^{[1]Reference 1Tülu 3: Pushing Frontiers in Open Language Model Post-Traininghttps://arxiv.org/abs/2411.15124}

What the PPO critic estimates

In standard PPO^{[14]Reference 14Proximal Policy Optimization Algorithms.https://arxiv.org/abs/1707.06347}, a critic (value model $V_\phi(s_t)$) estimates expected future cumulative return from token state $s_t$.

Under terminal reward settings common to LLM post-training, the Monte Carlo advantage with discount 1 is:

$R$ is the terminal reward in this simplified setting, with no intermediate charges. If rewards include tokenwise reference penalties or discounts, use the corresponding remaining return instead. Implementations often use Generalized Advantage Estimation (GAE).

A learned critic adds parameters, optimizer state, activations, and execution work. Its share of peak memory depends on model size, sharding, offload, reference residency, and rollout buffers. Removing it doesn't universally halve VRAM. A critic predicts expected return, not the outcome of every sampled continuation: one later arithmetic error isn't proof that an early expectation was wrong.

The GRPO critic-free mechanism

GRPO removes the separate critic network. Instead of estimating a global value function, it computes advantages by comparing rollouts generated for the exact same prompt in the current batch.

For our 14×12 prompt, correctly extracted 168 earns 1 and 148 earns 0. The group mean supplies a local baseline without a value network. Reward variation supplies the comparative signal; it doesn't ensure an optimizer step improves correctness on new prompts.

For prompt $q$, sample a group of $G$ responses $\{o_1, \dots, o_G\}$. The advantage $A_i$ for rollout $i$ is its normalized z-score within the group:

Positive advantages encourage the sampled actions under the unclipped reward term; negative advantages encourage the opposite direction. Clipping, regularizers, and shared parameters can alter the actual update. Group members share a prompt, though reward-standard-deviation scaling can change how prompts are weighted against one another.

When all rollouts achieve the same score (such as rewards [1, 1, 1, 1] or [0, 0, 0, 0]), the group standard deviation evaluates to zero. Division by zero yields undefined advantages; training engines must mask uniform groups or guard the denominator.

The table below works through $G = 8$ candidate rollouts for our GPU cluster prompt, where boxed 168 receives 1.0 and wrong answers receive 0.0:

Four candidates output 168 ($r=1.0$), and four miss ($r=0.0$). The group mean is $4/8 = 0.5$, and the population standard deviation (dividing by $G$, not $G-1$) is 0.5. Sample 1's advantage evaluates to $(1.0 - 0.5) / 0.5 = +1.0$, encouraging its generated tokens under the active reward surrogate. Sample 2's advantage is $(0.0 - 0.5) / 0.5 = -1.0$, supplying the opposite incentive. These are loss contributions, not guarantees about the final probabilities.

Correct completions share identical positive advantage, and incorrect completions share identical negative advantage. GRPO doesn't understand why sample 1 succeeded; it recognizes only that sample 1 outperformed the group baseline on this prompt.

The GRPO objective applies a PPO-clipped surrogate per generated token, averages over tokens within each response, and averages across the group:^{[13]Reference 13DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelshttps://arxiv.org/abs/2402.03300}

$T_i$ counts generated tokens included in the loss, excluding padding and any additional masked positions. The ratio $\rho_{i,t}=\exp(\log\pi_\theta(o_{i,t}\mid q,o_{i,<t})-\log\pi_{old}(o_{i,t}\mid q,o_{i,<t}))$ compares the training policy with the frozen rollout policy. $\epsilon$ sets clipping thresholds; 0.2 is a common choice, not a hard bound on all probability changes.

DeepSeekMath's sample reference term is $k_{i,t}=\exp(\ell_{i,t})-\ell_{i,t}-1$, where $\ell_{i,t}=\log\pi_{ref}-\log\pi_\theta$ at that token/prefix.^{[13]Reference 13DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelshttps://arxiv.org/abs/2402.03300} Each value is non-negative, but one sample isn't the full distribution's KL. Its expectation equals forward KL under current-policy sampling and shared full support. Rollouts from an older policy need attention to sampling weights and gradient conventions. Modern trainers expose correction choices; $\beta=0$ disables this reference term.

Even an unbiased estimate of the value needn't supply the exact KL gradient when sampled tokens are held fixed during backpropagation. At one fixed prefix, let the current token distribution be $p=(0.8,0.2)$ and reference $q=(0.5,0.5)$. Enumerating tokens with weights $p$ gives the exact KL value. Detaching those weights reproduces the expected gradient from directly differentiating fixed-token sample terms, which differs from differentiating the full categorical KL.

This CPU check isolates that distinction without sampling noise:

import torch

logits = torch.tensor([0.8, 0.2], dtype=torch.float64).log().requires_grad_()
log_p = logits.log_softmax(dim=0)
p = log_p.exp()
log_q = torch.tensor([0.5, 0.5], dtype=torch.float64).log()
log_ratio = log_q - log_p
k = log_ratio.exp() - log_ratio - 1
exact_kl = (p * (log_p - log_q)).sum()
fixed_token_estimate = (p.detach() * k).sum()
enumerated_identity = (p * k).sum()
true_gradient, = torch.autograd.grad(exact_kl, logits, retain_graph=True)
fixed_gradient, = torch.autograd.grad(fixed_token_estimate, logits, retain_graph=True)
identity_gradient, = torch.autograd.grad(enumerated_identity, logits)
assert torch.allclose(exact_kl, fixed_token_estimate)
assert torch.allclose(true_gradient, identity_gradient)
assert not torch.allclose(true_gradient, fixed_gradient)
print(f"both KL values={exact_kl.item():.6f}")
print("full KL gradient:", [round(v, 6) for v in true_gradient.tolist()])
print("fixed-token gradient:", [round(v, 6) for v in fixed_gradient.tolist()])

both KL values=0.192745
full KL gradient: [0.221807, -0.221807]
fixed-token gradient: [0.3, -0.3]

The full expectation changes both token probabilities and per-token terms. A sampled estimator needs the corresponding treatment of that distribution change to recover the full gradient. A training implementation may deliberately use a fixed-token regularizer instead. Record its sampling and differentiation conventions; equal logged KL values don't establish equal policy updates.

For outcome rewards, each included token inherits $A_i$. The displayed objective is DeepSeekMath's token-level form. R1's report uses response-level notation; a raw product of token ratios and tokenwise clipping define different surrogates. DAPO instead divides the accumulated token loss by the total active-token count and separates upper/lower clip thresholds.^{[15]Reference 15DAPO: An Open-Source LLM Reinforcement Learning System at Scalehttps://arxiv.org/abs/2503.14476} Sequence-based variants also exist; identify the formula your implementation actually runs.

For a uniform reward group, guarded normalization sets $A_i=0$. That removes this group's reward-surrogate gradient; enabled KL, entropy, or other terms can still contribute, and other prompts can update shared parameters. DeepSeekMath used $G=64$; more draws can increase the chance of seeing mixed outcomes, not guarantee it.^{[13]Reference 13DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelshttps://arxiv.org/abs/2402.03300} DAPO dynamically filters uniform groups and samples more to fill an informative batch.^{[15]Reference 15DAPO: An Open-Source LLM Reinforcement Learning System at Scalehttps://arxiv.org/abs/2503.14476} This changes task weighting and costs extra generation. Bound the sampling budget and report discarded groups instead of looping indefinitely on a policy that never succeeds.

Variance insight: Three failures and one success produce nonzero reward variance. This supplies nonzero advantages; it doesn't guarantee a nonzero final parameter gradient. Clipped branches, gradient cancellation, or other terms still matter.

Our diagnostic script detects which prompt groups supply comparative gradient signals:

def population_std(values: list[float]) -> float:
    mean = sum(values) / len(values)
    variance = sum((value - mean) ** 2 for value in values) / len(values)
    return variance ** 0.5

groups = {
    "mixed": [0.0, 0.0, 0.0, 1.0],
    "all_fail": [0.0, 0.0, 0.0, 0.0],
    "all_pass": [1.0, 1.0, 1.0, 1.0],
}

for name, rewards in groups.items():
    std = population_std(rewards)
    informative = std > 0.0
    print(f"{name}: std={std:.3f}, informative={informative}")

# Under independent binary outcomes, mixed-group probability is 1-p**G-(1-p)**G.
p, size = 0.01, 64
mixed_probability = 1 - p ** size - (1 - p) ** size
print(f"p={p:.0%}, G={size}: mixed-group chance={mixed_probability:.1%}")
assert 0.47 < mixed_probability < 0.48
assert 1 - 0.0 ** size - (1 - 0.0) ** size == 0

mixed: std=0.433, informative=True
all_fail: std=0.000, informative=False
all_pass: std=0.000, informative=False
p=1%, G=64: mixed-group chance=47.4%

This PyTorch exercise enumerates eight authored categorical outcomes. It doesn't generate text, run a transformer, or perform a complete RLVR training loop. At the initial uniform distribution ($1/8$), its sample reference term and gradient are zero. One small reward update increases rewarded mass in this fixture; four independent scalar probes check clipping branches.

import torch

def group_advantages(rewards):
    if rewards.numel() < 2 or not torch.isfinite(rewards).all():
        raise ValueError("Need at least two finite rewards")
    std = rewards.std(correction=0)
    return torch.zeros_like(rewards) if std == 0 else (rewards - rewards.mean()) / std

def clipped_surrogate(ratio, advantage, epsilon=0.2):
    return torch.minimum(ratio * advantage, ratio.clamp(1-epsilon, 1+epsilon) * advantage)

rewards = torch.tensor([1., 0., 0., 1., 0., 1., 0., 1.], dtype=torch.float64)
advantages = group_advantages(rewards)
logits = torch.zeros(8, dtype=torch.float64, requires_grad=True)
old_log_probs = logits.detach().log_softmax(dim=0)
reference_log_probs = old_log_probs.clone()
log_probs = logits.log_softmax(dim=0)
ratios = (log_probs - old_log_probs).exp()
log_ref_ratio = reference_log_probs - log_probs
kl_samples = log_ref_ratio.exp() - log_ref_ratio - 1
loss = -clipped_surrogate(ratios, advantages).mean() + 0.01 * kl_samples.mean()
loss.backward()
assert torch.allclose(logits.grad, -advantages / 8)
with torch.no_grad():
    logits -= 0.1 * logits.grad
    probabilities = logits.softmax(dim=0)
assert torch.isclose(probabilities.sum(), torch.tensor(1., dtype=torch.float64))
assert (probabilities.log() <= 0).all()
assert probabilities[rewards == 1].sum() > 0.5
assert torch.equal(group_advantages(torch.ones(8)), torch.zeros(8))
# Positive advantages plateau above 1.2; negative ones plateau below 0.8.
for value, advantage, gradient in [(1.3, 1., 0.), (0.7, -1., 0.), (0.7, 1., 1.), (1.3, -1., -1.)]:
    ratio = torch.tensor(value, requires_grad=True)
    clipped_surrogate(ratio, advantage).backward()
    assert ratio.grad.item() == gradient
print("advantages:", advantages.tolist())
print(f"initial_loss={loss.item():.6f}")
print(f"rewarded_probability_before=0.500000; after={probabilities[rewards == 1].sum().item():.6f}")
print("normalization, all-equal rewards, and four clipping branches: PASS")

advantages: [1.0, -1.0, -1.0, 1.0, -1.0, 1.0, -1.0, 1.0]
initial_loss=0.000000
rewarded_probability_before=0.500000; after=0.506250
normalization, all-equal rewards, and four clipping branches: PASS

A trainer name doesn't specify the objective

As checked October 10, 2026, TRL's GRPO guide defaults to loss_type="dapo", scale_rewards="group", token-level importance sampling, and beta=0.0.^{[16]Reference 16GRPO Trainer.https://huggingface.co/docs/trl/grpo_trainer} That default has no loaded reference model or reference-KL term. The guide also supports sequence-based ratios and other normalizers. Our worked population-standard-deviation calculation is an explicit teaching convention, not a claim that every library uses it. Record version, reduction, scaling, masking, clipping, sampling distribution, and KL settings with your result. Switching one can change the experiment even when the class name stays GRPOTrainer.

DeepSeek-R1 training pipeline

DeepSeek-R1's original January 2025 report describes four SFT/RL stages.^{[5]Reference 5DeepSeek-R1: Original January 2025 Report (v1)https://arxiv.org/html/2501.12948v1} This is a historical case study, not a mandatory contemporary recipe. Distinguish it from R1-Zero, which started RL directly from DeepSeek-V3-Base without a preliminary reasoning SFT stage.

The original report's R1-Zero training curve rose from 15.6% to 71.0% on AIME 2024. Its figure estimates single-answer accuracy by averaging 16 sampled responses per question; Table 2 reports 86.7% for consensus over 64 samples.^{[5]Reference 5DeepSeek-R1: Original January 2025 Report (v1)https://arxiv.org/html/2501.12948v1} The expanded report updates the training-curve endpoint to 77.9%.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} These are version-specific reported results. Averaging correctness labels to estimate pass@1 isn't the same operation as voting to choose one answer.

Caption: DeepSeek-R1's reported pipeline starts with readable cold-start behavior, then applies GRPO with rule-based rewards on reasoning tasks, including a language-consistency reward. Stage 2 supplies filtered traces for stage 3, which initializes from the base model and also uses general instruction data. Final RL combines reasoning rewards with general-behavior reward models.

Stage 1: cold start (often useful, not mandatory)

DeepSeek-R1-Zero demonstrated that large base models can discover reasoning behaviors through pure RLVR without initial supervised fine-tuning.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

R1-Zero had readability and language-mixing problems. For R1, the team used several thousand curated traces to fine-tune DeepSeek-V3-Base before RL.^{[5]Reference 5DeepSeek-R1: Original January 2025 Report (v1)https://arxiv.org/html/2501.12948v1} A cold start can increase valid outputs or useful exploration; it doesn't guarantee a format, target language, or nonzero success. Diagnose whether low reward comes from syntax, inadequate capability, task difficulty, or a broken reference before choosing the intervention.

Stage 2: reasoning RL (GRPO)

The original report names coding, mathematics, science, and logical reasoning with well-defined solutions. It adds a language-consistency reward, the proportion of target-language words in the chain of thought, summed with accuracy reward.^{[5]Reference 5DeepSeek-R1: Original January 2025 Report (v1)https://arxiv.org/html/2501.12948v1} The paper reports a slight capability loss in its ablation in exchange for readability. This targets language mixing without guaranteeing its elimination.

Under our outcome-gated design, a wrong final answer earns zero. A composite reward that pays independently for format or language can behave differently; inspect its components.

Stage 3: rejection sampling and SFT

Once the RL policy produces high-accuracy reasoning traces, those rollouts become training data for subsequent supervised runs. DeepSeek-R1 sampled hundreds of thousands of candidate solutions, filtering them through programmatic verifiers and DeepSeek-V3 judges.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

The original report collected about 600k reasoning samples and 200k non-reasoning samples. Some reasoning samples were accepted using DeepSeek-V3 judgments rather than programmatic proof. It fine-tuned DeepSeek-V3-Base for two epochs on the combined 800k; it wasn't simply an SFT continuation from the preceding RL checkpoint.^{[5]Reference 5DeepSeek-R1: Original January 2025 Report (v1)https://arxiv.org/html/2501.12948v1}

Stage 4: final RL

The final stage targets helpfulness and harmlessness alongside reasoning across varied prompts.^{[5]Reference 5DeepSeek-R1: Original January 2025 Report (v1)https://arxiv.org/html/2501.12948v1} Reasoning-only specialization can leave gaps in chat behavior; it doesn't inevitably cause overthinking or loss of tone.

This stage blends two distinct reward streams:

1. Rule-based rewards to preserve competitive math, coding, and logical reasoning accuracy.
2. Preference reward models for general behavior. The original report evaluates helpfulness on the final summary and harmlessness on the whole response, including reasoning. Those rewards target desired behavior; they don't certify safety.

Observed reasoning patterns

Outcome-based RL doesn't use step-level annotations. During DeepSeek-R1-Zero training, responses lengthened significantly while displaying verification, backtracking, and reflection patterns.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} Treat these as emergent output patterns reinforced by terminal rewards.

Self-verification patterns

Trained models frequently pause to audit intermediate calculations:

"Wait, let me double-check that calculation. $14 \times 12$ is $168$, not $148$. I made a mistake."

The expanded report observes reflective words becoming more frequent during RL, including later spikes in “wait.”^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} R1-Zero skipped preliminary reasoning SFT; that doesn't establish that pretraining never included such demonstrations or words. The GPU sentence above is an authored illustration. Word frequency alone doesn't prove a particular token caused correction, nor that the written explanation reveals the model's internal mechanism.

Backtracking patterns

Models frequently abandon unpromising reasoning directions:

"Counting slots as $14 + 12$ seems wrong. Let me multiply instead."

While this mimics tree search, the model remains an autoregressive generator updating token probabilities via group advantage. The verifier credits the final answer; it can't verify whether the pivot was necessary.

Extended thought and the reasoning tax

Response length expanded dramatically during R1-Zero training as accuracy improved.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} This dynamic links RLVR directly to test-time compute.

Longer responses consume generation work and can increase latency and cost, depending on serving and accounting. Measure lengths and utility by prompt class. Compare shortened-budget or thinking-mode settings on your tasks rather than assuming every extra token helps.

Research caution: Shao et al. reported MATH-500 accuracy gains of 21.4 percentage points with random rewards on Qwen2.5-Math-7B, versus 29.1 with ground-truth rewards.^{[17]Reference 17Spurious Rewards: Rethinking Training Signals in RLVRhttps://arxiv.org/abs/2506.10947} The same study generally found weak or harmful spurious-reward gains on tested Llama 3 and OLMo 2 variants. Its clipping ablations matter too. This is model/setup-specific evidence, not an independent replication or proof that reward truth is irrelevant. Compare reward baselines where useful and measure pass@1 and pass@k at stated budgets and sampling settings.

Reward hacking and failure modes

When you optimize a reward metric aggressively, policies exploit any discrepancy between the proxy and real intent. RLVR shifts exploitation from the learned reward model to the deterministic verifier.

Adversarially probe verifiers before starting large RL runs. A policy can learn to exploit parser loopholes or execution quirks that receive reward; optimization doesn't guarantee it discovers every flaw.

Format gaming

Format rewards can overshadow correctness if weighted improperly. Awarding generous credit for wrapping answers in \boxed{...} teaches models to emit beautifully formatted incorrect answers. Parsers that accept arbitrary lines risk rewarding outputs containing multiple conflicting guesses.

For a product contract that gives correctness priority, auxiliary credit mustn't make an incorrect answer outrank a correct one. Gating is one design; weighted independent shaping is another with tradeoffs. Predict what goes wrong in this deliberately broken reward:

import re

def boxed_value(response: str) -> int | None:
    if response.count(r"\boxed") != 1:
        return None
    matches = re.findall(r"\\boxed\{([0-9]{1,20})\}", response)
    return int(matches[0]) if len(matches) == 1 else None

def broken_reward(response: str, expected: int) -> float:
    value = boxed_value(response)
    return 0.7 if value is not None else (0.3 if str(expected) in response else 0.0)

def gated_reward(response: str, expected: int) -> float:
    value = boxed_value(response)
    return 1.0 if value == expected else 0.0

wrong_boxed = r"The answer is \boxed{148}."
right_plain = "The answer is 168."

print(f"broken_prefers_wrong={broken_reward(wrong_boxed, 168) > broken_reward(right_plain, 168)}")
print(f"gated_wrong_boxed={gated_reward(wrong_boxed, 168)}")

broken_prefers_wrong=True
gated_wrong_boxed=0.0

Shortcut exploitation

Biased prompt distributions create trivial shortcuts (such as multiple-choice answers clustering on option C). In coding tasks, candidate code that exploits unisolated environments or returns dummy objects can trick naive test harnesses into reporting passing status.

Mitigation strategies

Defend RLVR training pipelines with these engineering practices:

• Parsing and execution contracts: specify the final-answer field, accepted syntax, resource budgets, and isolation. Under our single-box contract, ambiguity earns zero; malformed gold data remains a separate fault.
• Difficulty and sampling: inspect pass/fail variation, duplicates, and failed measurements. DAPO-style filtering can help, but changes sampling weights and needs a generation budget.^{[15]Reference 15DAPO: An Open-Source LLM Reinforcement Learning System at Scalehttps://arxiv.org/abs/2503.14476}
• Independent evaluation: reserve tasks/cases from optimization and final-test reuse. A second solver can expose disagreement, though correlated bugs can survive both.
• Length shaping: evaluate truncation and utility by length. DAPO's soft overlong penalty grows in a buffer near the maximum; it can also discourage valid long solutions. It removes KL in its reported setup.^{[15]Reference 15DAPO: An Open-Source LLM Reinforcement Learning System at Scalehttps://arxiv.org/abs/2503.14476}

When RLVR breaks

Set evaluation criteria before training. Investigate reward gains paired with held-out or product regressions, using repeated measurements and uncertainty. A small noisy score change doesn't establish a regression; a required gate's confirmed failure does require repair before promotion.

Forcing a checker onto a subjective task

A three-bullet checker can verify format for a creative answer without judging its quality. A mixed task can combine executable and preference signals. State which requirements remain outside the checker rather than claiming a weak rule establishes taste, nuance, or policy compliance.

Treating cold-start SFT as always required or never useful

R1-Zero is an empirical example of RL without preliminary reasoning SFT, not a theorem about every foundation model.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} R1 used a cold start for readability and performance. Near-zero reward warrants a diagnosis: validate the checker first, inspect generated syntax, then compare warmer starts, easier tasks, and sampling settings. SFT can help but isn't guaranteed to create successful exploration.

Celebrating format accuracy

High tag compliance with low answer accuracy shows the two properties differ. Track reward components separately. If your contract requires correctness to outrank formatting, choose a reward composition that enforces that priority rather than inferring competence from tag compliance.

Ignoring general capability drift

Specialization can change instruction following or chat utility; it doesn't inevitably increase false refusal. Broad drift also isn't diagnosed by a reasoning score alone. Compare held-out product evaluations against a comparable baseline. KL regularization, data mixtures, or another training stage are interventions to test, not guarantees of preservation; some reasoning recipes deliberately omit KL.

The following scores and thresholds are authored. This arithmetic gate illustrates a product rule; it doesn't establish statistical significance, actual model quality, or a causal effect of RLVR:

baseline = {"checked_reasoning": 0.61, "instruction_following": 0.92, "false_refusal": 0.04}
candidate = {"checked_reasoning": 0.74, "instruction_following": 0.81, "false_refusal": 0.13}

violations = []
if candidate["checked_reasoning"] <= baseline["checked_reasoning"]:
    violations.append("no reasoning gain")
if candidate["instruction_following"] < baseline["instruction_following"] - 0.03:
    violations.append("instruction following regressed")
if candidate["false_refusal"] > baseline["false_refusal"] + 0.02:
    violations.append("false refusals increased")

print(f"reasoning_gain_pp={100 * (candidate['checked_reasoning'] - baseline['checked_reasoning']):+.0f}")
print(f"ship={not violations}")
print(violations)

reasoning_gain_pp=+13
ship=False
['instruction following regressed', 'false refusals increased']

Treating RLVR and distillation as rivals

RLVR uses executable reward feedback for policy updates. R1's original distillation stage used its approximately 800k curated samples for smaller Qwen/Llama students, including non-reasoning data.^{[5]Reference 5DeepSeek-R1: Original January 2025 Report (v1)https://arxiv.org/html/2501.12948v1} This was response-level supervised training, not an online student reward loop. More general distillation also transfers teacher distributions or other signals.

A tiny verifier lab

To solidify these mechanics, build a standalone verifier. You don't need a compute cluster to understand verification contracts; a Python script and elementary arithmetic problems expose the core dynamics.

Treat verifier modifications with the same rigor as model changes: freeze held-out prompt sets, replay adversarial edge cases, and ensure fail-closed behavior before resuming RL runs.

Checkpoint: Here we require one boxed numeric field. Which errors belong to the candidate, and which mean the reference or checking process must be repaired?

Exercise: construct a verifier that accepts raw model text and returns 1.0 for correct answers and 0.0 otherwise, evaluated on these pairs:

Step 1: Write extract_answer(text: str) -> str | None requiring exactly one \boxed{...} marker. Reject missing or duplicated markers.

Step 2: Write verify(prompt, response, ground_truth, abs_tol="0") returning 1.0 or 0.0. Keep reference and tolerance as bounded decimal strings (or integer references). Job counts use tolerance zero; any nonzero tolerance must come from the task specification.

Step 3: Test your verifier against these three completions:

1. "There are 14 workers and 12 slots each. 14 × 12 = 168. The answer is $\boxed{168}$." (Expected: 1.0)
2. "The total is $\boxed{148}$." (Expected: 0.0)
3. "It might be $\boxed{168}$ or $\boxed{148}$." (Expected: 0.0)

import re
from decimal import Decimal
from fractions import Fraction

def extract_answer(text: str) -> str | None:
    if len(text) > 4096 or text.count(r"\boxed") != 1:
        return None
    boxed = re.findall(r"\\boxed\{(-?[0-9]{1,20}(?:\.[0-9]{1,20})?)\}", text)
    return boxed[0] if len(boxed) == 1 else None

def parse_reference(value: int | str) -> Fraction:
    if type(value) is not int and not isinstance(value, str):
        raise ValueError("Use an integer or bounded decimal string")
    text = str(value)
    if re.fullmatch(r"-?[0-9]{1,20}(?:\.[0-9]{1,20})?", text) is None:
        raise ValueError("Reference is outside the declared numeric contract")
    return Fraction(Decimal(text))

def verify(prompt: str, response: str, ground_truth: int | str, abs_tol: str = "0") -> float:
    # Invalid references raise before evaluating candidate syntax.
    expected, tolerance = parse_reference(ground_truth), parse_reference(abs_tol)
    if tolerance < 0:
        raise ValueError("Tolerance must be nonnegative")
    answer = extract_answer(response)
    if answer is None:
        return 0.0
    # Fraction subtraction avoids Decimal context rounding and binary float error.
    difference = abs(Fraction(Decimal(answer)) - expected)
    return float(difference <= tolerance)

prompt = "A GPU batch runner has 14 workers with 12 slots each. How many concurrent jobs fit?"
correct_score = verify(prompt, r"There are 14 workers and 12 slots each. 14 * 12 = 168. The answer is \boxed{168}.", 168)
wrong_score = verify(prompt, r"The total is \boxed{148}.", 168)
ambiguous_score = verify(prompt, r"It might be \boxed{168} or \boxed{148}.", 168)

print(correct_score)
print(wrong_score)
print(ambiguous_score)
print(f"correct_passes={correct_score == 1.0}")
print(f"wrong_rejected={wrong_score == 0.0}")
print(f"ambiguous_rejected={ambiguous_score == 0.0}")
assert verify("overhead", r"\boxed{50.35}", "50.35") == 1.0
assert verify("overhead", r"\boxed{50.36}", "50.35", "0.01") == 1.0
assert verify("overhead", r"\boxed{50.361}", "50.35", "0.01") == 0.0
assert verify("exact", r"\boxed{12345678901234567890.12345678901234567891}",
              "12345678901234567890.12345678901234567890") == 0.0
try:
    verify("reference fault", "malformed candidate", "NaN")
except ValueError:
    pass
else:
    raise AssertionError("Invalid references need a separate fault, not reward zero")

1.0
0.0
0.0
correct_passes=True
wrong_rejected=True
ambiguous_rejected=True

RLVR and distillation

RLVR uses reward feedback; distillation supplies teacher-derived supervision. R1 combined these interventions. Its 800k-sample distillation mix included roughly 600k reasoning samples and 200k general samples in the original report.^{[5]Reference 5DeepSeek-R1: Original January 2025 Report (v1)https://arxiv.org/html/2501.12948v1} It wasn't 800k programmatically proven reasoning traces.

In the original report's Table 6, the distilled Qwen2.5-32B scored 72.6% AIME 2024 pass@1 versus 47.0% for its directly RL-trained 32B comparison.^{[5]Reference 5DeepSeek-R1: Original January 2025 Report (v1)https://arxiv.org/html/2501.12948v1} This is evidence for that teacher/data/training setup, not a budget-matched theorem that distillation always wins. A weak teacher can limit transferred examples; RL exploration can also fail when the policy cannot reach passing outcomes. Compare both interventions with held-out tasks and complete compute accounting.