RLVR & Verifiable Rewards

The last chapter trained against written constitutional principles. A critic can rank two answers using those rules, but two reviewers can still disagree, and a constitution doesn't execute.

Some targets don't need that kind of judge. Ask a model how many concurrent jobs fit on 14 GPU workers with 12 slots each. You don't need a preference label. You need 168.

Reinforcement Learning with Verifiable Rewards (RLVR) trains from rewards assigned by executable checks rather than a learned preference model.^{[1]Reference 1Tülu 3: Pushing Frontiers in Open Language Model Post-Traininghttps://arxiv.org/abs/2411.15124} Use it when an answer, program, schema, or constraint can be tested against a precise contract.

Two outputs show the boundary. One drafts an incident update, where tone and completeness are partly subjective. The other writes a Python function that either passes unit tests or fails them. Tülu 3 introduced the name RLVR for this post-training stage. DeepSeek-R1 used the same idea under the label rule-based rewards inside a broader multi-stage pipeline.^{[1]Reference 1Tülu 3: Pushing Frontiers in Open Language Model Post-Traininghttps://arxiv.org/abs/2411.15124[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

Keep the 14×12 prompt nearby. A candidate that boxes 168 can earn reward. A candidate that boxes 148 can't. Whether the written reasoning was valid is a separate question, and that gap comes back whenever the verifier only sees the boxed number.

A verifier can fail in the opposite direction. If it awards a point for seeing \boxed{168} anywhere, a model can learn to print the token without solving the problem. Training reward rises; held-out correctness stays flat. The checker, not the prose around it, defines the pressure.

The alignment space

RLVR sits in the post-training stack next to SFT, RLHF, and DPO. Large language models usually learn instruction following from demonstrations and preference signals first. The design question is which of those signals the task can honestly provide.

Supervised Fine-Tuning (SFT) imitates selected demonstrations. For the GPU prompt, SFT can show a worked 14×12 solution. It copies patterns in those traces. It doesn't score a newly sampled 168 against an external checker.

Reinforcement Learning from Human Feedback (RLHF)^{[3]Reference 3Training Language Models to Follow Instructions with Human Feedback (InstructGPT).https://arxiv.org/abs/2203.02155} trains a reward model to predict preference labels. That fits tone or helpfulness, and it depends on label collection plus reward-model validity outside the labeled set.

Direct Preference Optimization (DPO)^{[4]Reference 4Direct Preference Optimization: Your Language Model is Secretly a Reward Model.https://arxiv.org/abs/2305.18290} removes the online reward-model RL loop and trains from chosen/rejected pairs. Those pairs can come from humans, constitutions, or other pipelines. They're still preference comparisons, not executed correctness checks.

RLVR specializes in tasks where success can be programmatically verified: a final math answer, code under tests, a JSON schema, or an instruction constraint such as "exactly three bullet points." You skip a preference label for each rollout. Validity is only as strong as the specification, parser, test suite, and sandbox that assign reward.

The trade is breadth for a directly executable signal. Start from what this checker can prove. A coding lab can verify unit tests, type checks, and exact-output constraints. It still needs a human for whether the solution is maintainable or clear.

That contract starts before sampling. A task generator needs checkable answers and a difficulty range that produces both passes and failures. An easy prompt gives every rollout the same reward; an impossible prompt gives every rollout zero. In either case, group-relative optimization has nothing to compare.

Defining verifiable rewards

In the simplest outcome-only setup, a verifiable reward function $r(x, y)$ takes a problem $x$ and a generated response $y$ and returns a binary signal:

$$r(x, y) = \begin{cases} 1 & \text{if } y \text{ is correct (verified)} \\ 0 & \text{if } y \text{ is incorrect} \end{cases}$$

That's the teaching form. Tülu 3 used the same shape with a constant success reward $\alpha = 10$ chosen in pilots and then left fixed, plus a $-10$ penalty when generation hit the length cap without an end-of-sequence token.^{[1]Reference 1Tülu 3: Pushing Frontiers in Open Language Model Post-Traininghttps://arxiv.org/abs/2411.15124} The scale is a hyperparameter. The contract isn't: only verifiably correct completions get the success reward.

A second tiny prompt makes the outcome-only blind spot obvious: "Is 97 a prime number? Format your answer as $\boxed{Yes}$ or $\boxed{No}$."

Why is a binary verifier attractive for RLVR?

Answer

It removes ambiguity from the reward signal. If code, tests, or a proof checker can mark outputs right or wrong, training doesn't need a human preference model for every sample.

• Iteration 1: "Yes, because it ends in 7." The final answer happens to be correct, so an outcome-only verifier returns 1.0, even though the stated reason is invalid.
• Iteration 2: "No, 97/3 = 32.3." The answer is wrong, so the verifier returns 0.0.
• Iteration 3: "97 is prime because it's not divisible by 2, 3, 5, or 7." The answer is correct and the reasoning is solid, so the verifier returns 1.0.

Iterations 1 and 3 get the same reward. An outcome verifier does not tell the optimizer which correct answer used valid reasoning. Across many problems, reasoning patterns that correlate with correct outcomes may become more likely, but that's an empirical training result, not information contained in a single final-answer reward.

This check verifies only the boxed verdict, so an unsupported guess and a valid divisibility check receive identical reward.

Many RLVR systems start with an all-or-nothing outcome signal because a deterministic checker can compute it repeatedly. Others add rule-based terms, so reward stays executable without being purely binary. Sparse credit assignment is the cost: if a response makes one late arithmetic error, a final-answer checker gives zero even if earlier steps were useful.

Checkable isn't complete: Verifiable means checkable, not complete. A unit-test verifier can prove whether one generated function matches the tested cases. It can't prove that a correct answer came from a general strategy unless the evaluation tests that generalization.

In practice, systems often mix correctness rewards with other rule-based terms. DeepSeek-R1-Zero combined accuracy rewards with format rewards so the thinking process sat in parseable tags before the final answer.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

Read verification as its own pipeline: extract, normalize, check, then record a reason for rejection. A rising zero-rate can mean harder prompts, parser drift, or genuine model regression. Without failure categories, those causes look identical in a training curve.

Outcome vs. process supervision

The simplest form is outcome supervision: score the final answer. An RLVR setup can do this with a rule-based checker. Related work has also trained learned verifiers to score final answers.^{[5]Reference 5Training Verifiers to Solve Math Word Problems (GSM8K).https://arxiv.org/abs/2110.14168} Feedback is sparse.

Process supervision scores intermediate steps instead of only the final result. Lightman et al. compare outcome and process reward models trained with human labels on mathematical reasoning steps. That's adjacent to RLVR, not itself proof that each process score is programmatically verifiable.^{[6]Reference 6Let's Verify Step by Step.https://arxiv.org/abs/2305.20050} A formal system can instead run a checker after each proof step. Either version offers more localized credit, but it needs a trustworthy step-level signal.

Caption: Outcome supervision scores the final answer. Process supervision supplies step-level feedback, either from labels or from a checker when one exists, at higher construction cost.

Examples of verifiers

Mathematics

Check whether the extracted final answer matches ground truth, even if the reasoning path is different. The verifier below requires exactly one boxed field and compares values with fractions.Fraction, so $1/2$ and $0.5$ match while missing or duplicated boxes fail closed.

This is small enough to run locally. Production math verifiers add more normalization, a computer-algebra fallback, timeouts, and adversarial parser tests. Don't treat Fraction as a full symbolic simplifier.

Code generation

Running generated code against a strong suite of hidden tests checks concrete behavior. It doesn't prove full correctness unless the suite is complete, but it's stronger than judging code by surface form. The local example runs candidate code in a separate Python process and times out slow attempts. It isn't a security sandbox. Production systems still need containers, seccomp, filesystem isolation, network controls, and resource limits.

Visible tests aren't enough when the policy can memorize examples. Here a candidate passes the two development cases and fails a held-out pair. A reward based only on visible tests would reinforce the wrong program.

Instruction constraints

Not every verifiable task is math or code. Tülu 3 also trained RLVR on precise instruction-following prompts whose constraints a program can check, such as required bullet counts or word limits in IFEval-style tasks.^{[1]Reference 1Tülu 3: Pushing Frontiers in Open Language Model Post-Traininghttps://arxiv.org/abs/2411.15124} The reward is still fail closed: if the constraint checker can't confirm the contract, the score is 0.

Formal logic

Proof assistants such as Lean or Isabelle can check that a candidate proof term satisfies a formal theorem under their kernel and imported definitions. In production, a verifier wraps the theorem statement and candidate proof into a script, then returns 1.0 only if the prover accepts the full artifact.

The prover call is infrastructure-specific: Lean version, package cache, sandbox, timeout. Keep the reward conversion boring and testable, and keep the theorem-prover runner behind a clear boundary.

What can't be verified?

RLVR is effective but narrow. It works best when the task has an objective spec and a verifier that fails closed. That rules out many open-ended tasks, or at least forces you to break them into smaller verifiable sub-problems. It's not a clean fit for:

• Open-ended writing: "Write three status-page headlines for an outage" has no objective truth.
• Subjective analysis: "Explain the trust impact of a release delay" has many valid answers.
• Safety alignment: deciding if a response is harmful usually requires policy judgment or an AI proxy, which is closer to RLHF or RLAIF than a deterministic correctness check.

Why must an RLVR verifier fail closed?

Answer

If parsing errors, timeouts, or malformed outputs accidentally receive reward, training will amplify those loopholes. A failed verifier should return 0 or no reward, not a best-effort pass.

Group relative policy optimization (GRPO)

DeepSeek-R1 used Group Relative Policy Optimization (GRPO), an algorithm introduced in DeepSeekMath that avoids a learned value model by estimating advantages from a group of samples for the same prompt.^{[7]Reference 7DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelshttps://arxiv.org/abs/2402.03300} DeepSeek-R1 later used GRPO in its reasoning-training stages.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} GRPO is an optimizer, not a synonym for RLVR. Tülu 3 trained its RLVR stage with PPO.^{[1]Reference 1Tülu 3: Pushing Frontiers in Open Language Model Post-Traininghttps://arxiv.org/abs/2411.15124}

The problem with PPO

In standard PPO^{[8]Reference 8Proximal Policy Optimization Algorithms.https://arxiv.org/abs/1707.06347}, a critic (value function) predicts how good a state is. The critic estimates expected future reward $V_\phi(s_t)$.

In the terminal-reward setting common to LLM training, where reward is observed only at the end of a sequence, the advantage often simplifies to:

$$A_t = R - V_\phi(s_t)$$

$R$ is the final reward and $V_\phi$ is the critic's value estimate. More generally, the advantage uses a temporal-difference error $\delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t)$, often computed via Generalized Advantage Estimation over the full sequence.

Training a critic alongside an LLM adds model memory and compute. With terminal rewards, it must also predict eventual success from partial generations, which gets hard when a long response can later revise an earlier mistake.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

The GRPO solution

GRPO drops the separate critic. Instead of learning a value function that tries to predict how good any reasoning trajectory is, it computes advantages relative to the other samples generated for the exact same prompt in the current batch.

For the 14×12 prompt, sample several outputs. Some reach 168 and receive high reward. Others miss it. The successful outputs are better than average for this sampled group. You don't need a learned critic that estimates difficulty across prompts. You do need reward variation inside each group.

For a prompt $q$ you sample a group of $G$ outputs $\{o_1, \dots, o_G\}$. The advantage for output $i$ is the z-score of its reward within that group:

$$A_i = \frac{r_i - \text{mean}(\{r_1, \dots, r_G\})}{\text{std}(\{r_1, \dots, r_G\})}$$

Positive advantages push the policy toward those trajectories. Negative advantages push it away. Local normalization compares attempts at the same prompt difficulty. It doesn't guarantee stable training: if every sampled output gets the same reward, the group contributes no comparative signal.

Handle that boundary before dividing. For rewards [1, 1, 1, 1] or [0, 0, 0, 0], the group standard deviation is zero, so the literal formula becomes 0 / 0; production implementations must mask the group, use a guarded denominator, or apply another explicitly defined normalization policy.

What does a positive GRPO advantage mean?

Answer

That sample scored above the group average for the same prompt. The optimizer should increase probability of the tokens in that trajectory relative to worse samples.

This concrete example uses $G = 8$ samples for the GPU prompt. The verifier returns 1.0 for boxed 168 and 0.0 otherwise:

Four samples boxed 168, so their rewards are 1.0. Four missed, so their rewards are 0.0. The group mean is $4/8 = 0.5$, and the population standard deviation (divide by $G$, not $G-1$) is 0.5. Sample 1's advantage is $(1.0 - 0.5) / 0.5 = +1.0$, so the optimizer increases the probability of the tokens that led there. Sample 2's advantage is $(0.0 - 0.5) / 0.5 = -1.0$, so those tokens get pushed down.

Every correct answer gets the same positive advantage, and every wrong answer gets the same negative advantage. GRPO doesn't know why sample 1 was correct. It only knows that sample 1 beat the average for this prompt. Over many prompts, the policy learns which reasoning patterns reliably produce above-average rewards.

The update resembles a PPO-clipped objective with group-relative advantage and a KL penalty. DeepSeek-R1 writes a sequence-level sketch.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} DeepSeekMath applies the clipped surrogate per generated token, averages inside each sequence, then averages across the group.^{[7]Reference 7DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelshttps://arxiv.org/abs/2402.03300} Keep the sequence-level form for intuition, then treat token-level ratios as the implementation contract:

$$\begin{aligned} \mathcal{L}_{\text{GRPO}} = -\frac{1}{G} \sum_{i=1}^{G} \Big[ &\min\left(\rho_i A_i,\; \text{clip}(\rho_i, 1-\epsilon, 1+\epsilon) A_i\right) \\ &- \beta D_{\text{KL}}(\pi_\theta \|\pi_{\text{ref}}) \Big] \end{aligned}$$

Here $\rho_i = \frac{\pi_\theta(o_i|q)}{\pi_{\text{old}}(o_i|q)}$ is a sequence-level policy-ratio shorthand, $A_i$ is the normalized group advantage, $\epsilon$ is the clipping threshold, and $\beta$ controls the drift penalty toward the reference policy.

Implementation contract: during rollout, store old-policy token log-probs. At train time, recompute new-policy token log-probs, form per-token ratios, share the same group advantage $A_i$ across tokens of that sample under outcome supervision, and mask padding. DAPO later argued that averaging loss first by sequence (sample-level) underweights long traces and proposed token-level loss, plus a higher policy-ratio clip on the upper bound to slow entropy collapse.^{[9]Reference 9DAPO: An Open-Source LLM Reinforcement Learning System at Scalehttps://arxiv.org/abs/2503.14476} Reimplementing GRPO from the sequence-level sketch alone will mis-state gradient scale and clipping.

The Kullback-Leibler (KL) term penalizes drift from a reference policy. It can constrain movement toward a verifier exploit. It can't prove that the verifier represents the intended behavior.

If every sample in the group gets the same reward, the standard deviation is 0 and vanilla GRPO gets no learning signal from that group. That's why prompt difficulty, group size, and reward shaping matter. DeepSeekMath reported $G = 64$ for its math RL run.^{[7]Reference 7DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelshttps://arxiv.org/abs/2402.03300} DeepSeek-R1's Figure 2 samples 16 responses per AIME question to stabilize evaluation averages. That isn't a published training group size $G$.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} DAPO later used 16 training rollouts per prompt and dynamic sampling: drop all-pass and all-fail groups and keep sampling until the batch has mixed outcomes.^{[9]Reference 9DAPO: An Open-Source LLM Reinforcement Learning System at Scalehttps://arxiv.org/abs/2503.14476}

Common mistake: Saying that three equal rewards out of four collapse the GRPO signal. They don't: the one different outcome creates reward variance. Signal disappears when every sampled output gets the same reward, such as all failures on a prompt that's too hard or all passes on a prompt that's too easy.

This diagnostic identifies which prompt groups supply comparative signal before an update. The mixed group is usable even though three out of four answers fail. The all-fail and all-pass groups carry no relative ranking information.

The next snippet is a scalar sketch of the GRPO math: each sampled trajectory is one log-probability, which is enough to test group-normalized advantage, PPO-style clipping, and the KL estimator published with GRPO, $\exp(\log \pi_{\text{ref}} - \log \pi_\theta) - (\log \pi_{\text{ref}} - \log \pi_\theta) - 1$.^{[7]Reference 7DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelshttps://arxiv.org/abs/2402.03300} A production trainer applies the objective at token level and stores old-policy log probabilities during rollout. Use population std (divide by $G$). PyTorch's default std divides by $G-1$ and will shrink $|A_i|$ on small groups unless you set it to unbiased/population mode.

Before the optimizer step, predict its sign. A positive advantage should increase a sampled trajectory's log-probability; a negative advantage should decrease it; an all-equal group should do neither. The scalar lab checks those three cases.

At initialization every ratio is 1, the clip is inactive, and the KL term is 0, so one SGD step on the mean clipped surrogate moves each log-probability by $\eta A_i / G$.

The batch loss is 0 at this initialization because the four $+1$ and four $-1$ advantages cancel in the mean. The update still moves: each trajectory's log-probability changes by $\eta A_i / G$. GRPO is an optimizer, not a synonym for RLVR. Tülu 3 trained its RLVR stage with PPO. DeepSeekMath introduced GRPO as a PPO variant, and DeepSeek-R1 used GRPO in its reasoning stages.^{[1]Reference 1Tülu 3: Pushing Frontiers in Open Language Model Post-Traininghttps://arxiv.org/abs/2411.15124[7]Reference 7DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelshttps://arxiv.org/abs/2402.03300[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} Both optimizers can consume rule-based verifier rewards.

Self-check: In the worked table above, what would happen to the advantages if all 8 samples got the same reward? Why does that make prompt difficulty and group size important in practice?

GRPO often fits RLVR for three engineering reasons. There's no critic to hold in GPU memory. The group mean is local to the current prompt, so you skip training a value head to predict returns from partial reasoning traces. Binary $\{0,1\}$ rewards are easy to read: when a group contains both passes and failures, passes get positive advantage and failures negative. If all candidates pass or all fail, advantages are zero, so prompt selection and exploration still have to produce mixed outcomes often enough to learn.

DeepSeek-R1 training pipeline

RLVR success depends on the pipeline as well as the algorithm. DeepSeek-R1 used a multi-stage process that alternates supervised fine-tuning and reinforcement learning to improve checked reasoning, readability, and broader assistant behavior.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

On AIME 2024, DeepSeek-R1-Zero's reported pass@1 rose from 15.6% to 77.9% during rule-based RL from DeepSeek-V3-Base. Figure 1 also reports 86.7% with consistency over 16 samples.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} Those numbers are the empirical stake for the pipeline below. They aren't a promise that every base model and verifier will move the same way.

Caption: DeepSeek-R1's reported pipeline starts with readable cold-start behavior, then applies GRPO with rule-based rewards on reasoning tasks, including a language-consistency reward. Later stages collect filtered traces into SFT data, mix in broader instruction data, and combine reasoning rewards with general-behavior reward models.

Stage 1: cold start (not mandatory, often useful)

DeepSeek-R1-Zero showed that DeepSeek-V3-Base could improve on reported reasoning evaluations through rule-based RL without a preliminary SFT phase.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} That result doesn't mean every base model or verifier will train usefully from zero-reward-heavy rollouts.

DeepSeek still added cold-start data for DeepSeek-R1 because R1-Zero had poor readability and mixed languages. The paper describes collecting thousands of long-CoT examples, filtering for a readable response pattern, and using them to initialize the model before RL.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} In other projects, cold start also helps when the base model's initial pass rate is so low that almost every rollout gets zero reward.

Stage 2: reasoning RL (GRPO)

This stage trains on math, code, and logic prompts using GRPO with rule-based rewards. In DeepSeek-R1-Zero, those rewards were accuracy plus format. In DeepSeek-R1, the first RL stage also added a language-consistency reward (the share of target-language words in the chain of thought) to reduce mixed-language traces, with the paper reporting a slight performance trade-off in its ablation.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

For rule-graded prompts, a response that fails the checked outcome doesn't receive the accuracy reward no matter how persuasive its text is. That still leaves coverage risk: a pass can reflect a shortcut if the verifier or evaluation set fails to test it.

Stage 3: rejection sampling and SFT

Once the RL policy is producing higher-scoring traces, they can become new demonstrations. DeepSeek-R1 used rejection sampling, retaining checked-correct responses for rule-gradable data and using DeepSeek-V3 judgments for some expanded reasoning data.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

That stage produced about 600k reasoning samples plus about 200k non-reasoning samples, for roughly 800k total.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} This turns exploratory RL behavior into a stable supervised dataset and helps recover capabilities that pure reasoning RL doesn't optimize for, such as writing quality and everyday instruction following.

Stage 4: final RL

Pure reasoning RL can produce a model that's strong on benchmarks but rough around the edges. In the final stage, DeepSeek runs RL again, this time on a broader mix of scenarios.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

This uses a mix of:

1. Rule-based rewards to maintain strong reasoning capabilities.
2. Reward models (RLHF) for helpfulness and harmlessness in general conversation.

This final stage is intended to combine reasoning performance with broader helpfulness and harmlessness objectives. Those outcomes still have to be measured rather than assumed.

Observed reasoning patterns

Outcome-checked RL doesn't label each reasoning step. In DeepSeek-R1-Zero, the authors report longer responses and visible patterns such as verification, reflection, and exploration of alternatives during training.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} Treat those as observed output behaviors, not proof of an internal reasoning mechanism.

Self-verification patterns

An output may pause and check its work.

"Wait, let me double-check that calculation. $14 \times 12$ is $168$, not $148$. I made a mistake."

DeepSeek-R1-Zero wasn't trained from step-level labels saying "write let me check here." Its paper reports a sharp increase in the use of "wait" during reflection later in training.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} The cautious interpretation is that RL changed the distribution of generated traces and that some reflective-looking traces co-occurred with improved checked outcomes. An outcome-only verifier doesn't establish why those traces improved.

Backtracking patterns

An output may abandon one approach and try another.

"Counting slots as $14 + 12$ seems wrong. Let me multiply instead."

This can look search-like on the surface, but the training loop is still autoregressive generation plus reward-weighted updates, not an explicit tree-search controller. A verifier credits the final checked outcome. It doesn't separately establish that the pivot was necessary.

Extended thought

In DeepSeek-R1-Zero, average response length jumped sharply during training alongside reported accuracy gains.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} This connects RLVR to test-time compute because trained policies may emit longer traces on reasoning tasks. Longer traces aren't automatically better. Their value has to be measured against correctness, latency, and token cost.

A useful self-check: suppose a model trained with outcome rewards starts saying "Wait, let me double-check that" before finalizing answers. What can you conclude from the output pattern, and what would require separate evaluation? The reward confirms correct final answers, not the necessity or faithfulness of the written reflection.

Research caution: It remains open how much RLVR creates new problem-solving behavior versus eliciting behavior already likely under the base model. Shao et al. report that on Qwen2.5-Math-7B, MATH-500 rose 21.4 points with random rewards and 13.8 points with format-only rewards, versus 29.1 points with ground-truth rewards. Comparable spurious rewards gave little benefit or harmed Llama3 and OLMo2 variants.^{[10]Reference 10Spurious Rewards: Rethinking Training Signals in RLVRhttps://arxiv.org/abs/2506.10947} They connect this result to GRPO clipping and model-specific high-prior behaviors. Practical takeaway: compare against weak or spurious-reward baselines and validate across model families and held-out tasks. Track pass@1 and pass@k. A rising training reward with flat held-out pass@k is usually hacking, memorization, or probability mass shifting among traces the base model could already sample.

Reward hacking and failure modes

Any time you optimize a metric, a policy can exploit gaps in that metric. RLVR is no exception. If the verifier checks only final answers, a memorized answer, leaked target, or weak parser can receive reward without demonstrating general solution ability. DeepSeek-R1 explicitly identifies reliable reward construction as a limitation once tasks can't be graded by dependable rules.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

Don't train against a verifier before adversarially testing it. Any malformed output or shortcut that earns reward can be reinforced by the optimization loop.

Format gaming

A format-heavy reward can favor output wrappers over correctness. If the verifier gives substantial credit for \boxed{...} before checking the value, it can reinforce neatly formatted wrong answers. A parser that accepts any matching line also risks rewarding output that contains several contradictory answers.

High format compliance with low task accuracy means the reward function prizes formatting before correctness. Make correctness dominant and fail closed on wrong or ambiguous contents, even if the wrapper is present.

This example compares a broken format-first reward with a correctness-gated version. A wrong boxed 148 should never beat an unboxed 168 just because it's easy to parse.

Shortcut exploitation

If a training set has shortcuts (for example, one multiple-choice position is correct much more often), optimizing checked training reward can favor that shortcut. A code verifier that treats execution errors as passing outcomes creates an even more direct loophole.

Mitigation strategies

To prevent policies from exploiting the verifier, several defensive practices help during training:

• Explicit verifier contracts: use isolated execution and fail closed on any parsing error. Symbolic checking belongs here when the answer isn't a plain integer.
• Difficulty calibration: include prompts where rollouts produce both passing and failing outputs often enough for group-relative learning. All-zero groups contribute no comparative advantage. DAPO-style dynamic sampling filters those groups out of the update batch.^{[9]Reference 9DAPO: An Open-Source LLM Reinforcement Learning System at Scalehttps://arxiv.org/abs/2503.14476}
• Held-out tests: evaluate on independent problems and, when possible, an independently implemented checker so a parser shortcut isn't mistaken for generalization.
• Length controls: cap rollout length or add carefully tuned penalties so the model doesn't learn to think forever without improving correctness. DAPO adds a soft overlong punishment near the generation cap rather than a hard truncate-and-fail that can punish otherwise valid traces.^{[9]Reference 9DAPO: An Open-Source LLM Reinforcement Learning System at Scalehttps://arxiv.org/abs/2503.14476}

When RLVR breaks

Set an evidence gate before training. Pause if training reward rises while held-out pass@k, format validity, or broad instruction metrics fall. Inspect verifier coverage, leakage, and reward components before adding steps or widening the task mix.

Forcing a checker onto a subjective task

A reward function that secretly depends on taste, safety judgment, or fuzzy grader text is forcing an objective RL method onto a subjective task. Ask whether a deterministic checker, theorem prover, unit test suite, database constraint, or schema validator can grade the output. If not, use RLHF, DPO, Reinforcement Learning from AI Feedback (RLAIF), or decompose the task into smaller verifiable checks.

Treating cold-start SFT as always required or never useful

DeepSeek-R1-Zero improved reported reasoning-task results from DeepSeek-V3-Base without a preliminary SFT phase.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} DeepSeek-R1 still used cold-start SFT because readability and language consistency mattered. Measure the base model's initial pass rate and output quality. If almost every rollout gets zero reward or the traces are unreadable, cold-start data may make RL more tractable.

Celebrating format accuracy

A verifier that rewards \boxed{} too strongly can teach box-writing instead of problem-solving. Keep correctness dominant, run adversarial parser tests, and track real task accuracy separately from format compliance.

Ignoring general capability drift

A model trained heavily on math or code RLVR can become better at those tasks while getting worse at normal instruction following. Use KL anchoring, mix in general instruction data during later SFT stages, and run broad evaluations such as writing, safety, and everyday chat alongside math or code metrics.

This release gate catches a reasoning gain that comes with an unacceptable instruction-following regression. The numbers are illustrative. Each team should define thresholds before training.

Treating RLVR and distillation as rivals

RLVR can improve checked outcomes for a teacher. Distillation transfers sampled teacher behavior into a smaller model. Decide which bottleneck you have: if the teacher fails verifiable evaluations, train against better checks; if a capable teacher is too expensive to serve, consider distillation.

A tiny verifier lab

To understand RLVR, build a verifier yourself. You don't need a GPU cluster or a billion-parameter model. A Python script and a small set of arithmetic problems are enough to see the mechanics.

Treat a verifier edit like a model edit. Freeze held-out prompts, replay malformed and adversarial outputs, compare failure categories, and resume RL only after checker behavior is stable.

Checkpoint: Before building the verifier, state the exact contract: require one answer field, extract one number, compare with tolerance, and fail closed on missing, duplicated, or malformed fields. If that contract feels vague, re-read the math verifier example above.

Exercise: write a verifier that takes a model's raw text output and returns 1.0 if the answer is correct and 0.0 otherwise. Use the following prompt and ground-truth pairs:

Step 1: Implement extract_answer(text: str) -> str | None that accepts exactly one \boxed{...} answer field. Reject missing or duplicated fields rather than guessing which number the model intended as final.

Step 2: Implement verify(prompt: str, response: str, ground_truth: float) -> float that extracts the predicted answer, compares it to the ground truth with a small tolerance (e.g., abs(predicted - expected) < 1e-3), and returns 1.0 or 0.0.

Step 3: Test your verifier against these three model outputs for the first prompt:

1. "There are 14 workers and 12 slots each. 14 × 12 = 168. The answer is $\boxed{168}$."
2. "The total is $\boxed{148}$."
3. "It might be $\boxed{168}$ or $\boxed{148}$."

Expected results: output 1 should return 1.0. Outputs 2 and 3 should return 0.0. The third response contains the correct value, but its final answer is ambiguous.

Start from this minimal solution:

Step 4 (optional): add a format reward. Give +0.2 for using \boxed{} correctly, and +0.8 for a correct answer inside the box. What behavior does this mixed reward encourage? Does the model still get a positive reward if the box is present but the answer is wrong? For a strict production verifier, decide whether that partial credit is worth the gaming risk or whether correctness should gate every positive reward.

DeepSeek-R1-Zero combined accuracy rewards with format rewards targeting a parseable response structure.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} Mixed rewards require careful testing so formatting doesn't dominate correctness.

RLVR and distillation

RLVR and distillation aren't mutually exclusive. DeepSeek-R1 used RL in its teacher pipeline, then fine-tuned smaller dense models on the same 800k generated training samples.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} The systems distinction is clean: RLVR optimizes a policy against checks, while distillation trains a student from teacher outputs.

DeepSeek-R1 makes this trade-off concrete. Distilling into Qwen2.5-32B outperformed a large-scale RL run on Qwen-32B-Base (DeepSeek-R1-Zero-Qwen-32B) on the paper's reasoning benchmarks.^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} That's evidence for testing distillation when a strong teacher exists, not a universal ranking of the two methods.

RLVR still matters because distillation doesn't optimize the student online against a verifier. If the teacher's checked outcomes are insufficient, improving that policy against well-tested verifiers is a different operation from transferring its sampled outputs.