Reasoning & Test-Time Compute

A familiar on-call trade-off arrives as capacity_math: estimate GPU memory for eight concurrent branches sharing a 4,000-token prompt, each producing 1,000 continuation tokens, while the queue is already busy. One pass is cheap, but it can omit a KV term. Eight candidates may expose the omission, then occupy the cache headroom needed by the next request. The real question isn't “can the model think harder?” It’s “which extra evidence earns its TTFT, memory, and cost?”

State-space models showed one way to keep decode state from growing with context length by compressing history into recurrent state. Reasoning models make a different trade: keep working longer, branch, or check an answer when task difficulty makes that useful. The system buys a chance to catch its own plausible mistake before it commits.

Count the r's in strawberry one position at a time. A quick association can stop at two; marking the letters gives three. A single-pass capacity estimate has the same failure mode: it can sound finished until someone walks through KV bytes, branch count, and the latency SLO.

Classic scaling work focused on train-time compute: more parameters, more data, and more pretraining FLOPs (floating-point operations).^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361}

Work from 2024 onward made a second axis hard to ignore. On hard reasoning tasks, extra compute during generation can improve the answer.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314[3]Reference 3Learning to reason with LLMshttps://openai.com/index/learning-to-reason-with-llms/} That is test-time compute scaling. OpenAI's o1 launch post and DeepSeek-R1 made the shift visible, while Snell et al. showed in evaluated math settings that a smaller model with a well-chosen inference policy can beat a much larger model answering once.^{[3]Reference 3Learning to reason with LLMshttps://openai.com/index/learning-to-reason-with-llms/[4]Reference 4DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314}

Provider APIs expose this choice as an effort or thinking control, with names and supported values that vary by model. A useful production skill follows: measure where another token, branch, or check changes correctness, then stop when its latency and cost outweigh that gain. More thinking isn't automatically better.^{[5]Reference 5Does Thinking More Always Help? Mirage of Test-Time Scaling in Reasoning Modelshttps://arxiv.org/abs/2506.04210[6]Reference 6Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning}

We’ll use capacity_math throughout. Easy lookups stay single-pass. For this request, extra samples, a verifier, or a longer trace might change the answer, but each option consumes budget. Keep the earlier picture of how transformers predict the next token, then choose among single-pass, best-of-N, and guided search by the evidence each path can add. Routing and token budgets are part of that choice, not cleanup after the algorithm.

Treat the values as a teaching fixture, not a hardware benchmark. A real sweep records model version, hardware, prompt distribution, sampling policy, verifier, correctness metric, and p50/p99 latency beside each budget.

What does test-time compute scaling add that train-time scaling alone doesn't?

Answer

Train-time scaling spends compute before deployment to improve the model. Test-time scaling spends extra compute during generation through longer traces, more samples, revision, or search, so hard tasks can get more deliberation than easy tasks.

Fast answers miss checks

Start with the operation, then attach Kahneman's labels. Read strawberry as a sequence of positions and mark each r: the answer is three. Reading it as a familiar word and blurting out “two” is the fast path.

In Kahneman's framework, that fast, pattern-driven response is System 1. The slower, deliberate count is System 2. The labels help us describe a serving trade-off, but they aren't an architecture taxonomy. A low-budget single-pass call can lean on learned patterns; a larger budget can buy checks, alternate samples, or tool calls.

The miss isn't a knowledge gap. Counting requires deliberate step-by-step verification, while pretraining's next-token objective rewards a fluent continuation more directly than a checked final answer. capacity_math fails in the same place: a plausible GPU count collapses once KV bytes are multiplied by sequence length.

Test-time compute addresses that failure mode only when it buys useful evidence. A model or wrapper can explore several approaches, inspect intermediate steps, and select a candidate before committing. If every branch repeats the same mistake, extra tokens add cost without adding confidence.

Why is counting the r letters in strawberry a System 2 task for a model?

Answer

The answer isn't a stored fact or a fluent association. The model must inspect positions and verify the count before answering, which is exactly the kind of work extra reasoning tokens can support.

When does extra inference compute help most?

Answer

It helps when the task requires verifiable multi-step reasoning, search, or correction. It helps much less for fluent pattern-matching tasks such as simple extraction, translation, or sentiment classification.

The two axes of compute scaling

Separate when the bill arrives. Train-time compute is spent before deployment to improve fixed weights. Test-time compute is spent during a live request: trying alternate fixes, checking evidence, and verifying an answer before committing.

Traditional scaling changes model parameters $N$ or training tokens $D$. Empirical studies fit loss with an approximate power law of compute $C$; Kaplan et al. use roughly $6ND$ as a training-compute estimate in their scaling analysis:^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361}

$L_{\text{train}}(C) \propto C^{-\alpha}, \quad C = 6ND$

Within the measured regime, that relationship is predictable enough to guide how a team spends pretraining compute. It says nothing by itself about which live request deserves a longer rollout.

That live decision adds a second axis, inference compute $C_{\text{infer}}$. A useful model treats accuracy as a function of compute, task shape, policy, and verifier quality:

$\text{accuracy} = g(C_{\text{infer}}, \text{task}, \text{policy}, \text{verifier})$

In an evaluated operating range, more inference compute can improve accuracy with diminishing returns. It can plateau or even reduce accuracy when a trace overthinks, samples remain correlated, or a verifier picks the wrong branch. Compute may arrive as a longer trace, repeated samples, a revision loop, or explicit search with a verifier.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314[5]Reference 5Does Thinking More Always Help? Mirage of Test-Time Scaling in Reasoning Modelshttps://arxiv.org/abs/2506.04210}

Snell et al. don't claim one law for every model and benchmark. Their operational result is narrower and more useful: in their math evaluations, a well-allocated inference budget let a smaller model outperform a larger model sampled once.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314}

Why does test-time compute have diminishing returns?

Answer

Early extra samples or reasoning steps often find obvious mistakes. Later compute tends to explore correlated candidates, repeat work, or hit verifier limits, so each additional token or branch buys less improvement.

That distinction changes how a system spends compute on a hard prompt:

The next cell turns that distinction into a selection rule. It reuses the opening eval table and chooses the highest measured accuracy that fits a 1000 ms latency limit.

The output answers a production question, not a theory question: among the tested budgets, which one gives the best measured quality while meeting the service-level objective? Here, 512 wins. A longer trace would need new evidence before it earns its extra latency.

How reasoning models work

Reasoning-oriented training and inference policies allocate additional tokens or branches before returning an answer. You may see decomposition, checks, or revision, but none of those behaviors is guaranteed on every request. Treat them as policies to measure.

Extended chain-of-thought

Reasoning models often generate a long scratchpad or intermediate trace around a final answer. A provider may expose that trace, return a summary, or hide it. Some APIs also interleave non-visible reasoning with visible output or tool calls. For this chapter, the important distinction is operational: can the model or wrapper spend additional inference compute before it commits?

Built-in reasoning also differs from prompted chain-of-thought (CoT)^{[7]Reference 7Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.https://arxiv.org/abs/2201.11903}. Training can shape the policy through Reinforcement Learning (RL), distillation, or both, instead of relying on prompt wording alone. A serving policy can vary the budget by request difficulty, and a trace may backtrack or revise a dead end. Providers can account for and bill that work without exposing raw intermediate tokens.

Current reasoning APIs make the accounting concrete. OpenAI's docs describe reasoning tokens as non-visible output tokens that still consume context budget and count as billed output tokens. They also describe interleaved thinking for some current models, where visible output or tool calls can appear between reasoning steps.^{[6]Reference 6Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning}

Thinking depth is now a control surface, but the contract is model- and endpoint-specific. OpenAI's reasoning.effort accepts a model-dependent subset of none, minimal, low, medium, high, xhigh, and max; lower effort favors speed and lower token use, while higher effort favors more complete reasoning.^{[6]Reference 6Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning}

Google's current Interactions API uses generation_config.thinking_level and lists supported levels per Gemini model. Its docs represent thoughts and signatures differently in the generateContent API, so a client should follow the selected endpoint's schema instead of copying a knob between APIs.^{[8]Reference 8Gemini thinkinghttps://ai.google.dev/gemini-api/docs/thinking}

Anthropic's docs distinguish adaptive thinking (thinking.type: "adaptive") on newer models from manual budget_tokens on models that support extended thinking. Manual mode is deprecated on Claude 4.6 and rejected on Claude 4.7 and later, while support remains model-specific.^{[9]Reference 9Building with extended thinkinghttps://docs.claude.com/en/docs/build-with-claude/extended-thinking}

Open-weight systems expose related controls, though their interfaces still differ. GLM-5.2 is documented as a 744B-A40B MoE with 1M text context, reasoning_effort values of high or max, and a thinking switch.^{[10]Reference 10GLM-5.2 Model Cardhttps://huggingface.co/zai-org/GLM-5.2[11]Reference 11GLM-5 Repositoryhttps://github.com/zai-org/GLM-5[12]Reference 12GLM-5.2 Model Documentationhttps://docs.z.ai/guides/llm/glm-5.2}

DeepSeek-V4-Flash-0731 documents low, high, and max effort with a 1M text context.^{[13]Reference 13DeepSeek-V4-Flash-0731 Model Cardhttps://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731[14]Reference 14Use DeepSeek with Codexhttps://api-docs.deepseek.com/quick_start/agent_integrations/codex/} The V4 paper sizes Flash at 284B total and 13B active; the 0731 checkpoint attaches a speculative module, which explains why hosted parameter counts can read higher.^{[15]Reference 15DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencehttps://arxiv.org/abs/2606.19348}

Across these systems, higher effort can improve difficult work, but it also increases latency, output volume, and GPU occupancy. Start with clear constraints, tune the supported knob, and evaluate the result instead of assuming a chain-of-thought prompt helps.

Test-time compute can mean a longer single trace or multiple sampled traces. OpenAI's o1 launch post reported one benchmark example on AIME 2024, a math competition: accuracy rose from one sample to consensus over 64 samples, then to learned reranking over 1000 samples.^{[3]Reference 3Learning to reason with LLMshttps://openai.com/index/learning-to-reason-with-llms/} Keep the scope attached to the number. That result belongs to one evaluation, not a promise for another task, model, or selector.

Why can provider-reported non-visible reasoning tokens affect cost and capacity even when users never see them?

Answer

When the provider accounts for non-visible reasoning tokens, they still consume billed output tokens and context budget. They can also add wall-clock work. Transformer runtimes that materialize them autoregressively retain temporary key-value (KV) state while generating them, so a short visible answer can still be expensive.

Deliberation and search at inference time

Test-time compute is an umbrella term, not a single algorithm. Some systems sample many complete answers and pick a winner. Some iteratively critique and revise one candidate. Others run explicit search over partial reasoning states using a verifier or reward model. All of these patterns branch from the same idea: spend extra compute to explore, score, and refine before returning an answer.

Not every reasoning model literally runs beam search or a PRM at inference time. The shared pattern is optional inference compute allocation. A good routing policy keeps easy requests cheap and assigns more tokens, branches, or verification only where evaluation shows a payoff.

What is the shared pattern behind longer traces, best-of-N, revision loops, and tree search?

Answer

They allocate more inference compute before committing to an answer. The system explores, verifies, revises, or selects among candidates instead of returning the first plausible completion.

Provider-reported hidden reasoning

Suppose the dashboard shows a short answer but a large output-token count. Some hosted APIs report non-visible reasoning in usage accounting while returning only final text; others interleave visible output or tool calls with non-visible work. OpenAI's reasoning-token documentation describes both patterns.^{[6]Reference 6Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} An open-weight deployment, explicit search wrapper, or provider with summarized thinking may expose and account for intermediate work differently, so inspect the actual response schema.

For an API that reports them, non-visible tokens consume context window space and may count toward billing even though users never see them.^{[6]Reference 6Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} A transformer runtime that materializes those tokens autoregressively also spends decode work and holds temporary key-value (KV) cache state while generation runs. Exact cache behavior belongs to the provider and engine, not to this accounting shortcut.

Why should prompts for a hosted reasoning model usually start simple and direct?

Answer

Clear task constraints, success criteria, and output requirements give a good evaluation baseline. Then follow the selected model's prompting guidance rather than assuming visible chain-of-thought instructions improve its non-visible reasoning.

The next cell bills the same 180 visible + 1,220 hidden split as the figure, at an illustrative $10 per million output tokens.

Use the provider's usage schema and pricing table for a real implementation. The example gives a capacity lesson: a dashboard that counts visible text alone can understate both spend and active KV state.

Test-time compute strategies

When capacity_math is uncertain, extra work can take three forms: parallel candidates, sequential repair, or a branching search that scores partial work. The right pattern follows from what can be checked and how much latency the request can spend.

1. Best-of-N sampling

Start with parallel alternatives. Sample $N$ candidates for the same capacity_math prompt, then keep the candidate with the best verifier score or the most common extracted answer. More attempts raise the chance of seeing a fix only when they add useful diversity and the selector can recognize it. Shared model biases make samples correlated, so the independence assumption is only a baseline.

Best-of-N is easy to parallelize, which keeps wall-clock time manageable. The cost remains high: every completion consumes tokens, memory, and capacity, even though only one candidate returns to the user.

The sketch below assumes a model.generate method. The testable math is in the next two cells.

Scaling behavior

Ask first what an ideal selector would achieve. With an oracle verifier and independent samples, the chance of generating at least one correct answer in $N$ tries follows:

$P_{\text{success}} = 1 - (1 - p)^N$

Here $p$ is the base probability that one attempt is correct. For the toy value $p=0.25$, four independent attempts give $1-(0.75)^4=68.4\%$. Real systems fall below that ideal because candidates correlate and selectors make mistakes, but the equation shows where the gain comes from. OpenAI's o1 launch post reported the same pattern on AIME 2024: 74% with one sample, 83% with 64-sample consensus, and 93% when a learned scorer reranked 1000 samples.^{[3]Reference 3Learning to reason with LLMshttps://openai.com/index/learning-to-reason-with-llms/}

The toy curve makes the trade-off visible: ideal success rises quickly while cost grows linearly with every completion. Correlation and selector errors pull the deployed curve down. The next cell applies an 80% selector recall after $N > 1$, matching the gap drawn in the figure.

The selector is deliberately simple. One candidate has no selection step. Once branches exist, generation and selection become separate failure surfaces: a correct answer can appear and still be discarded.

What must be true for best-of-N to pay off?

Answer

The base model must have a non-trivial chance of producing a correct candidate, the samples must add useful diversity, and the verifier or selection rule must reliably identify better answers.

Train for the test-time policy you plan to use

Training and serving need to agree on what counts as success. Supervised fine-tuning with cross-entropy (CE) puts more probability on each demonstrated target, which fits a pass@1 system. Best-of-N needs coverage across samples. If CE concentrates probability on one trace family, extra draws become near-duplicates and the search budget stops buying exploration.

Chen et al. report this mismatch in mathematical reasoning and theorem-proving settings: longer CE training can raise confidence while lowering pass@N, while a confidence-limiting objective better matches later search.^{[16]Reference 16Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoninghttps://proceedings.neurips.cc/paper_files/paper/2025/hash/e8f4eae0a41cab67fdead3aa6b77f083-Abstract-Conference.html} CE isn't a failed objective. Evaluate it against the policy you will ship: next-token loss, pass@1, candidate diversity, and pass@N at the actual sample budget.

Why can cross-entropy SFT improve pass@1 yet hurt a best-of-N search policy?

Answer

CE can concentrate probability on one demonstrated trace and make samples less diverse. Pass@N needs coverage of at least one correct candidate, so training should be evaluated against the deployed sample budget and selection policy.

2. Sequential revision

Now make the work sequential. Draft the capacity_math answer, check it against the KV formula and latency SLO, then repair the weak step. Each pass sees the previous answer, unlike best-of-N's independent attempts.

The loop is draft, critique, and regenerate when needed. The sketch stops when the critique reports "no errors found"; a deployed system would replace that string match with a trained verifier or structured critique schema. Every round adds dependency and latency, so a plateau should end the loop.

When is sequential revision a better fit than best-of-N?

Answer

Use sequential revision when each pass can improve a draft through critique, editing, or repair. Use best-of-N when candidates are independent and the answer can be scored directly.

The next cell applies that stopping rule to measured scores. Once a round gains less than 0.01, later rounds would spend budget on a plateau.

3. Tree search with process reward models

Tree search spends compute before a path is complete. For capacity_math, one branch might multiply KV bytes correctly, another might keep the wrong branch count, and a third might violate the SLO. A process reward model scores each intermediate step, so the controller can prune a bad path before paying for its remaining steps. An outcome score waits for the final answer.

The simplified beam search below starts with one prompt state. Each round proposes several next steps, scores them with the PRM, and keeps the highest-scoring unfinished paths. A branch whose next step starts with Final: moves into a separate completed pool, where it can't be expanded or displaced by an unfinished trace that keeps accumulating score. Search continues until no active branch remains or the step budget runs out.

The return value makes completion explicit. It selects the highest-scoring completed chain when one exists. If every branch exhausts the budget without a final answer, it returns the best partial chain with status="incomplete"; the caller can retry, increase the budget, or escalate instead of presenting partial reasoning as an answer. model.generate_step and prm.score_step remain conceptual interfaces here.

The tagged fallback is part of the API contract, not a cosmetic field. A high process score means a partial trace looked promising; it doesn't turn that trace into a final answer. The sketch sums step scores to keep the mechanics visible. A deployed scorer also needs a cross-length rule such as a calibrated mean, minimum-step score, or length penalty, or a longer chain can win merely because it collected more terms.

Beam search also tends to collapse onto near-duplicate paths. More capable systems inject diversity, backtrack, or use Monte Carlo Tree Search-style expansion. The tree is easier to write than a verifier that reliably knows which partial step deserves more budget.

The deterministic fixture below exercises both boundaries. In the first run, an unfinished branch reaches a higher score than either final branch, but the selector still returns the best completed answer. In the second run, no branch finishes, so the result is visibly incomplete.

Why can't the controller return the highest-scoring unfinished beam after the search budget expires?

Answer

A process score ranks partial progress; it doesn't certify that a branch produced an answer. The controller must choose from completed chains, or tag the best partial chain as incomplete so the caller can retry or escalate.

Why is verifier quality the bottleneck in search-based reasoning?

Answer

Search only helps if bad branches are pruned and good branches survive. A weak verifier can reward fluent wrong steps, discard correct paths, or collapse the search into near-duplicate traces.

Process reward models vs. outcome reward models

The reward model determines when search gets evidence. If a branch fails only at its final answer, all earlier work is sunk. If partial steps predict failure reliably, the controller can spend the next token on a different branch.

Final scoring and step scoring answer different questions. An ORM asks whether the completed answer works: “the GPU count matches” or “it doesn't.” No intermediate failure location reaches the search controller. A PRM asks about intermediate decisions: “KV formula used,” “branch count held at eight,” “prefix sharing applied.” That denser feedback costs more to build and can discard a good path when wrong. Reliable step scores can still save work by redirecting search early.

Outcome reward models (ORMs)

Outcome Reward Models (ORMs) evaluate a completed trajectory rather than its intermediate steps. The outcome signal may be binary correctness, a probability or scalar score, or a preference-derived comparison. Exact-answer math makes binary grading convenient, but that isn't part of the ORM definition.

Their supervision boundary is relatively simple: labels attach to completed outputs rather than every reasoning step. Verifiable tasks can supply labels automatically; subjective tasks may need learned or human preference signals. During search, though, the ORM has to wait. A generator can make a fatal error in step 1 and produce 100 more steps before the final score rejects the trajectory, so no early branch pruning is available.

How PRMs work

Process Reward Models (PRMs) evaluate the validity of each intermediate step in a chain of thought. A PRM scores a step conditioned on the problem and preceding steps, which gives search a chance to abandon a path when it first becomes weak. That can improve efficiency only when the intermediate score predicts final correctness.

PRMs also need step-level supervision. Lightman et al. trained their strongest PRM with human step labels, while methods such as Math-Shepherd derive process targets from rollout outcomes.^{[17]Reference 17Let's Verify Step by Step.https://arxiv.org/abs/2305.20050[18]Reference 18Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationshttps://arxiv.org/abs/2312.08935} A process score is earlier evidence for keeping or abandoning a branch, not proof that a kept step is correct.

This side-by-side pseudocode shows when each model can act. The labels are qualitative so they don't imply calibrated probabilities from an unevaluated scorer:

Snell et al. reported that compute-optimal test-time scaling can use up to 4× less test-time compute than a best-of-N baseline in their math-reasoning evaluation, and that a smaller model plus test-time compute can outperform a model about 14× larger answering once on problems where the small model already has a non-trivial hit rate.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314} Lightman et al. reported that process supervision outperformed outcome supervision on MATH.^{[17]Reference 17Let's Verify Step by Step.https://arxiv.org/abs/2305.20050} These results motivate testing PRM-guided pruning on verifiable workloads. They don't establish that every PRM or task benefits.

Not every reasoning system uses a learned PRM or ORM. DeepSeek-R1-Zero trained with rule-based accuracy and format rewards rather than a neural reward model, which is one reason you should treat test-time compute as a family of techniques, not a single stack.^{[4]Reference 4DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

What is the practical difference between an ORM and a PRM during search?

Answer

An ORM scores only the final answer, so it can't prune based on intermediate evidence. A PRM scores intermediate steps, so a search policy can prune low-scoring branches early when the process scores are reliable.

The next cell counts generated steps when three of four branches are pruned early. That's the upside case. A deployment eval also has to count false pruning of branches that would have ended correct.

DeepSeek-R1-Zero and DeepSeek-R1

DeepSeek's contribution separates into two experiments. DeepSeek-R1-Zero tested whether large-scale RL with verifiable rewards could induce strong reasoning without supervised cold start data.^{[4]Reference 4DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} DeepSeek-R1 kept the reasoning RL idea, then added data and post-training to make the behavior more stable and readable.^{[4]Reference 4DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

Training pipeline

The final DeepSeek-R1 pipeline therefore isn't “pure RL” end to end. The paper describes two supervised fine-tuning (SFT) stages around two RL stages. Each stage answers a concrete product problem: seed readable traces, reinforce checkable reasoning, filter useful rollouts, then broaden behavior.

1. Base model: Start with DeepSeek-V3-Base, a 671B Mixture-of-Experts (MoE) model with 37B active parameters per token.^{[4]Reference 4DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}
2. Cold-start Supervised Fine-Tuning (SFT): Collect thousands of long CoT examples and fine-tune the base model. The seed traces address the readability and language-mixing issues reported for R1-Zero.
3. Reasoning-oriented RL: Run GRPO (Group Relative Policy Optimization)-based RL on reasoning tasks with verifiable, rule-based rewards, keeping the feedback tied to checkable outcomes.^{[4]Reference 4DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}
4. Rejection sampling + second SFT: Use the RL checkpoint to generate high-quality reasoning traces, mix them with supervised data from DeepSeek-V3 for non-reasoning domains, and retrain the base model. This recovers broader language behavior around the specialized reasoning policy.
5. Final RL for all scenarios: Run another RL stage across a broader prompt mix to produce DeepSeek-R1, rather than stopping with a math-only specialist.

The sequence matters when you diagnose a result. “Pure RL” describes R1-Zero, not R1. R1's readability and breadth came from the supervised stages around its RL stages.^{[4]Reference 4DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

Why is "DeepSeek-R1 was pure RL" an inaccurate summary?

Answer

DeepSeek-R1-Zero was trained with large-scale RL without a supervised cold start. DeepSeek-R1 added cold-start SFT, reasoning RL, rejection sampling plus another SFT stage, and final broad RL.

Emergent reasoning

R1-Zero displayed self-verification, backtracking, and longer rollouts, while the paper also reports readability issues and occasional language mixing.^{[4]Reference 4DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} Both observations belong in the same diagnosis: verifiable rewards can induce useful reasoning, but raw RL rollouts still need cleanup before they serve as a general-purpose product.

What did R1-Zero prove, and what did it not solve by itself?

Answer

It showed that verifiable rewards can induce reasoning behaviors like self-verification and backtracking. It didn't by itself solve readability, language mixing, or broad product polish.

Compute-optimal inference: when to think harder

More thinking changes different workloads in different ways. Ghosal et al. found an inverted-U pattern across reasoning models: accuracy rose with some extra thinking, then fell as traces grew.^{[5]Reference 5Does Thinking More Always Help? Mirage of Test-Time Scaling in Reasoning Modelshttps://arxiv.org/abs/2506.04210} Their analysis attributes part of the early gain to higher output variance, not only better reasoning, and recommends parallel sampling over extending one trace indefinitely.

An Anthropic study found a related inverse-scaling effect on several tasks, where longer reasoning amplified distractions and errors.^{[19]Reference 19Inverse Scaling in Test-Time Computehttps://alignment.anthropic.com/2025/inverse-scaling/} Budget is therefore an evaluated operating point, not a slider to max out.

Use task shape to choose a starting experiment, then check quality and latency on your workload:

The cross-over point

A task family may have a measured compute budget where a smaller model plus a test-time policy beats a larger single-pass model:

$Q_{\text{small+policy}}(C^*) > Q_{\text{large,single}}$

Here, $Q$ is measured quality and $C^*$ is one evaluated inference budget. A crossover at $C^*$ isn't a promise that every larger budget keeps helping. Quality can plateau or fall, and the preferred path changes with task difficulty, latency limits, verifier quality, and request volume.

Snell et al. observed in FLOPs-matched evaluations that a smaller model with additional test-time compute can outperform a model roughly 14× larger answering in one pass, when the smaller model already has a non-trivial success rate.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314} Treat longer thinking as a candidate for that evaluated task family, not a general replacement for a larger model.

What is the compute-optimal routing question for a reasoning workload?

Answer

Ask whether extra inference compute on a smaller or cheaper model beats a larger model answering once for this task family, latency budget, verifier quality, and target accuracy.

The router turns those measurements into a product decision. It can inspect task signals, stakes, and the remaining latency budget, then send easy work to one pass and uncertain, checkable work to a more expensive path. The values in the next cell are illustrative measurements, not a benchmark claim.

That route avoids an intelligence tax: extra latency and cost for a reasoning path when a single-pass baseline already matches measured quality. Sentiment classification and ticket-ID extraction usually have little to gain. capacity_math earns the expensive path only if its measured gain survives the SLO.

Why should a production system route tasks before invoking an expensive reasoning path?

Answer

Reasoning paths add latency, output tokens, KV cache pressure, and cost. Routing reserves that budget for tasks where extra reasoning is likely to improve verifiable quality.

Production considerations

Once a request takes the expensive route, its hidden work becomes a serving problem. Longer traces and more branches raise cache residency, first-token delay, and cost even when the visible answer stays short. A quality gain that exhausts the queue is not a free gain.

KV cache pressure and prefix sharing

Reasoning workloads stress an inference engine differently from ordinary chat because they can generate many intermediate tokens before or between visible outputs. Start with the cache, not the final answer length.

For every generated token, the server appends a Key vector and a Value vector for every layer:

$\text{KV bytes per token} = 2 \times L \times n_{kv} \times d_h \times b$

Here $L$ is the number of layers, $n_{kv}$ the number of KV heads, $d_h$ the head dimension, and $b$ bytes per value. The scaling fact to carry forward is KV cache memory grows linearly with sequence length. A rollout that spends 10,000 tokens thinking creates roughly 10,000 tokens of KV state, which can collapse batch size before raw FLOPs become the bottleneck.

PagedAttention stores KV cache in fixed-size blocks to reduce fragmentation and make long-context scheduling practical.^{[20]Reference 20Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} Prefix-sharing runtimes reuse a cached prompt or partial trace when branches still share it, rather than duplicating that state. SGLang's RadixAttention is one example.^{[21]Reference 21SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104} Best-of-N, self-consistency, and tree search can benefit because candidates often diverge only near their leaves. The next cell measures that saving for capacity_math: eight branches, a 4,000-token prompt, and 1,000 continuation tokens per branch.

Why do reasoning workloads stress KV cache more than ordinary chat?

Answer

Non-visible-token modes and branching search can generate many extra tokens before or between visible outputs. In transformer serving paths that materialize those tokens, each one appends K/V state across layers, reducing batch size and concurrency unless the runtime pages, shares, or evicts cache efficiently.

TTFT vs. inter-token latency

Test-time compute changes which latency the user feels first. A provider or wrapper that generates non-visible reasoning before revealing an answer can push TTFT (time to first token) up substantially.^{[6]Reference 6Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} An interleaved mode can pause between visible chunks or tool calls. ITL (inter-token latency) may still look healthy once answer text streams, while the request already feels slow.

Hidden reasoning is still decode load for the fleet. Under continuous batching it occupies KV residency and memory bandwidth, raises other tenants' ITL, and reduces goodput (offered request rate under joint TTFT and decode SLOs). A reasoning router therefore controls capacity as well as quality: a long rollout must leave headroom for co-tenant latency and for its own TTFT budget.

Why can a reasoning model feel slow even if decode tokens per second are healthy?

Answer

In a mode that generates non-visible reasoning first, those tokens raise TTFT. In an interleaved mode, additional thinking can also create pauses between visible chunks or tool calls. ITL during answer streaming can still look normal.

The next cell converts a thinking-token budget into a simple TTFT estimate at 80 decode tokens per second. It rejects budgets that miss a 3000 ms first-token SLO. Treat the arithmetic as a capacity screen, not a provider benchmark.

Latency vs. quality trade-off

Reasoning models add latency, but the exact numbers depend on hardware, batching policy, and provider implementation:

For strict interactive latency budgets, a long reasoning path needs measured gains to justify its delay. Larger budgets fit better in:

• Batch processing (code review, data analysis)
• High-stakes analyst workflows (incident review, safety analysis, compliance analysis)
• Asynchronous workflows (software engineering, research)

Cost implications

Extended reasoning is expensive even when the final answer is short. Hosted APIs may charge directly for hidden reasoning tokens. OpenAI's reasoning docs, for example, count them as output tokens even though the API doesn't return them verbatim.^{[6]Reference 6Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} Open-weight deployments pay through wall-clock time, lower throughput, and higher KV residency. Either path reduces how many concurrent requests the same GPU can serve.

Three controls keep that bill tied to evidence:

1. Routing: send only hard or high-stakes tasks to expensive reasoning paths
2. Token budgets: cap how long a rollout may think before the marginal gain stops being worth it
3. Early stopping: terminate search when verifier scores stop improving

Without them, an ambiguous or unsolvable prompt can burn inference budget without producing a better answer.

What three controls keep reasoning costs bounded in production?

Answer

Route only hard or high-stakes tasks to reasoning paths, cap reasoning tokens or branches, and stop early when verifier scores or expected gains plateau.

Distillation: making reasoning affordable

Distillation moves work from serving to training. A larger reasoner writes detailed capacity_math traces; a smaller student trains on them and can solve similar sizing problems with less live compute. The student may inherit a solution style without inheriting the teacher's full search behavior.

DeepSeek-R1 showed that reasoning capabilities can be distilled into much smaller models:

In DeepSeek's reported AIME evaluation, the 7B and 32B distilled models retain substantial benchmark accuracy at lower parameter counts than R1. Parameter count is only a proxy: deployment cost also depends on active parameters, quantization, batch size, and engine choice. Distillation can lower reasoning cost, but routing and token budgets still decide the serving bill.

What can reasoning distillation transfer, and what can it fail to transfer?

Answer

It can transfer solution traces, formats, and many task patterns from a stronger teacher. It may not transfer the teacher's full search policy, verifier behavior, or reliability on unfamiliar tasks.

Trace cloning has a scaling ceiling. Setlur et al. compare verifier-free methods that distill or clone successful traces with verifier-based methods using rewards, RL, or search.^{[22]Reference 22Scaling Test-Time Compute Without Verification or RL is Suboptimalhttps://arxiv.org/abs/2502.12118} Under their assumptions and evaluated reasoning tasks, verifier-free methods become increasingly suboptimal as rollout length and test-time budget grow: the student sees successful traces, but not which alternative traces a verifier would reward.

That result doesn't make distillation useless. The DeepSeek numbers show that a smaller student can gain substantial capability. It sets a boundary: more teacher traces won't reproduce verifier-guided improvement indefinitely. Filter traces with reliable checks, retain a verifier during evaluation, or use verifier-guided RL when the workload supplies trustworthy rewards.

Why doesn't cloning more reasoning traces guarantee the same scaling as verifier-guided RL or search?

Answer

Cloning teaches the student to imitate observed successful traces. It doesn't directly teach a reward over alternative traces, so the gap can widen as rollout length and search budget grow. Distillation remains useful, but its ceiling should be measured with a verifier-backed evaluation.

The next cell turns that boundary into a tiny serving gate. Send extra reasoning only when the request is verifiable and its predicted quality gain clears a threshold. capacity_math passes; extraction and unverifiable copy stay single-pass.

What to remember and where to go next

1. Inference is a second scaling axis: on hard reasoning tasks, spending more compute at request time can beat a larger model answering once.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314}

2. The axis has several policies: a longer scratchpad, repeated samples, revision, and explicit search all spend compute before commitment.

3. Evidence sets the ceiling: Snell et al. report up to 4× less test-time compute than a best-of-N baseline in their evaluation, while a reliable PRM can prune a weak path before its final answer.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314[17]Reference 17Let's Verify Step by Step.https://arxiv.org/abs/2305.20050}

4. Training history changes the behavior: R1-Zero used pure RL with verifiable rewards; DeepSeek-R1 added cold-start data, another SFT stage, and broader RL to make the behavior more readable and usable.^{[4]Reference 4DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

5. Budget is a measured dial: provider controls vary by model and endpoint, and accuracy can follow an inverted-U as a trace grows. Tune the supported control against quality, latency, and cost.^{[6]Reference 6Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning[8]Reference 8Gemini thinkinghttps://ai.google.dev/gemini-api/docs/thinking[9]Reference 9Building with extended thinkinghttps://docs.claude.com/en/docs/build-with-claude/extended-thinking[5]Reference 5Does Thinking More Always Help? Mirage of Test-Time Scaling in Reasoning Modelshttps://arxiv.org/abs/2506.04210}

6. Serving decides whether gains ship: long traces grow KV state linearly, so prefix sharing, routing, and token budgets belong in the design.^{[20]Reference 20Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180[21]Reference 21SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104}

Return to the opening checks: count the r's instead of trusting a quick association; ask whether traces, search, or distillation recover measured math quality; then route capacity_math only when its evidence justifies another GPU millisecond.