Reasoning & Test-Time Compute

State Space Models showed one way to keep decode state from growing with context length by compressing history into recurrent state. Reasoning models move in the opposite direction on purpose: spend more inference compute when the task is hard enough to justify it.

Think about two incident agents handling the same failed deployment. The first accepts the first rollback plan that looks plausible. The second checks failing tests, deploy diff, error budget, database migration state, and rollback blast radius before committing. When those checks catch a real mistake, the second plan performs better, not because the agent has different data, but because it spends more compute on the decision.

The same idea now shapes how frontier large language models (LLMs) are built and evaluated. Classic scaling work focused on train-time compute: more parameters, more data, and more pretraining FLOPs (floating-point operations).^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} Work from 2024 onward made a second axis impossible to ignore: on hard reasoning tasks, you can often get better answers by spending more compute during generation itself.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314[3]Reference 3Learning to reason with LLMshttps://openai.com/index/learning-to-reason-with-llms/} This is test-time compute scaling. Reasoning-model APIs from OpenAI and open-weight systems such as DeepSeek-R1 made this shift visible, and Snell et al. showed that, on tasks where a smaller model already has a non-trivial chance of success, extra inference-time compute can beat a much larger single-pass model.^{[3]Reference 3Learning to reason with LLMshttps://openai.com/index/learning-to-reason-with-llms/[4]Reference 4Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning[5]Reference 5DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314}

Current open-weight systems expose this choice directly. GLM-5.2 supports thinking and effort controls on a 744B-A40B MoE with 1M text context. DeepSeek V4 Flash 0731 offers low, high, and max effort on a 284B/13B core with 1M text context. Both are MIT-licensed, cluster-scale models. More effort can improve difficult work, but it also increases latency, output volume, and GPU occupancy.^{[6]Reference 6GLM-5.2 Model Cardhttps://huggingface.co/zai-org/GLM-5.2[7]Reference 7GLM-5 Repositoryhttps://github.com/zai-org/GLM-5[8]Reference 8GLM-5.2 Model Documentationhttps://docs.z.ai/guides/llm/glm-5.2[9]Reference 9Use DeepSeek with Codexhttps://api-docs.deepseek.com/quick_start/agent_integrations/codex/}

Provider controls are model-specific and they change. OpenAI documents reasoning.effort; Google's Gemini docs distinguish Interactions API generation_config.thinking_level from GenerateContent thinkingConfig.thinkingLevel for Gemini 3 and thinkingConfig.thinkingBudget for Gemini 2.5; Anthropic's current Claude Opus 4.8 and 4.7 docs use adaptive thinking with an effort parameter, while older Claude models use manual extended thinking with budget_tokens.^{[4]Reference 4Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning[10]Reference 10Gemini thinkinghttps://ai.google.dev/gemini-api/docs/thinking[11]Reference 11Building with extended thinkinghttps://docs.claude.com/en/docs/build-with-claude/extended-thinking} The durable skill is deciding how much compute a request deserves, because more thinking isn't always better.^{[12]Reference 12Does Thinking More Always Help? Mirage of Test-Time Scaling in Reasoning Modelshttps://arxiv.org/abs/2506.04210}

Some problems need deliberate reasoning instead of fast pattern matching, and test-time compute scaling turns that extra work into a production design choice. You'll need a basic mental model of how transformers predict the next token (covered earlier in the preparation path). The practical goal is to choose between single-pass, best-of-N, and guided-search strategies, then explain why routing and token budgets matter as much as the algorithms.

What does test-time compute scaling add that train-time scaling alone doesn't?

Answer

Train-time scaling spends compute before deployment to improve the model. Test-time scaling spends extra compute during generation through longer traces, more samples, revision, or search, so hard tasks can get more deliberation than easy tasks.

System 1 and System 2 thinking

Psychologist Daniel Kahneman's framework is a useful starting point. System 1 is fast, instinctive, and pattern-driven. When you read the word "strawberry" and immediately know it's a fruit, that's System 1. System 2 is slow, deliberate, and step-by-step. When you count how many times the letter "r" appears in "strawberry," you have to switch to System 2 because your gut reaction ("two?") is often wrong. The correct answer is three, but you only get there by checking each letter deliberately.

The System 1/System 2 distinction is an analogy for serving behavior, not an architectural taxonomy. A low-budget single-pass call may answer from strong patterns; a model or surrounding system with more budget can check intermediate work, sample alternatives, or run tools. On tasks that require counting, lookahead, or multi-step deduction, that extra verification path can matter.

Reasoning-oriented models and systems are designed to make deliberate behavior easier to buy at inference time. Depending on the model and wrapper, they may spend non-visible reasoning tokens, sample alternatives, run an external verifier, or revise a candidate. Compare an agent that accepts the first patch with one that reproduces the failing test, checks the diff, verifies side effects, and only then commits.

This shift from pure pattern matching to deliberate reasoning is what makes test-time compute scaling possible.

Why is counting the "r" letters in "strawberry" a System 2 task for a model?

Answer

The answer isn't a stored fact or a fluent association. The model must deliberately inspect positions and verify the count before answering, which is exactly the kind of work extra reasoning tokens can support.

Why low-budget generation can fail on hard problems

Here's a concrete example. Ask a model: "How many times does the letter 'r' appear in the word 'strawberry'?" A rushed answer can be "two," because the third "r" in "strawberry" is easy to miss without a check.

The problem isn't that the model lacks knowledge. It's that the task requires deliberate step-by-step verification, not fast association. Pretraining optimizes next-token prediction, which rewards fluent continuations more directly than checked final answers. On problems like complex math, code debugging, or incident triage, the same dynamic appears: the model produces a plausible-looking answer that collapses under scrutiny.

Test-time compute scaling can address this failure mode by spending budget before committing. Instead of one shot, the model or the system around it can explore multiple approaches, verify intermediate steps, and select a candidate. The extra compute only helps when exploration or verification changes the result for the better.

When does extra inference compute help most?

Answer

It helps when the task requires verifiable multi-step reasoning, search, or correction. It helps much less for fluent pattern-matching tasks such as simple extraction, translation, or sentiment classification.

The two axes of compute scaling

The operational distinction is simple: train-time compute is spent before deployment to improve the model. Test-time compute is spent during a live request: trying alternate fixes, checking evidence, and verifying the answer before committing.

Classic scaling emphasized the pre-season tuning. Evaluations on reasoning tasks now show that inference-time allocation can sometimes compete with moving to a larger single-pass model.

Traditional scaling focuses on train-time compute, increasing model parameters $N$ or training tokens $D$. The relationship between compute and loss follows a power law:

$L_{\text{train}}(C) \propto C^{-\alpha}, \quad C = 6ND$

Empirical scaling studies fit loss with an approximate power law of compute $C$, where training compute is often estimated as roughly 6 × model parameters $N$ × training tokens $D$.^{[1]Reference 1Scaling Laws for Neural Language Modelshttps://arxiv.org/abs/2001.08361} Within the measured regime, increasing training compute reduces loss predictably enough to guide training decisions.

Test-time compute scaling adds a second axis, inference compute $C_{\text{infer}}$. Treat accuracy as a function of inference compute, task shape, policy, and verifier quality:

$\text{accuracy} = g(C_{\text{infer}}, \text{task}, \text{policy}, \text{verifier})$

In a measured operating range, more inference compute can improve accuracy with diminishing returns. It can also plateau or reduce accuracy when a trace overthinks, sampled candidates are correlated, or a verifier selects the wrong branch. That extra compute can come from longer reasoning traces, repeated sampling, revision loops, or explicit search with a verifier.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314[12]Reference 12Does Thinking More Always Help? Mirage of Test-Time Scaling in Reasoning Modelshttps://arxiv.org/abs/2506.04210}

Snell et al.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314} don't claim one universal law for every model and every benchmark. Their more useful result is operational: if you allocate inference compute well, test-time scaling is strong enough that it can outperform buying a larger model and sampling once.

Why does test-time compute have diminishing returns?

Answer

Early extra samples or reasoning steps often find obvious mistakes. Later compute tends to explore correlated candidates, repeat work, or hit verifier limits, so each additional token or branch buys less improvement.

This changes how systems spend compute on hard prompts:

This evaluation table encodes the production question: choose the best measured quality under the service-level objective, rather than assuming the longest trace is best.

How reasoning models work

Reasoning-oriented training and inference policies allocate additional tokens or branches before returning an answer. The visible behavior can include decomposition, checks, or revision, but those behaviors are capabilities to evaluate rather than guarantees of every response.

Extended chain-of-thought

Reasoning models often generate long scratchpads or intermediate traces around a final answer. Sometimes that trace is exposed, sometimes it's summarized, and sometimes it's hidden entirely by provider policy. Some current APIs can interleave non-visible reasoning with visible output or tool calls. What matters isn't whether the user sees every token, but whether the model is allowed to spend additional inference-time compute while solving the task.

This built-in reasoning process differs from prompted chain-of-thought (CoT)^{[13]Reference 13Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.https://arxiv.org/abs/2201.11903} in several key ways:

1. Usually learned during post-training via Reinforcement Learning (RL), distillation, or both, rather than relying on prompt wording alone.
2. Potentially variable-length: the API or serving policy can permit different budgets by request
3. Can include self-correction: a trace may backtrack, recognize dead ends, and revise earlier steps
4. Often hidden or summarized: providers may not expose the raw reasoning tokens directly

Current reasoning APIs make this concrete. OpenAI's reasoning docs describe reasoning tokens as non-visible output tokens that still consume context budget and count as billed output tokens. The docs also describe interleaved thinking for some current models, where visible output or tool calls can appear between reasoning steps.^{[4]Reference 4Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning}

The thinking budget is now a primary control surface, even though providers expose it with model-specific knobs. OpenAI's reasoning.effort sets an effort level for reasoning models.^{[4]Reference 4Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} Gemini's Interactions API uses generation_config.thinking_level; GenerateContent uses thinkingConfig.thinkingLevel for Gemini 3 and thinkingConfig.thinkingBudget for Gemini 2.5, with SDK-specific casing such as Python thinking_config.thinking_level and thinking_budget.^{[10]Reference 10Gemini thinkinghttps://ai.google.dev/gemini-api/docs/thinking} Claude Opus 4.8 and 4.7 use adaptive thinking plus effort; Anthropic documents budget_tokens for older Claude extended-thinking models.^{[11]Reference 11Building with extended thinkinghttps://docs.claude.com/en/docs/build-with-claude/extended-thinking} Provider prompting guidance also differs by model, so start with a clear task and constraints, tune the supported knob, and evaluate instead of assuming a chain-of-thought prompt helps.

Test-time compute can mean either a longer single trace or multiple sampled traces. OpenAI's o1 launch post made this visible at the benchmark level: on AIME 2024 (a math competition benchmark), reported accuracy improved when the system moved from a single sample to consensus over 64 samples and then to learned reranking over 1000 samples.^{[3]Reference 3Learning to reason with LLMshttps://openai.com/index/learning-to-reason-with-llms/} Treat this as one reported evaluation result, not a guarantee for other tasks or selection rules.

Why can provider-reported non-visible reasoning tokens affect cost and capacity even when users never see them?

Answer

When the provider accounts for non-visible reasoning tokens, they still consume billed output tokens and context budget. They can also add wall-clock work. Transformer runtimes that materialize them autoregressively retain temporary key-value (KV) state while generating them, so a short visible answer can still be expensive.

Deliberation and search at inference time

Test-time compute is an umbrella term, not a single algorithm. Some systems sample many complete answers and pick a winner. Some iteratively critique and revise one candidate. Others run explicit search over partial reasoning states using a verifier or reward model. All of these patterns branch from the same idea: spend extra compute to explore, score, and refine before returning an answer.

Not every reasoning model literally runs beam search or a PRM at inference time. The shared pattern is optional inference compute allocation. A good routing policy keeps easy requests cheap and assigns more tokens, branches, or verification only where evaluation shows a payoff.

What is the shared pattern behind longer traces, best-of-N, revision loops, and tree search?

Answer

They allocate more inference compute before committing to an answer. The system explores, verifies, revises, or selects among candidates instead of returning the first plausible completion.

Provider-reported hidden reasoning

Some hosted reasoning APIs report non-visible reasoning tokens in usage accounting while returning only a final visible answer. Others can interleave visible output or tool calls with non-visible work. OpenAI's reasoning-token documentation describes both patterns.^{[4]Reference 4Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} Don't generalize this into one universal architecture: an open-weight deployment, explicit search wrapper, or provider with summarized thinking can expose and account for intermediate work differently.

For such APIs, non-visible tokens matter because they consume context window space and may count toward billing even though users never see them.^{[4]Reference 4Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} In transformer runtimes that materialize those tokens autoregressively, they also add decode work and temporary key-value (KV) cache state while generation is in progress. Exact cache behavior is provider- and engine-specific.

Why should prompts for a hosted reasoning model usually start simple and direct?

Answer

Clear task constraints, success criteria, and output requirements give a good evaluation baseline. Then follow the selected model's prompting guidance rather than assuming visible chain-of-thought instructions improve its non-visible reasoning.

Use the provider's usage schema and pricing table when implementing this calculation; the example demonstrates why billing and capacity dashboards can't count visible text alone.

Test-time compute strategies

Inference-time compute usually appears in a few patterns:

1. Best-of-N sampling

The simplest pattern is to generate 16 candidate fixes for a failed test and keep the one with the best verifier score. Each candidate is sampled separately, and more attempts raise your ceiling as long as they add useful diversity and your verifier or selection rule can reliably identify the best one. Shared model biases make the attempts correlated, not truly independent.

Best-of-N samples a generative model $N$ times on the same prompt, then returns one winner by reward-model score or self-consistency (majority vote on extracted final answers).

• Cost note: This method is easily parallelizable (all N attempts can run simultaneously), making it simple to implement but potentially expensive since you pay for all N completions.

Scaling behavior

With an oracle verifier and independent samples, the chance of generating at least one correct answer in $N$ tries follows:

$P_{\text{success}} = 1 - (1 - p)^N$

Where $p$ is the base probability of the model generating a correct answer on a single attempt. Real systems do worse than this idealized formula because samples are correlated and verifiers make mistakes, but the equation explains why best-of-N works at all. OpenAI's o1 launch post reports the same pattern on AIME 2024: o1 improved from 74% with one sample to 83% with 64-sample consensus and 93% when reranking 1000 samples with a learned scorer.^{[3]Reference 3Learning to reason with LLMshttps://openai.com/index/learning-to-reason-with-llms/}

The toy calculation shows why best-of-N is attractive and dangerous. The idealized success curve rises quickly, but cost rises linearly with every sampled completion. Real systems also plateau earlier because samples share model biases and verifiers make mistakes.

This toy selector is deliberately simple. With one candidate, there's nothing to select. Once multiple candidates exist, generating a correct candidate and selecting it become separate failure surfaces.

What must be true for best-of-N to pay off?

Answer

The base model must have a non-trivial chance of producing a correct candidate, the samples must add useful diversity, and the verifier or selection rule must reliably identify better answers.

Train for the test-time policy you plan to use

Ordinary supervised fine-tuning with cross-entropy (CE) rewards putting more probability on each demonstrated target. That is a sensible pass@1 objective, but a best-of-N system also needs useful coverage across samples. If training makes the model too confident in one narrow trace family, more samples can become near-duplicates and pass@N stops buying exploration.

Chen et al. show this mismatch on mathematical reasoning and theorem-proving settings: longer CE training can increase overconfidence while reducing pass@N, and a confidence-limiting objective better aligned training with later search.^{[14]Reference 14Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoninghttps://proceedings.neurips.cc/paper_files/paper/2025/hash/e8f4eae0a41cab67fdead3aa6b77f083-Abstract-Conference.html} CE doesn't always hurt reasoning. Co-design training and search by evaluating the objective against the deployed metric, sample budget, next-token loss, and pass@1.

Why can cross-entropy SFT improve pass@1 yet hurt a best-of-N search policy?

Answer

CE can concentrate probability on one demonstrated trace and make samples less diverse. Pass@N needs coverage of at least one correct candidate, so training should be evaluated against the deployed sample budget and selection policy.

2. Sequential revision

This is like drafting a rollback plan, checking it against logs and migration state, then revising the weak steps before committing. Unlike best-of-N (where each attempt is independent), each revision builds on the previous one.

Sequential revision starts with one draft, then loops: critique the draft, and if the critique finds issues, regenerate an improved version. The sketch stops when the critique reports "no errors found". Production systems usually replace that string match with a trained verifier or structured critique schema.

When is sequential revision a better fit than best-of-N?

Answer

Use sequential revision when each pass can improve a draft through critique, editing, or repair. Use best-of-N when candidates are independent and the answer can be scored directly.

3. Tree search with process reward models

Tree search is heavier: a repair agent considers multiple patch paths, evaluates each action, and prunes bad branches early. Instead of scoring just the final answer ("did the fix pass?"), a process reward model scores each intermediate step ("was the failing test reproduced?" or "does this diff touch the right module?").

A simplified beam search using a PRM accepts a model and a PRM, taking a prompt as the initial state. At each step, it generates multiple possible next steps (the beam width), scores them with the PRM, and keeps only the highest-scoring paths. The function outputs the best completed reasoning chain. This is a conceptual interface sketch; it assumes model.generate_step, prm.score_step, and is_final_answer exist.

Beam search is easiest to explain, but it has a well-known weakness for reasoning: beams can collapse onto near-duplicate traces. Stronger systems inject diversity, allow backtracking, or use Monte Carlo Tree Search-style expansion policies. Building the tree is straightforward compared with building a verifier good enough to prune bad branches early.

Why is verifier quality the bottleneck in search-based reasoning?

Answer

Search only helps if bad branches are pruned and good branches survive. A weak verifier can reward fluent wrong steps, discard correct paths, or collapse the search into near-duplicate traces.

Process reward models vs. outcome reward models

The choice of reward model changes search efficiency:

Final CI audit and step audit answer different questions. An ORM only checks whether the final answer works: "tests passed" or "tests failed." You have no idea where the plan broke. A PRM checks each intermediate decision: "failure reproduced," "right module selected," "unsafe migration path rejected." The per-step feedback is much more expensive to provide, and its own mistakes can discard a good path. When it's reliable enough, it can catch errors early and avoid wasting time on low-quality paths.

Outcome reward models (ORMs)

Outcome Reward Models (ORMs) evaluate a completed trajectory rather than its intermediate steps. The outcome signal can be binary correctness, a probability or scalar score, or a preference-derived comparison. Exact-answer math makes binary grading convenient, but that isn't part of the ORM definition.

• Score completed trajectories: The model receives an outcome signal after the answer or trajectory is complete, such as exact correctness, a scalar reward, or a preference comparison.
• Simpler supervision boundary: Labels attach to completed outputs rather than every reasoning step. Verifiable tasks can supply these labels automatically; subjective tasks may need learned or human preference signals.
• Search limitation: They can't guide intermediate reasoning steps. A generator might make a fatal error in step 1 but continue generating 100 more steps before the ORM finally rejects it.
• Delayed feedback: The system must generate complete solutions before any evaluation can happen, heavily limiting the usefulness of tree search.

How PRMs work

Process Reward Models (PRMs), in contrast, evaluate the validity of each intermediate step in a chain of thought. By providing step-by-step guidance, they allow search algorithms to quickly abandon incorrect reasoning paths before wasting compute on them.

• Granular evaluation: They score each reasoning step conditioned on the problem and preceding steps, identifying where a path first becomes weak or invalid.
• Early pruning: They can prune low-scoring paths before completion, improving search efficiency when their intermediate scores predict final correctness.
• Complex data requirements: They need step-level supervision from some source. Lightman et al. trained their strongest PRM with human step labels, while methods such as Math-Shepherd derive process targets automatically from rollout outcomes.^{[15]Reference 15Let's Verify Step by Step.https://arxiv.org/abs/2305.20050[16]Reference 16Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationshttps://arxiv.org/abs/2312.08935}
• Denser guidance: A process score gives a search controller earlier evidence for keeping or abandoning a branch, but it's not proof that a kept step is correct.

This side-by-side pseudocode shows when each model can act. The labels are qualitative so they don't imply calibrated probabilities from an unevaluated scorer:

Snell et al.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314} reported that compute-optimal test-time scaling can be more than 4× as efficient as a best-of-N baseline in their math-reasoning evaluation. Lightman et al.^{[15]Reference 15Let's Verify Step by Step.https://arxiv.org/abs/2305.20050} reported that process supervision outperformed outcome supervision on MATH. These results motivate testing PRM-guided pruning on verifiable workloads; they don't establish that every PRM or task benefits.

Not every reasoning system uses a learned PRM or ORM. DeepSeek-R1-Zero trained with rule-based accuracy and format rewards rather than a neural reward model, which is one reason you should treat test-time compute as a family of techniques, not a single stack.^{[5]Reference 5DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

What is the practical difference between an ORM and a PRM during search?

Answer

An ORM scores only the final answer, so it cannot prune based on intermediate evidence. A PRM scores intermediate steps, so a search policy can prune low-scoring branches early when the process scores are reliable.

This is the upside case in which the process scorer prunes the right paths. A deployment evaluation must also count false pruning of branches that would have ended correct.

DeepSeek-R1-Zero and DeepSeek-R1

DeepSeek's contribution is easiest to understand as two related results. DeepSeek-R1-Zero showed that large-scale RL with verifiable rewards can induce strong reasoning behaviors without a supervised cold start.^{[5]Reference 5DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} DeepSeek-R1 then added cold-start data and additional post-training to make those behaviors more stable and readable.^{[5]Reference 5DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

Training pipeline

The final DeepSeek-R1 pipeline isn't "pure RL" end to end. The paper explicitly describes two supervised fine-tuning (SFT) stages and two RL stages:

1. Base model: Start with DeepSeek-V3-Base^{[5]Reference 5DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}, a 671B Mixture-of-Experts (MoE) model with 37B active parameters per token.
2. Cold-start Supervised Fine-Tuning (SFT): Collect thousands of long CoT examples and fine-tune the base model. This helps avoid the readability and language-mixing issues reported for R1-Zero.
3. Reasoning-oriented RL: Run GRPO (Group Relative Policy Optimization)-based RL on reasoning tasks with verifiable, rule-based rewards, similar in spirit to R1-Zero.^{[5]Reference 5DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}
4. Rejection sampling + second SFT: Use the RL checkpoint to generate high-quality reasoning traces, mix them with supervised data from DeepSeek-V3 for non-reasoning domains, and retrain the base model.
5. Final RL for all scenarios: Run another RL stage across a broader prompt mix to produce DeepSeek-R1.

That sequence matters because people often summarize R1 as "pure RL." Only R1-Zero fits that description.^{[5]Reference 5DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

Why is "DeepSeek-R1 was pure RL" an inaccurate summary?

Answer

DeepSeek-R1-Zero was trained with large-scale RL without a supervised cold start. DeepSeek-R1 added cold-start SFT, reasoning RL, rejection sampling plus another SFT stage, and final broad RL.

Emergent reasoning

R1-Zero displayed behaviors such as self-verification, backtracking, and longer rollouts, but the paper also reports readability issues and occasional language mixing.^{[5]Reference 5DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} That detail matters. It means verifiable rewards can induce useful reasoning behavior, but raw RL rollouts still need cleanup before they become a general-purpose product.

What did R1-Zero prove, and what did it not solve by itself?

Answer

It showed that verifiable rewards can induce reasoning behaviors like self-verification and backtracking. It didn't by itself solve readability, language mixing, or broad product polish.

Compute-optimal inference: when to think harder

Not all problems benefit equally from extended reasoning, and on some tasks extra thinking actively hurts. Ghosal et al. studied this directly: across reasoning models, accuracy often rises with a little more thinking and then falls as the trace grows, an inverted-U rather than a monotonic climb.^{[12]Reference 12Does Thinking More Always Help? Mirage of Test-Time Scaling in Reasoning Modelshttps://arxiv.org/abs/2506.04210} They attribute the apparent early gains partly to higher output variance instead of genuinely better reasoning, and they recommend parallel sampling (best-of-N) over endlessly extending one trace. An Anthropic study found the same inverse-scaling effect on several tasks, where longer reasoning amplified distractions and errors rather than fixing them.^{[17]Reference 17Inverse Scaling in Test-Time Computehttps://alignment.anthropic.com/2025/inverse-scaling/} The lesson for production is that thinking budget is a real tuning parameter with a sweet spot, not a slider you turn to maximum.

The optimal test-time compute allocation depends on problem difficulty:

The cross-over point

A task family may have a measured compute budget where a smaller model plus a test-time policy beats a larger single-pass model:

$Q_{\text{small+policy}}(C^*) > Q_{\text{large,single}}$

Here, $Q$ is measured quality and $C^*$ is one evaluated inference budget. Finding a crossover at $C^*$ doesn't mean every larger budget keeps helping. Quality can plateau or fall, and the preferred path can change with task difficulty, latency limits, verifier quality, and how many requests you expect to serve.

Snell et al.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314} observed that in FLOPs-matched evaluations, a smaller model with additional test-time compute can outperform a model roughly 14× larger answering in a single pass on problems where the small model already achieves non-trivial success rates. This makes longer thinking a candidate for compensating for model size on similar evaluated tasks, not a general replacement for larger models.

What is the compute-optimal routing question for a reasoning workload?

Answer

Ask whether extra inference compute on a smaller or cheaper model beats a larger model answering once for this task family, latency budget, verifier quality, and target accuracy.

Production systems often use a routing layer to handle varying difficulty without exploding cost. The router evaluates task complexity and directs the request to the appropriate model and test-time path, balancing latency, quality, and computational cost.

Why should a production system route tasks before invoking an expensive reasoning path?

Answer

Reasoning paths add latency, output tokens, KV cache pressure, and cost. Routing reserves that budget for tasks where extra reasoning is likely to improve verifiable quality.

Production considerations

Reasoning traffic changes capacity planning. Longer traces and more branches raise cache residency, first-token delay, and cost even when the visible answer stays short.

KV cache pressure and prefix sharing

Reasoning workloads stress inference engines differently from ordinary chat because they can generate far more intermediate tokens before or between visible outputs.

For every generated token, the server appends a Key vector and a Value vector for every layer:

$\text{KV bytes per token} = 2 \times L \times n_{kv} \times d_h \times b$

Where $L$ is the number of layers, $n_{kv}$ is the number of KV heads, $d_h$ is the head dimension, and $b$ is bytes per value. The important scaling fact is simple: KV cache memory grows linearly with sequence length. A rollout that spends 10,000 tokens thinking creates roughly 10,000 tokens' worth of KV state, which can collapse batch size long before raw FLOPs become the bottleneck.

PagedAttention stores KV cache in fixed-size blocks to reduce fragmentation and make scheduling practical at long context lengths.^{[18]Reference 18Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} When multiple rollouts share the same prompt or partial trace, prefix-sharing runtimes can reuse that cached prefix instead of duplicating it across every branch. SGLang's RadixAttention is a good example.^{[19]Reference 19SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104} This matters a lot for best-of-N, self-consistency, and tree search, where many candidates share the same long prompt and diverge only near the leaves.

Why do reasoning workloads stress KV cache more than ordinary chat?

Answer

Non-visible-token modes and branching search can generate many extra tokens before or between visible outputs. In transformer serving paths that materialize those tokens, each one appends K/V state across layers, reducing batch size and concurrency unless the runtime pages, shares, or evicts cache efficiently.

TTFT vs. inter-token latency

Test-time compute distorts user-facing latency metrics. For a provider or wrapper that generates non-visible reasoning before revealing the answer, TTFT (time to first token) can increase substantially.^{[4]Reference 4Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} An interleaved mode can also add pauses between visible chunks or tool calls. ITL (inter-token latency) can still be fine once answer text starts streaming. A reasoning system can feel slow even when decode throughput is healthy.

Those reasoning tokens are still decode load for the fleet, even when the user never sees them. Under continuous batching they occupy KV residency and memory bandwidth, raise other tenants' ITL, and reduce goodput (offered request rate under joint TTFT and decode SLOs). A reasoning router is therefore also a capacity-control device: routing hard queries to long rollouts must leave headroom for co-tenant latency as well as the request's own TTFT budget.

Why can a reasoning model feel slow even if decode tokens per second are healthy?

Answer

In a mode that generates non-visible reasoning first, those tokens raise TTFT. In an interleaved mode, additional thinking can also create pauses between visible chunks or tool calls. ITL during answer streaming can still look normal.

Latency vs. quality trade-off

Reasoning models add latency, but the exact numbers depend on hardware, batching policy, and provider implementation:

For strict interactive latency budgets, a long reasoning path may be unacceptable unless measured gains justify it. Candidates for larger budgets include:

• Batch processing (code review, data analysis)
• High-stakes analyst workflows (incident review, safety analysis, compliance analysis)
• Asynchronous workflows (software engineering, research)

Cost implications

Extended reasoning is expensive even when the final answer is short. Hosted APIs may charge directly for hidden reasoning tokens. OpenAI's reasoning docs, for example, note that these tokens are billed as output tokens even though they aren't returned verbatim in the API response.^{[4]Reference 4Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} Open-weight deployments pay through longer wall-clock time, lower throughput, and higher KV cache residency. Either way, longer rollouts reduce how many concurrent requests the same GPU can serve.

Production systems usually combine three controls:

1. Routing: send only hard or high-stakes tasks to expensive reasoning paths
2. Token budgets: cap how long a rollout may think before the marginal gain stops being worth it
3. Early stopping: terminate search when verifier scores stop improving

Without those controls, ambiguous or unsolvable prompts can burn a large amount of inference budget without producing a better answer.

What three controls keep reasoning costs bounded in production?

Answer

Route only hard or high-stakes tasks to reasoning paths, cap reasoning tokens or branches, and stop early when verifier scores or expected gains plateau.

Distillation: making reasoning affordable

Distillation is like having a senior incident engineer write detailed recovery traces, then training a smaller model to solve similar outages the same way. The smaller model won't match the source model everywhere, but it can inherit much of the solution style while being much cheaper to serve.

DeepSeek-R1 showed that reasoning capabilities can be distilled into much smaller models:

In DeepSeek's reported AIME evaluation, the 7B and 32B distilled models retain substantial benchmark accuracy at lower parameter counts than R1. Parameter count is only a proxy, though. Real deployment cost still depends on active parameters, quantization, batch size, and engine choice. Distillation can make reasoning cheaper, but it doesn't remove the need for routing and token budgets.

What can reasoning distillation transfer, and what can it fail to transfer?

Answer

It can transfer solution traces, formats, and many task patterns from a stronger teacher. It may not transfer the teacher's full search policy, verifier behavior, or reliability on unfamiliar tasks.

Trace cloning also has a scaling ceiling. Setlur et al. compare verifier-free methods that distill or clone successful traces with verifier-based methods that use rewards, RL, or search.^{[20]Reference 20Scaling Test-Time Compute Without Verification or RL is Suboptimalhttps://arxiv.org/abs/2502.12118} Under the paper's assumptions and evaluated reasoning tasks, verifier-free methods become increasingly suboptimal as rollout length and test-time budget grow because the student learns observed traces without learning which alternative traces a verifier would reward.

That result doesn't make distillation useless. The DeepSeek numbers above show that a smaller student can gain substantial capability. It means long-rollout scaling should not assume more teacher traces will reproduce verifier-guided improvement indefinitely. Filter traces with reliable checks, retain a verifier at evaluation time, or train with verifier-guided RL when the workload supports trustworthy rewards.

Why doesn't cloning more reasoning traces guarantee the same scaling as verifier-guided RL or search?

Answer

Cloning teaches the student to imitate observed successful traces. It doesn't directly teach a reward over alternative traces, so the gap can widen as rollout length and search budget grow. Distillation remains useful, but its ceiling should be measured with a verifier-backed evaluation.

What to remember and where to go next

1. Test-time compute adds a second scaling axis: spending more compute at inference can outperform moving to a larger single-pass model on hard reasoning tasks.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314}

2. Test-time compute is an umbrella, not one algorithm: longer scratchpads, repeated sampling, revision loops, and explicit search all fit under the same idea.

3. Good verifiers and compute-optimal policies matter: Snell et al. report more than 4× better test-time compute efficiency than a best-of-N baseline in their evaluation, and PRMs can support earlier pruning because they score intermediate steps instead of waiting for the end.^{[2]Reference 2Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314[15]Reference 15Let's Verify Step by Step.https://arxiv.org/abs/2305.20050}

4. DeepSeek-R1-Zero and DeepSeek-R1 aren't the same result: R1-Zero showed emergent reasoning from pure RL, while DeepSeek-R1 added cold-start data plus additional SFT and RL stages to make that behavior readable and broadly usable.^{[5]Reference 5DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}

5. Thinking is now a dial, and the sweet spot isn't always the maximum: current providers expose model- and API-specific effort or thinking controls, and accuracy can follow an inverted-U as the trace grows, so tune the supported knob rather than maxing it.^{[4]Reference 4Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning[10]Reference 10Gemini thinkinghttps://ai.google.dev/gemini-api/docs/thinking[11]Reference 11Building with extended thinkinghttps://docs.claude.com/en/docs/build-with-claude/extended-thinking[12]Reference 12Does Thinking More Always Help? Mirage of Test-Time Scaling in Reasoning Modelshttps://arxiv.org/abs/2506.04210}

6. Serving bottlenecks matter as much as algorithms: KV cache memory grows linearly with long reasoning traces, so prefix sharing, routing, and token budgets are core production tools.^{[18]Reference 18Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180[19]Reference 19SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104}

Explain why a single-pass model can miss "How many r's in strawberry," why a smaller model with traces, search, or distillation can recover much of a larger model's math performance, and why a production stack needs a routing layer before it needs a bigger GPU.