Reasoning & Test-Time Compute

State Space Models showed one way to keep decode state from growing with context length by compressing history into recurrent state. Reasoning models move in the opposite direction on purpose: spend more inference compute when the task is hard enough to justify it.

Think about two dispatch planners handling the same delayed-truck problem. The first planner accepts the first reroute that looks plausible. The second planner checks carrier capacity, hub cutoffs, refund risk, and customer priority before committing. When those checks catch a real mistake, the second plan performs better, not because the planner has different data, but because it spends more compute on the decision.

The same idea now shapes how frontier large language models (LLMs) are built and evaluated. Classic scaling work focused on train-time compute: more parameters, more data, and more pretraining FLOPs (floating-point operations).^[1] Work from 2024 onward made a second axis impossible to ignore: on hard reasoning tasks, you can often get better answers by spending more compute during generation itself.^[2][3] This is test-time compute scaling. Reasoning-model APIs from OpenAI and open-weight systems such as DeepSeek-R1 made this shift visible, and Snell et al. showed that, on tasks where a smaller model already has a non-trivial chance of success, extra inference-time compute can beat a much larger single-pass model.^[3][4][5][2]

Provider controls are model-specific and they change. OpenAI documents reasoning.effort; Google's Gemini docs use thinkingLevel for Gemini 3 models and thinkingBudget for Gemini 2.5 models; Anthropic's current Claude Opus 4.7 docs use adaptive thinking with an effort parameter, while older Claude models use manual extended thinking with budget_tokens.^[4][6][7] The durable skill is deciding how much compute a request deserves, because more thinking is not always better.^[8]

In this article you'll learn why some problems need deliberate reasoning instead of fast pattern matching, how test-time compute scaling works, and what it means for production systems. You'll need a basic mental model of how transformers predict the next token (covered earlier in the preparation path). By the end, you'll be able to choose between single-pass, best-of-N, and guided-search strategies, and you'll know why routing and token budgets are as important as the algorithms themselves.

System 1 and System 2 thinking

Psychologist Daniel Kahneman's framework is a useful starting point. System 1 is fast, instinctive, and pattern-driven. When you read the word "strawberry" and immediately know it's a fruit, that's System 1. System 2 is slow, deliberate, and step-by-step. When you count how many times the letter "r" appears in "strawberry," you have to switch to System 2 because your gut reaction ("two?") is often wrong. The correct answer is three, but you only get there by checking each letter deliberately.

The System 1/System 2 distinction is an analogy for serving behavior, not an architectural taxonomy. A low-budget single-pass call may answer from strong patterns; a model or surrounding system with more budget can check intermediate work, sample alternatives, or run tools. On tasks that require counting, lookahead, or multi-step deduction, that extra verification path can matter.

Reasoning-oriented models and systems are designed to make deliberate behavior easier to buy at inference time. Depending on the model and wrapper, they may spend non-visible reasoning tokens, sample alternatives, run an external verifier, or revise a candidate. Think of the difference between a routing system that accepts the first carrier option and one that checks cutoff times, capacity, destination risk, and refund impact before committing.

This shift from pure pattern matching to deliberate reasoning is what makes test-time compute scaling possible.

Why low-budget generation can fail on hard problems

Here's a concrete example. Ask a model: "How many times does the letter 'r' appear in the word 'strawberry'?" A rushed answer can be "two," because the third "r" in "strawberry" is easy to miss without a check.

The problem isn't that the model lacks knowledge. It's that the task requires deliberate step-by-step verification, not fast association. Pretraining optimizes next-token prediction, which rewards fluent continuations more directly than checked final answers. On problems like complex math, code debugging, or logistics planning, the same dynamic appears: the model produces a plausible-looking answer that collapses under scrutiny.

Test-time compute scaling can address this failure mode by spending budget before committing. Instead of one shot, the model or the system around it can explore multiple approaches, verify intermediate steps, and select a candidate. The extra compute only helps when exploration or verification changes the result for the better.

The two axes of compute scaling

Here's a useful analogy: think of a fulfillment network. Train-time compute is like months spent tuning forecasts, routes, and warehouse procedures before peak season. Test-time compute is the effort spent during a live incident: trying alternate routes, checking cutoff times, and verifying the plan before committing.

Classic scaling emphasized the pre-season tuning. Evaluations on reasoning tasks now show that inference-time allocation can sometimes compete with moving to a larger single-pass model.

Traditional scaling focuses on train-time compute, increasing model parameters $N$ or training tokens $D$. The relationship between compute and loss follows a power law:

$L_{\text{train}}(C) \propto C^{-\alpha}, \quad C = 6ND$

Empirical scaling studies fit loss with an approximate power law of compute $C$, where training compute is often estimated as roughly 6 × model parameters $N$ × training tokens $D$.^[1] Within the measured regime, increasing training compute reduces loss predictably enough to guide training decisions.

Test-time compute scaling adds a second axis, inference compute $C_{\text{infer}}$. A practical mental model is:

$\text{accuracy} = g(C_{\text{infer}}, \text{task}, \text{policy}, \text{verifier})$

In a measured operating range, more inference compute can improve accuracy with diminishing returns. It can also plateau or reduce accuracy when a trace overthinks, sampled candidates are correlated, or a verifier selects the wrong branch. That extra compute can come from longer reasoning traces, repeated sampling, revision loops, or explicit search with a verifier.^[2][8]

Snell et al.^[2] don't claim one universal law for every model and every benchmark. Their more useful result is operational: if you allocate inference compute well, test-time scaling is strong enough that it can outperform simply buying a larger model and sampling once.

This represents a fundamental shift in how we approach problem-solving with AI models:

This evaluation table encodes the production question: choose the best measured quality under the service-level objective, rather than assuming the longest trace is best.

How reasoning models work

At a high level, reasoning-oriented training and inference policies aim to allocate additional tokens or branches before returning an answer. The visible behavior can include decomposition, checks, or revision, but those behaviors are capabilities to evaluate rather than guarantees of every response.

Extended chain-of-thought

Reasoning models often generate long scratchpads or intermediate traces before producing a final answer. Sometimes that trace is exposed, sometimes it's summarized, and sometimes it's hidden entirely by provider policy. What matters isn't whether the user sees every token, but whether the model is allowed to spend additional inference-time compute before answering.

This built-in reasoning process differs from prompted chain-of-thought (CoT)^[9] in several key ways:

1. Usually learned during post-training via Reinforcement Learning (RL), distillation, or both, rather than relying on prompt wording alone.
2. Potentially variable-length: the API or serving policy can permit different budgets by request
3. Can include self-correction: a trace may backtrack, recognize dead ends, and revise earlier steps
4. Often hidden or summarized: providers may not expose the raw reasoning tokens directly

Current reasoning APIs make this concrete. OpenAI's reasoning docs describe reasoning tokens as non-visible output tokens that still consume context budget, and they recommend giving the task, constraints, and desired output format while treating reasoning effort as a tuning knob.^[4]

The thinking budget is now a primary control surface, even though providers expose it with model-specific knobs. OpenAI's reasoning.effort sets an effort level for reasoning models.^[4] Gemini 3 models use thinkingLevel, while Gemini 2.5 models retain a numeric thinkingBudget.^[6] Claude Opus 4.7 uses adaptive thinking plus effort; Anthropic documents budget_tokens for older Claude extended-thinking models.^[7] Provider prompting guidance also differs by model, so start with a clear task and constraints, tune the supported knob, and evaluate instead of assuming a chain-of-thought prompt helps.

The important point is that test-time compute can mean either a longer single trace or multiple sampled traces. OpenAI's o1 launch post made this visible at the benchmark level: on AIME 2024 (a math competition benchmark), reported accuracy improved when the system moved from a single sample to consensus over 64 samples and then to learned reranking over 1000 samples.^[3] Treat this as one reported evaluation result, not a guarantee for other tasks or selection rules.

Deliberation and search at inference time

Test-time compute is an umbrella term, not a single algorithm. Some systems sample many complete answers and pick a winner. Some iteratively critique and revise one candidate. Others run explicit search over partial reasoning states using a verifier or reward model. The diagram below shows these patterns as branches of the same idea: spend extra compute to explore, score, and refine before returning an answer.

Not every reasoning model literally runs beam search or a PRM at inference time. The shared pattern is optional inference compute allocation. A good routing policy keeps easy requests cheap and assigns more tokens, branches, or verification only where evaluation shows a payoff.

Provider-reported hidden reasoning

Some hosted reasoning APIs report non-visible reasoning tokens in usage accounting while returning only the final visible answer. OpenAI's reasoning-token documentation is one concrete example.^[4] Do not generalize this into one universal architecture: an open-weight deployment, explicit search wrapper, or provider with summarized thinking can expose and account for intermediate work differently.

For such APIs, the non-visible tokens matter for production because they consume context window space, increase key-value (KV) cache pressure, and may count toward billing even though users never see them.^[4]

Use the provider's usage schema and pricing table when implementing this calculation; the example demonstrates why billing and capacity dashboards cannot count visible text alone.

Test-time compute strategies

Several strategies allocate compute at inference time, each with different characteristics:

1. Best-of-N sampling

This is the simplest strategy: generate 16 candidate recovery plans for a delayed order and keep the one with the best verifier score. Each attempt is independent, and more attempts raise your ceiling as long as your verifier or selection rule can reliably identify the best one.

The conceptual interface sketch below demonstrates this approach. It takes a generative model and a prompt as inputs, alongside the number of desired completions ($N$). It returns the single best response by either scoring them with a reward model or falling back to self-consistency (majority vote).

Cost note: This method is easily parallelizable (all N attempts can run simultaneously), making it simple to implement but potentially expensive since you pay for all N completions.

Scaling behavior

With an oracle verifier and independent samples, the chance of generating at least one correct answer in $N$ tries follows:

$P_{\text{success}} = 1 - (1 - p)^N$

Where $p$ is the base probability of the model generating a correct answer on a single attempt. Real systems do worse than this idealized formula because samples are correlated and verifiers make mistakes, but the equation explains why best-of-N works at all. OpenAI's o1 launch post reports the same pattern on AIME 2024: o1 improved from 74% with one sample to 83% with 64-sample consensus and 93% when reranking 1000 samples with a learned scorer.^[3]

The toy calculation shows why best-of-N is attractive and dangerous. The idealized success curve rises quickly, but cost rises linearly with every sampled completion. Real systems also plateau earlier because samples share model biases and verifiers make mistakes.

This toy selector is deliberately simple: it shows that generating a correct candidate and selecting it are separate failure surfaces.

2. Sequential revision

This is like drafting a carrier-recovery plan, checking it against policy and capacity, then revising the weak steps before committing to the customer. Unlike best-of-N (where each attempt is independent), each revision builds on the previous one.

The conceptual interface below takes an initial prompt and a model. It generates an initial response and then enters a loop, asking the model to critique its own output. If errors are found, it uses the critique to generate an improved version, returning the final refined text. The error detection ("no errors found") is intentionally simplified; production systems use a trained verifier model or structured critique format.

3. Tree search with process reward models

This is the most sophisticated strategy, like a fulfillment planner who considers multiple recovery paths, evaluates each action, and prunes bad branches early. Instead of scoring just the final answer ("did you choose refund or reship?"), a process reward model scores each intermediate step ("is this policy check correct?").

Here's how a simplified beam search using a PRM operates. It accepts a model and a PRM, taking a prompt as the initial state. At each step, it generates multiple possible next steps (the beam width), scores them with the PRM, and keeps only the highest-scoring paths. The function outputs the best completed reasoning chain. This is a conceptual interface sketch; it assumes model.generate_step, prm.score_step, and is_final_answer exist.

Beam search is easiest to explain, but it has a well-known weakness for reasoning: beams can collapse onto near-duplicate traces. Stronger systems inject diversity, allow backtracking, or use Monte Carlo Tree Search-style expansion policies. The hard part isn't building a tree. It's having a verifier good enough to prune bad branches early.

Process reward models vs. outcome reward models

The choice of reward model fundamentally affects search efficiency:

Think of it as final delivery audit versus checkpoint audit. An ORM only checks whether the final plan works: "delivered on time" or "missed SLA." You have no idea where the plan broke. A PRM checks each intermediate decision: "route chosen correctly," "hub cutoff valid," "carrier capacity wrong here." The per-step feedback is much more expensive to provide, and its own mistakes can discard a good path. When it is reliable enough, it can catch errors early and avoid wasting time on low-quality paths.

Outcome reward models (ORMs)

Outcome Reward Models (ORMs) evaluate the final generated output as a single, complete result. They act as a binary judge at the end of the generation process, determining only whether the final answer matches the expected solution.

• Score only the final answer: The model receives a pass/fail grade based entirely on the conclusion (e.g., whether the final number is correct).
• Simpler training: They're easier to train because they only need (question, answer, correctness) labels, which are readily available in many datasets.
• Search limitation: They cannot guide intermediate reasoning steps. A generator might make a fatal error in step 1 but continue generating 100 more steps before the ORM finally rejects it.
• Delayed feedback: The system must generate complete solutions before any evaluation can happen, heavily limiting the usefulness of tree search.

How PRMs work

Process Reward Models (PRMs), in contrast, evaluate the validity of each intermediate step in a chain of thought. By providing step-by-step guidance, they allow search algorithms to quickly abandon incorrect reasoning paths before wasting compute on them.

• Granular evaluation: They score each reasoning step independently, identifying exactly where logic breaks down.
• Early pruning: They can prune low-scoring paths before completion, improving search efficiency when their intermediate scores predict final correctness.
• Complex data requirements: They require step-level annotations for training, which are notoriously expensive and difficult for humans to collect at scale.
• Denser guidance: A process score gives a search controller earlier evidence for keeping or abandoning a branch, but it is not proof that a kept step is correct.

Here is a conceptual side-by-side comparison of how each model scores the same problem:

Snell et al.^[2] reported that compute-optimal test-time scaling can be more than 4× as efficient as a best-of-N baseline in their math-reasoning evaluation. Lightman et al.^[10] reported that process supervision outperformed outcome supervision on MATH. These results motivate testing PRM-guided pruning on verifiable workloads; they do not establish that every PRM or task benefits.

That said, not every reasoning system uses a learned PRM or ORM. DeepSeek-R1-Zero trained with rule-based accuracy and format rewards rather than a neural reward model, which is one reason you should treat test-time compute as a family of techniques, not a single stack.^[5]

This is the upside case in which the process scorer prunes the right paths. A deployment evaluation must also count false pruning of ultimately correct branches.

DeepSeek-R1-Zero and DeepSeek-R1

DeepSeek's contribution is easiest to understand as two related results. DeepSeek-R1-Zero showed that large-scale RL with verifiable rewards can induce strong reasoning behaviors without a supervised cold start.^[5] DeepSeek-R1 then added cold-start data and additional post-training to make those behaviors more stable and readable.^[5]

Training pipeline

The final DeepSeek-R1 pipeline isn't "pure RL" end to end. The paper explicitly describes two supervised fine-tuning (SFT) stages and two RL stages:

1. Base model: Start with DeepSeek-V3-Base^[5], a 671B Mixture-of-Experts (MoE) model with 37B active parameters per token.
2. Cold-start Supervised Fine-Tuning (SFT): Collect thousands of long CoT examples and fine-tune the base model. This helps avoid the readability and language-mixing issues reported for R1-Zero.
3. Reasoning-oriented RL: Run GRPO (Group Relative Policy Optimization)-based RL on reasoning tasks with verifiable, rule-based rewards, similar in spirit to R1-Zero.^[5]
4. Rejection sampling + second SFT: Use the RL checkpoint to generate high-quality reasoning traces, mix them with supervised data from DeepSeek-V3 for non-reasoning domains, and retrain the base model.
5. Final RL for all scenarios: Run another RL stage across a broader prompt mix to produce DeepSeek-R1.

That sequence matters because people often summarize R1 as "pure RL." Only R1-Zero fits that description.^[5]

Emergent reasoning

R1-Zero displayed behaviors such as self-verification, backtracking, and longer rollouts, but the paper also reports readability issues and occasional language mixing.^[5] That detail matters. It means verifiable rewards can induce sophisticated reasoning behavior, but raw RL rollouts still need cleanup before they become a general-purpose product.

Compute-optimal inference: when to think harder

Not all problems benefit equally from extended reasoning, and on some tasks extra thinking actively hurts. Ghosal et al. studied this directly: across reasoning models, accuracy often rises with a little more thinking and then falls as the trace grows, an inverted-U rather than a monotonic climb.^[8] They attribute the apparent early gains partly to higher output variance instead of genuinely better reasoning, and they recommend parallel sampling (best-of-N) over endlessly extending one trace. An Anthropic study found the same inverse-scaling effect on several tasks, where longer reasoning amplified distractions and errors rather than fixing them.^[11] The lesson for production is that thinking budget is a real tuning parameter with a sweet spot, not a slider you turn to maximum.

The optimal test-time compute allocation depends on problem difficulty:

The cross-over point

A task family may have a compute budget threshold where test-time scaling becomes more efficient than using a larger single-pass model. Below or above that point, the preferred path must be established with quality, latency, and cost measurements:

$L_{\text{small+search}}(C) < L_{\text{large,single}} \quad \text{for some } C \ge C^*$

There's a compute budget $C^*$ above which a small model that searches through multiple solutions can outperform a large model answering in one shot. This isn't universal across all tasks, but it appears clearly on hard reasoning problems where the smaller model already has a non-zero hit rate.

Snell et al.^[2] observed that in FLOPs-matched evaluations, a smaller model with additional test-time compute can outperform a model roughly 14× larger answering in a single pass on problems where the small model already achieves non-trivial success rates. This makes longer thinking a candidate for compensating for model size on similar evaluated tasks, not a general replacement for larger models.

To handle varying levels of difficulty in production without exploding costs, systems often implement a routing layer. This architecture evaluates an incoming task's complexity and directs it to the appropriate model and test-time strategy, balancing latency, quality, and computational cost.

Production considerations

Deploying reasoning models in a real-world environment requires careful planning. While the capability gains are substantial, they come with operational challenges that engineering teams must proactively address.

KV cache pressure and prefix sharing

Reasoning workloads stress inference engines differently from ordinary chat because they often generate far more intermediate tokens before the visible answer arrives.

For every generated token, the server appends a Key vector and a Value vector for every layer:

$\text{KV bytes per token} = 2 \times L \times n_{kv} \times d_h \times b$

Where $L$ is the number of layers, $n_{kv}$ is the number of KV heads, $d_h$ is the head dimension, and $b$ is bytes per value. The important scaling fact is simple: KV cache memory grows linearly with sequence length. A rollout that spends 10,000 tokens thinking creates roughly 10,000 tokens' worth of KV state, which can collapse batch size long before raw FLOPs become the bottleneck.

PagedAttention stores KV cache in fixed-size blocks to reduce fragmentation and make scheduling practical at long context lengths.^[12] When multiple rollouts share the same prompt or partial trace, prefix-sharing runtimes can reuse that cached prefix instead of duplicating it across every branch. SGLang's RadixAttention is a good example.^[13] This matters a lot for best-of-N, self-consistency, and tree search, where many candidates share the same long prompt and diverge only near the leaves.

TTFT vs. inter-token latency

Test-time compute distorts user-facing latency metrics. For a provider or wrapper that generates non-visible reasoning before revealing the answer, TTFT (time to first token) can increase substantially.^[4] ITL (inter-token latency) can still be fine once the answer starts streaming. In other words, a reasoning system can feel slow even when decode throughput is healthy.

Latency vs. quality trade-off

Reasoning models introduce significant latency, but the exact numbers depend on hardware, batching policy, and provider implementation:

For strict interactive latency budgets, a long reasoning path may be unacceptable unless measured gains justify it. Candidates for larger budgets include:

• Batch processing (code review, data analysis)
• High-stakes analyst workflows (chargeback review, fraud review, compliance analysis)
• Asynchronous workflows (software engineering, research)

Cost implications

Extended reasoning is expensive even when the final answer is short. Hosted APIs may charge directly for hidden reasoning tokens. OpenAI's reasoning docs, for example, note that these tokens are billed as output tokens even though they aren't returned verbatim in the API response.^[4] Open-weight deployments pay through longer wall-clock time, lower throughput, and higher KV cache residency. Either way, longer rollouts reduce how many concurrent requests the same GPU can serve.

Production systems usually combine three controls:

1. Routing: send only hard or high-stakes tasks to expensive reasoning paths
2. Token budgets: cap how long a rollout may think before the marginal gain stops being worth it
3. Early stopping: terminate search when verifier scores stop improving

Without those controls, ambiguous or unsolvable prompts can burn a large amount of inference budget without producing a better answer.

Distillation: making reasoning affordable

Distillation is like having a senior logistics planner write detailed recovery traces, then training a smaller model to solve similar incidents the same way. The smaller model won't match the source model everywhere, but it can inherit much of the solution style while being much cheaper to serve.

DeepSeek-R1 showed that reasoning capabilities can be distilled into much smaller models:

In DeepSeek's reported AIME evaluation, the 7B and 32B distilled models retain substantial benchmark accuracy at lower parameter counts than R1. Parameter count is only a proxy, though. Real deployment cost still depends on active parameters, quantization, batch size, and engine choice. Distillation can make reasoning cheaper, but it doesn't remove the need for routing and token budgets.

Mastery check

Evaluation rubric

• Scaling view: train-time compute improves the model before deployment, while test-time compute spends extra budget during generation.
• Strategy view: longer traces, best-of-N, sequential revision, and verifier-guided tree search are different ways to buy more deliberation.
• Verifier view: ORMs score final answers, while reliable PRMs let a search policy prune low-scoring branches early.
• Systems view: when an API uses and reports non-visible reasoning tokens, they still affect context budget, KV cache, TTFT, throughput, and billing.
• Routing view: reasoning paths are candidates when the task is hard, verifiable, and important enough to justify measured latency and cost.
• Budget view: provider/model knobs differ (reasoning.effort, Gemini thinkingLevel or thinkingBudget, Claude adaptive effort or legacy budget_tokens), and more thinking is not always better because accuracy can follow an inverted-U.
• DeepSeek view: R1-Zero showed pure-RL reasoning behavior; R1 added cold-start data, SFT, and additional RL to make the behavior more usable.

Follow-up questions

When should you choose a reasoning model over a standard model?

Evaluate a reasoning path when the task needs multi-step deduction and you have a way to check whether the answer is good. Math, formal verification, code debugging, and structured planning are promising candidates. Start with a single-pass baseline for summarization, translation, extraction, and conversational UX, then promote only workloads where measured correctness gains justify latency and cost.

How does test-time compute interact with KV-cache optimization?

Reasoning workloads generate longer rollouts or multiple branches, which increases KV-cache pressure because each new token adds per-layer K/V state. KV-cache memory grows linearly with sequence length, so long reasoning traces reduce batch size and throughput. Engines like vLLM use paging to manage that memory, and prefix-sharing runtimes can reuse the shared prefix across best-of-N or search branches instead of duplicating it.

What are the failure modes of extended reasoning?

Common failure modes include loops, error propagation, overthinking, and weak self-verification. If the same model generates and judges, its blind spots are correlated, so it may confidently approve a bad trace. PRMs, task-specific checkers, explicit token budgets, and early stopping help by scoring intermediate steps and terminating unproductive branches earlier.

How can you distill reasoning into a smaller model?

Distillation uses traces or verifier-filtered answers from a stronger reasoning system to train a smaller student. A student can learn useful solution patterns, as the DeepSeek results illustrate, but the gain must be measured against a comparable baseline. The catch is that the student may imitate output format more easily than the teacher's actual search policy, so good benchmark scores don't always mean the smaller model learned the same underlying reasoning process.

How would you set a first-pass router for mixed workloads?

Start by separating tasks into three buckets: easy pattern matching, medium tasks with occasional reasoning upside, and hard tasks with clear verification signals. Route the first bucket to a fast single-pass model, route the second to a bounded reasoning budget or brief best-of-N, and reserve verifier-guided search for the third. Then instrument latency, total generated tokens, and verified win rate so the router can move requests down when extra thinking stops paying off.

Common pitfalls

Even experienced engineers misapply reasoning models. Here are frequent pitfalls, with symptoms to watch for and concrete fixes.

The intelligence tax

Symptom: A production task takes 30 seconds to complete and costs five times more than the standard model, but the output quality is identical.

Cause: You're using a reasoning model for a task that doesn't show measurable benefit from extra deduction. Summarization, translation, simple extraction, and sentiment analysis are often strong single-pass baselines. Extra thinking time may add latency and cost without an accuracy gain.

Fix: Route easy tasks to fast single-pass models. Reserve reasoning models for math, code, planning, verification, and other tasks where you can actually verify whether the answer is correct.

Confusing prompting with architecture

Symptom: You assume a reasoning model is just a standard model with a hidden system prompt like "You are a careful thinker..."

Cause: The API hides the reasoning tokens, so it's tempting to imagine they're produced by a wrapper around a normal model. That mental model misses the training difference.

Fix: Reasoning models may be post-trained, often with RL or distillation, to use extra inference compute productively. Treat this as different model behavior and serving accounting, not a prompt hack or necessarily a new transformer architecture.

Treating visible chain-of-thought as the whole system

Symptom: You assume a visible "think step by step" trace, a hidden reasoning trace, best-of-N, PRM search, and RL with verifiable rewards are interchangeable.

Cause: They all spend extra inference or training budget around reasoning, but they solve different problems.

Fix: Name the mechanism. Prompted CoT changes the prompt. Hidden reasoning changes model behavior and serving accounting. Best-of-N changes sampling. PRM search changes branch selection. Verifiable-reward RL changes post-training.

Ignoring KV cache pressure

Symptom: A reasoning rollout looks cheap because the final answer is short, but batch size collapses under load.

Cause: Hidden and candidate tokens still create K/V state across layers.

Fix: Track total generated tokens, not just visible tokens. Use paging, prefix sharing, routing, and hard budgets before scaling reasoning traffic.

Infinite loops and overthinking

Symptom: Token usage spikes on certain prompts, the model spins in circles repeating the same deduction, or accuracy drops on simple prompts when the budget is set high.

Cause: No reasoning budget was set, or the budget was set too high. On ambiguous or simple prompts, more thinking can amplify distractions and errors rather than fix them, so accuracy follows an inverted-U as the trace grows.^[8][11]

Fix: Set a reasoning budget tuned to the task instead of maximizing it, and add a hard cap. Implement early stopping when verifier confidence plateaus. For simple prompts, lower the effort level. Monitor per-request token distributions and alert on outliers.

Practice

Before you move on, try this short audit to make the trade-offs concrete.

The Efficiency Audit

Consider these five production tasks. For each one, predict whether it benefits from test-time compute scaling, then check your reasoning against the explanation.

The warehouse calculation and microservice debugging are the strongest candidates because they provide verification signals. The core idea is that test-time compute is worth testing when the task requires verifiable, multi-step reasoning, not merely because a task sounds difficult.

What to remember and where to go next

1. Test-time compute adds a second scaling axis: spending more compute at inference can outperform moving to a larger single-pass model on hard reasoning tasks.^[2]

2. Test-time compute is an umbrella, not one algorithm: longer scratchpads, repeated sampling, revision loops, and explicit search all fit under the same idea.

3. Good verifiers and compute-optimal policies matter: Snell et al. report more than 4× better test-time compute efficiency than a best-of-N baseline in their evaluation, and PRMs can support earlier pruning because they score intermediate steps instead of waiting for the end.^[2][10]

4. DeepSeek-R1-Zero and DeepSeek-R1 aren't the same result: R1-Zero showed emergent reasoning from pure RL, while DeepSeek-R1 added cold-start data plus additional SFT and RL stages to make that behavior readable and broadly usable.^[5]

5. Thinking is now a dial, and the sweet spot is not always the maximum: current providers expose model-specific effort or thinking controls, and accuracy can follow an inverted-U as the trace grows, so tune the supported knob rather than maxing it.^[4][6][7][8]

6. Serving bottlenecks matter as much as algorithms: KV cache memory grows linearly with long reasoning traces, so prefix sharing, routing, and token budgets are core production tools.^[12][13]

You should now be able to explain why a single-pass model can miss "How many r's in strawberry," why a smaller model with traces, search, or distillation can recover much of a larger model's math performance, and why your production stack needs a routing layer before it needs a bigger GPU.