Reasoning Agent System Design

Real-time voice agents optimize for split-second streaming, interruption, and media state. This final capstone studies the opposite tradeoff: when a task is hard enough, the system may spend extra inference compute before it releases an answer or irreversible effect.

A reasoning agent can spend extra test-time compute on planning, checking, and tool use before its final answer. This design chapter explains when that checked work yields measured value and how to bound it with evidence contracts, evaluators, and budgets.

A release owner asks an AI deployment platform: "The canary is at 10%, smoke tests passed, but one latency panel spiked after a config flag changed. Should we roll back, hold, or continue the rollout?"

A human SRE wouldn't shout the first guess that comes to mind. They would check the rollout state, verify the smoke-test run, read the latency panel, inspect the config diff, and only then recommend hold, promote, or rollback. A standard large language model (LLM) generates each continuation left to right. It can't rewrite tokens already emitted, though later tokens can state a correction. External verification still requires a controller to call tools, request a revision, or generate competing attempts. On tricky problems, the product question is whether that additional work is justified and verifiable.

Reasoning models such as OpenAI's earlier o-series reasoning models, including o1 ^{[1]Reference 1o1 Preview Modelhttps://developers.openai.com/api/docs/models/o1-preview}, and DeepSeek-R1 ^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} spend additional compute before returning an answer. Don't conflate that native internal reasoning with a product controller that explicitly samples candidates, calls tools, scores partial states, or performs tree search: those are separable designs with different evidence and serving contracts. The controller below routes easy questions to bounded answers and hard, verifiable questions to deeper work.

Fast thinking and slow thinking

Fast/slow-thinking terminology gives us a useful analogy. Fast thinking is automatic and intuitive. When you read "2 + 2 =", you instantly know the answer. Slow thinking is deliberate and sequential. When you multiply 17 by 34 in your head, you work through steps, catch yourself if you miscarry a digit, and restart if the intermediate result looks wrong. In a product, however, enforceable controls are external candidate state, tool evidence, evaluator behavior, and release gates, not a promise about hidden mental steps.

Single-pass decoding looks more like fast thinking. At every step the model samples the next token conditioned on everything so far; without a controller or tool it can emit a plausible answer without validating it. A reasoning-enabled product can instead allocate a bounded check, use an exact tool when one exists, and release only a result supported by that check.

The useful shift is to treat some inference requests as a budgeted search and verification problem, rather than one left-to-right decoding loop. A controller may explore candidate actions, score partial solutions, consult external evidence, and spend extra compute only when evaluation justifies it.

The extra compute is called test-time compute: the floating-point operations (FLOPs) and tokens you spend during inference, after training is done. In a FLOPs-matched study, Snell et al. show that a smaller model plus the right inference-time strategy can beat a 14x larger model on easy-to-intermediate prompts where the smaller model already has a non-trivial chance of succeeding ^{[3]Reference 3Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314}. The same paper warns that the gain reverses on the hardest prompts and when you serve many inference tokens per query, so test-time compute isn't a free substitute for a stronger base model. The lever you control is no longer just model size; it's measured inference budget.

What is the product difference between training compute and test-time compute?

Answer

Training compute is spent before deployment to build the model. Test-time compute is spent per request to search, verify, rerank, or allocate more internal reasoning budget. Production systems can vary test-time compute by task difficulty instead of making every request pay for the same model size.

The illustration above shows three budget strategies. A fixed budget spends the same tokens on every request. An escalation rule adds more compute only after a weak first attempt. An adaptive budget uses a difficulty router to match the strategy to the task before the expensive search begins. The remaining sections build that adaptive system from the ground up.

The simplest form of extra compute: one deliberate path

Start with the cheapest inference control before building a tree-search engine: allocate one candidate a bounded reasoning budget. On supported reasoning APIs, reasoning.effort is the budget control. A structured response format is only the outward interface for checked milestones and a final answer; XML doesn't make the model think longer. Classic Chain of Thought prompts asked a model to "think step by step" ^{[4]Reference 4Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.https://arxiv.org/abs/2201.11903}, but production controls should separate hidden reasoning budget from evidence the application can validate.

On reasoning-trained APIs, direct task instructions are generally preferred over requests to reveal step-by-step thinking, because supported models already perform internal reasoning ^{[5]Reference 5Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning}. Your job is to request only outward artifacts the application needs: tool evidence, concise checked milestones, and a final answer. Hidden scratchpad text isn't a proof or an audit log.

One common budget knob on OpenAI reasoning models is a reasoning.effort setting ^{[6]Reference 6GPT-5.5 Modelhttps://developers.openai.com/api/docs/models/gpt-5.5}. Supported values vary by model family, but the operational lesson is the same. Lower effort favors latency and fewer reasoning tokens, while higher effort permits more internal reasoning before answering. This is the cheapest form of test-time compute control: raise effort only when evals show a measurable quality gain that justifies extra latency and cost, and keep the lowest useful setting for fast, deterministic tasks. Treat it as the single-path equivalent of the difficulty router you build below.

Here's a lightweight parser for the interface you want the model to satisfy. It turns a compact XML response into visible checkpoints plus a final answer:

Notice what this code does. The RESPONSE_TEMPLATE constrains output to major evidence-backed milestones. It doesn't make the claims true by itself; downstream code still needs to require tool results for external facts. Hidden token-by-token deliberation stays inside the model runtime, while the application receives a compact interface it can validate.

For our canary-release problem, a single-path trace might look like this:

• Checkpoint 1: Deploy tool reports that the canary is still at 10%.
• Checkpoint 2: Smoke-test tool reports that the release passed.
• Verification summary: Metrics tool shows elevated latency below the rollback threshold.
• Final answer: Hold the canary, keep rollback armed, and inspect the changed config flag before promotion.

A single bounded path is the cheapest candidate strategy when tool evidence or an exact check can validate the result. But what if multiple plans remain plausible?

Why should a reasoning API expose checkpoints instead of raw chain-of-thought?

Answer

Checkpoints give users and downstream systems useful milestones without exposing long internal scratchpads. They are shorter, safer to audit, easier to parse, and less likely to leak policy reasoning or hidden verifier details.

Asking several agents: Best-of-N sampling

Generate multiple candidate answers and rank them when a single path isn't reliable enough. For an incident-analysis workflow, that means several independent candidates enter a policy check or validated reviewer before the system selects a response.

This is called Best-of-N sampling. You run the single-path generator N times with a diversity-producing configuration, rank candidates with the most reliable evaluator available, and release an eligible winner. Temperature is one knob: low temperature tends to collapse diversity, while a higher value can produce different plans. Neither setting guarantees diversity or correctness.

Here's a deterministic version of the selector. In production, the candidates come from separate model calls; the selection rule stays the same:

Why this helps: a worked example

Suppose the base model has a 30% chance of producing a correct mitigation plan for a complex canary incident. With one sample, your success rate is 30%. With five independent samples, the probability that at least one is correct is:

$$P(\text{at least one success}) = 1 - (1 - 0.30)^5 = 1 - 0.168 = 0.83$$

That's 83% for the event that a correct candidate exists under the independence assumption. It's not an 83% success rate for the returned answer: selection succeeds only when a checker or verifier recognizes the correct candidate. Extra sampling is wasted if candidates are correlated or the verifier can't identify the useful one.

Worse, a reward model is only a proxy for correctness, so pushing N too high can backfire. Gao et al. show that as you optimize harder against a learned reward model, true (gold) quality eventually drops even while the proxy score keeps climbing, a Goodhart-style effect they call reward-model overoptimization ^{[7]Reference 7Scaling Laws for Reward Model Overoptimizationhttps://arxiv.org/abs/2210.10760}. In practice you cap N, ensemble or regularize the reward model, and watch a held-out gold metric rather than trusting the verifier score alone.

When an exact checker exists, it should outrank a persuasive learned score. This selector chooses a tested dependency path rather than the answer with the highest style score:

Self-consistency: Best-of-N without a reward model

When the answer is a comparable string (a number, a label, a parse), you can sometimes skip a learned reward model. Self-consistency samples several traces and returns the most frequent answer ^{[8]Reference 8Self-Consistency Improves Chain of Thought Reasoning in Language Models.https://arxiv.org/abs/2203.11171}. It can improve selected reasoning benchmarks when correct answers concentrate more than incorrect ones, but majority vote isn't verification: correlated wrong answers still win. Prefer an exact checker whenever available; use voting when endpoints are comparable and empirical evaluation supports it.

When does Best-of-N help, and when is it wasted?

Answer

It helps when the base model has a meaningful chance of producing at least one correct trace and the verifier can identify it. It's wasted when samples are near-identical, the base model has near-zero task competence, or the reward model can't distinguish good traces from bad ones.

Where beam search fits

Beam search is the classic decoding baseline: keep the top-$k$ most likely partial continuations at each step. It's easy to batch, but on tasks requiring distinct hypotheses its high-probability branches can become near-duplicates rather than useful alternatives. Evaluate it as a baseline or pruning mechanism against sampling and search on the task distribution you serve.

Why can beam search underperform on reasoning tasks?

Answer

Beam search keeps high-probability continuations, which can become small variations of the same path. For tasks needing diverse hypotheses and backtracking, compare it with sampling or broader search and use it as a pruning mechanism only when evaluation supports that choice.

When one path isn't enough: exploring a tree

Some problems force you to backtrack. Imagine a dependency resolver that must choose package versions while respecting API compatibility, Python version floors, and security advisories. If the resolver pins parser-v2 and later discovers that parser-v2 conflicts with the deployed runtime, it needs to undo that decision and try another branch.

Tree of Thoughts (ToT), proposed by Yao et al. ^{[9]Reference 9Tree of Thoughts: Deliberate Problem Solving with Large Language Models.https://arxiv.org/abs/2305.10601}, treats reasoning as a tree search. At each step, the model generates multiple possible next actions, scores them, and explores the most promising branches while pruning weak ones. For dependency resolution, each branch is a candidate version assignment and each evaluator check tests whether that partial assignment still satisfies known constraints.

The dependency-resolution problem forms a search tree. From the initial state, three version choices are possible. Thought B is pruned immediately because it violates a runtime constraint. Thought A looks promising, so it expands into two sub-choices. A.2 turns out to break an API-compatibility constraint and gets pruned. A.1 and C.1 both reach valid solutions.

To implement this, we maintain a search beam that keeps only candidate branches active. At each depth, the system generates multiple next steps, scores them with a task-specific evaluator, and prunes paths that fall below its threshold. That evaluator may be an exact constraint checker, a learned reward model, or a combination:

This is a beam-search realization of Tree of Thoughts: simple to batch, simple to reason about, and often the first version teams ship. The algorithm starts with the problem, expands each active node into branching_factor children, evaluates every child, and keeps only the beam_width highest-ranked candidates for the next round.

What makes Tree of Thoughts different from Best-of-N?

Answer

Best-of-N samples full traces independently and reranks them at the end. Tree of Thoughts evaluates partial states, can prune low-ranked branches early, and can backtrack before spending tokens on a candidate that fails its evaluator or constraints.

How does Tree of Thoughts handle non-deterministic outputs?

Answer

Use a diversity-producing generation policy at each node, then apply the most reliable evaluator available: exact constraints first, learned scoring where exact checks are unavailable. If every generation call returns the same continuation, tree search loses its main advantage.

When you want true MCTS

Tree of Thoughts is a general framework. If you need adaptive revisits instead of level-by-level pruning, you can swap the beam for Monte Carlo Tree Search. In LLM settings, a common variant is PUCT (Predictor + Upper Confidence bounds applied to Trees), the AlphaZero-style selection rule that combines a value estimate with the model's next-step prior ^{[10]Reference 10Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm.https://arxiv.org/abs/1712.01815}:

$$\text{PUCT}(s, a) = Q(s, a) + c_{\text{puct}} \cdot P(s, a) \cdot \frac{\sqrt{N(s)}}{1 + N(s, a)}$$

Where:

• $Q(s, a)$ is the estimated downstream value for taking action $a$ in state $s$ (from rollouts, a value model, or another evaluator)
• $P(s, a)$ is the policy prior for the next thought
• $N(s)$ is the total number of visits to the parent node
• $N(s, a)$ is the number of visits to the child node
• $c_{\text{puct}}$ is the exploration constant that balances exploration vs exploitation

The first term exploits high-value branches. The second term favors branches that either have a strong model prior or have not been explored much yet. Beam search is usually easier to run on GPUs because you can batch every frontier expansion together. MCTS can become attractive when evaluator calls are expensive and you want to revisit promising nodes instead of expanding every branch evenly.

When would you choose MCTS over level-by-level beam expansion?

Answer

Choose MCTS when revisiting promising branches beats expanding every frontier node evenly, especially when verifier calls are expensive or search depth varies by branch. Beam expansion is simpler to batch on GPUs and is often the first production version.

Who judges the plans? Verifiers and reward models

Generating many candidate plans is useless if you can't identify eligible results. Use exact checkers for objective constraints wherever possible. When exact checking is unavailable and a learned verifier scores candidate traces, two common forms are:

An Outcome Reward Model (ORM) scores only the final answer. It's like an audit that checks whether the final dependency plan is valid without inspecting the intermediate version choices. An ORM is cheap to train because it needs only one label per chain, but it's sparse: if the answer is wrong, you get no signal about where the reasoning broke down.

A Process Reward Model (PRM) scores individual reasoning steps, not the final answer alone ^{[11]Reference 11Let's Verify Step by Step.https://arxiv.org/abs/2305.20050}. When step quality is labelable and the PRM is validated on the served domain, it can help prune low-scored branches before paying for complete traces. It remains a learned proxy, not proof that a step is correct.

Why can a validated PRM be more useful than an ORM for search?

Answer

An ORM scores only the final answer. A PRM can score intermediate steps, so a search controller can prune low-scored branches earlier and avoid spending more tokens on them. Exact checkers still outrank either learned score when available.

The PRM estimates risk at each dependency step. When a candidate calls a removed API, a validated PRM may give that step a low score. If an exact compatibility checker is available, use it as the authoritative rejection; otherwise the score is a pruning signal whose false positives and negatives need evaluation.

How a PRM works

A PRM is typically a language model with a scalar reward head, trained to predict the correctness of a step $s_t$ given the prompt and the previous steps $s_{1...t-1}$. Lightman et al. released PRM800K, a dataset with 800,000 step-level human feedback labels, because step supervision is the bottleneck for building these verifiers well ^{[11]Reference 11Let's Verify Step by Step.https://arxiv.org/abs/2305.20050}.

$$P(\text{correct} \mid \text{context}, s_{1...t}) = \sigma(W \cdot h_t)$$

We take the hidden representation of the current step ($h_t$), map it through a learned linear layer ($W$), and squash the result with a sigmoid ($\sigma$). That gives a score between 0.0 and 1.0; treating it as a calibrated probability requires separate calibration and held-out validation.

Training data is collected via three main paths:

1. Human feedback: Experts label individual steps as Positive, Negative, or Neutral.
2. Monte Carlo estimation: Roll out $N$ completions from a step. If $M$ of $N$ lead to the correct answer, the step score is roughly $M/N$.
3. Teacher-model supervision: Use a stronger verifier or frontier model to pre-label steps, then audit those labels before trusting them at scale.

How do you train the Process Reward Model effectively?

Answer

Use dense step-level supervision when you can: human labels for partial steps, Monte Carlo rollouts that estimate whether a step leads to correct completions, and audited teacher-model labels for scale. An ORM can bootstrap early experiments, but a real PRM needs examples of correct, incorrect, and neutral intermediate reasoning steps.

Here's a practical scoring interface. In production the scoring call is a model with a reward head; this runnable version uses deterministic rules so you can see how weak steps pull down the whole trace:

The score_full_trace method uses the minimum score as a conservative pruning heuristic. In production, set thresholds against held-out task outcomes and prefer exact checks for constraints such as weight limits or policy eligibility.

ORM vs PRM at a glance

Lightman et al. (2023) ^{[11]Reference 11Let's Verify Step by Step.https://arxiv.org/abs/2305.20050} found that process supervision outperformed outcome supervision in their challenging math setting. Carrying that result into release-policy, planning, or code tasks requires domain-specific data and validation.

A correct answer can hide a broken evaluator

An evaluator may accept invalid reasoning because the final answer looks right. Sun et al. isolate this failure with Valid-Answer-Invalid-Reasoning (VAIR) math examples: frontier reasoning models that produced solutions well still scored as low as 48% when asked to judge solutions containing simple reasoning flaws but valid answers.^{[12]Reference 12An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Modelshttps://arxiv.org/abs/2606.01462} Their analysis finds evidence of answer confirmation bias, where the model works backward from the correct endpoint and rationalizes flawed steps.

That result comes from one controlled math setting, so don't turn 48% into expected production accuracy. Use it as an evaluation design warning:

For release decisions, exact tool evidence remains authoritative. When learned process scoring is necessary, score steps before revealing the final candidate where the interface permits, calibrate on held-out adversarial cases, and track false acceptance separately from ordinary solver accuracy. Solver pass rate and evaluator error rate are different production metrics.

Why isn't a correct final answer enough to validate the reasoning trace or its evaluator?

Answer

The correct endpoint can coexist with invalid steps, accidental cancellation, or unsupported evidence. Test the evaluator on valid-answer-invalid-reasoning cases and compare step verdicts before the final answer can influence the judgment.

Training reasoning models with RL

Modern reasoning models don't rely only on external search controllers. DeepSeek-R1 shows a complementary path where reinforcement learning makes the base policy itself better at producing long, structured reasoning traces ^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}.

Group Relative Policy Optimization (GRPO) is one important algorithm in that line of work, and DeepSeek-R1 made it especially visible in open-model practice ^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}. Compared with PPO (Proximal Policy Optimization), GRPO removes the need for a separate critic model:

1. Sample a group of outputs from the policy for each problem
2. Compute rewards for each output (using task rewards, exact verifiers where available, or validated reward models)
3. Calculate the relative advantage within the group (baseline = group mean)
4. Update the policy to favor higher-reward outputs

This cuts RL training overhead because you don't need a separate critic network. It can make the base model more sample-efficient at test time, but it doesn't remove the need for routing, budget limits, or verifier design in production.

What does GRPO improve, and what does it not replace?

Answer

GRPO can train the base policy to produce better reasoning traces with lower RL overhead than critic-based PPO. It doesn't replace production routing, token budgets, verifier quality, safe streaming, or serving capacity planning.

Putting it together: a production reasoning agent

So far we've looked at individual techniques. A real production system wires them into a pipeline that classifies the incoming problem, picks a strategy, runs the search, verifies the results, and streams progress back to the user.

What the system must handle

Before drawing boxes, list the concrete responsibilities:

• Complex reasoning: Accept problems that need multi-step logic (canary incident triage, dependency resolution, multi-file debugging).
• Candidate state: Maintain external candidate actions, tool observations, and evaluator results needed for search and audit, without assuming access to a model's hidden scratchpad.
• Adaptive compute: Scale inference compute proportional to problem difficulty. A "What is the API key retention policy?" question shouldn't trigger a 20-second tree search.
• Streaming: Emit status updates or safe progress summaries while the solver is working, without exposing unverified conclusions as completed checks.
• Budget control: Enforce configurable reasoning budgets in tokens, time, branches, and cost; product tiers may change caps but must not weaken evidence requirements.

On the non-functional side, declare measurable objectives: concurrency for a target workload, quality lift over single pass on representative verifiable tasks, p95 latency, and per-request cost by route. A 3x or 10x budget is an experiment parameter, not a production guarantee.

What are the five core responsibilities of a production reasoning agent?

Answer

It must classify difficulty, choose a compute strategy, run search or sampling, verify candidates, and stream safe progress plus the final answer while enforcing cost, latency, and token budgets.

Architecture overview

This diagram shows the full pipeline. Requests enter through a gateway, get classified by a difficulty router, and then flow through either a fast path, a medium path (Best-of-N), or a deep path (Tree of Thoughts). Branching paths should use the best available evaluator and reuse key-value (KV) cache prefixes only where the serving runtime safely supports compatible prefix sharing.

The request flow below shows the sequence in more detail:

Notice the loop in the request flow. The solver generates candidates, reuses shared KV prefixes, scores with the verifier, and streams progress events back. This loop is where the extra test-time compute lives.

The difficulty router

Not every problem needs extended thinking. Simple factual queries waste compute if treated as reasoning tasks. The router evaluates input complexity and assigns a strategy plus token budget. The lab below uses keyword matches only as a transparent demo. A production router is a cheap classifier trained on features (length, tool need, historical difficulty, verifier fail rate) and judged on held-out under-routing and over-routing cost, not a hardcoded substring list.

Here's a minimal implementation:

The router should be cheap relative to solving work. Its job is to avoid expensive search on requests that don't benefit while escalating tasks whose risk and verifiability justify more work. Misclassification isn't harmless: under-routing can release an incorrect answer, while over-routing can increase cost, latency, and exposure to verifier overoptimization.

Why should the router be cheap and conservative?

Answer

The router runs before every expensive reasoning decision, so it must not dominate latency. Evaluate both under-routing quality failures and over-routing cost or proxy-score failures; high-stakes cases may require tools, exact checks, or human review regardless of router confidence.

How would you optimize latency for the easy path?

Answer

Use a cheap router, direct instructions, low reasoning effort, and a small output schema. Bypass learned scoring and branch management when exact evidence already answers the request. Decode optimizations may help after profiling, but avoiding unnecessary reasoning stages is usually the first latency lever to measure.

Compute-optimal scheduling

Snell et al. ^{[3]Reference 3Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314} show that no single test-time strategy dominates every prompt. The best move depends on problem difficulty and on whether the base model already has a plausible path to the correct answer. A production scheduler should start cheap and escalate only when an exact check or a validated evaluation signal says the first attempt is weak:

This creates an adaptive system that spends small budgets on easy, checkable problems and escalates only when expected quality lift is worth extra latency and KV-cache pressure. It doesn't treat "unverifiable" as permission to make high-impact decisions.

What signal should trigger escalation from single path to Best-of-N or tree search?

Answer

Use an exact-check result or a learned evaluator validated against task outcomes, not user impatience or arbitrary prompt length alone. Escalate when the first path performs weakly under that signal, the task is verifiable, and measured quality lift is worth the extra latency and KV-cache pressure.

The serving bottleneck nobody talks about

Long reasoning traces can stress memory as well as floating-point operations (FLOPs). Every live generated token adds a new key and value vector for every transformer layer, so key-value (KV) cache pressure grows linearly with live context length and multiplies across unshared branches.

KV cache pressure

A useful back-of-the-envelope formula is:

$$\text{KV bytes per token} \approx 2 \cdot L \cdot H_{kv} \cdot d_{\text{head}} \cdot \text{bytes per element}$$

Where $L$ is the number of layers, $H_{kv}$ is the number of KV heads (not always the full attention-head count if the model uses grouped-query attention (GQA)), and $d_{\text{head}}$ is the head dimension. For an 80-layer model with 8 KV heads, head dimension 128, and bfloat16 (BF16) activations:

$$2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327{,}680 \text{ bytes per token} = 320 \text{ KiB/token}$$

($327{,}680 / 1024 = 320$ KiB; label it binary kibibytes, not decimal KB.) At 16k live tokens, that's roughly 5.0 GiB of KV cache for one branch. Without any prefix reuse, a tree search with eight active 16k-token branches would need about 40 GiB of KV cache before you count model weights or temporary activations.

Prefix sharing and paged KV storage

Best-of-N and tree search create many branches that share the same prompt and often the same early reasoning prefix. If you duplicate that prefix for every branch, memory usage grows with the number and length of branches. Systems like vLLM use PagedAttention to manage KV memory in blocks, which cuts fragmentation and makes long contexts much easier to serve ^{[13]Reference 13Efficient Memory Management for Large Language Model Serving with PagedAttentionhttps://arxiv.org/abs/2309.06180}.

PagedAttention and radix-style prefix caches solve related but different problems. PagedAttention makes allocation cheaper. RadixAttention-style caches, as used in SGLang, index shared prefixes in a tree so sibling branches can reuse the same cached prefix instead of copying it branch by branch ^{[14]Reference 14SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104}. In a reasoning agent you usually want both: block-level memory management plus prefix-aware reuse.

That changes the capacity math materially. If eight branches share a 12k-token prefix and each branch adds only a 4k-token unique suffix, the effective footprint is closer to one 12k prefix plus eight 4k suffixes, or about 14 GiB, not the 40 GiB worst case above. The exact savings depend on how quickly branches diverge and on whether your runtime can share prefixes at token-level or block-level granularity.

This is one reason test-time search is an infrastructure problem, not a prompting trick alone. A search policy may be sound on paper while its branch width, trace length, or lack of compatible prefix reuse exhausts the serving memory budget before useful search completes.

Why does tree search stress KV cache more than normal chat?

Answer

Each live branch carries keys and values for every generated token. Without prefix sharing, branch count multiplies memory usage. Paged KV allocation and prefix-aware reuse let sibling branches share common prompt and early-reasoning blocks.

Compression needs value awareness; recurrent depth changes architecture

Prefix sharing removes duplicate stems, but each surviving suffix still grows. Token eviction can cap memory by dropping old KV entries, yet a reasoning trace may depend on a small earlier detail. Value-Aware Stochastic KV Cache Eviction (VaSE) reports that evicting rare, large-magnitude value states can trigger repetitive failure loops; its mitigation protects those states and adds stochasticity to eviction.^{[15]Reference 15Value-Aware Stochastic KV Cache Eviction for Reasoning Modelshttps://arxiv.org/abs/2606.03928} Treat that as one 2026 method evaluated on its reported models and tasks, not a universal importance rule.

An eviction policy should therefore compare:

• Exact retention: pin prompt constraints, tool observations, branch decisions, and other application-known critical spans.
• Model signal: use measured attention or value-state signals for remaining tokens.
• Search lifecycle: free losing suffix pages immediately; never evict a prefix while live children reference it.
• Quality evidence: replay long reasoning tasks at the target compression ratio and measure loop rate, exact-check pass rate, and final quality against the uncompressed baseline.

Recurrent-depth models attack a different source of growth. Instead of expressing every extra unit of compute as more visible chain-of-thought tokens, a recurrent block can unroll additional latent depth while sharing model parameters and KV state. Geiping et al. present a 3.5B-parameter proof-of-concept with per-token adaptive compute, KV-cache sharing, and speculative decoding.^{[16]Reference 16Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachhttps://proceedings.neurips.cc/paper_files/paper/2025/hash/3b01972cf31e6fa0fe29e4b8b5c2a0a1-Abstract-Conference.html} It's a model-architecture choice, not a cache flag you can add to a conventional autoregressive checkpoint. A capacity plan should separate mitigations available now from changes that require training and serving a different architecture.

Why can't recurrent depth replace KV eviction as a drop-in runtime optimization?

Answer

It changes how the model allocates computation and represents iterative state. Existing token-reasoning checkpoints still append KV per generated token; the recurrent-depth benefit requires a model and runtime designed for that architecture.

First update vs final-answer latency

Extended reasoning changes the latency contract. At the model-serving layer, a long hidden search phase can inflate time to first token (TTFT). At the product layer, a quick status event can make first-update latency look healthy while time-to-final-answer and total budget still fail. Measure all three separately.

That's why production systems can stream progress events instead of raw scratchpads. Safe events include selected strategy, budget usage, branch counts, or summaries of externally confirmed checks. Treat learned verifier scores as internal diagnostics unless they are calibrated and presented with an appropriate contract.

Why do deep reasoning workflows need both first-update and final-answer latency metrics?

Answer

Without progress events, hidden search can make first visible output unacceptably slow. With progress events, first-update latency can be fast while final-answer latency or cost still exceeds its budget. Track both while keeping raw scratchpads private.

A sample capacity plan

This table uses the 320 KiB/token estimate to show how quickly memory usage grows as you add longer traces or more live branches:

Those are worst-case branch-multiplication numbers. Strong prefix reuse can cut the effective KV footprint substantially.

Streaming safe updates

Users need feedback while the system is spending extra inference budget, but that doesn't mean you should dump raw internal scratchpad text or expose an uncalibrated reward score as confidence. A safer pattern is to stream strategy selection, budget usage, and externally grounded summaries while the solver keeps search details private:

The solver still keeps internal_score for routing and diagnostics, but stream_reasoning strips that field before the event reaches the client. The user sees confirmed work, not an uncalibrated proxy.

When reasoning is the wrong tool

Not every problem benefits from test-time scaling. Applying tree search or Best-of-N to the wrong use case results in unnecessary latency and wasted resources. You should bypass extended reasoning for:

• Simple source-of-truth queries (e.g., "What is the API key retention policy?"). Query the policy source directly; extra internal reasoning won't recover a missing or current fact.
• Retrieval failures. If the problem requires external facts or tools, longer internal reasoning is the wrong fix. You need retrieval, search, or tool use.
• Open-ended marketing copy (e.g., "Write three brand voice options for a product page"). There is no single objective answer unless the application defines a rubric and validates it, so deep PRM-driven search is usually unjustified.
• Latency-sensitive conversational turns. Live products need measured response objectives; a deep search path that exceeds that objective should acknowledge, defer, or avoid the search.
• Cost-sensitive batch processing. When running millions of documents through a classification pipeline, a large inference multiplier can erase the value of a small quality gain. Escalate only request cohorts whose measured lift pays for the extra cost.

Allocate substantial test-time compute when evaluation shows gain for the request class and evidence or verification can keep the answer trustworthy. Creative revision may still use a small candidate budget, but it isn't proof-oriented reasoning.

Overthinking: when more thinking lowers accuracy

Longer reasoning isn't monotonically better. Empirical 2025 work finds that accuracy often plateaus and can decline once a trace runs past a problem-specific sweet spot: extra steps add error accumulation, and a model can talk itself out of a correct answer it already found ^{[17]Reference 17Does Thinking More Always Help? Mirage of Test-Time Scaling in Reasoning Modelshttps://arxiv.org/abs/2506.04210}. This overthinking failure mode is also a cost and latency problem, because a model can burn thousands of reasoning tokens on a trivial question. The defenses are the same controls you already have: a router that keeps trivial tasks on the fast path, a capped reasoning_effort or token budget, and a stop rule that exits once exact results are sufficient or validated evaluation stops improving. Treat reasoning budget as a tuned hyperparameter, not a dial you turn to maximum.

Why can adding more test-time compute hurt accuracy, and how do you defend against it?

Answer

Beyond a problem-specific sweet spot, extra steps accumulate errors and can override a correct answer the model already reached, while inflating cost and latency. Defend with difficulty routing, a capped reasoning-effort or token budget, and a stop rule based on exact completion or validated evaluation improvement.

Why a bigger model isn't enough

It's tempting to think that deploying a bigger frontier model should dominate every reasoning workload. In practice, the trade-off is more subtle:

Snell et al. make the practical point: on prompts where a smaller model already has some chance of success, extra inference-time compute can beat switching to a much larger model at matched compute ^{[3]Reference 3Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314}. But if the base model has near-zero task competence, reranking or sampling is unlikely to manufacture the missing capability. Test-time compute amplifies evaluated candidate quality; it doesn't supply missing knowledge or tools.

When is switching to a bigger model better than adding test-time search?

Answer

Use a stronger model when the current base model has near-zero competence, lacks required domain knowledge, or fails before search can produce useful candidates. Test-time search magnifies an existing capability; it doesn't create missing knowledge or tools.

Common pitfalls

"Higher temperature will reveal a great answer if I sample enough"

• Symptom: Best-of-N produces more varied traces, but quality barely improves and the chosen winner still feels random.
• Cause: Diversity alone isn't enough. Without a verifier that tracks correctness, more samples mostly add noise.
• Fix: Increase temperature only when you also have a PRM, ORM, or exact checker that can reliably separate strong traces from fluent wrong ones.

"Tree search is always better than single-path reasoning"

• Symptom: Easy requests suddenly become slower and more expensive even though accuracy barely moves.
• Cause: Search budget was applied to every request instead of only the hard, verifiable ones.
• Fix: Route by difficulty. Keep trivial and easy requests on the fast path, then escalate only when the first trace scores weakly.

"More thinking always improves the answer"

• Symptom: The system finds a decent first answer, then keeps exploring until it talks itself into a worse one.
• Cause: Overthinking. Past a task-specific sweet spot, extra search compounds errors instead of fixing them.
• Fix: Stop when exact checks pass or a validated evaluator plateaus, cap reasoning effort, and keep the best eligible fallback that can survive later noisy branches.

"One budget policy can serve every request"

• Symptom: Cheap FAQ traffic burns deep-search budget while hard debugging tasks run out of depth and tokens too early.
• Cause: Compute was allocated uniformly instead of matching task difficulty and business value.
• Fix: Separate routing, branch limits, token caps, and wall-clock caps by request class. Budget policy is part of product design, not model tuning alone.

"Users need raw chain-of-thought to trust the system"

• Symptom: Streams become long, unstable, and hard to parse, while downstream tools can't tell what is final.
• Cause: Internal scratchpad text was exposed instead of a stable progress interface.
• Fix: Stream strategy, budget usage, and short evidence-backed summaries. Keep hidden search traces and uncalibrated evaluator scores inside the runtime.

"Reasoning quality comes only from prompts"

• Symptom: Prompt tweaks plateau, but the team still has no reliable gain on hard tasks.
• Cause: Model capability, verifier quality, cache reuse, and stopping logic were collapsed into one prompt-design problem.
• Fix: Treat base model, search controller, verifier, and serving runtime as separate levers. Improve whichever layer is bottlenecking quality or cost.

"Stop criteria are an optional cleanup detail"

• Symptom: Branches revisit the same semantic state until they exhaust the token budget or hit wall-clock limits.
• Cause: The controller has no hard policy for repeated states, budget exhaustion, or diminishing verifier returns.
• Fix: Enforce token, depth, wall-clock, and repeated-state limits per branch, and return the highest-ranked surviving trace that satisfies required checks when the budget ends.

What happens if the reasoning trace enters an infinite loop?

Answer

The controller stops it. Enforce hard token, depth, and wall-clock budgets; reject repeated or circular states; and prune branches that revisit the same semantic state. Return the highest-ranked surviving trace that satisfies required checks when the budget expires.

Final design checklist

Use this checklist to judge whether your final design is defensible:

• Choose a routing strategy for trivial, medium, and hard verifiable tasks, and explain why each class gets that budget.
• Defend when a task needs a single path, Best-of-N, tree search, or a bigger base model instead of more search.
• Pick the right judge for the job: exact checker, ORM, or PRM, and explain what failure appears if you choose the wrong one.
• Test learned evaluators with correct-answer flawed-reasoning cases so endpoint confirmation doesn't masquerade as process verification.
• Check that arithmetic and scheduling constraints are satisfiable before generating branches; reject or clarify impossible inputs instead of optimizing a fabricated solution.
• Estimate rough KV-cache pressure for one branch, many branches, and a shared-prefix layout.
• Choose between lossless prefix reuse, evaluated value-aware eviction, or a recurrent-depth model without treating them as interchangeable knobs.
• Define hard stop criteria for depth, tokens, wall-clock time, and repeated semantic states.
• Describe what the user sees while the solver works, and why that stream should expose progress summaries instead of raw scratchpads.