Reasoning & Test-Time Compute

Real-time voice agents optimize for split-second streaming, interruption, and media state. This final capstone studies the opposite tradeoff: when a task is hard enough, the system may spend extra inference compute before it releases an answer or irreversible effect.

A reasoning agent can spend extra test-time compute on planning, checking, and tool use before its final answer. This design chapter explains when that checked work yields measured value and how to bound it with evidence contracts, evaluators, and budgets.

A release owner asks an AI deployment platform: "The canary is at 10%, smoke tests passed, but one latency panel spiked after a config flag changed. Should we roll back, hold, or continue the rollout?"

A human SRE wouldn't shout the first guess that comes to mind. They would check the rollout state, verify the smoke-test run, read the latency panel, inspect the config diff, and only then recommend hold, promote, or rollback. A standard large language model (LLM) generates each continuation left to right. It can't rewrite tokens already emitted, though later tokens can state a correction. External verification still requires a controller to call tools, request a revision, or generate competing attempts. On tricky problems, the product question is whether that additional work is justified and verifiable.

Reasoning models such as OpenAI's earlier o-series reasoning models, including o1 ^{[1]Reference 1o1 Preview Modelhttps://developers.openai.com/api/docs/models/o1-preview}, and DeepSeek-R1 ^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} spend additional compute before returning an answer. Don't conflate that native internal reasoning with a product controller that explicitly samples candidates, calls tools, scores partial states, or performs tree search: those are separable designs with different evidence and serving contracts. The controller below routes easy questions to bounded answers and hard, verifiable questions to deeper work.

Fast thinking and slow thinking

Fast/slow-thinking terminology gives us a useful analogy. Fast thinking is automatic and intuitive. When you read "2 + 2 =", you instantly know the answer. Slow thinking is deliberate and sequential. When you multiply 17 by 34 in your head, you work through steps, catch yourself if you miscarry a digit, and restart if the intermediate result looks wrong. In a product, however, enforceable controls are external candidate state, tool evidence, evaluator behavior, and release gates, not a promise about hidden mental steps.

Single-pass decoding looks more like fast thinking. At every step the model samples the next token conditioned on everything so far; without a controller or tool it can emit a plausible answer without validating it. A reasoning-enabled product can instead allocate a bounded check, use an exact tool when one exists, and release only a result supported by that check.

The useful shift is to treat some inference requests as a budgeted search and verification problem, rather than one left-to-right decoding loop. A controller may explore candidate actions, score partial solutions, consult external evidence, and spend extra compute only when evaluation justifies it.

The extra compute is called test-time compute: the floating-point operations (FLOPs) and tokens you spend during inference, after training is done. In a FLOPs-matched study, Snell et al. show that a smaller model plus the right inference-time strategy can beat a 14x larger model on easy-to-intermediate prompts where the smaller model already has a non-trivial chance of succeeding ^{[3]Reference 3Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314}. The same paper warns that the gain reverses on the hardest prompts and when you serve many inference tokens per query, so test-time compute isn't a free substitute for a stronger base model. The lever you control is no longer just model size; it's measured inference budget.

The illustration above shows three budget strategies. A fixed budget spends the same tokens on every request. An escalation rule adds more compute only after a weak first attempt. An adaptive budget uses a difficulty router to match the strategy to the task before the expensive search begins. The remaining sections build that adaptive system from the ground up.

The simplest form of extra compute: one deliberate path

Start with the cheapest inference control before building a tree-search engine: allocate one candidate a bounded reasoning budget. On supported reasoning APIs, reasoning.effort is the budget control. A structured response format is only the outward interface for checked milestones and a final answer; XML doesn't make the model think longer. Classic Chain of Thought prompts asked a model to "think step by step" ^{[4]Reference 4Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.https://arxiv.org/abs/2201.11903}, but production controls should separate hidden reasoning budget from evidence the application can validate.

On reasoning-trained APIs, direct task instructions are generally preferred over requests to reveal step-by-step thinking, because supported models already perform internal reasoning ^{[5]Reference 5Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning}. Your job is to request only outward artifacts the application needs: tool evidence, concise checked milestones, and a final answer. Hidden scratchpad text isn't a proof or an audit log.

One common budget knob on OpenAI reasoning models is a reasoning.effort setting ^{[6]Reference 6GPT-5.5 Modelhttps://developers.openai.com/api/docs/models/gpt-5.5}. Supported values vary by model family, but the operational lesson is the same. Lower effort favors latency and fewer reasoning tokens, while higher effort permits more internal reasoning before answering. This is the cheapest form of test-time compute control: raise effort only when evals show a measurable quality gain that justifies extra latency and cost, and keep the lowest useful setting for fast, deterministic tasks. Treat it as the single-path equivalent of the difficulty router you build below.

Here's a lightweight parser for the interface you want the model to satisfy. It turns a compact XML response into visible checkpoints plus a final answer:

Notice what this code does. The RESPONSE_TEMPLATE constrains output to major evidence-backed milestones. It doesn't make the claims true by itself; downstream code still needs to require tool results for external facts. Hidden token-by-token deliberation stays inside the model runtime, while the application receives a compact interface it can validate.

For our canary-release problem, a single-path trace might look like this:

• Checkpoint 1: Deploy tool reports that the canary is still at 10%.
• Checkpoint 2: Smoke-test tool reports that the release passed.
• Verification summary: Metrics tool shows elevated latency below the rollback threshold.
• Final answer: Hold the canary, keep rollback armed, and inspect the changed config flag before promotion.

A single bounded path is the cheapest candidate strategy when tool evidence or an exact check can validate the result. But what if multiple plans remain plausible?

Asking several agents: Best-of-N sampling

Generate multiple candidate answers and rank them when a single path isn't reliable enough. For an incident-analysis workflow, that means several independent candidates enter a policy check or validated reviewer before the system selects a response.

This is called Best-of-N sampling. You run the single-path generator N times with a diversity-producing configuration, rank candidates with the most reliable evaluator available, and release an eligible winner. Temperature is one knob: low temperature tends to collapse diversity, while a higher value can produce different plans. Neither setting guarantees diversity or correctness.

Here's a deterministic version of the selector. In production, the candidates come from separate model calls; the selection rule stays the same:

Why this helps: a worked example

Suppose the base model has a 30% chance of producing a correct mitigation plan for a complex canary incident. With one sample, your success rate is 30%. With five independent samples, the probability that at least one is correct is:

$$P(\text{at least one success}) = 1 - (1 - 0.30)^5 = 1 - 0.168 = 0.83$$

That's 83% for the event that a correct candidate exists under the independence assumption. It's not an 83% success rate for the returned answer: selection succeeds only when a checker or verifier recognizes the correct candidate. Extra sampling is wasted if candidates are correlated or the verifier can't identify the useful one.

Worse, a reward model is only a proxy for correctness, so pushing N too high can backfire. Gao et al. show that as you optimize harder against a learned reward model, true (gold) quality eventually drops even while the proxy score keeps climbing, a Goodhart-style effect they call reward-model overoptimization ^{[7]Reference 7Scaling Laws for Reward Model Overoptimizationhttps://arxiv.org/abs/2210.10760}. In practice you cap N, ensemble or regularize the reward model, and watch a held-out gold metric rather than trusting the verifier score alone.

When an exact checker exists, it should outrank a persuasive learned score. This selector chooses a tested dependency path rather than the answer with the highest style score:

Self-consistency: Best-of-N without a reward model

When the answer is a comparable string (a number, a label, a parse), you can sometimes skip a learned reward model. Self-consistency samples several traces and returns the most frequent answer ^{[8]Reference 8Self-Consistency Improves Chain of Thought Reasoning in Language Models.https://arxiv.org/abs/2203.11171}. It can improve selected reasoning benchmarks when correct answers concentrate more than incorrect ones, but majority vote isn't verification: correlated wrong answers still win. Prefer an exact checker whenever available; use voting when endpoints are comparable and empirical evaluation supports it.

Where beam search fits

Beam search is the classic decoding baseline: keep the top-$k$ most likely partial continuations at each step. It's easy to batch, but on tasks requiring distinct hypotheses its high-probability branches can become near-duplicates rather than useful alternatives. Evaluate it as a baseline or pruning mechanism against sampling and search on the task distribution you serve.

When one path isn't enough: exploring a tree

Some problems force you to backtrack. Imagine a dependency resolver that must choose package versions while respecting API compatibility, Python version floors, and security advisories. If the resolver pins parser-v2 and later discovers that parser-v2 conflicts with the deployed runtime, it needs to undo that decision and try another branch.

Tree of Thoughts (ToT), proposed by Yao et al. ^{[9]Reference 9Tree of Thoughts: Deliberate Problem Solving with Large Language Models.https://arxiv.org/abs/2305.10601}, treats reasoning as a tree search. At each step, the model generates multiple possible next actions, scores them, and explores the most promising branches while pruning the weak ones. It's similar to playing chess: you consider several moves, evaluate the board, explore the promising lines, and abandon the bad ones.

The dependency-resolution problem forms a search tree. From the initial state, three version choices are possible. Thought B is pruned immediately because it violates a runtime constraint. Thought A looks promising, so it expands into two sub-choices. A.2 turns out to break an API-compatibility constraint and gets pruned. A.1 and C.1 both reach valid solutions.

To implement this, we maintain a search beam that keeps only candidate branches active. At each depth, the system generates multiple next steps, scores them with a task-specific evaluator, and prunes paths that fall below its threshold. That evaluator may be an exact constraint checker, a learned reward model, or a combination:

This is a beam-search realization of Tree of Thoughts: simple to batch, simple to reason about, and often the first version teams ship. The algorithm starts with the problem, expands each active node into branching_factor children, evaluates every child, and keeps only the beam_width highest-ranked candidates for the next round.

When you want true MCTS

Tree of Thoughts is a general framework. If you need adaptive revisits instead of level-by-level pruning, you can swap the beam for Monte Carlo Tree Search. In LLM settings, a common variant is PUCT (Predictor + Upper Confidence bounds applied to Trees), the AlphaZero-style selection rule that combines a value estimate with the model's next-step prior ^{[10]Reference 10Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm.https://arxiv.org/abs/1712.01815}:

$$\text{PUCT}(s, a) = Q(s, a) + c_{\text{puct}} \cdot P(s, a) \cdot \frac{\sqrt{N(s)}}{1 + N(s, a)}$$

Where:

• $Q(s, a)$ is the estimated downstream value for taking action $a$ in state $s$ (from rollouts, a value model, or another evaluator)
• $P(s, a)$ is the policy prior for the next thought
• $N(s)$ is the total number of visits to the parent node
• $N(s, a)$ is the number of visits to the child node
• $c_{\text{puct}}$ is the exploration constant that balances exploration vs exploitation

The first term exploits high-value branches. The second term favors branches that either have a strong model prior or have not been explored much yet. Beam search is usually easier to run on GPUs because you can batch every frontier expansion together. MCTS can become attractive when evaluator calls are expensive and you want to revisit promising nodes instead of expanding every branch evenly.

Who judges the plans? Verifiers and reward models

Generating many candidate plans is useless if you can't identify eligible results. Use exact checkers for objective constraints wherever possible. When exact checking is unavailable and a learned verifier scores candidate traces, two common forms are:

An Outcome Reward Model (ORM) scores only the final answer. It's like an audit that checks whether the final dependency plan is valid without inspecting the intermediate version choices. An ORM is cheap to train because it needs only one label per chain, but it's sparse: if the answer is wrong, you get no signal about where the reasoning broke down.

A Process Reward Model (PRM) scores individual reasoning steps, not the final answer alone ^{[11]Reference 11Let's Verify Step by Step.https://arxiv.org/abs/2305.20050}. When step quality is labelable and the PRM is validated on the served domain, it can help prune low-scored branches before paying for complete traces. It remains a learned proxy, not proof that a step is correct.

The PRM estimates risk at each dependency step. When a candidate calls a removed API, a validated PRM may give that step a low score. If an exact compatibility checker is available, use it as the authoritative rejection; otherwise the score is a pruning signal whose false positives and negatives need evaluation.

How a PRM works

A PRM is typically a language model with a scalar reward head, trained to predict the correctness of a step $s_t$ given the prompt and the previous steps $s_{1...t-1}$. Lightman et al. released PRM800K, a dataset with 800,000 step-level human feedback labels, because step supervision is the bottleneck for building these verifiers well ^{[11]Reference 11Let's Verify Step by Step.https://arxiv.org/abs/2305.20050}.

$$P(\text{correct} \mid \text{context}, s_{1...t}) = \sigma(W \cdot h_t)$$

We take the hidden representation of the current step ($h_t$), map it through a learned linear layer ($W$), and squash the result with a sigmoid ($\sigma$). That gives a score between 0.0 and 1.0; treating it as a calibrated probability requires separate calibration and held-out validation.

Training data is collected via three main paths:

1. Human feedback: Experts label individual steps as Positive, Negative, or Neutral.
2. Monte Carlo estimation: Roll out $N$ completions from a step. If $M$ of $N$ lead to the correct answer, the step score is roughly $M/N$.
3. Teacher-model supervision: Use a stronger verifier or frontier model to pre-label steps, then audit those labels before trusting them at scale.

Here's a practical scoring interface. In production the scoring call is a model with a reward head; this runnable version uses deterministic rules so you can see how weak steps pull down the whole trace:

The score_full_trace method uses the minimum score as a conservative pruning heuristic. In production, set thresholds against held-out task outcomes and prefer exact checks for constraints such as weight limits or policy eligibility.

ORM vs PRM at a glance

Lightman et al. (2023) ^{[11]Reference 11Let's Verify Step by Step.https://arxiv.org/abs/2305.20050} found that process supervision outperformed outcome supervision in their challenging math setting. Carrying that result into release-policy, planning, or code tasks requires domain-specific data and validation.

Training reasoning models with RL

Modern reasoning models don't rely only on external search controllers. DeepSeek-R1 shows a complementary path where reinforcement learning makes the base policy itself better at producing long, structured reasoning traces ^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}.

Group Relative Policy Optimization (GRPO) is one important algorithm in that line of work, and DeepSeek-R1 made it especially visible in open-model practice ^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}. Compared with PPO (Proximal Policy Optimization), GRPO removes the need for a separate critic model:

1. Sample a group of outputs from the policy for each problem
2. Compute rewards for each output (using task rewards, exact verifiers where available, or validated reward models)
3. Calculate the relative advantage within the group (baseline = group mean)
4. Update the policy to favor higher-reward outputs

This cuts RL training overhead because you don't need a separate critic network. It can make the base model more sample-efficient at test time, but it doesn't remove the need for routing, budget limits, or verifier design in production.

Putting it together: a production reasoning agent

So far we've looked at individual techniques. A real production system wires them into a pipeline that classifies the incoming problem, picks a strategy, runs the search, verifies the results, and streams progress back to the user.

What the system must handle

Before drawing boxes, list the concrete responsibilities:

• Complex reasoning: Accept problems that need multi-step logic (canary incident triage, dependency resolution, multi-file debugging).
• Candidate state: Maintain external candidate actions, tool observations, and evaluator results needed for search and audit, without assuming access to a model's hidden scratchpad.
• Adaptive compute: Scale inference compute proportional to problem difficulty. A "What is the API key retention policy?" question shouldn't trigger a 20-second tree search.
• Streaming: Emit status updates or safe progress summaries while the solver is working, without exposing unverified conclusions as completed checks.
• Budget control: Enforce configurable reasoning budgets in tokens, time, branches, and cost; product tiers may change caps but must not weaken evidence requirements.

On the non-functional side, declare measurable objectives: concurrency for a target workload, quality lift over single pass on representative verifiable tasks, p95 latency, and per-request cost by route. A 3x or 10x budget is an experiment parameter, not a production guarantee.

Architecture overview

This diagram shows the full pipeline. Requests enter through a gateway, get classified by a difficulty router, and then flow through either a fast path, a medium path (Best-of-N), or a deep path (Tree of Thoughts). Branching paths should use the best available evaluator and reuse key-value (KV) cache prefixes only where the serving runtime safely supports compatible prefix sharing.

The request flow below shows the sequence in more detail:

Notice the loop in the request flow. The solver generates candidates, reuses shared KV prefixes, scores with the verifier, and streams progress events back. This loop is where the extra test-time compute lives.

The difficulty router

Not every problem needs extended thinking. Simple factual queries waste compute if treated as reasoning tasks. The router evaluates input complexity and assigns a strategy plus token budget. Here's a minimal implementation:

The router should be cheap relative to solving work. Its job is to avoid expensive search on requests that don't benefit while escalating tasks whose risk and verifiability justify more work. Misclassification isn't harmless: under-routing can release an incorrect answer, while over-routing can increase cost, latency, and exposure to verifier overoptimization.

Compute-optimal scheduling

Snell et al. ^{[3]Reference 3Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314} show that no single test-time strategy dominates every prompt. The best move depends on problem difficulty and on whether the base model already has a plausible path to the correct answer. A production scheduler should start cheap and escalate only when an exact check or a validated evaluation signal says the first attempt is weak:

This creates an adaptive system that spends small budgets on easy, checkable problems and escalates only when expected quality lift is worth extra latency and KV-cache pressure. It doesn't treat "unverifiable" as permission to make high-impact decisions.

The serving bottleneck nobody talks about

Long reasoning traces can stress memory as well as floating-point operations (FLOPs). Every live generated token adds a new key and value vector for every transformer layer, so key-value (KV) cache pressure grows linearly with live context length and multiplies across unshared branches.

KV cache pressure

A useful back-of-the-envelope formula is:

$$\text{KV bytes per token} \approx 2 \cdot L \cdot H_{kv} \cdot d_{\text{head}} \cdot \text{bytes per element}$$

Where $L$ is the number of layers, $H_{kv}$ is the number of KV heads (not always the full attention-head count if the model uses grouped-query attention (GQA)), and $d_{\text{head}}$ is the head dimension. For an 80-layer model with 8 KV heads, head dimension 128, and bfloat16 (BF16) activations:

$$2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327{,}680 \text{ bytes per token} \approx 320 \text{ KB/token}$$

At 16k live tokens, that's roughly 5.0 GiB of KV cache for one branch. Without any prefix reuse, a tree search with eight active 16k-token branches would need about 40 GiB of KV cache before you count model weights or temporary activations.

Prefix sharing and paged KV storage

Best-of-N and tree search create many branches that share the same prompt and often the same early reasoning prefix. If you duplicate that prefix for every branch, memory usage grows with the number and length of branches. Systems like vLLM use PagedAttention to manage KV memory in blocks, which cuts fragmentation and makes long contexts much easier to serve ^{[12]Reference 12Efficient Memory Management for Large Language Model Serving with PagedAttentionhttps://arxiv.org/abs/2309.06180}.

PagedAttention and radix-style prefix caches solve related but different problems. PagedAttention makes allocation cheaper. RadixAttention-style caches, as used in SGLang, index shared prefixes in a tree so sibling branches can reuse the same cached prefix instead of copying it branch by branch ^{[13]Reference 13SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104}. In a reasoning agent you usually want both: block-level memory management plus prefix-aware reuse.

That changes the capacity math materially. If eight branches share a 12k-token prefix and each branch adds only a 4k-token unique suffix, the effective footprint is closer to one 12k prefix plus eight 4k suffixes, or about 14 GiB, not the 40 GiB worst case above. The exact savings depend on how quickly branches diverge and on whether your runtime can share prefixes at token-level or block-level granularity.

This is one reason test-time search is an infrastructure problem, not a prompting trick alone. A search policy may be sound on paper while its branch width, trace length, or lack of compatible prefix reuse exhausts the serving memory budget before useful search completes.

First update vs final-answer latency

Extended reasoning changes the latency contract. At the model-serving layer, a long hidden search phase can inflate time to first token (TTFT). At the product layer, a quick status event can make first-update latency look healthy while time-to-final-answer and total budget still fail. Measure all three separately.

That's why production systems can stream progress events instead of raw scratchpads. Safe events include selected strategy, budget usage, branch counts, or summaries of externally confirmed checks. Treat learned verifier scores as internal diagnostics unless they are calibrated and presented with an appropriate contract.

A sample capacity plan

This table uses the 320 KB/token estimate to show how quickly memory usage grows as you add longer traces or more live branches:

Those are worst-case branch-multiplication numbers. Strong prefix reuse can cut the effective KV footprint substantially.

Streaming safe updates

Users need feedback while the system is spending extra inference budget, but that doesn't mean you should dump raw internal scratchpad text or expose an uncalibrated reward score as confidence. A safer pattern is to stream strategy selection, budget usage, and externally grounded summaries while the solver keeps search details private:

The solver still keeps internal_score for routing and diagnostics, but stream_reasoning strips that field before the event reaches the client. The user sees confirmed work, not an uncalibrated proxy.

When reasoning is the wrong tool

Not every problem benefits from test-time scaling. Applying tree search or Best-of-N to the wrong use case results in unnecessary latency and wasted resources. You should bypass extended reasoning for:

• Simple source-of-truth queries (e.g., "What is the API key retention policy?"). Query the policy source directly; extra internal reasoning won't recover a missing or current fact.
• Retrieval failures. If the problem requires external facts or tools, longer internal reasoning is the wrong fix. You need retrieval, search, or tool use.
• Open-ended marketing copy (e.g., "Write three brand voice options for a product page"). There is no single objective answer unless the application defines a rubric and validates it, so deep PRM-driven search is usually unjustified.
• Latency-sensitive conversational turns. Live products need measured response objectives; a deep search path that exceeds that objective should acknowledge, defer, or avoid the search.
• Cost-sensitive batch processing. When running millions of documents through a classification pipeline, a large inference multiplier can erase the value of a small quality gain. Escalate only request cohorts whose measured lift pays for the extra cost.

Allocate substantial test-time compute when evaluation shows gain for the request class and evidence or verification can keep the answer trustworthy. Creative revision may still use a small candidate budget, but it isn't proof-oriented reasoning.

Overthinking: when more thinking lowers accuracy

Longer reasoning isn't monotonically better. Empirical 2025 work finds that accuracy often plateaus and can decline once a trace runs past a problem-specific sweet spot: extra steps add error accumulation, and a model can talk itself out of a correct answer it already found ^{[14]Reference 14Does Thinking More Always Help? Mirage of Test-Time Scaling in Reasoning Modelshttps://arxiv.org/abs/2506.04210}. This overthinking failure mode is also a cost and latency problem, because a model can burn thousands of reasoning tokens on a trivial question. The defenses are the same controls you already have: a router that keeps trivial tasks on the fast path, a capped reasoning_effort or token budget, and a stop rule that exits once exact results are sufficient or validated evaluation stops improving. Treat reasoning budget as a tuned hyperparameter, not a dial you turn to maximum.

Why a bigger model isn't enough

It's tempting to think that deploying a bigger frontier model should dominate every reasoning workload. In practice, the trade-off is more subtle:

Snell et al. make the practical point: on prompts where a smaller model already has some chance of success, extra inference-time compute can beat switching to a much larger model at matched compute ^{[3]Reference 3Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314}. But if the base model has near-zero task competence, reranking or sampling is unlikely to manufacture the missing capability. Test-time compute amplifies evaluated candidate quality; it doesn't supply missing knowledge or tools.

Common pitfalls

"Higher temperature will reveal a great answer if I sample enough"

• Symptom: Best-of-N produces more varied traces, but quality barely improves and the chosen winner still feels random.
• Cause: Diversity alone isn't enough. Without a verifier that tracks correctness, more samples mostly add noise.
• Fix: Increase temperature only when you also have a PRM, ORM, or exact checker that can reliably separate strong traces from fluent wrong ones.

"Tree search is always better than single-path reasoning"

• Symptom: Easy requests suddenly become slower and more expensive even though accuracy barely moves.
• Cause: Search budget was applied to every request instead of only the hard, verifiable ones.
• Fix: Route by difficulty. Keep trivial and easy requests on the fast path, then escalate only when the first trace scores weakly.

"More thinking always improves the answer"

• Symptom: The system finds a decent first answer, then keeps exploring until it talks itself into a worse one.
• Cause: Overthinking. Past a task-specific sweet spot, extra search compounds errors instead of fixing them.
• Fix: Stop when exact checks pass or a validated evaluator plateaus, cap reasoning effort, and keep the best eligible fallback that can survive later noisy branches.

"One budget policy can serve every request"

• Symptom: Cheap FAQ traffic burns deep-search budget while hard debugging tasks run out of depth and tokens too early.
• Cause: Compute was allocated uniformly instead of matching task difficulty and business value.
• Fix: Separate routing, branch limits, token caps, and wall-clock caps by request class. Budget policy is part of product design, not model tuning alone.

"Users need raw chain-of-thought to trust the system"

• Symptom: Streams become long, unstable, and hard to parse, while downstream tools can't tell what is final.
• Cause: Internal scratchpad text was exposed instead of a stable progress interface.
• Fix: Stream strategy, budget usage, and short evidence-backed summaries. Keep hidden search traces and uncalibrated evaluator scores inside the runtime.

"Reasoning quality comes only from prompts"

• Symptom: Prompt tweaks plateau, but the team still has no reliable gain on hard tasks.
• Cause: Model capability, verifier quality, cache reuse, and stopping logic were collapsed into one prompt-design problem.
• Fix: Treat base model, search controller, verifier, and serving runtime as separate levers. Improve whichever layer is bottlenecking quality or cost.

"Stop criteria are an optional cleanup detail"

• Symptom: Branches revisit the same semantic state until they exhaust the token budget or hit wall-clock limits.
• Cause: The controller has no hard policy for repeated states, budget exhaustion, or diminishing verifier returns.
• Fix: Enforce token, depth, wall-clock, and repeated-state limits per branch, and return the highest-ranked surviving trace that satisfies required checks when the budget ends.

Follow-up questions

Use these checkpoints to test your routing instinct. Decide first, then compare against the answer sketch.

Final design checklist

Use this checklist to judge whether your final design is defensible:

• Choose a routing strategy for trivial, medium, and hard verifiable tasks, and explain why each class gets that budget.
• Defend when a task needs a single path, Best-of-N, tree search, or a bigger base model instead of more search.
• Pick the right judge for the job: exact checker, ORM, or PRM, and explain what failure appears if you choose the wrong one.
• Check that arithmetic and scheduling constraints are satisfiable before generating branches; reject or clarify impossible inputs instead of optimizing a fabricated solution.
• Estimate rough KV-cache pressure for one branch, many branches, and a shared-prefix layout.
• Define hard stop criteria for depth, tokens, wall-clock time, and repeated semantic states.
• Describe what the user sees while the solver works, and why that stream should expose progress summaries instead of raw scratchpads.

What this completes

Additional inference work can outperform a fixed single attempt only when the task has enough uncertainty and the search budget is controlled. A practical design combines difficulty routing, Best-of-N sampling, Tree of Thoughts, partial-trace scoring with a Process Reward Model, and a bias toward exact evidence when available. KV-cache growth, prefix sharing, and final-answer latency determine whether that search design is affordable.

The roadmap ends here because earlier chapters taught the serving stack, retrieval layer, agent loop, evaluation harness, and safety controls one piece at a time. This capstone asks you to connect them: a reasoning agent is useful only when search policy, evidence contract, evaluator quality, KV-cache budget, streaming behavior, and fallback plan all work together.

Hold onto these design decisions:

1. Test-time compute buys additional attempts or checks; keep it only where held-out evaluation shows quality gains worth its cost and latency.
2. A strategy ladder covers most systems: single path with tuned reasoning effort -> self-consistency or Best-of-N -> Tree of Thoughts / MCTS. More compute helps only up to a problem-specific sweet spot; past it, overthinking degrades accuracy.
3. Process Reward Models (PRM) can score individual steps and enable earlier pruning, but exact checkers and calibrated evaluation remain authoritative where available.
4. Compute-optimal scheduling uses quick attempts first and only escalates for hard problems.
5. KV-cache design is a first-class constraint. Prefix sharing and paged KV storage decide whether search is affordable.
6. Streaming should focus on grounded progress updates and verified summaries, not raw internal scratchpads or uncalibrated confidence.

This capstone closes the system-design arc by asking you to connect the algorithm, verifier, cache budget, streaming behavior, and fallback plan into one defensible system design. Your final deliverable is a proof-of-skill checklist.