Reasoning & Test-Time Compute

The Real-Time Voice AI Agent chapter focused on split-second streaming, interruption, and media state. This final capstone studies the opposite tradeoff: when a task is hard enough, the system may spend extra inference compute before it releases an answer or irreversible effect.

A reasoning agent can spend extra test-time compute on planning, checking, and tool use before its final answer. This design chapter explains when that checked work yields measured value and how to bound it with evidence contracts, evaluators, and budgets.

Imagine you manage a small fulfillment center. A customer sends a message: "I ordered a laptop, a monitor stand, and a cable on Monday with express shipping. The stand just arrived, but the laptop and cable haven't. The tracking number for the laptop says it's at the regional hub. Can I return the stand if I don't like it, and when will the rest arrive?"

A human support agent would not shout the first guess that comes to mind. They would check the order split, verify the tracking status, read the return policy, and only then compose an answer. A standard large language model (LLM) generates each continuation left to right; by itself, one answer attempt does not revisit earlier choices or consult missing evidence. A controller can still ask it to revise, call tools, or generate competing attempts. On tricky problems, the product question is whether that additional work is justified and verifiable.

Reasoning models such as OpenAI's earlier o-series reasoning models, including o1 ^[1], and DeepSeek-R1 ^[2] spend additional compute before returning an answer. Do not conflate that native internal reasoning with a product controller that explicitly samples candidates, calls tools, scores partial states, or performs tree search: those are separable designs with different evidence and serving contracts. This article builds the controller around those contracts, routing easy questions to bounded answers and hard, verifiable questions to deeper work.

Fast thinking and slow thinking

Fast/slow-thinking terminology gives us a useful analogy. Fast thinking is automatic and intuitive. When you read "2 + 2 =", you instantly know the answer. Slow thinking is deliberate and sequential. When you multiply 17 by 34 in your head, you work through steps, catch yourself if you miscarry a digit, and restart if the intermediate result looks wrong. In a product, however, enforceable controls are external candidate state, tool evidence, evaluator behavior, and release gates, not a promise about hidden mental steps.

Single-pass decoding looks more like fast thinking. At every step the model samples the next token conditioned on everything so far; without a controller or tool it can emit a plausible answer without validating it. A reasoning-enabled product can instead allocate a bounded check, use an exact tool when one exists, and release only a result supported by that check.

The useful shift is to treat some inference requests as a budgeted search and verification problem, not just one left-to-right decoding loop. A controller may explore candidate actions, score partial solutions, consult external evidence, and spend extra compute only when evaluation justifies it.

The extra compute is called test-time compute: the floating-point operations (FLOPs) and tokens you spend during inference, after training is done. In a FLOPs-matched study, Snell et al. show that a smaller model plus the right inference-time strategy can beat a 14x larger model on easy-to-intermediate prompts where the smaller model already has a non-trivial chance of succeeding ^[3]. The same paper warns that the gain reverses on the hardest prompts and when you serve many inference tokens per query, so test-time compute is not a free substitute for a stronger base model. The lever you control is no longer just model size; it is measured inference budget.

The illustration above shows three budget strategies. A fixed budget spends the same tokens on every request. An escalation rule adds more compute only after a weak first attempt. An adaptive budget uses a difficulty router to match the strategy to the task before the expensive search begins. The rest of this article builds that adaptive system from the ground up.

The simplest form of thinking longer: one deliberate path

Before we build a tree-search engine, let's look at the cheapest way to add reasoning. You generate one candidate solution, but you structure the output so the model shows its checkpoints. This is the modern successor to the classic "Chain of Thought" prompt, where you explicitly asked the model to "think step by step" ^[4].

On reasoning-trained APIs, direct task instructions are generally preferred over requests to reveal step-by-step thinking, because supported models already perform internal reasoning ^[5]. Your job is to request only outward artifacts the application needs: tool evidence, concise checked milestones, and a final answer. Hidden scratchpad text is not a proof or an audit log.

One common budget knob on OpenAI reasoning models is a reasoning.effort setting ^[6]. Supported values vary by model family, but the operational lesson is the same. Lower effort favors latency and fewer reasoning tokens, while higher effort permits a larger internal reasoning budget before answering. This is the cheapest form of test-time compute control: raise effort only when evals show a measurable quality gain that justifies the extra latency and cost, and keep the lowest useful setting for fast, deterministic tasks. Treat it as the single-path equivalent of the difficulty router you build later in this article.

Here's a lightweight parser for the interface you want the model to satisfy. It turns a compact XML response into visible checkpoints plus a final answer:

Notice what this code does. The RESPONSE_TEMPLATE constrains output to major evidence-backed milestones. It does not make the claims true by itself; downstream code still needs to require tool results for external facts. Hidden token-by-token deliberation stays inside the model runtime, while the application receives a compact interface it can validate.

For our fulfillment-center problem, a single-path trace might look like this:

• Checkpoint 1: Order tool reports that the stand shipped separately.
• Checkpoint 2: Carrier tool reports the current laptop ETA as tomorrow.
• Verification summary: Return-policy tool permits this return within its stated window.
• Final answer: You can return the stand. The current laptop ETA is tomorrow.

A single bounded path is the cheapest candidate strategy when tool evidence or an exact check can validate the result. But what if multiple plans remain plausible?

Asking several agents: Best-of-N sampling

A simple way to pursue better reliability is to generate multiple candidate answers and rank them. Think of it like asking five support agents to solve the same ticket independently, then requiring a policy check or a validated reviewer before selecting a response.

This is called Best-of-N sampling. You run the single-path generator N times with a diversity-producing configuration, rank candidates with the most reliable evaluator available, and release an eligible winner. Temperature is one knob: low temperature tends to collapse diversity, while a higher value can produce different plans. Neither setting guarantees diversity or correctness.

Here's a deterministic version of the selector. In production, the candidates come from separate model calls; the selection rule stays the same:

Why this helps: a worked example

Suppose the base model has a 30% chance of producing a correct routing plan for a complex multi-stop delivery problem. With one sample, your success rate is 30%. With five independent samples, the probability that at least one is correct is:

$$P(\text{at least one success}) = 1 - (1 - 0.30)^5 = 1 - 0.168 = 0.83$$

That's 83% for the event that a correct candidate exists under the independence assumption. It is not an 83% success rate for the returned answer: selection succeeds only when a checker or verifier recognizes the correct candidate. Extra sampling is wasted if candidates are correlated or the verifier cannot identify the useful one.

Worse, a reward model is only a proxy for correctness, so pushing N too high can backfire. Gao et al. show that as you optimize harder against a learned reward model, true (gold) quality eventually drops even while the proxy score keeps climbing, a Goodhart-style effect they call reward-model overoptimization ^[7]. In practice you cap N, ensemble or regularize the reward model, and watch a held-out gold metric rather than trusting the verifier score alone.

When an exact checker exists, it should outrank a persuasive learned score. The example below selects a tested route rather than the answer with the highest style score:

Self-consistency: Best-of-N without a reward model

When the answer is a comparable string (a number, a label, a parse), you can sometimes skip a learned reward model. Self-consistency samples several traces and returns the most frequent answer ^[8]. It can improve selected reasoning benchmarks when correct answers concentrate more than incorrect ones, but majority vote is not verification: correlated wrong answers still win. Prefer an exact checker whenever available; use voting when endpoints are comparable and empirical evaluation supports it.

Where beam search fits

Beam search is the classic decoding baseline: keep the top-$k$ most likely partial continuations at each step. It is easy to batch, but on tasks requiring distinct hypotheses its high-probability branches can become near-duplicates rather than useful alternatives. Evaluate it as a baseline or pruning mechanism against sampling and search on the task distribution you actually serve.

When one path isn't enough: exploring a tree

Some problems force you to backtrack. Imagine a warehouse robot that must pack orders into vans while respecting weight limits, fragile-item rules, and delivery time windows. If the robot assigns the heavy pallet to Van A and later discovers Van A can't cross the bridge, it needs to undo that decision and try Van C instead.

Tree of Thoughts (ToT), proposed by Yao et al. ^[9], treats reasoning as a tree search. At each step, the model generates multiple possible next actions, scores them, and explores the most promising branches while pruning the weak ones. It's similar to playing chess: you consider several moves, evaluate the board, explore the promising lines, and abandon the bad ones.

The diagram shows a search tree for our warehouse problem. From the initial state, three van assignments are possible. Thought B is pruned immediately because it violates a weight limit. Thought A looks promising, so it expands into two sub-assignments. A.2 turns out to break a time-window constraint and gets pruned. A.1 and C.1 both reach valid solutions.

To implement this, we maintain a search beam that keeps only candidate branches active. At each depth, the system generates multiple next steps, scores them with a task-specific evaluator, and prunes paths that fall below its threshold. That evaluator may be an exact constraint checker, a learned reward model, or a combination:

This is a beam-search realization of Tree of Thoughts: simple to batch, simple to reason about, and often the first version teams ship. The algorithm starts with the problem, expands each active node into branching_factor children, evaluates every child, and keeps only the beam_width highest-ranked candidates for the next round.

When you want true MCTS

Tree of Thoughts is a general framework. If you need adaptive revisits instead of level-by-level pruning, you can swap the beam for Monte Carlo Tree Search. In LLM settings, a common variant is PUCT (Predictor + Upper Confidence bounds applied to Trees), the AlphaZero-style selection rule that combines a value estimate with the model's next-step prior ^[10]:

$$\text{PUCT}(s, a) = Q(s, a) + c_{\text{puct}} \cdot P(s, a) \cdot \frac{\sqrt{N(s)}}{1 + N(s, a)}$$

Where:

• $Q(s, a)$ is the estimated downstream value for taking action $a$ in state $s$ (from rollouts, a value model, or another evaluator)
• $P(s, a)$ is the policy prior for the next thought
• $N(s)$ is the total number of visits to the parent node
• $N(s, a)$ is the number of visits to the child node
• $c_{\text{puct}}$ is the exploration constant that balances exploration vs exploitation

The first term exploits high-value branches. The second term favors branches that either have a strong model prior or have not been explored much yet. Beam search is usually easier to run on GPUs because you can batch every frontier expansion together. MCTS can become attractive when evaluator calls are expensive and you want to revisit promising nodes instead of expanding every branch evenly.

Who judges the plans? Verifiers and reward models

Generating many candidate plans is useless if you cannot identify eligible results. Use exact checkers for objective constraints wherever possible. When exact checking is unavailable and a learned verifier scores candidate traces, two common forms are:

An Outcome Reward Model (ORM) scores only the final answer. It's like an audit that checks whether the final shipment plan is valid without inspecting the intermediate routing steps. An ORM is cheap to train because it needs only one label per chain, but it's sparse: if the answer is wrong, you get no signal about where the reasoning broke down.

A Process Reward Model (PRM) scores individual reasoning steps, not just the final answer ^[11]. When step quality is labelable and the PRM is validated on the served domain, it can help prune low-scored branches before paying for complete traces. It remains a learned proxy, not proof that a step is correct.

Think of the PRM like a fulfillment auditor who estimates risk at each routing step. When a candidate assigns the heavy pallet to Van A, a validated PRM may give that step a low score. If an exact bridge-rule checker is available, use it as the authoritative rejection; otherwise the score is a pruning signal whose false positives and negatives need evaluation.

How a PRM works

A PRM is typically a language model with a scalar reward head, trained to predict the correctness of a step $s_t$ given the prompt and the previous steps $s_{1...t-1}$. Lightman et al. released PRM800K, a dataset with 800,000 step-level human feedback labels, because step supervision is the bottleneck for building these verifiers well ^[11].

$$P(\text{correct} \mid \text{context}, s_{1...t}) = \sigma(W \cdot h_t)$$

We take the hidden representation of the current step ($h_t$), map it through a learned linear layer ($W$), and squash the result with a sigmoid ($\sigma$). That gives a score between 0.0 and 1.0; treating it as a calibrated probability requires separate calibration and held-out validation.

Training data is collected via three main paths:

1. Human feedback: Experts label individual steps as Positive, Negative, or Neutral.
2. Monte Carlo estimation: Roll out $N$ completions from a step. If $M$ of $N$ lead to the correct answer, the step score is roughly $M/N$.
3. Teacher-model supervision: Use a stronger verifier or frontier model to pre-label steps, then audit those labels before trusting them at scale.

Here's a practical scoring interface. In production the scoring call is a model with a reward head; this runnable version uses deterministic rules so you can see how weak steps pull down the whole trace:

The score_full_trace method uses the minimum score as a conservative pruning heuristic. In production, set thresholds against held-out task outcomes and prefer exact checks for constraints such as weight limits or policy eligibility.

ORM vs PRM at a glance

Lightman et al. (2023) ^[11] found that process supervision outperformed outcome supervision in their challenging math setting. Carrying that result into routing, support policy, or code tasks requires domain-specific data and validation.

Training reasoning models with RL

Modern reasoning models don't rely only on external search controllers. DeepSeek-R1 shows a complementary path where reinforcement learning makes the base policy itself better at producing long, structured reasoning traces ^[2].

Group Relative Policy Optimization (GRPO) is one important algorithm in that line of work, and DeepSeek-R1 made it especially visible in open-model practice ^[2]. Compared with PPO (Proximal Policy Optimization), GRPO removes the need for a separate critic model:

1. Sample a group of outputs from the policy for each problem
2. Compute rewards for each output (using task rewards, exact verifiers where available, or validated reward models)
3. Calculate the relative advantage within the group (baseline = group mean)
4. Update the policy to favor higher-reward outputs

This cuts RL training overhead because you don't need a separate critic network. It can make the base model more sample-efficient at test time, but it doesn't remove the need for routing, budget limits, or verifier design in production.

Putting it together: a production reasoning agent

So far we've looked at individual techniques. A real production system wires them into a pipeline that classifies the incoming problem, picks a strategy, runs the search, verifies the results, and streams progress back to the user.

What the system must handle

Before drawing boxes, let's list the concrete responsibilities:

• Complex reasoning: Accept problems that need multi-step logic (delivery routing, refund eligibility checks, inventory allocation).
• Candidate state: Maintain external candidate actions, tool observations, and evaluator results needed for search and audit, without assuming access to a model's hidden scratchpad.
• Adaptive compute: Scale inference compute proportional to problem difficulty. A "What is your return policy?" question shouldn't trigger a 20-second tree search.
• Streaming: Emit status updates or safe progress summaries while the solver is working, without exposing unverified conclusions as completed checks.
• Budget control: Enforce configurable reasoning budgets in tokens, time, branches, and cost; product tiers may change caps but must not weaken evidence requirements.

On the non-functional side, declare measurable objectives: concurrency for a target workload, quality lift over single pass on representative verifiable tasks, p95 latency, and per-request cost by route. A 3x or 10x budget is an experiment parameter, not a production guarantee.

Architecture overview

The diagram below shows the full pipeline. Requests enter through a gateway, get classified by a difficulty router, and then flow through either a fast path, a medium path (Best-of-N), or a deep path (Tree of Thoughts). Branching paths should use the best available evaluator and reuse key-value (KV) cache prefixes only where the serving runtime safely supports compatible prefix sharing.

The request flow below shows the sequence in more detail:

Notice the loop in the request flow. The solver generates candidates, reuses shared KV prefixes, scores with the verifier, and streams progress events back. This loop is where the extra test-time compute lives.

The difficulty router

Not every problem needs extended thinking. Simple factual queries waste compute if treated as reasoning tasks. The router evaluates input complexity and assigns a strategy plus token budget. Here's a minimal implementation:

The router should be cheap relative to solving work. Its job is to avoid expensive search on requests that do not benefit while escalating tasks whose risk and verifiability justify more work. Misclassification is not harmless: under-routing can release an incorrect answer, while over-routing can increase cost, latency, and exposure to verifier overoptimization.

Compute-optimal scheduling

The key insight from Snell et al. ^[3] is that no single test-time strategy dominates every prompt. The best move depends on problem difficulty and on whether the base model already has a plausible path to the correct answer. A production scheduler should start cheap and escalate only when an exact check or a validated evaluation signal says the first attempt is weak:

This creates an adaptive system that spends small budgets on easy, checkable problems and escalates only when expected quality lift is worth extra latency and KV-cache pressure. It does not treat "unverifiable" as permission to make high-impact decisions.

The serving bottleneck nobody talks about

Long reasoning traces can stress memory as well as floating-point operations (FLOPs). Every live generated token adds a new key and value vector for every transformer layer, so key-value (KV) cache pressure grows linearly with live context length and multiplies across unshared branches.

KV cache pressure

A useful back-of-the-envelope formula is:

$$\text{KV bytes per token} \approx 2 \cdot L \cdot H_{kv} \cdot d_{\text{head}} \cdot \text{bytes per element}$$

Where $L$ is the number of layers, $H_{kv}$ is the number of KV heads (not always the full attention-head count if the model uses grouped-query attention (GQA)), and $d_{\text{head}}$ is the head dimension. For an 80-layer model with 8 KV heads, head dimension 128, and bfloat16 (BF16) activations:

$$2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327{,}680 \text{ bytes per token} \approx 320 \text{ KB/token}$$

At 16k live tokens, that's roughly 5.0 GiB of KV cache for one branch. Without any prefix reuse, a tree search with eight active 16k-token branches would need about 40 GiB of KV cache before you count model weights or temporary activations.

Prefix sharing and paged KV storage

Best-of-N and tree search create many branches that share the same prompt and often the same early reasoning prefix. If you duplicate that prefix for every branch, memory usage grows with the number and length of branches. Systems like vLLM use PagedAttention to manage KV memory in blocks, which cuts fragmentation and makes long contexts much easier to serve ^[12].

PagedAttention and radix-style prefix caches solve related but different problems. PagedAttention makes allocation cheaper. RadixAttention-style caches, as used in SGLang, index shared prefixes in a tree so sibling branches can reuse the same cached prefix instead of copying it branch by branch ^[13]. In a reasoning agent you usually want both: block-level memory management plus prefix-aware reuse.

That changes the capacity math materially. If eight branches share a 12k-token prefix and each branch adds only a 4k-token unique suffix, the effective footprint is closer to one 12k prefix plus eight 4k suffixes, or about 14 GiB, not the 40 GiB worst case above. The exact savings depend on how quickly branches diverge and on whether your runtime can share prefixes at token-level or block-level granularity.

This is one reason test-time search is an infrastructure problem, not just a prompting trick. A search policy may be sound on paper while its branch width, trace length, or lack of compatible prefix reuse exhausts the serving memory budget before useful search completes.

TTFT vs inter-token latency

Extended reasoning changes the latency contract. If the product emits no status until the answer is ready, a long hidden search phase inflates time to first user-visible token (TTFT). If it emits a quick progress event, TTFT can look healthy while time-to-final-answer and total budget still fail. Measure both.

That is why production systems can stream progress events instead of raw scratchpads. Safe events include selected strategy, budget usage, branch counts, or summaries of externally confirmed checks. Treat learned verifier scores as internal diagnostics unless they are calibrated and presented with an appropriate contract.

A sample capacity plan

The table below uses the 320 KB/token estimate to show how quickly memory usage grows as you add longer traces or more live branches:

Those are worst-case branch-multiplication numbers. Strong prefix reuse can cut the effective KV footprint substantially.

Streaming safe updates

Users need feedback while the system is spending extra inference budget, but that does not mean you should dump raw internal scratchpad text or expose an uncalibrated reward score as confidence. A safer pattern is to stream strategy selection, budget usage, and externally grounded summaries while the solver keeps search details private:

When reasoning is the wrong tool

Not every problem benefits from test-time scaling. Applying tree search or Best-of-N to the wrong use case results in unnecessary latency and wasted resources. You should bypass extended reasoning for:

• Simple source-of-truth queries (e.g., "What is your return policy?"). Query the policy source directly; extra internal reasoning will not recover a missing or current fact.
• Retrieval failures. If the problem requires external facts or tools, longer internal reasoning is the wrong fix. You need retrieval, search, or tool use.
• Open-ended marketing copy (e.g., "Write three brand voice options for a product page"). There is no single objective answer unless the application defines a rubric and validates it, so deep PRM-driven search is usually unjustified.
• Latency-sensitive conversational turns. Live products need measured response objectives; a deep search path that exceeds that objective should acknowledge, defer, or avoid the search.
• Cost-sensitive batch processing. When running millions of documents through a classification pipeline, a large inference multiplier can erase the value of a small quality gain. Escalate only request cohorts whose measured lift pays for the extra cost.

The guiding heuristic is simple: allocate substantial test-time compute when evaluation shows gain for the request class and evidence or verification can keep the answer trustworthy. Creative revision may still use a small candidate budget, but it is not proof-oriented reasoning.

Overthinking: when more thinking lowers accuracy

Longer reasoning is not monotonically better. Empirical 2025 work finds that accuracy often plateaus and can decline once a trace runs past a problem-specific sweet spot: extra steps add error accumulation, and a model can talk itself out of a correct answer it already found ^[14]. This overthinking failure mode is also a cost and latency problem, because a model can burn thousands of reasoning tokens on a trivial question. The defenses are the same controls you already have: a router that keeps trivial tasks on the fast path, a capped reasoning_effort or token budget, and a stop rule that exits once exact results are sufficient or validated evaluation stops improving. Treat reasoning budget as a tuned hyperparameter, not a dial you turn to maximum.

Why not just use a bigger model?

It's tempting to think that deploying a bigger frontier model should dominate every reasoning workload. In practice, the trade-off is more subtle:

Snell et al. make the practical point: on prompts where a smaller model already has some chance of success, extra inference-time compute can beat simply switching to a much larger model at matched compute ^[3]. But if the base model has near-zero task competence, reranking or sampling is unlikely to manufacture the missing capability. Test-time compute amplifies evaluated candidate quality; it does not supply missing knowledge or tools.

Common pitfalls

"Higher temperature will reveal a great answer if I sample enough"

Symptom: Best-of-N produces more varied traces, but quality barely improves and the chosen winner still feels random. Cause: Diversity alone is not enough. Without a verifier that tracks correctness, more samples mostly add noise. Fix: Increase temperature only when you also have a PRM, ORM, or exact checker that can reliably separate strong traces from fluent wrong ones.

"Tree search is always better than single-path reasoning"

Symptom: Easy requests suddenly become slower and more expensive even though accuracy barely moves. Cause: Search budget was applied to every request instead of only the hard, verifiable ones. Fix: Route by difficulty. Keep trivial and easy requests on the fast path, then escalate only when the first trace scores weakly.

"More thinking always improves the answer"

Symptom: The system finds a decent first answer, then keeps exploring until it talks itself into a worse one. Cause: Overthinking. Past a task-specific sweet spot, extra search compounds errors instead of fixing them. Fix: Stop when exact checks pass or a validated evaluator plateaus, cap reasoning effort, and keep the best eligible fallback that can survive later noisy branches.

"One budget policy can serve every request"

Symptom: Cheap FAQ traffic burns deep-search budget while hard debugging tasks run out of depth and tokens too early. Cause: Compute was allocated uniformly instead of matching task difficulty and business value. Fix: Separate routing, branch limits, token caps, and wall-clock caps by request class. Budget policy is part of product design, not only model tuning.

"Users need raw chain-of-thought to trust the system"

Symptom: Streams become long, unstable, and hard to parse, while downstream tools cannot tell what is final. Cause: Internal scratchpad text was exposed instead of a stable progress interface. Fix: Stream strategy, budget usage, and short evidence-backed summaries. Keep hidden search traces and uncalibrated evaluator scores inside the runtime.

"Reasoning quality comes only from prompts"

Symptom: Prompt tweaks plateau, but the team still has no reliable gain on hard tasks. Cause: Model capability, verifier quality, cache reuse, and stopping logic were collapsed into one prompt-design problem. Fix: Treat base model, search controller, verifier, and serving runtime as separate levers. Improve whichever layer is bottlenecking quality or cost.

"Stop criteria are an optional cleanup detail"

Symptom: Branches revisit the same semantic state until they exhaust the token budget or hit wall-clock limits. Cause: The controller has no hard policy for repeated states, budget exhaustion, or diminishing verifier returns. Fix: Enforce token, depth, wall-clock, and repeated-state limits per branch, and return the highest-ranked surviving trace that satisfies required checks when the budget ends.

Follow-up questions

Use these checkpoints to test your routing instinct. Decide first, then compare against the answer sketch.

Mastery checklist

Use this checklist to judge whether your final design is defensible:

• Choose a routing strategy for trivial, medium, and hard verifiable tasks, and explain why each class gets that budget.
• Defend when a task needs a single path, Best-of-N, tree search, or a bigger base model instead of more search.
• Pick the right judge for the job: exact checker, ORM, or PRM, and explain what failure appears if you choose the wrong one.
• Estimate rough KV-cache pressure for one branch, many branches, and a shared-prefix layout.
• Define hard stop criteria for depth, tokens, wall-clock time, and repeated semantic states.
• Describe what the user sees while the solver works, and why that stream should expose progress summaries instead of raw scratchpads.

What this completes

You now understand when allocating additional inference work can outperform a fixed single attempt, and when it cannot. You can build a difficulty router, run Best-of-N sampling, explore a Tree of Thoughts, and score partial traces with a Process Reward Model while still preferring exact evidence when available. You also know the serving constraints (KV cache growth, prefix sharing, and final-answer latency) that determine whether a search design is affordable.

This is why the roadmap ends here. Earlier chapters taught the serving stack, retrieval layer, agent loop, evaluation harness, and safety controls one piece at a time. This capstone asks you to connect them: a reasoning agent is useful only when search policy, evidence contract, evaluator quality, KV-cache budget, streaming behavior, and fallback plan all work together.

Key takeaways:

1. Test-time compute buys additional attempts or checks; keep it only where held-out evaluation shows quality gains worth its cost and latency.
2. A strategy ladder covers most systems: single path with tuned reasoning effort -> self-consistency or Best-of-N -> Tree of Thoughts / MCTS. More compute helps only up to a problem-specific sweet spot; past it, overthinking degrades accuracy.
3. Process Reward Models (PRM) can score individual steps and enable earlier pruning, but exact checkers and calibrated evaluation remain authoritative where available.
4. Compute-optimal scheduling uses quick attempts first and only escalates for hard problems.
5. KV-cache design is a first-class constraint. Prefix sharing and paged KV storage decide whether search is affordable.
6. Streaming should focus on grounded progress updates and verified summaries, not raw internal scratchpads or uncalibrated confidence.

This capstone closes the system-design arc by asking you to connect the algorithm, verifier, cache budget, streaming behavior, and fallback plan into one defensible system design. Your final deliverable is a proof-of-skill checklist.