Reasoning Agent System Design

A release owner sees a canary at 10%: smoke tests passed, but one latency panel spiked after a config flag changed. The agent must choose rollback, hold, or continue the rollout. A plausible recommendation is cheap. A defensible one needs the rollout state, smoke-test result, latency evidence, and config diff before it spends more search.

That is the running incident. A human SRE would gather those observations, compare them with policy, and only then recommend hold, promote, or rollback. This design spends extra test-time compute on that checked work, then asks whether the extra budget earned a safer release.

A standard large language model (LLM) generates each continuation left to right. It can't rewrite tokens already emitted, though later tokens can state a correction. Extra checking still needs a controller that can call tools, request a revision, or generate competing attempts.

Reasoning models such as OpenAI's earlier o-series, including o1 ^{[1]Reference 1o1 Preview Modelhttps://developers.openai.com/api/docs/models/o1-preview}, and DeepSeek-R1 ^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948} spend additional compute before returning an answer. That native internal reasoning is not the same as a product controller that samples candidates, calls tools, scores partial states, or searches a tree. They have different evidence and serving contracts.

The controller has two jobs: apply a known release policy, then investigate a possible dependency fix. Applying policy needs fresh observations and arithmetic; repair planning may benefit from competing plans. All Python examples below run locally with the standard library. Their tool records, model responses, and evaluator scores are synthetic fixtures, not measurements from a deployment or a trained judge.

Fast answers aren't a release contract

Suppose the model says "hold" before it has seen the metrics result. What can the release gate prove? Single-pass decoding only samples the next token from everything so far. Without a controller or tool, it can emit a fluent hold/rollback recommendation that never touched the deploy API, smoke-test log, or latency panel. Hidden "slow thinking" isn't an enforceable control. The shippable controls are external candidate state, tool evidence, evaluator behavior, and release gates.

Treat this request as a budgeted search and verification problem, not one left-to-right decoding loop. A controller can explore candidate actions, score partial solutions, consult external evidence, and spend extra compute only when evaluation justifies it.

The extra work is called test-time compute: the floating-point operations (FLOPs) and tokens spent during inference, after training is done. Snell et al. show that a smaller model plus a suitable inference-time strategy can beat a 14x larger model on some math prompts ^{[3]Reference 3Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314}. Their comparison matches estimated pretraining plus inference FLOPs across a workload, not a per-request token allowance. The result depends on prompt difficulty and expected inference load: harder prompts and heavy inference workloads can favor the larger pretrained model. The study also trains its revision models and verifiers; prompting an arbitrary small model doesn't reproduce the result.

An initial route is only a spending ceiling. The controller must reserve capacity before dispatching work and can stop well below that ceiling. A threshold lookup needs no competing plans; a dependency repair may need several. Whether that extra search improves quality must be measured on held-out requests.

The simplest form of extra compute: one deliberate path

What if the supplied tool observations already settle the canary decision? Start with one candidate and a bounded output allowance. On supported reasoning APIs, reasoning.effort adjusts relative internal effort; it isn't a hard token, time, or cost cap. A structured response format is only an outward interface for checked milestones and a final answer. XML doesn't make the model think longer. Classic Chain of Thought prompts asked a model to "think step by step" ^{[4]Reference 4Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.https://arxiv.org/abs/2201.11903}, but a production controller should separate hidden reasoning budget from evidence the application can validate.

OpenAI's guidance, checked October 10, 2026, treats reasoning.effort as a tuning knob alongside a clear task, constraints, and output contract ^{[5]Reference 5Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning}. Request tool evidence, concise checked milestones, and a final answer rather than a hidden scratchpad. Hidden reasoning tokens aren't proof or an audit log.

Supported effort values are model-dependent. GPT-5.5, for example, documents none, low, medium (default), high, and xhigh ^{[6]Reference 6GPT-5.5 Modelhttps://developers.openai.com/api/docs/models/gpt-5.5}. Lower effort favors latency and fewer reasoning tokens; higher effort permits more internal work before answering. Raise effort only when evals show a quality gain that justifies extra latency and cost, and keep the lowest useful setting for fast, deterministic tasks. Treat it as the single-path version of the difficulty router introduced later.

The current guide separates four controls. Check support on the actual model; GPT-6 Astra, for example, rejects effort=none.^{[5]Reference 5Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning}

The guide documents all_turns as GPT-5.6's default; it requires access to prior response items and cannot reuse reasoning across incompatible model families.^{[5]Reference 5Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} Keep authenticated tool observations and policy versions in application state even when opaque model reasoning persists.

Use a lightweight parser for the interface you want the model to satisfy. It turns a compact XML response into visible checkpoints plus a final answer:

from dataclasses import asdict, dataclass
import json
import xml.etree.ElementTree as ET

@dataclass
class ReasoningStep:
    kind: str
    content: str

@dataclass
class ReasoningTrace:
    steps: list[ReasoningStep]
    final_answer: str

class SinglePathReasoner:
    RESPONSE_TEMPLATE = """
    Answer using supplied tool observations only.

    Return XML with this structure:
    <response>
      <checkpoint>major milestone only</checkpoint>
      <checkpoint>major milestone only</checkpoint>
      <verification_summary>brief check against supplied evidence</verification_summary>
      <final_answer>final answer</final_answer>
    </response>

    Keep checkpoints concise. Do not dump a full scratchpad.
    """

    def parse(self, content: str) -> ReasoningTrace:
        if len(content) > 16_000 or "<!" in content:
            raise ValueError("Response exceeds limit or contains declarations")
        root = ET.fromstring(content)
        allowed = {"checkpoint", "verification_summary", "final_answer"}
        if root.tag != "response" or root.attrib or (root.text or "").strip():
            raise ValueError("Expected a response containing only child fields")
        if any(node.tag not in allowed or node.attrib or len(node)
               or not (node.text or "").strip() or (node.tail or "").strip()
               for node in root):
            raise ValueError("Invalid or empty response field")
        if any(len(root.findall(tag)) != 1
               for tag in ("verification_summary", "final_answer")):
            raise ValueError("Expected one verification summary and one final answer")
        checkpoints = [node.text.strip() for node in root.findall("checkpoint") if node.text]
        steps = [ReasoningStep("checkpoint", cp) for cp in checkpoints if cp]
        verification = self._extract_text(root, "verification_summary")
        if verification:
            steps.append(ReasoningStep("verification", verification))
        return ReasoningTrace(
            steps=steps,
            final_answer=self._extract_text(root, "final_answer"),
        )

    def _extract_text(self, root: ET.Element, tag: str) -> str:
        text = root.findtext(tag, default="")
        return text.strip()

model_response = """
<response>
  <checkpoint>Deploy API reports canary still at 10%.</checkpoint>
  <checkpoint>Smoke-test tool reports the release passed.</checkpoint>
  <verification_summary>Metrics tool shows latency above watch threshold but below rollback threshold.</verification_summary>
  <final_answer>Hold the canary at 10%, keep rollback armed, and inspect the config flag before promotion.</final_answer>
</response>
"""

trace = SinglePathReasoner().parse(model_response)
print(json.dumps({
    "steps": [asdict(step) for step in trace.steps],
    "final_answer": trace.final_answer,
}, indent=2))

{
  "steps": [
    {
      "kind": "checkpoint",
      "content": "Deploy API reports canary still at 10%."
    },
    {
      "kind": "checkpoint",
      "content": "Smoke-test tool reports the release passed."
    },
    {
      "kind": "verification",
      "content": "Metrics tool shows latency above watch threshold but below rollback threshold."
    }
  ],
  "final_answer": "Hold the canary at 10%, keep rollback armed, and inspect the config flag before promotion."
}

The split matters. RESPONSE_TEMPLATE constrains output to major evidence-backed milestones, but it doesn't make claims true. Downstream code still needs to require tool results for external facts. Hidden token-by-token deliberation stays inside the model runtime; the application receives a compact interface it can validate.

For the canary, one bounded path should expose only the observations needed to justify its action:

1. The deploy tool reports that the canary is still at 10%.
2. The smoke-test tool reports that the release passed.
3. The metrics tool shows elevated latency below the rollback threshold.
4. The final answer holds the canary, keeps rollback armed, and inspects the changed config flag before promotion.

That is enough when tool evidence or an exact check settles the result. If multiple plans remain plausible, the next move is a small set of candidates, not a tree by default.

from dataclasses import dataclass, replace
import json
import math

@dataclass(frozen=True)
class Observation:
    deploy_id: str
    revision: int
    observed_at: int  # seconds in the fixture clock
    canary_percent: int
    smoke_passed: bool
    p95_ms: float

def release_answer(observation: Observation | None, now: int) -> dict[str, object]:
    # Authenticated adapters, not model output, construct this record.
    if observation is None:
        return {"release": False, "reason": "missing_observation"}
    if any(type(value) is not int for value in
           (observation.revision, observation.observed_at, now)):
        return {"release": False, "reason": "invalid_observation"}
    if (observation.deploy_id, observation.revision) != ("checkout-42", 7):
        return {"release": False, "reason": "wrong_deploy_or_revision"}
    if not 0 <= now - observation.observed_at <= 30:
        return {"release": False, "reason": "stale_or_future_observation"}
    if (type(observation.canary_percent) is not int or observation.canary_percent != 10
            or type(observation.smoke_passed) is not bool
            or type(observation.p95_ms) not in (int, float)
            or not math.isfinite(observation.p95_ms) or observation.p95_ms < 0):
        return {"release": False, "reason": "invalid_observation"}
    # Fixture policy v3: >=300 ms or failed smoke => rollback;
    # >=200 ms => hold; otherwise eligible to continue the rollout.
    if not observation.smoke_passed or observation.p95_ms >= 300:
        action = "rollback"
    elif observation.p95_ms >= 200:
        action = "hold"
    else:
        action = "continue"
    return {"release": True, "recommendation": action,
            "evidence": "checkout-42/revision-7", "policy": "canary-v3"}

observed = Observation("checkout-42", 7, 100, 10, True, 240)
attempts = [None, observed, replace(observed, p95_ms=300),
            replace(observed, revision=6), replace(observed, revision=7.0)]
assert not release_answer(replace(observed, observed_at=100.5), now=110)["release"]
print(json.dumps([release_answer(item, now=110) for item in attempts], indent=2))

[
  {
    "release": false,
    "reason": "missing_observation"
  },
  {
    "release": true,
    "recommendation": "hold",
    "evidence": "checkout-42/revision-7",
    "policy": "canary-v3"
  },
  {
    "release": true,
    "recommendation": "rollback",
    "evidence": "checkout-42/revision-7",
    "policy": "canary-v3"
  },
  {
    "release": false,
    "reason": "wrong_deploy_or_revision"
  },
  {
    "release": false,
    "reason": "invalid_observation"
  }
]

At 240 ms, this policy recommends hold; exactly 300 ms means rollback. Those thresholds are invented for the exercise, not SRE guidance. The model can't change the record, policy version, or decision. release=True permits an advisory response, not a production write. An actuator would need separate authorization, an idempotency key, and an atomic check that deployment revision 7 is still current. The 30-second freshness bound also needs a product-specific justification.

Python annotations don't validate incoming records: 7.0 == 7, so the original equality test admitted a float revision. The fifth attempt now rejects it. This fixture also requires whole-second integer clock values; a real adapter needs a defined timestamp format and unit, not an assumption that every valid timestamp is an integer.

A checkpoint is a parseable milestone. The release gate still needs the matching tool observation:

Comparing complete candidates: Best-of-N sampling

What changes when both "hold" and "rollback" look plausible after the first pass? Generate a small set of candidate answers and rank them before selecting a response. In an incident-analysis workflow, separately generated candidates enter a policy check or validated reviewer.

This is Best-of-N sampling. Run the single-path generator N times with a diversity-producing configuration, rank candidates with the most reliable evaluator available, and release an eligible winner. Temperature is one knob: low temperature tends to collapse diversity, while a higher value can produce different plans. Neither setting guarantees diversity or correctness.

Start with ranking alone. These invented scores select B; that selection is not permission to release it. The evidence gate still decides eligibility. Separate model calls would produce the candidates in a live system, but shared prompts and training mean their errors needn't be independent.

from dataclasses import asdict, dataclass
import json
import math

@dataclass
class CandidateTrace:
    candidate_id: str
    answer: str
    verifier_score: float

class BestOfNSampler:
    @staticmethod
    def probability_at_least_one_success(base_success_rate: float, n: int) -> float:
        if (not math.isfinite(base_success_rate) or not 0 <= base_success_rate <= 1
                or type(n) is not int or n < 0):
            raise ValueError("Expected a probability and nonnegative integer count")
        if n == 0 or base_success_rate == 0:
            return 0.0
        if base_success_rate == 1:
            return 1.0
        return -math.expm1(n * math.log1p(-base_success_rate))

    def choose(self, traces: list[CandidateTrace]) -> CandidateTrace | None:
        if any(not math.isfinite(trace.verifier_score) for trace in traces):
            raise ValueError("Non-finite verifier score")
        return max(traces, key=lambda trace: trace.verifier_score, default=None)

traces = [
    CandidateTrace("A", "Rollback immediately; latency panel is unknown.", 0.61),
    CandidateTrace("B", "Hold canary; tests passed and latency is below rollback threshold.", 0.93),
    CandidateTrace("C", "Promote to 100% because smoke tests passed.", 0.28),
    CandidateTrace("D", "Disable all traffic before checking metrics.", 0.44),
    CandidateTrace("E", "Ignore the spike because rollout is partial.", 0.19),
]

sampler = BestOfNSampler()
winner = sampler.choose(traces)
rare_coverage = sampler.probability_at_least_one_success(1e-20, 100)
assert math.isclose(rare_coverage, 1e-18, rel_tol=1e-15)
assert sampler.probability_at_least_one_success(1, 0) == 0
assert sampler.probability_at_least_one_success(1, 5) == 1

print(json.dumps({
    "n": len(traces),
    "base_success_rate": 0.30,
    "at_least_one_success": round(
        sampler.probability_at_least_one_success(0.30, len(traces)),
        3,
    ),
    "selected": asdict(winner) if winner else None,
    "rare_case_100_attempts_coverage": format(rare_coverage, ".1e"),
}, indent=2))

{
  "n": 5,
  "base_success_rate": 0.3,
  "at_least_one_success": 0.832,
  "selected": {
    "candidate_id": "B",
    "answer": "Hold canary; tests passed and latency is below rollback threshold.",
    "verifier_score": 0.93
  },
  "rare_case_100_attempts_coverage": "1.0e-18"
}

Why this helps: a worked example

Suppose the base model has a 30% chance of producing a correct mitigation plan for a complex canary incident. Before looking at the selector, predict what five independent attempts buy you. With one sample, success is 30%. With five, the probability that at least one is correct is:

That 83% describes the event that a correct candidate exists under the independence assumption. It's not an 83% success rate for the returned answer: selection succeeds only when a checker or verifier recognizes the correct candidate. Extra sampling is wasted if candidates are correlated or the verifier can't identify the useful one.

The rare-case value is still effectively useless coverage. Why use log1p and expm1? With p=1e-20, floating-point 1-p rounds to 1, making the naive formula return zero. The stable expression preserves about 1e-18 for 100 hypothetical attempts. This fixes arithmetic, not base-model competence, independence, or selection quality.

A reward model is only a proxy, so pushing N too high can backfire. Gao et al.'s synthetic setup uses a separate “gold” reward model to stand in for human preferences; its score can fall while optimization raises the learned proxy score.^{[7]Reference 7Scaling Laws for Reward Model Overoptimizationhttps://proceedings.mlr.press/v202/gao23h.html} Gold score here isn't an exact correctness oracle. Cap N and evaluate returned outcomes on held-out tasks.

When an exact checker exists, it outranks a persuasive learned score. The next selector chooses a tested dependency path rather than the answer with the highest style score:

import json

candidates = [
    {"id": "A", "path": ["api", "legacy_adapter", "database"], "proxy_score": 0.96},
    {"id": "B", "path": ["api", "feature_flag", "database"], "proxy_score": 0.82},
]
blocked_edges = {("api", "legacy_adapter")}
allowed_edges = {("api", "legacy_adapter"), ("legacy_adapter", "database"),
                 ("api", "feature_flag"), ("feature_flag", "database")}

def passes_constraints(path: list[str]) -> bool:
    if len(path) < 2 or path[0] != "api" or path[-1] != "database":
        return False
    if len(set(path)) != len(path):
        return False
    edges = set(zip(path, path[1:]))
    return edges <= allowed_edges and not bool(edges & blocked_edges)

checked = [
    {**candidate, "exact_check": passes_constraints(candidate["path"])}
    for candidate in candidates
]
eligible = [candidate for candidate in checked if candidate["exact_check"]]
winner = max(eligible, key=lambda candidate: candidate["proxy_score"], default=None)

print(json.dumps({"checked": checked, "selected": winner["id"] if winner else None}, indent=2))

{
  "checked": [
    {
      "id": "A",
      "path": [
        "api",
        "legacy_adapter",
        "database"
      ],
      "proxy_score": 0.96,
      "exact_check": false
    },
    {
      "id": "B",
      "path": [
        "api",
        "feature_flag",
        "database"
      ],
      "proxy_score": 0.82,
      "exact_check": true
    }
  ],
  "selected": "B"
}

Self-consistency: Best-of-N without a reward model

When the answer is a comparable string, such as a number, label, or parse, you can sometimes skip a learned reward model. Self-consistency samples several traces and returns the most frequent answer ^{[8]Reference 8Self-Consistency Improves Chain of Thought Reasoning in Language Models.https://arxiv.org/abs/2203.11171}. It can improve selected reasoning benchmarks when correct answers concentrate more than incorrect ones, but majority vote isn't verification: correlated wrong answers still win. Prefer an exact checker whenever available; use voting only when endpoints are comparable and evaluation supports it.

Where beam search fits

Beam search is the classic decoding baseline: keep the top-$k$ most likely partial continuations at each step. It's easy to batch, but on tasks requiring distinct hypotheses its high-probability branches can become near-duplicates rather than useful alternatives. Evaluate it as a baseline or pruning mechanism against sampling and search on the task distribution you serve.

When one path isn't enough: exploring a tree

The canary is held. The changed flag enabled parser-v2, but correlation doesn't establish the cause of the latency spike. Suppose a compatibility check now finds a removed API call. Repair planning is a separate problem from proving the incident's cause. A single tool-using agent can revise a plan; an explicit tree makes competing partial plans, their constraints, and their remaining budgets inspectable.

Diagnosis is now a search problem. The resolver must choose package versions while respecting API compatibility, Python version floors, and security advisories. If it pins parser-v2 and later learns that parser-v2 calls an API the deployed runtime removed, it needs to backtrack.

Tree of Thoughts (ToT), proposed by Yao et al. ^{[9]Reference 9Tree of Thoughts: Deliberate Problem Solving with Large Language Models.https://arxiv.org/abs/2305.10601}, treats reasoning as tree search. At each step, the model generates possible next actions, scores them, and explores promising branches while pruning weak ones. For dependency resolution, each branch is a candidate version assignment; each evaluator check asks whether that partial assignment still satisfies known constraints.

The canary's config flag turns into that tree. From the initial state, three version choices are possible. Predict which branch should die before reading the scores: nightly violates the runtime constraint. Parser-v2 looks usable, so it expands into two sub-choices. Calling the removed API breaks compatibility and gets pruned. The compatibility shim and the parser-v1 sanitizer patch both reach valid solutions.

This finite fixture uses a breadth-first beam. Hard constraint failures are rejected before proxy ranking. Completed solutions are retained separately, so another search round can't erase them. Beam pruning is irreversible here: unlike a backtracking search, it won't revisit a discarded node or guarantee an optimal solution.

from dataclasses import dataclass
import json

EXPANSIONS = {
    "start": ["Pin parser v2", "Pin parser nightly", "Keep parser v1"],
    "Pin parser v2": ["A.1 add compatibility shim", "A.2 call removed API"],
    "Keep parser v1": ["C.1 patch sanitizer", "C.2 defer security fix"],
}

SCORES = {
    "Pin parser v2": 0.74,
    "Pin parser nightly": 0.12,
    "Keep parser v1": 0.81,
    "A.1 add compatibility shim": 0.70,
    "A.2 call removed API": 0.18,
    "C.1 patch sanitizer": 0.96,
    "C.2 defer security fix": 0.39,
}
INVALID = {"Pin parser nightly": "runtime_floor",
           "A.2 call removed API": "removed_api",
           "C.2 defer security fix": "unfixed_advisory"}
COMPLETE = {"A.1 add compatibility shim", "C.1 patch sanitizer"}

@dataclass
class ThoughtNode:
    content: str
    depth: int
    score: float
    parent: "ThoughtNode | None" = None

    def trace(self) -> str:
        if self.parent is None:
            return self.content
        return f"{self.parent.trace()}\n{self.content}"

class TreeOfThoughts:
    def __init__(self, beam_width: int = 2, min_score: float = 0.35):
        if type(beam_width) is not int or beam_width < 1 or not 0 <= min_score <= 1:
            raise ValueError("Invalid search limits")
        self.beam_width = beam_width
        self.min_score = min_score

    def search(self, max_depth: int = 2) -> dict[str, object]:
        if type(max_depth) is not int or max_depth < 0:
            raise ValueError("Invalid depth")
        root = ThoughtNode("start", depth=0, score=1.0)
        beam = [root]
        rounds = []
        completed = []

        for depth in range(max_depth):
            candidates = []
            for node in beam:
                for thought in EXPANSIONS.get(node.content, []):
                    candidates.append(ThoughtNode(thought, depth + 1, SCORES[thought], node))

            survivors = [node for node in candidates
                         if node.content not in INVALID and node.score >= self.min_score]
            survivors.sort(key=lambda node: node.score, reverse=True)
            completed.extend(node for node in survivors if node.content in COMPLETE)
            unfinished = [node for node in survivors if node.content not in COMPLETE]
            beam = unfinished[:self.beam_width]
            rounds.append({
                "depth": depth + 1,
                "kept": [node.content for node in beam],
                "completed": [node.content for node in completed],
                "pruned": [
                    node.content for node in candidates
                    if node.content in INVALID or node.score < self.min_score
                ] + [node.content for node in unfinished[self.beam_width:]],
            })
            if not beam:
                break

        best = max(completed, key=lambda node: node.score, default=None)
        return {"winner": best.trace().split("\n") if best else None, "rounds": rounds}

print(json.dumps(TreeOfThoughts().search(), indent=2))

{
  "winner": [
    "start",
    "Keep parser v1",
    "C.1 patch sanitizer"
  ],
  "rounds": [
    {
      "depth": 1,
      "kept": [
        "Keep parser v1",
        "Pin parser v2"
      ],
      "completed": [],
      "pruned": [
        "Pin parser nightly"
      ]
    },
    {
      "depth": 2,
      "kept": [],
      "completed": [
        "C.1 patch sanitizer",
        "A.1 add compatibility shim"
      ],
      "pruned": [
        "C.2 defer security fix",
        "A.2 call removed API"
      ]
    }
  ]
}

The fixture's constraints and terminal states come from explicit tables, not package metadata or a real solver. It demonstrates search bookkeeping: depth zero or one returns no completed winner; depth two finds the sanitizer patch; extra depth preserves that result. A live resolver must derive those verdicts from manifests and compatibility tests. C.2 defer security fix is rejected even though its 0.39 score clears the 0.35 proxy threshold.

When revisiting branches beats a fixed beam

Tree of Thoughts is a general framework. If you need adaptive revisits instead of level-by-level pruning, you can swap the beam for Monte Carlo tree search (MCTS). In LLM settings, a common variant is PUCT (Predictor + Upper Confidence bounds applied to Trees), the AlphaZero-style selection rule that combines a value estimate with the model's next-step prior ^{[10]Reference 10Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm.https://arxiv.org/abs/1712.01815}:

Where:

• $Q(s, a)$ is the estimated downstream value for taking action $a$ in state $s$ (from rollouts, a value model, or another evaluator)
• $P(s, a)$ is the policy prior for the next thought
• $N(s)$ is the total number of visits to the parent node
• $N(s, a)$ is the number of visits to the child node
• $c_{\text{puct}}$ is the exploration constant that balances exploration vs exploitation

For example, with $Q=0.5$, $c_{\text{puct}}=1$, $P=0.2$, $N(s)=16$, and $N(s,a)=1$, the score is $0.5 + 0.2(4)/2 = 0.9$. At a previously unvisited parent, this expression's exploration bonus is zero; an implementation needs an initialization and tie-breaking rule. Neither Q nor the combined score is proof of correctness. Beam frontiers offer batchable work; MCTS can allocate visits unevenly when that benefits the evaluator/task pair.

Who judges the plans? Verifiers and reward models

The search only helps if the controller can identify an eligible result. Use exact checkers for objective constraints wherever possible. When exact checking is unavailable, a learned verifier can score candidate traces, but its job must remain explicit.

An Outcome Reward Model (ORM) learns from labels on completed outcomes. It may read the whole solution, not just the final answer, but receives no direct label identifying the first faulty step. One label per solution can reduce annotation work; total training cost depends on how those labels are obtained.

A Process Reward Model (PRM) scores individual reasoning steps, not only the final answer ^{[11]Reference 11Let's Verify Step by Step.https://arxiv.org/abs/2305.20050}. When step quality is labelable and the PRM is validated on the served domain, it can prune low-scored branches before complete traces are generated. It remains a learned proxy, not proof that a step is correct. Exact compatibility checks still outrank it for the parser pin.

The PRM estimates risk at each dependency step. When a candidate calls a removed API, a validated PRM may give that step a low score. If an exact compatibility checker exists, it makes the authoritative rejection. Otherwise, the score is only a pruning signal whose false positives and negatives need measurement.

How a PRM works

Lightman et al. train a step classifier over human labels: Positive, Negative, or Neutral. They released PRM800K, about 800,000 step-level labels ^{[11]Reference 11Let's Verify Step by Step.https://arxiv.org/abs/2305.20050}. A search controller can reduce that three-way distribution to a scalar such as $P(\text{positive})$, then prune below a held-out threshold. For a simpler binary-label task, an illustrative sigmoid head would be:

$h_t$ is the hidden representation of the current step, $W$ is a learned linear layer, and $\sigma$ is a sigmoid that maps the result into $(0, 1)$. Treating that number as a calibrated probability still requires separate calibration and held-out validation.

Experts can label individual steps. Alternatively, roll out $N>0$ completions from a partial state and use $M/N$ when $M$ reach a checked correct outcome. That estimates downstream success under the rollout policy, not the probability that the current step is logically valid: a later correction can rescue an invalid step. Teacher-model labels need a separate audit before they drive pruning.

The following fixture tests when pruning happens, not whether a learned judge is accurate. Each yielded step represents a generator call. Scores are supplied by a fixed table; the loop must stop before requesting the fourth step. Runtime compatibility remains an exact check, independent of the score.

import json
import math

generated = []
def generate_steps():
    for step, score, exact_ok in [
        ("Parse manifest", 0.92, True),
        ("Pin parser v2", 0.74, True),
        ("Call removed API", 0.15, False),
        ("Recommend deployment", 0.85, True),
    ]:
        generated.append(step)
        yield step, score, exact_ok

def consume_until_pruned(steps):
    checked = []
    for step, score, exact_ok in steps:
        if type(exact_ok) is not bool:
            raise ValueError("Exact-check verdict must be a boolean")
        if type(score) not in (int, float) or not math.isfinite(score) or not 0 <= score <= 1:
            raise ValueError("Expected a finite score in [0, 1]")
        checked.append({"step": step, "score": score, "exact_ok": exact_ok})
        if not exact_ok or score < 0.35:
            return {"decision": "prune", "checked": checked}
    return {"decision": "needs_final_check", "checked": checked}

result = consume_until_pruned(generate_steps())
assert len(generated) == 3
for malformed in [("Call removed API", 0.99, "false"),
                  ("Pin parser v2", float("nan"), True)]:
    try:
        consume_until_pruned(iter([malformed]))
    except ValueError:
        pass
    else:
        raise AssertionError("Malformed verifier input reached the search policy")
print(json.dumps(result, indent=2))

{
  "decision": "prune",
  "checked": [
    {
      "step": "Parse manifest",
      "score": 0.92,
      "exact_ok": true
    },
    {
      "step": "Pin parser v2",
      "score": 0.74,
      "exact_ok": true
    },
    {
      "step": "Call removed API",
      "score": 0.15,
      "exact_ok": false
    }
  ]
}

Only three steps were requested. A post-hoc minimum over an already generated trace wouldn't save the fourth step. In a live system, step scoring adds its own latency and token cost, and a mistaken low score can prune a valid solution. Measure net savings and false rejection; don't infer either from this fixture.

ORM vs PRM at a glance

Lightman et al. ^{[11]Reference 11Let's Verify Step by Step.https://arxiv.org/abs/2305.20050} found that process supervision outperformed outcome supervision in their challenging math setting. Carrying that result into release-policy, planning, or code tasks requires domain-specific data and validation.

#A checker adapter returning "false" as a string is a concrete trap: Python treats it as true, so the removed-API step with score 0.99 would reach needs_final_check in an unvalidated loop. A NaN score also evades a simple score < 0.35 comparison. The fixture rejects both before accepting a step. A malformed verdict requires repair or abstention; it is not a low-quality candidate to rank.

A correct answer can hide a broken evaluator

An evaluator may accept invalid reasoning because the final answer looks right. Sun et al. isolate this failure with Valid-Answer-Invalid-Reasoning (VAIR) math examples: GPT-5.4 scored 47.9% when asked to grade solutions that contained simple reasoning flaws but valid answers, despite near-perfect production on the same problems.^{[12]Reference 12An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Modelshttps://arxiv.org/abs/2606.01462} Their analysis finds evidence of answer confirmation bias, where the model works backward from the correct endpoint and rationalizes flawed steps.

That result is from one controlled math setting, so don't turn 47.9% into expected production accuracy. Use it as an evaluation design warning:

For release decisions, exact tool evidence remains authoritative. When learned process scoring is necessary, score steps before revealing the final candidate where the interface permits, calibrate on held-out adversarial cases, and track false acceptance separately from ordinary solver accuracy. Solver pass rate and evaluator error rate are different production metrics.

Training reasoning models with RL

External search controllers aren't the only route. DeepSeek-R1 describes a complementary path where reinforcement learning improves the base policy's ability to produce structured reasoning traces ^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}.

Group Relative Policy Optimization (GRPO) is one algorithm in that work. Compared with PPO (Proximal Policy Optimization), GRPO removes the need for a separate critic model ^{[2]Reference 2DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learninghttps://arxiv.org/abs/2501.12948}. For one problem, the policy samples a group of outputs, scores each with task rewards, exact verifiers, or validated reward models, and compares each reward with the group mean. The update then favors outputs with higher relative reward.

That removes a separate critic network from this training setup. It can make the base model more effective at test time, but it doesn't remove production routing, budget limits, or verifier design. Training-time improvement and request-time control still solve different problems.

Putting it together: a production reasoning agent

We now have the pieces: one bounded path, candidate sampling, partial-state search, judges, and stop rules. A production system connects them in that order: classify the problem, choose a strategy, search, verify, and stream only safe progress.

What the system must handle

Before drawing boxes, make responsibilities concrete. The system must accept multi-step work such as canary triage, dependency resolution, and multi-file debugging. It must keep candidate actions, tool observations, and evaluator results outside the model's hidden scratchpad so search and audit have a shared state.

It also needs adaptive compute. A request asking for the rollback threshold for checkout-api shouldn't trigger a 20-second tree search, while a compatibility graph may justify one. Streaming should expose status or safe progress summaries, never an unverified conclusion as a completed check. Finally, token, time, branch, and cost budgets can vary by product tier, but none may weaken evidence requirements.

On the non-functional side, declare measurable objectives: concurrency for a target workload, quality lift over single pass on representative verifiable tasks, p95 latency, and per-request cost by route. A 3x or 10x budget is an experiment parameter, not a production guarantee.

Architecture overview

The full pipeline has one gateway, one difficulty router, and three lanes: a fast path, a medium Best-of-N path, and a deep Tree-of-Thoughts path. Branching lanes should use the best available evaluator and reuse key-value (KV) cache prefixes only where the serving runtime safely supports compatible sharing.

The difficulty router

Not every problem needs extended thinking. A simple factual query wastes compute if treated as a reasoning task. The router evaluates input complexity and assigns a strategy plus token budget. Predict the route first: a threshold lookup should stay on the fast path, a canary diagnosis may need candidates, and a compatibility graph may need backtracking.

This small lab uses keyword matches as a transparent demo. A production router is a cheap classifier trained on features such as length, tool need, historical difficulty, and verifier fail rate. Judge it on held-out under-routing and over-routing cost, not on a hardcoded substring list.

from dataclasses import asdict, dataclass
import json

@dataclass
class Route:
    difficulty: str
    strategy: str
    max_tokens: int

class DifficultyRouter:
    CONFIGS = {
        "trivial": Route("trivial", "single_pass", 500),
        "easy": Route("easy", "single_path", 2_000),
        "medium": Route("medium", "best_of_3", 4_000),
        "hard": Route("hard", "tree_search", 8_000),
    }

    def classify(self, problem: str) -> Route:
        text = problem.lower()
        if "satisfies" in text or "constraints after" in text:
            return self.CONFIGS["hard"]
        if "canary" in text or "debug" in text:
            return self.CONFIGS["medium"]
        if "rollback threshold" in text or "policy" in text:
            return self.CONFIGS["easy"]
        return self.CONFIGS["trivial"]

router = DifficultyRouter()
problems = [
    "What is the rollback threshold for checkout-api?",
    "Debug this canary: smoke tests passed, but p95 latency spiked after the parser_v2 flag.",
    "Find a parser pin that satisfies runtime, API, and security constraints after the flag change.",
]

print(json.dumps([asdict(router.classify(problem)) for problem in problems], indent=2))

[
  {
    "difficulty": "easy",
    "strategy": "single_path",
    "max_tokens": 2000
  },
  {
    "difficulty": "medium",
    "strategy": "best_of_3",
    "max_tokens": 4000
  },
  {
    "difficulty": "hard",
    "strategy": "tree_search",
    "max_tokens": 8000
  }
]

Here max_tokens means the total generated-token allowance across generator and learned-verifier calls, not an allowance for each branch. It excludes input tokens and isn't a dollar or FLOPs budget. Account for input/cache/output pricing separately. Under-routing can miss a useful answer or cause an abstention; it must never bypass the same evidence gate used by deeper routes. Over-routing increases cost, latency, and opportunities to exploit a flawed evaluator.

Escalation is a measured policy

Snell et al. ^{[3]Reference 3Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314} show that no single test-time strategy dominates every prompt. The best move depends on problem difficulty and on whether the base model already has a plausible path to the correct answer. Ask what the first score can change: a strong exact result should stop cheaply, a weak but verifiable result can justify more candidates, and an unverifiable high-impact claim needs a tool or human. A production scheduler turns those signals into escalation:

import json

class EscalationPolicy:
    def choose_strategy(
        self, evaluation_score: float, verifiable: bool, high_impact: bool = False,
        exact_complete: bool = False, needs_backtracking: bool = False,
    ) -> dict[str, object]:
        if type(evaluation_score) not in (int, float) or not 0 <= evaluation_score <= 1:
            raise ValueError("Expected a finite score in [0, 1]")
        if any(type(flag) is not bool for flag in
               (verifiable, high_impact, exact_complete, needs_backtracking)):
            raise ValueError("Escalation flags must be booleans")
        if high_impact and not verifiable:
            return {"strategy": "tool_or_human_review", "reason": "high-impact claim lacks verifier"}
        if not verifiable:
            return {"strategy": "clarify_task", "reason": "no validated acceptance criterion"}
        if exact_complete:
            return {"strategy": "stop", "reason": "required evidence and checks complete"}
        if needs_backtracking:
            return {"strategy": "tree_search", "reason": "partial-state alternatives matter"}
        return {"strategy": "best_of_3", "reason": "compare bounded complete candidates"}

requests = [
    {"task": "Rollback-threshold FAQ", "evaluation_score": 0.94, "verifiable": True, "exact_complete": True},
    {"task": "Ambiguous canary hold", "evaluation_score": 0.72, "verifiable": True},
    {"task": "Parser compatibility graph", "evaluation_score": 0.41, "verifiable": True, "needs_backtracking": True},
    {"task": "Incident-postmortem titles", "evaluation_score": 0.52, "verifiable": False},
    {"task": "Approve production write without policy record", "evaluation_score": 0.82, "verifiable": False, "high_impact": True},
]

scheduler = EscalationPolicy()
try:
    scheduler.choose_strategy(0.99, verifiable="false", exact_complete="false")
except ValueError:
    pass
else:
    raise AssertionError("String flags must not select the stop strategy")
for request in requests:
    request.update(scheduler.choose_strategy(**{key: value for key, value in request.items() if key != "task"}))

print(json.dumps(requests, indent=2))

[
  {
    "task": "Rollback-threshold FAQ",
    "evaluation_score": 0.94,
    "verifiable": true,
    "exact_complete": true,
    "strategy": "stop",
    "reason": "required evidence and checks complete"
  },
  {
    "task": "Ambiguous canary hold",
    "evaluation_score": 0.72,
    "verifiable": true,
    "strategy": "best_of_3",
    "reason": "compare bounded complete candidates"
  },
  {
    "task": "Parser compatibility graph",
    "evaluation_score": 0.41,
    "verifiable": true,
    "needs_backtracking": true,
    "strategy": "tree_search",
    "reason": "partial-state alternatives matter"
  },
  {
    "task": "Incident-postmortem titles",
    "evaluation_score": 0.52,
    "verifiable": false,
    "strategy": "clarify_task",
    "reason": "no validated acceptance criterion"
  },
  {
    "task": "Approve production write without policy record",
    "evaluation_score": 0.82,
    "verifiable": false,
    "high_impact": true,
    "strategy": "tool_or_human_review",
    "reason": "high-impact claim lacks verifier"
  }
]

This is a policy sketch, not a compute-optimal scheduler. Its branches depend on task structure and checked completion, not a magical 0.90 confidence cutoff. A weak score alone doesn't imply that tree search helps. An offline evaluation must establish whether any escalation is worth its cost; a low-competence model may need replacement or abstention instead. Creative tasks can use a separate user-approved rubric, but "unverifiable" isn't synonymous with "creative."

Escalation also needs an exit. A branch may be looping, capped, or producing no new evaluator signal. Check those conditions before expanding another child, then return the best trace that still satisfies hard evidence requirements:

import json
import math

def stop_reason(
    tokens_used: int,
    token_cap: int,
    elapsed_ms: int,
    wall_clock_cap_ms: int,
    repeated_state: bool,
    score_history: list[float],
) -> str:
    if (any(type(value) is not int or value < 0 for value in
            (tokens_used, token_cap, elapsed_ms, wall_clock_cap_ms))
            or any(not math.isfinite(score) or not 0 <= score <= 1 for score in score_history)):
        raise ValueError("Invalid usage, limit, or score")
    if repeated_state:
        return "repeated_state"
    if tokens_used >= token_cap:
        return "token_cap"
    if elapsed_ms >= wall_clock_cap_ms:
        return "wall_clock_cap"
    if len(score_history) >= 4 and max(score_history[-3:]) <= max(score_history[:-3]):
        return "no_recent_score_improvement"
    return "continue"

branches = [
    {"tokens_used": 1200, "token_cap": 4000, "elapsed_ms": 800, "wall_clock_cap_ms": 3000,
     "repeated_state": True, "score_history": [0.54, 0.55]},
    {"tokens_used": 2500, "token_cap": 4000, "elapsed_ms": 1800, "wall_clock_cap_ms": 3000,
     "repeated_state": False, "score_history": [0.72, 0.70, 0.71, 0.69]},
    {"tokens_used": 900, "token_cap": 4000, "elapsed_ms": 500, "wall_clock_cap_ms": 3000,
     "repeated_state": False, "score_history": [0.40, 0.55]},
]

print(json.dumps([stop_reason(**branch) for branch in branches], indent=2))

[
  "repeated_state",
  "no_recent_score_improvement",
  "continue"
]

Those are post-step stop signals, not enforcement of the next call's budget. A three-step plateau is only a heuristic and can discard a useful continuation. Before dispatch, reserve the next call's maximum allowance against the shared request budget. At a wall-clock deadline, stop dispatching, cancel in-flight work where supported, and retain its reservation until usage is known.

Reserving the request budget

Before counting cache bytes, make the compute ceiling real. With 4,000 generated tokens available, two 1,500-token calls can start together. A third 1,500-token reservation must wait. If the first call finishes using 900, its unused 600 becomes available. Generator, retry, and learned-verifier calls all use the same ledger.

This single-owner ledger models atomic admission. Its invariant is spent + reserved <= cap. A concurrent service would serialize these operations or use a transactional store, and reconcile usage by unique call ID after retries or crashes.

import json

class TokenBudget:
    def __init__(self, cap: int):
        if type(cap) is not int or cap < 0:
            raise ValueError("Invalid cap")
        self.cap, self.spent = cap, 0
        self.pending: dict[str, int] = {}
        self.finished: dict[str, int] = {}

    @property
    def available(self) -> int:
        return self.cap - self.spent - sum(self.pending.values())

    def reserve(self, call_id: str, maximum: int) -> bool:
        if not call_id or type(maximum) is not int or maximum <= 0:
            raise ValueError("Invalid reservation")
        if call_id in self.pending or call_id in self.finished:
            raise ValueError("Call ID already used")
        if maximum > self.available:
            return False
        self.pending[call_id] = maximum
        return True

    def settle(self, call_id: str, used: int) -> None:
        if type(used) is not int or used < 0:
            raise ValueError("Invalid usage")
        if call_id in self.finished:
            if self.finished[call_id] != used:
                raise ValueError("Conflicting duplicate receipt")
            return
        maximum = self.pending[call_id]
        if used > maximum:
            # Keep the reservation and surface a backend contract violation.
            raise ValueError("Backend exceeded reserved maximum")
        self.pending.pop(call_id)
        self.finished[call_id] = used
        self.spent += used

budget = TokenBudget(4_000)
admitted = [budget.reserve("candidate-a", 1_500),
            budget.reserve("candidate-b", 1_500),
            budget.reserve("candidate-c", 1_500)]
budget.settle("candidate-a", 900)
budget.settle("candidate-a", 900)  # duplicate receipt is idempotent
admitted.append(budget.reserve("candidate-c", 1_500))
assert budget.spent + sum(budget.pending.values()) <= budget.cap
print(json.dumps({"admitted": admitted, "spent": budget.spent,
                  "reserved": sum(budget.pending.values()),
                  "available": budget.available}, indent=2))

{
  "admitted": [
    true,
    true,
    false,
    true
  ],
  "spent": 900,
  "reserved": 3000,
  "available": 100
}

The remaining 100 tokens aren't enough for another 1,500-token call. A verifier needing more than that must wait for released capacity, use a previously reserved allowance, or cause abstention. Don't release a timed-out call's reservation merely because the client disconnected: the backend may still be running. A provider hard output limit must include hidden reasoning tokens where applicable. A cap violation means the budget guarantee failed and requires reconciliation; raising an exception can't undo spent compute.

An output cap can also leave an unfinished answer. OpenAI documents status=incomplete with reason max_output_tokens, sometimes before any visible text; nonempty partial text is still possible.^{[5]Reference 5Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning} Would the word “Hold” alone prove the candidate finished its checks? This advisory adapter requires both terminal completion and eligibility from the separate evidence gate:

import json

def advisory(response: dict[str, object], evidence_eligible: bool) -> dict[str, object]:
    if response.get("status") != "completed":
        return {"release": False, "reason": "response_not_completed"}
    if evidence_eligible is not True:
        return {"release": False, "reason": "evidence_gate_failed"}
    text = response.get("output_text")
    if type(text) is not str or not text.strip():
        return {"release": False, "reason": "empty_advisory"}
    return {"release": True, "advisory": text}

partial = {"status": "incomplete", "output_text": "Hold",
           "incomplete_details": {"reason": "max_output_tokens"}}
complete = {"status": "completed", "output_text": "Hold the canary at 10%."}
results = [advisory(partial, True), advisory(complete, True), advisory(complete, False)]
assert not results[0]["release"] and results[1]["release"]
assert not results[2]["release"]
print(json.dumps(results, indent=2))

[
  {
    "release": false,
    "reason": "response_not_completed"
  },
  {
    "release": true,
    "advisory": "Hold the canary at 10%."
  },
  {
    "release": false,
    "reason": "evidence_gate_failed"
  }
]

These are synthetic projections of response status and the SDK's output_text convenience value, not raw API messages or a full output-item parser. Eligibility comes from authenticated evidence and policy checks, never from a model-supplied flag. Handle refusals, tools, and other output types in their own adapters. Preserve usage receipts even when text is incomplete, and reserve any retry against the same request ledger.

KV cache limits live branch width

Long reasoning traces can stress memory as well as floating-point operations (FLOPs). In a full-attention transformer, each live token contributes key and value vectors at every transformer layer. The key-value (KV) cache therefore grows with live context length and multiplies across unshared branches.

KV cache pressure

Before choosing a branch width, estimate what one live token costs. A back-of-the-envelope formula is:

Here $L$ is the number of layers, $H_{kv}$ is the number of KV heads, and $d_{\text{head}}$ is the head dimension. $H_{kv}$ isn't always the full attention-head count when the model uses grouped-query attention (GQA). For an 80-layer model with 8 KV heads, head dimension 128, and bfloat16 (BF16) activations:

($327{,}680 / 1024 = 320$ KiB; these are binary kibibytes, not decimal KB.) Here and in the figures, k means 1,024 tokens. At 16,384 live tokens, one branch needs 5.0 GiB of KV cache. Without prefix reuse, eight active branches need 40 GiB before model weights, temporary activations, and allocation overhead. This formula assumes full attention with uniform KV dimensions, not sliding-window or latent-attention caches.

Prefix sharing and paged KV storage

Best-of-N and tree search create many branches that share the same prompt and often the same early reasoning prefix. If you duplicate that prefix for every branch, memory usage grows with the number and length of branches. Systems like vLLM use PagedAttention to manage KV memory in blocks, which cuts fragmentation and makes long contexts much easier to serve ^{[13]Reference 13Efficient Memory Management for Large Language Model Serving with PagedAttentionhttps://arxiv.org/abs/2309.06180}.

PagedAttention and radix-style prefix caches solve related but different problems. PagedAttention permits non-contiguous block allocation and supports sharing with copy-on-write. RadixAttention-style caches, as used in SGLang, index shared prefixes in a tree ^{[14]Reference 14SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104}. Reuse requires compatible tokens, positions, model/adapters, and cache configuration, plus the intended tenant isolation. Identical-looking prompt text alone isn't sufficient.

Now redo the arithmetic with sharing. Eight branches sharing 12,288 tokens and adding 4,096 tokens each occupy $12{,}288 + 8(4{,}096) = 45{,}056$ distinct token slots. At 320 KiB per slot, that's 13.75 GiB, approximately 14 GiB. The saving depends on branch divergence and the runtime's sharing granularity; page rounding and metadata add overhead.

This is one reason test-time search is an infrastructure problem, not a prompting trick alone. A search policy may be sound on paper while its branch width, trace length, or lack of compatible prefix reuse exhausts the serving memory budget before useful search completes.

Compression needs value awareness; recurrent depth changes architecture

Prefix sharing removes duplicate stems, but each surviving suffix still grows. Token eviction can cap memory by dropping old KV entries, yet a reasoning trace may depend on a small earlier detail. Value-Aware Stochastic KV Cache Eviction (VaSE) reports that evicting rare, large-magnitude value states can trigger repetitive failure loops; its mitigation protects those states and adds stochasticity to eviction.^{[15]Reference 15Value-Aware Stochastic KV Cache Eviction for Reasoning Modelshttps://arxiv.org/abs/2606.03928} Treat that as one 2026 method evaluated on its reported models and tasks, not a universal importance rule.

If the runtime supports protecting selected spans, retain application-critical constraints and tool observations before applying a lossy policy. Keeping raw text in an external record doesn't preserve its KV representation. Reclaim losing suffix pages only after in-flight readers finish and reference counts reach zero. Replay long tasks at the target compression ratio and measure loops, exact-check pass rate, and final quality against the uncompressed baseline.

Recurrent-depth models attack a different source of growth. Instead of expressing every extra unit of compute as more visible chain-of-thought tokens, a recurrent block can unroll additional latent depth while sharing model parameters and KV state. Geiping et al. present a 3.5B-parameter proof-of-concept with per-token adaptive compute, KV-cache sharing, and speculative decoding.^{[16]Reference 16Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachhttps://proceedings.neurips.cc/paper_files/paper/2025/hash/3b01972cf31e6fa0fe29e4b8b5c2a0a1-Abstract-Conference.html} It's a model-architecture choice, not a cache flag you can add to a conventional autoregressive checkpoint. A capacity plan should separate mitigations available now from changes that require training and serving a different architecture.

First update vs final-answer latency

Extended reasoning changes the latency contract. At the model-serving layer, a long hidden search phase can inflate time to first token (TTFT). At the product layer, a quick status event can make first-update latency look healthy while final-answer latency and total budget still fail. Measure all three separately.

That's why production systems can stream progress events instead of raw scratchpads. Safe events include selected strategy, budget usage, branch counts, or summaries of externally confirmed checks. Treat learned verifier scores as internal diagnostics unless they're calibrated and presented with an appropriate contract.

A sample capacity plan

This table uses the 320 KiB/token estimate to show how quickly memory usage grows as you add longer traces or more live branches:

These are logical KV bytes, not GPU deployment recommendations. Available capacity also depends on weights, precision, parallelism, workspaces, allocator overhead, and other requests. Prefix reuse changes the count of distinct live slots, not the bytes per slot.

Streaming safe updates

Users need feedback while the system is spending extra inference budget, but that doesn't mean you should dump raw internal scratchpad text or expose an uncalibrated reward score as confidence. A safer pattern is to stream strategy selection, budget usage, and externally grounded summaries while the solver keeps search details private:

from dataclasses import dataclass
import json

@dataclass
class StreamEvent:
    type: str
    content: str
    internal_score: float | None = None

class ReasoningSolver:
    def solve_stream(self) -> list[StreamEvent]:
        return [
            StreamEvent("progress", "checked canary status", 0.82),
            StreamEvent("progress", "verified rollback threshold", 0.91),
            StreamEvent("final", "Hold canary at 10% and inspect the config flag.", 0.88),
        ]

def stream_reasoning(config: dict[str, object], solver: ReasoningSolver) -> list[dict[str, object]]:
    events = [
        {"type": "status", "content": "Analyzing problem difficulty..."},
        {
            "type": "config",
            "content": f"Using {config['strategy']} with budget {config['max_tokens']} tokens",
        },
    ]

    for event in solver.solve_stream():
        if event.type == "progress":
            events.append({
                "type": "progress",
                "content": event.content,
            })
        elif event.type == "final":
            events.append({
                "type": "final_answer",
                "content": event.content,
            })
    return events

config = {"strategy": "best_of_3", "max_tokens": 4_000}
print(json.dumps(stream_reasoning(config, ReasoningSolver()), indent=2))

[
  {
    "type": "status",
    "content": "Analyzing problem difficulty..."
  },
  {
    "type": "config",
    "content": "Using best_of_3 with budget 4000 tokens"
  },
  {
    "type": "progress",
    "content": "checked canary status"
  },
  {
    "type": "progress",
    "content": "verified rollback threshold"
  },
  {
    "type": "final_answer",
    "content": "Hold canary at 10% and inspect the config flag."
  }
]

This is an event-projection fixture, not a network stream or a tool verifier. It strips internal_score, but the producer is still responsible for deriving progress from completed observations and sending a final event only after the release gate passes. A model-generated "verified" string must not enter that trusted channel. A real stream also needs disconnect/cancellation handling and access controls on retained traces.

When reasoning is the wrong tool

Ask first whether more inference can create evidence or only more text. Not every canary question needs a search tree. Tree search or Best-of-N on the rollback FAQ just burns tokens.

Simple source-of-truth queries such as "What is the rollback threshold for checkout-api?" belong against the runbook or policy source. Extra internal reasoning won't recover a missing or current fact.

Retrieval failures need retrieval, search, or tool use. Longer internal reasoning can't invent the external observation the task requires.

Open-ended copy such as "Write three titles for the incident postmortem" has no single objective answer unless the application defines and validates a rubric. Deep PRM-driven search is usually unjustified.

Latency-sensitive conversational turns need measured response objectives. If a deep path exceeds the objective, acknowledge, defer, or avoid the search.

Cost-sensitive batch processing changes the arithmetic again. Across millions of documents, a large inference multiplier can erase a small quality gain. Escalate only request cohorts whose measured lift pays for the extra cost.

Allocate substantial test-time compute when evaluation shows gain for the request class and evidence or verification can keep the answer trustworthy. Creative revision may still use a small candidate budget, but it isn't proof-oriented reasoning.

Overthinking: when more thinking lowers accuracy

Longer reasoning isn't monotonically better. Ghosal et al. study forced “Wait” extensions on DeepSeek-R1 distilled models and math benchmarks: accuracy can rise then fall, and their analysis associates extension with greater output uncertainty.^{[17]Reference 17Does Thinking More Always Help? Mirage of Test-Time Scaling in Reasoning Modelshttps://arxiv.org/abs/2506.04210} Parallel traces with majority vote outperform that sequential policy at matched thinking-token budgets. That comparison isn't matched total serving cost, and doesn't establish how every model's native effort policy behaves.

Keep trivial work on the fast route, choose a useful effort setting, cap generated tokens and elapsed time, and stop when exact results are sufficient or validated evaluation stops improving. Measure all candidate outputs and verifier work when comparing total cost.

Why a bigger model isn't enough

It's tempting to deploy a bigger frontier model for every reasoning workload. Quality may move, but its cost pattern differs from request-time search:

If the base model has near-zero task competence, reranking or sampling is unlikely to manufacture the missing capability. Test-time compute amplifies evaluated candidate quality; it doesn't supply missing knowledge or tools. Snell's FLOPs-matched comparison puts numbers on that boundary: extra inference can beat a much larger model on prompts the small model can already sometimes solve, but the advantage stops on the hardest ones ^{[3]Reference 3Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314}.

Common pitfalls

"Higher temperature will reveal a great answer if I sample enough"

If Best-of-N produces varied traces but quality barely improves, diversity isn't the missing ingredient. Without a verifier that tracks correctness, more samples mostly add noise. Increase temperature only when a PRM, ORM, or exact checker can separate strong traces from fluent wrong ones.

"Tree search is always better than single-path reasoning"

If easy requests become slower and more expensive while accuracy barely moves, search budget reached requests that didn't need it. Route by difficulty, keep trivial and easy work on the fast path, and escalate only when the first trace scores weakly.

"More thinking always improves the answer"

If a decent first answer degrades while the system keeps exploring, stop when required exact checks pass or a validated evaluator plateaus. Tune effort, enforce token/time caps, and keep the best eligible fallback.

"One budget policy can serve every request"

If cheap FAQ traffic burns deep-search budget while hard debugging tasks run out of depth, compute was allocated uniformly. Separate routing, branch limits, token caps, and wall-clock caps by request class. Budget policy belongs to product design, not model tuning alone.

"Users need raw chain-of-thought to trust the system"

If streams become long and unstable while downstream tools can't tell what is final, internal scratchpad text has replaced a progress interface. Stream strategy, budget usage, and short evidence-backed summaries. Keep hidden search traces and uncalibrated evaluator scores inside the runtime.

"Reasoning quality comes only from prompts"

If prompt tweaks plateau on hard tasks, model capability, verifier quality, cache reuse, and stopping logic may have been collapsed into one prompt-design problem. Treat the base model, search controller, verifier, and serving runtime as separate levers, then improve the bottleneck.

"Stop criteria are an optional cleanup detail"

If branches revisit the same semantic state until they exhaust tokens or wall-clock time, the controller lacks a hard policy for repetition, budget exhaustion, or diminishing verifier returns. Enforce token, depth, wall-clock, and repeated-state limits per branch, then return the highest-ranked surviving trace that satisfies required checks.

Defending the design

Evaluation rubric

• Foundational: Derive the 240 ms hold decision from policy v3 and fresh revision-7 observations. Distinguish advisory output from authority to change a deployment.
• Intermediate: Keep hard eligibility separate from proxy ranking, preserve completed search results, and enforce spent + reserved <= cap across candidates and verifier calls.
• Advanced: Derive 40 GiB versus 13.75 GiB of logical KV, explain compatibility and reference-count requirements for sharing, and design a cost-matched evaluation that could reject the proposed escalation policy.

Follow-up questions