CoT, ToT & Self-Consistency Prompting

Picture a release candidate with passing tests, a clear security scan, and an approved review. Policy allows an automatic merge only when all three facts hold. A direct answer can still miss one of them.

The last chapter used a compressed embedding index to put the right policy clause in context, including an auto-merge rule. Retrieval isn't the end: an assistant can still skip a condition, mishandle a check, or commit too early to one plan. The question isn't whether a model can produce a longer explanation; it's how much extra test-time compute buys a better decision under token, latency, and safety budgets. Start with one prompt-elicited chain of thought (CoT), compare several candidates with self-consistency, and move to an external Tree of Thoughts (ToT) search only when early choices can dead-end.

Start with a failure you can audit

Put those facts on one row: unit tests pass, the security scan is clear, and a reviewer has approved the diff. The policy permits auto-merge only when every field is true. A model can still skip a field or suggest a force-merge.

A useful outward artifact is a short decision record containing those facts, the checks applied, and the final action. It's smaller than an open-ended rationale, easy to score, and safe to compare against deterministic policy logic. It isn't raw chain of thought. It's a task-specific output contract that the application can validate.

This code isn't an LLM. It's the oracle that your prompt variants must match. Define the output contract and scorer before increasing model compute.

When to skip CoT entirely: structured machine state (booleans, enum fields, and runtime-owned policy predicates) belongs in code. Sampling five CoT traces over three known flags adds cost without evidence. Use model reasoning for unstructured evidence (diff text, free-form review comments, or incident narratives) when no deterministic scorer exists yet. These fixtures teach scoring and controllers; they aren't a recommendation to put all(checks) behind a model.

If a required fact isn't in the prompt at all, a longer rationale can't invent it. The missing value has to come from the source system. That boundary leads to the typed tool request in the next lesson.

The Policy route handles known flags, while the Tool route obtains a live observation. This chapter takes the middle route: extra inference over facts already present, scored against a contract.

Why define an oracle and decision record before testing reasoning prompts?

Answer

Without a scored outcome and required checks, a longer response can look thoughtful without being more correct. The oracle lets you compare strategies on the same task.

Chain-of-Thought: one decomposed candidate

Wei et al. introduced few-shot CoT by placing worked intermediate steps in prompt examples. On GSM8K (a grade-school math benchmark), PaLM 540B rose from 17.9% with standard prompting to 56.9% with eight CoT exemplars, with further gains on commonsense and symbolic tasks once models were large enough for the scratchpad to matter.^{[1]Reference 1Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.https://arxiv.org/abs/2201.11903} Kojima et al. later showed that the zero-shot trigger "Let's think step by step" could improve several benchmark reasoning tasks without worked examples.^{[2]Reference 2Large Language Models are Zero-Shot Reasoners.https://arxiv.org/abs/2205.11916}

Those papers establish techniques to evaluate, not a production law. On your endpoint and task, a visible scratchpad may help, do nothing, or add cost. Meincke et al. tested GPQA Diamond and found that an explicit "think step by step" request often added little accuracy for reasoning-tuned models while increasing time and tokens; for ordinary chat models the average gain was small and sometimes came with more errors on questions the model already answered reliably.^{[3]Reference 3Prompting Science Report 2: The Decreasing Value of Chain of Thought in Promptinghttps://arxiv.org/abs/2506.07142}

🔬 Research insight: More reasoning text isn't always better. Zheng et al. tested CoT and its variants (including ToT and ReAct, or Reasoning and Acting) on nine pattern-based in-context-learning benchmarks across 16 models and found they consistently underperformed plain direct answering, with the gap widening as more demonstrations were added. Even long-CoT reasoning models that spent far more tokens didn't overcome the effect.^{[4]Reference 4The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning.https://arxiv.org/abs/2504.05081} When a task is pattern matching from examples, a visible scratchpad can add noise instead of signal. Measure that tradeoff rather than assuming a longer response will help.

When one prompt decomposition isn't enough, least-to-most prompting handles a different shape of problem. It breaks the task into ordered sub-questions, answers the simplest one first, and passes each intermediate answer into the next sub-prompt.^{[5]Reference 5Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelshttps://arxiv.org/abs/2205.10625} That sequential handoff can improve compositional generalization without the latency and memory overhead of branching search.

Keep four reasoning artifacts separate

The word reasoning now covers several different artifacts. Mixing them creates false audit claims and brittle integrations, so name the artifact before deciding what it proves.

Provider disclosure rules and API surfaces differ. OpenAI's API documentation says raw reasoning tokens aren't exposed, although supported models can return an opt-in reasoning summary. Those hidden tokens still count toward context and billing.^{[6]Reference 6Reasoning modelshttps://developers.openai.com/api/docs/guides/reasoning}

Anthropic still returns thinking blocks, including summarized thinking. Its control surface is model-specific: older Claude 4.x models use a fixed budget_tokens extended-thinking mode, while newer models reject that setting and expect adaptive thinking with an effort control.^{[7]Reference 7Building with extended thinkinghttps://docs.claude.com/en/docs/build-with-claude/extended-thinking}

Recheck provider documentation when you integrate. Don't write controller logic that assumes every model returns a visible scratchpad or that one thinking API works across Claude generations.

This distinction changes the build-versus-buy decision. An external CoT, voting, or tree-search controller gives you explicit sample and search state, but generated rationales still aren't guaranteed to be faithful.

A native reasoning model owns more of the internal inference process, while its controls, visibility, and accounting follow the provider contract. Put both behind the same task-level evaluator and compare outcomes, latency, token use, and parse failures.

Ask for reviewable task artifacts at the workflow boundary rather than text that claims to reveal private internal reasoning:

The required checks make a missed predicate observable. They still aren't proof of faithful hidden cognition. Turpin et al. showed that CoT explanations can rationalize outputs influenced by hidden biasing features without mentioning those features.^{[8]Reference 8Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptinghttps://arxiv.org/abs/2305.04388} Log inputs, actions, validations, and outcomes; don't treat eloquent reasoning text or a provider summary as an audit guarantee.

Here, the trace is useful because it meets the scored contract. It doesn't win merely because it contains more words.

Zero-shot or few-shot?

Zero-shot CoT supplies an instruction and lets the model choose its intermediate format. Few-shot CoT gives one or more solved examples, so it can teach both decomposition and the final answer shape. Native structured output enforcement is better when available. Examples still help when the prompt must communicate task-specific checks.

When is few-shot CoT worth more prompt tokens than zero-shot CoT?

Answer

When examples teach task-specific checks or output fields that your evaluator needs, and held-out results justify the added tokens.

Stable formatting solves parsing, not sampling variance. A well-shaped candidate can still choose the wrong action.

Self-Consistency: sample answers, then vote

One structured trace can fail because generation takes an unlucky path. Self-Consistency replaces reliance on one path with several sampled paths and chooses the most consistent final answer.^{[9]Reference 9Self-Consistency Improves Chain of Thought Reasoning in Language Models.https://arxiv.org/abs/2203.11171} The vote operates on extracted answers, not on whose rationale sounds best.

Let $y_i$ be the final text from sample $i$, and let $c(y_i)$ map that text to an allowed action or reject it as invalid. With $N$ total samples, the winning action and its share are:

$\hat{a} = \arg\max_a \sum_{i=1}^{N} \mathbf{1}[c(y_i)=a], \qquad s = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[c(y_i)=\hat{a}]$

The denominator remains $N$, even when some outputs fail to parse. Emit $\hat{a}$ only when the winner is unique and $s$ meets a configured threshold $\tau$; otherwise abstain.

To obtain different candidates, use stochastic decoding, usually with a nonzero temperature, rather than rerunning one deterministic decode. The earlier Decoding Algorithms lesson explains that control. Keep the setting fixed in your experiment and log it with the sample count.

In the paper's PaLM-540B setting, 40 independently sampled outputs per run raised GSM8K accuracy from 56.5% with greedy CoT to 74.4% with self-consistency, a 17.9 percentage-point gain.^{[9]Reference 9Self-Consistency Improves Chain of Thought Reasoning in Language Models.https://arxiv.org/abs/2203.11171} That result belongs to its model, prompts, sampling setup, and benchmark. Five samples here aren't expected to reproduce it.

Wang et al. also observed that the share of samples agreeing with the aggregated answer correlated with GSM8K accuracy. That's a research observation, not a calibrated confidence score. Treat agreement as a vote statistic you threshold, then still run ordinary policy checks.

Normalize before counting

Model outputs rarely use exactly one spelling. Map harmless variants to one allowed action and reject unknown outputs before voting.

A winner isn't always confident enough

A 2 to 2 tie, zero accepted outputs, or a narrow plurality with many rejected outputs shouldn't silently become an automated action. Add an abstention rule.

The winning share must use all sampled outputs as its denominator, including strings that failed to parse. A passing vote chooses a candidate; ordinary runtime policy still decides whether that action is allowed.

Measure gains against call cost

Use a held-out fixture set before calling the strategy ready. Each fixture represents five already-canonicalized final actions for one case. Unknown values are rejected, but they remain in the denominator. The controller compares first-sample accuracy with five-sample voting accuracy.

This fixture is intentionally small and deterministic, so it tests controller logic rather than model quality. A real release decision needs representative labeled cases, real model samples, token counts, latency, refusal rates, and cost.

What should a self-consistency controller do when votes don't form a strong majority?

Answer

Abstain or route to review. More samples are useful only when a measured gain justifies their cost and the action remains safe.

Tree-of-Thoughts: search when branches can dead-end

Voting helps when independent paths tend to converge on the same short answer. It doesn't deliberately revisit an earlier choice. Tree-of-Thoughts (ToT) represents partial solutions as search states, then generates alternatives, evaluates those states, and preserves only branches worth extending.^{[10]Reference 10Tree of Thoughts: Deliberate Problem Solving with Large Language Models.https://arxiv.org/abs/2305.10601}

Yao et al. evaluated ToT on tasks built for planning and search. A beam width is the number of partial states retained after each scoring step. On Game of 24 with GPT-4, ToT with beam width 5 solved 74% of tasks, while CoT solved 4.0% and CoT self-consistency with 100 samples solved 9.0%.^{[10]Reference 10Tree of Thoughts: Deliberate Problem Solving with Large Language Models.https://arxiv.org/abs/2305.10601} Independent voting didn't rescue a search problem.

The narrower conclusion isn't "use trees everywhere." When a problem has verifiable partial states and meaningful backtracking, search can recover from a bad early move.

Beyond tree branching, Graph-of-Thoughts (GoT) stores reasoning states on a directed acyclic graph (DAG) instead of a tree.^{[11]Reference 11Graph of Thoughts: Solving Elaborate Problems with Large Language Modelshttps://arxiv.org/abs/2308.09687} ToT grows each branch from a single parent. GoT can merge independent paths into one state (thought aggregation), refine a thought after feedback, and rewrite the graph as search proceeds.

That helps when separate sub-tasks, such as reading two logs or drafting two candidate clauses, have to join before a final decision.

Search states in a release plan

For a release gate, consider a workflow whose final recommendation must be supported by two observations: a current security scan and a reviewer approval record. A controller can expand legal steps instead of letting a model invent a final action before evidence is present.

In the original ToT framework, a language model can propose and evaluate thoughts while an external search algorithm owns state, branching, and backtracking. The deterministic code below isolates that controller skeleton; it doesn't simulate an LLM evaluator.

The two evidence-gathering orders converge on the same four complete evidence states. Only a clear scan plus an approved review can merge; any failed scan or rejected review blocks the release. The next lesson will populate those outcomes from real APIs through tool calls.

A fully runnable search example

Game of 24 gives us an exact evaluator: arithmetic either reaches 24 using each input once or it doesn't. The solver below explores partial equations with breadth-first search and returns a verified solution.

Pruning is a source of failure

An LLM evaluator isn't an arithmetic oracle. If it scores an apparently simple but dead branch above a non-obvious solvable branch, an aggressive beam can remove the answer before expansion.

exact_score is a hand-labeled fixture oracle, not a capability you get for free. Real search tasks often need executable validators, environment feedback, or a separately evaluated scorer.

The production implications are concrete. Keep ToT for tasks with real branch structure, not ordinary classification. Prefer deterministic validators when a partial state can be checked in code. Measure solver recall at each beam width alongside final successes, then cap expansions and latency before an open-ended search reaches users.

Why can Tree-of-Thoughts perform worse when the evaluator is weak and the beam is narrow?

Answer

Search spends compute only on retained states. If an incorrect evaluator prunes the valid branch early, later expansions can't recover it.

Search can recover from bad early choices, but each expansion multiplies calls and latency. Choose a strategy through one quality-and-cost experiment rather than assuming a bigger tree is better.

Choose compute with an eval gate

Direct prompting, one trace, voting, and tree search aren't maturity levels. They are candidates with different accuracy and serving cost. Start with the cheapest candidate, then promote a more expensive strategy through an eval gate only when held-out results require it. Track p95 latency, the time under which 95% of requests finish, alongside quality. Snell et al. found that the best test-time allocation can depend on problem difficulty, so a fixed extra-sample budget isn't automatically optimal.^{[12]Reference 12Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.https://arxiv.org/abs/2408.03314}

Treat these as example evaluation results, not a benchmark claim. For a real system, keep a table with:

When not to use CoT (or heavier search)

Use this checklist before you promote scaffolding or multi-sample voting:

Adding a visible CoT scaffold to a reasoning-tuned model can add tokens and latency with little accuracy gain. On ordinary chat models it can also inject errors on questions the model already answered reliably.^{[3]Reference 3Prompting Science Report 2: The Decreasing Value of Chain of Thought in Promptinghttps://arxiv.org/abs/2506.07142} Treat that prompt as another experimental arm, not as a required wrapper around every reasoning endpoint.

Where reasoning ends and tools begin

All runnable experiments above operate on provided facts or deterministic state. A real release decision requires current CI status, security scan output, and review state from source systems. Reasoning alone can't obtain those observations.

ReAct interleaves reasoning traces and task-specific actions so new observations can update the next decision.^{[13]Reference 13ReAct: Synergizing Reasoning and Acting in Language Models.https://arxiv.org/abs/2210.03629} This loop doesn't require raw private reasoning. It needs a validated action request, a controlled execution result, and a bounded next decision:

The next lesson implements that action boundary with typed function calls, schemas, errors, and safe execution.

What to remember

• Define the scorer first. A decision record lets you test whether extra inference work improves outcomes.
• Keep artifacts distinct. Prompt-elicited CoT, hidden reasoning tokens, provider summaries, and application decision records aren't interchangeable.
• One trace is one candidate. CoT can expose missed steps, but a plausible rationale isn't a faithful audit log.^{[8]Reference 8Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptinghttps://arxiv.org/abs/2305.04388}
• Vote on normalized outcomes. Self-consistency is useful only when its measured gain beats its sample cost.^{[9]Reference 9Self-Consistency Improves Chain of Thought Reasoning in Language Models.https://arxiv.org/abs/2203.11171}
• Search only with branch structure. ToT and GoT need meaningful states, evaluators, pruning limits, and failure measurements.^{[10]Reference 10Tree of Thoughts: Deliberate Problem Solving with Large Language Models.https://arxiv.org/abs/2305.10601 [11]Reference 11Graph of Thoughts: Solving Elaborate Problems with Large Language Modelshttps://arxiv.org/abs/2308.09687}
• Promote strategies through evals. Direct, trace, vote, and search should compete under quality and latency gates.
• Skip CoT when policy is already machine state. Deterministic predicates, pattern-ICL, and tasks already above the accuracy SLO need no scratchpad.