CoT, ToT & Self-Consistency Prompting

Compressed embeddings can retrieve the right policy evidence and still leave the assistant with a decision to make. Retrieval isn't the end of the task. An assistant may have the right facts and still choose the wrong action because it skips a condition, mishandles arithmetic, or commits too early to one plan.

Treat extra inference work as an engineering decision. You'll build a small release-resolution controller for a risky code change:

• Chain-of-Thought (CoT): request one decomposed candidate decision.
• Self-Consistency: sample several candidates, normalize final actions, and vote.
• Tree-of-Thoughts (ToT): expand and prune branches when a decision needs backtracking.

Long rationales aren't the target. The target is better measurable decision accuracy under a token, latency, and safety budget.

Start with a failure you can audit

Suppose a repository has a candidate release. Unit tests pass, the security scan is clear, and a reviewer has approved the diff. Policy permits auto-merge only when all three facts are true. A direct response may still overlook one condition and suggest a force-merge or escalation unnecessarily.

A useful outward artifact is a short decision record: facts used, checks applied, and final action. It's smaller than an open-ended rationale, easy to score, and safe to compare against deterministic policy logic.

This code isn't an LLM. It's the oracle that your prompt variants must match. Before increasing model compute, define an output contract and a scorer.

When to skip CoT entirely: if the inputs are already structured machine state (booleans, enum fields, policy predicates the runtime owns), evaluate the policy in code. Sampling five CoT traces over three known flags adds cost without evidence. Use LLM reasoning only over unstructured evidence (diff text, free-form review comments, incident narratives) where there is no deterministic scorer yet. The fixtures below teach scoring and controllers; they aren't a product recommendation to put all(checks) behind a model.

Why define an oracle and decision record before testing reasoning prompts?

Answer

Without a scored outcome and required checks, a longer response can look thoughtful without being more correct. The oracle lets you compare strategies on the same task.

Chain-of-Thought: one decomposed candidate

Wei et al. introduced few-shot CoT by placing worked intermediate steps in prompt examples. Their experiments found gains on arithmetic, commonsense, and symbolic reasoning benchmarks for sufficiently large models, including a strong GSM8K result with PaLM 540B.^{[1]Reference 1Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.https://arxiv.org/abs/2201.11903} Kojima et al. later showed that the zero-shot trigger "Let's think step by step" could improve several benchmark reasoning tasks without worked examples.^{[2]Reference 2Large Language Models are Zero-Shot Reasoners.https://arxiv.org/abs/2205.11916}

Those papers establish techniques to evaluate, not a production law. On your endpoint and task, a visible scratchpad may help, do nothing, or add cost. If an API offers a native reasoning control, evaluate that option as another strategy rather than assuming that extra visible text helps.

🔬 Research insight: More reasoning text isn't always better. Zheng et al. tested CoT and its variants (including ToT and ReAct) on nine pattern-based in-context-learning benchmarks across 16 models and found they consistently underperformed plain direct answering, with the gap widening as more demonstrations were added. Even long-CoT reasoning models that spent far more tokens didn't overcome the effect.^{[3]Reference 3The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning.https://arxiv.org/abs/2504.05081} When a task is really pattern matching from examples, a visible scratchpad can add noise instead of signal. That's the point of the eval gate: measure the win, don't assume it.

This tradeoff also shapes a build-versus-buy decision you'll face in production. You can orchestrate reasoning yourself with prompt scaffolding (the CoT, self-consistency, and tree-search controllers in this chapter), or call a native reasoning model that performs its own internal reasoning before answering. Scaffolding keeps each step visible, auditable, and cheap to swap, but you own the token and latency budget. A native reasoning model can lift accuracy on genuinely multi-step problems, yet it hides its reasoning, bills you for it, and gives you fewer control points to gate. Treat the two as competing candidates behind the same eval gate rather than assuming the newer option wins.

For a production workflow, don't ask the model to reveal unrestricted inner reasoning. Ask it to produce reviewable artifacts:

The checks are useful because a missed predicate becomes observable. They aren't proof of faithful hidden cognition. Turpin et al. showed that CoT explanations can rationalize outputs influenced by hidden biasing features without mentioning those features.^{[4]Reference 4Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptinghttps://arxiv.org/abs/2305.04388} Log inputs, actions, validations, and outcomes; don't treat eloquent reasoning text as an audit guarantee.

Here, the trace is useful because it meets the scored contract. It doesn't win merely because it contains more words.

Zero-shot or few-shot?

Zero-shot CoT supplies an instruction and lets the model choose its intermediate format. Few-shot CoT gives one or more solved examples, so it can teach both decomposition and the final answer shape. Native structured output enforcement is better when available; examples are still useful when the prompt must communicate task-specific checks.

When is few-shot CoT worth more prompt tokens than zero-shot CoT?

Answer

When examples teach task-specific checks or output fields that your evaluator needs, and held-out results justify the added tokens.

Self-Consistency: sample answers, then vote

One structured trace can fail because generation takes an unlucky path. Self-Consistency replaces reliance on one path with several sampled paths and chooses the most consistent final answer.^{[5]Reference 5Self-Consistency Improves Chain of Thought Reasoning in Language Models.https://arxiv.org/abs/2203.11171} The vote operates on extracted answers, not on whose rationale sounds best.

To obtain different candidates, use stochastic decoding, usually with a nonzero temperature, rather than rerunning one deterministic decode. The earlier Decoding Strategies lesson explains that control. Keep it fixed in your experiment and log it with the sample count.

On the original benchmark setting, Wang et al. reported a 17.9 percentage-point GSM8K improvement over CoT prompting for PaLM 540B.^{[5]Reference 5Self-Consistency Improves Chain of Thought Reasoning in Language Models.https://arxiv.org/abs/2203.11171} That number is evidence for the method on those benchmarks. It's not your expected support-resolution gain. Measure your own cases and sample cost.

Normalize before counting

Model outputs rarely use exactly one spelling. Your controller should map harmless variants to one allowed action and reject unknown outputs before voting.

A winner isn't always confident enough

A 2 to 2 tie, zero accepted outputs, or a narrow plurality with many rejected outputs shouldn't silently become an automated action. Add an abstention rule. The winning share must use all sampled outputs as its denominator, including strings that failed to parse.

Measure gains against call cost

Use a held-out fixture set before calling the strategy ready. The fixture represents five sampled final actions returned for each case; the controller compares first-sample accuracy with five-sample voting accuracy.

This fixture is intentionally small and deterministic: it tests controller logic. A real release decision needs representative labeled cases, real model samples, token counts, latency, refusal rates, and cost.

What should a self-consistency controller do when votes do not form a strong majority?

Answer

Abstain or route to review. More samples are useful only when a measured gain justifies their cost and the action remains safe.

Tree-of-Thoughts: search when branches can dead-end

Voting helps when independent paths tend to converge on the same short answer. It doesn't deliberately revisit earlier choices. Tree-of-Thoughts (ToT) represents partial solutions as search states, generates alternatives, evaluates the states, and preserves only branches worth extending.^{[6]Reference 6Tree of Thoughts: Deliberate Problem Solving with Large Language Models.https://arxiv.org/abs/2305.10601}

Yao et al. evaluated ToT on tasks built for planning and search. On Game of 24, their GPT-4 ToT setup solved 74% of tasks while their CoT baseline solved 4%.^{[6]Reference 6Tree of Thoughts: Deliberate Problem Solving with Large Language Models.https://arxiv.org/abs/2305.10601} The narrower conclusion is not "use trees everywhere." If a problem has verifiable partial states and meaningful backtracking, search can rescue a bad early move.

Search states in a release plan

For a release gate, consider a workflow whose final recommendation must be supported by two observations: a current security scan and a reviewer approval record. A controller can expand legal steps instead of letting a model invent a final action before evidence is present.

The two evidence-gathering orders converge on outcome-bearing states. Only the state with a clear scan and an approved review can merge; a failed scan or rejected review blocks the release. In the next lesson, tool calls will populate those outcomes from real APIs.

A fully runnable search example

Game of 24 is useful because the evaluator is exact: arithmetic either reaches 24 using each input once or it doesn't. The solver below explores partial equations with breadth-first search and returns a verified solution.

Pruning is a source of failure

An LLM evaluator isn't an arithmetic oracle. If it scores an apparently simple but dead branch above a non-obvious solvable branch, an aggressive beam can remove the answer before expansion.

The production implications are concrete:

• Keep ToT for tasks with real branch structure, not ordinary classification.
• Prefer deterministic validators when a partial state can be checked in code.
• Measure solver recall at each beam width alongside final successes.
• Cap expansions and latency before an open-ended search reaches users.

Why can Tree-of-Thoughts perform worse when the evaluator is weak and the beam is narrow?

Answer

Search spends compute only on retained states. If an incorrect evaluator prunes the valid branch early, later expansions cannot recover it.

Choose compute with an eval gate

Direct prompting, one trace, voting, and tree search aren't maturity levels. They are candidates with different accuracy and serving cost. Start with the cheapest candidate, then promote a more expensive strategy through an eval gate only when held-out results require it.

These numbers are example evaluation results, not a benchmark claim. In your system, keep a table with:

When not to use CoT (or heavier search)

Use this checklist before you promote scaffolding or multi-sample voting:

Forcing a visible CoT scaffold on a model that already does hidden long-CoT reasoning can add cost without accuracy gain. Treat that as another arm in the same release table.

Where reasoning ends and tools begin

All runnable experiments above operate on provided facts or deterministic state. A real release decision requires current CI status, security scan output, and review state from source systems. Reasoning alone can't obtain those observations.

ReAct interleaves reasoning traces and task-specific actions so new observations can update the next decision.^{[7]Reference 7ReAct: Synergizing Reasoning and Acting in Language Models.https://arxiv.org/abs/2210.03629} The production handoff isn't a saved inner monologue. It's a validated action request, a controlled execution result, and a bounded next decision:

The next lesson implements that action boundary with typed function calls, schemas, errors, and safe execution.

What to remember

• Define the scorer first. A decision record lets you test whether extra inference work improves outcomes.
• One trace is one candidate. CoT can reveal missed steps, but a plausible rationale isn't a faithful audit log.^{[4]Reference 4Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptinghttps://arxiv.org/abs/2305.04388}
• Vote on normalized outcomes. Self-consistency is useful only when its measured gain beats its sample cost.^{[5]Reference 5Self-Consistency Improves Chain of Thought Reasoning in Language Models.https://arxiv.org/abs/2203.11171}
• Search only with branch structure. ToT needs meaningful states, evaluators, pruning limits, and failure measurements.^{[6]Reference 6Tree of Thoughts: Deliberate Problem Solving with Large Language Models.https://arxiv.org/abs/2305.10601}
• Promote strategies through evals. Direct, trace, vote, and search should compete under quality and latency gates.
• Skip CoT when policy is already machine state. Deterministic predicates, pattern-ICL, and tasks already above the accuracy SLO need no scratchpad.