CoT, ToT & Self-Consistency Prompting

The previous lesson measured whether compressed embeddings still retrieved the right policy evidence. Retrieval is not the end of the task. An assistant may now have the right facts and still choose the wrong action because it skips a condition, mishandles arithmetic, or commits too early to one plan.

This lesson turns extra inference work into an engineering decision. You will build a small support-resolution controller for a delayed parcel:

• Chain-of-Thought (CoT): request one decomposed candidate decision.
• Self-Consistency: sample several candidates, normalize final actions, and vote.
• Tree-of-Thoughts (ToT): expand and prune branches when a decision needs backtracking.

The goal is not to collect long rationales. The goal is to improve measurable decision accuracy under a token, latency, and safety budget.

Start with a failure you can audit

Suppose an order has been delayed for six days. A replacement is in stock, the carrier scan confirms a delivery exception, and policy permits expedited replacement after five delayed days. A direct response may still overlook one condition and offer a refund or escalation unnecessarily.

A useful outward artifact is a short decision record: facts used, checks applied, and final action. It is smaller than an open-ended rationale, easy to score, and safe to compare against deterministic policy logic.

This code is not an LLM. It is the oracle that your prompt variants must match. Before increasing model compute, define an output contract and a scorer.

Chain-of-Thought: one decomposed candidate

Wei et al. introduced few-shot CoT by placing worked intermediate steps in prompt examples. Their experiments found gains on arithmetic, commonsense, and symbolic reasoning benchmarks for sufficiently large models, including a strong GSM8K result with PaLM 540B.^[1] Kojima et al. later showed that the zero-shot trigger "Let's think step by step" could improve several benchmark reasoning tasks without worked examples.^[2]

Those papers establish techniques to evaluate, not a production law. On your endpoint and task, a visible scratchpad may help, do nothing, or add cost. If an API offers a native reasoning control, evaluate that option as another strategy rather than assuming that extra visible prose helps.

For a customer-facing workflow, do not ask the model to reveal unrestricted inner reasoning. Ask it to produce reviewable artifacts:

The checks are useful because a missed predicate becomes observable. They are not proof of faithful hidden cognition. Turpin et al. showed that CoT explanations can rationalize outputs influenced by hidden biasing features without mentioning those features.^[3] Log inputs, actions, validations, and outcomes; do not treat eloquent reasoning text as an audit guarantee.

Here, the trace is useful because it meets the scored contract. It does not win merely because it contains more words.

Zero-shot or few-shot?

Zero-shot CoT supplies an instruction and lets the model choose its intermediate format. Few-shot CoT gives one or more solved examples, so it can teach both decomposition and the final answer shape. Native structured-output enforcement is better when available; examples are still useful when the prompt must communicate task-specific checks.

Self-Consistency: sample answers, then vote

One structured trace can fail because generation takes an unlucky path. Self-Consistency replaces reliance on one path with several sampled paths and chooses the most consistent final answer.^[4] The vote operates on extracted answers, not on whose rationale sounds best.

On the original benchmark setting, Wang et al. reported a 17.9 percentage-point GSM8K improvement over CoT prompting for PaLM 540B.^[4] That number is evidence for the method on those benchmarks. It is not your expected support-resolution gain. Measure your own cases and sample cost.

Normalize before counting

Model outputs rarely use exactly one spelling. Your controller should map harmless variants to one allowed action and reject unknown outputs before voting.

A winner is not always confident enough

A 2 to 2 tie, zero accepted outputs, or a narrow plurality with many rejected outputs should not silently become a customer action. Add an abstention rule. The winning share must use all sampled outputs as its denominator, not only the strings that parsed successfully.

Measure gains against call cost

Use a held-out fixture set before calling the strategy ready. The code below represents five sampled final actions returned for each fixture; the controller compares first-sample accuracy with five-sample voting accuracy.

This fixture is intentionally small and deterministic: it tests controller logic. A real release decision needs representative labeled cases, real model samples, token counts, latency, refusal rates, and cost.

Tree-of-Thoughts: search when branches can dead-end

Voting helps when independent paths tend to converge on the same short answer. It does not deliberately revisit earlier choices. Tree-of-Thoughts (ToT) represents partial solutions as search states, generates alternatives, evaluates the states, and preserves only branches worth extending.^[5]

Yao et al. evaluated ToT on tasks built for planning and search. On Game of 24, their GPT-4 ToT setup solved 74% of tasks while their CoT baseline solved 4%.^[5] The lesson is narrower than "use trees everywhere": if a problem has verifiable partial states and meaningful backtracking, search can rescue a bad early move.

Search states in a support plan

For a delayed parcel, consider a workflow whose final recommendation must be supported by two observations: a confirmed carrier exception and a replacement inventory check. A controller can expand legal steps instead of letting a model invent a final action before evidence is present.

The two legal evidence-gathering orders converge on the same verified state before the final action. In the next lesson, tool calls will populate that state from real APIs.

A fully runnable search example

Game of 24 is useful because the evaluator is exact: arithmetic either reaches 24 using each input once or it does not. The solver below explores partial equations with breadth-first search and returns a verified solution.

Pruning is a source of failure

An LLM evaluator is not an arithmetic oracle. If it scores an apparently simple but dead branch above a non-obvious solvable branch, an aggressive beam can remove the answer before expansion.

The production implications are concrete:

• Keep ToT for tasks with real branch structure, not ordinary classification.
• Prefer deterministic validators when a partial state can be checked in code.
• Measure solver recall at each beam width, not only final successes.
• Cap expansions and latency before an open-ended search reaches users.

Choose compute with a release gate

Direct prompting, one trace, voting, and tree search are not maturity levels. They are candidates with different accuracy and serving cost. Start with the cheapest candidate, then promote a more expensive strategy only when held-out results require it.

These numbers are example evaluation results, not a benchmark claim. In your system, keep a table with:

Where reasoning ends and tools begin

All runnable experiments above operate on provided facts or deterministic state. A real delayed-parcel request requires current carrier scans and inventory. Reasoning alone cannot obtain those observations.

ReAct interleaves reasoning traces and task-specific actions so new observations can update the next decision.^[6] For production systems, the important handoff is not a saved inner monologue. It is a validated action request, a controlled execution result, and a bounded next decision:

The next lesson implements that action boundary with typed function calls, schemas, errors, and safe execution.

What to remember

• Define the scorer first. A decision record lets you test whether extra inference work improves outcomes.
• One trace is one candidate. CoT can reveal missed steps, but a plausible rationale is not a faithful audit log.^[3]
• Vote on normalized outcomes. Self-consistency is useful only when its measured gain beats its sample cost.^[4]
• Search only with branch structure. ToT needs meaningful states, evaluators, pruning limits, and failure measurements.^[5]
• Promote strategies through evals. Direct, trace, vote, and search should compete under quality and latency gates.

Mastery check

Key concepts

• Chain-of-Thought as one structured candidate trace
• Zero-shot CoT versus few-shot contract examples
• Reasoning trace faithfulness limits
• Self-Consistency with canonicalization and abstention
• Tree-of-Thoughts as bounded, evaluator-guided search
• Beam pruning failure modes
• Inference-time cost, latency, and release gates
• ReAct as the handoff from decisions to observations

Evaluation rubric

• Foundational: Defines an expected action and the checks required to justify it.
• Intermediate: Implements final-answer normalization, majority voting, and an abstention threshold.
• Intermediate: Compares direct, trace, and voting strategies on labeled fixtures with cost accounting.
• Advanced: Implements a searchable state space and explains how evaluator errors interact with pruning.
• Advanced: Designs a release gate using task success, safety, token cost, and tail latency.

Common pitfalls

• Scoring prose instead of outcomes: Require final actions and policy-check fields that an evaluator can validate.
• Logging unrestricted rationale as evidence: Trace text can be unfaithful. Log retrieved inputs, validated actions, observations, and results.
• Voting on raw strings: Normalize final decisions into allowed enums before counting.
• Treating every plurality as safe: Add abstention for ties, weak majorities, and rejected outputs.
• Using ToT for a lookup: Search cost is wasted when no branch can backtrack or improve the answer.
• Pruning with an unmeasured evaluator: A confident score can remove the only valid branch. Evaluate beam recall and cap the budget.

Practice extension

Create twenty delayed-parcel fixtures with policy-grounded final actions. Collect direct, structured-trace, and five-sample candidate outputs from one model endpoint. Canonicalize the final action, abstain on weak votes, and produce a comparison table with accuracy, unsafe actions, abstentions, tokens, and p95 latency. Add search only for cases where choosing an action requires sequencing multiple checks or backing out of a failed plan.