Self-Consistency: Why Sampling Multiple Reasoning Paths Beats Greedy Decoding in Chain-of-Thought Prompting

Chain-of-thought prompting, introduced by Wei et al. in 2022, gave language models a powerful capability: instead of answering directly, the model generates intermediate reasoning steps before producing its final answer. A question like 'Roger has 5 tennis balls. He buys 2 more cans with 3 balls each. How many tennis balls does he have now?' becomes an explicit calculation rather than a direct lookup. Chain-of-thought dramatically improved performance on multi-step reasoning tasks, but it relied on a mechanism that a follow-up team identified as a meaningful bottleneck: greedy decoding. Greedy decoding picks the highest-probability token at each generation step, which produces exactly one reasoning chain per question. That chain might be the model's best single guess, but it is not necessarily the most reliably correct. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou at Google Research Brain Team submitted 'Self-Consistency Improves Chain of Thought Reasoning in Language Models' to arXiv on March 21, 2022 (arXiv:2203.11171), and the paper was published at ICLR 2023. It introduced a decoding strategy called self-consistency that replaces the single greedy chain with a diverse sample of reasoning paths aggregated by majority vote, adding up to 17.9 percentage points of absolute accuracy on GSM8K over standard chain-of-thought prompting.

Why one greedy reasoning path is not enough

Greedy decoding is deterministic. Given the same prompt and question, a language model with temperature set to zero will always generate the same chain of thought and the same final answer. If that chain contains a reasoning error, no amount of re-prompting will catch it, because the model will repeat the same error. More subtly, greedy decoding optimizes for local probability at each token rather than global quality of the full reasoning sequence. A token that is slightly more likely in isolation might lead the chain down a path where the final answer is less likely to be correct. These two limitations compound on tasks involving multiple reasoning steps, where an error early in the chain propagates through to the end. The self-consistency paper observes that this is a structural problem with the decoding strategy, not a fundamental limitation of chain-of-thought prompting itself. Complex reasoning tasks, the authors note, typically admit multiple valid paths to a correct answer. A math problem can often be solved by direct calculation or by working backward from estimates. A commonsense inference can follow from several independent observations. When many different reasoning strategies all arrive at the same final answer, that convergence is strong evidence the answer is correct.

The self-consistency method: sample, then vote

Self-consistency operates in three steps. First, the language model is prompted with the same chain-of-thought few-shot exemplars used in standard chain-of-thought prompting. No additional template or special format is required. Second, instead of taking the single greedy output, the model samples multiple reasoning paths from its decoder using temperature sampling, top-k sampling, or nucleus sampling. Each sampled output includes both a reasoning chain and a final answer. Because sampling is stochastic, different runs through the same prompt produce different chains, some correct and some incorrect. Third, the final answers are aggregated by majority vote. Formally, if the model generates m outputs with final answers drawn from a fixed set A, self-consistency selects the answer a that maximizes the count of outputs producing it: argmax over a of the sum of indicator functions across all sampled outputs. The reasoning chains are discarded after voting; only the final answers matter for aggregation. The paper also evaluates a weighted variant where each output is weighted by its generation probability normalized by output length, but unweighted majority vote performs nearly as well in practice and requires no additional computation beyond counting.

Why the vote reveals the correct answer

The intuition behind majority vote is grounded in the structure of reasoning errors. Correct reasoning paths tend to converge on the same final answer regardless of which specific argument they use. Incorrect reasoning paths scatter: a model that makes an arithmetic error in step two will reach a different wrong answer than a model that misapplies a commonsense rule in step three. The correct answer is a common fixed point of multiple reasoning strategies; incorrect answers are spread across many different alternatives. As the number of sampled paths grows, the correct answer accumulates votes while any specific incorrect answer remains outvoted. The paper describes this as 'self-ensemble': unlike a conventional model ensemble, which trains multiple separate models and aggregates their outputs, self-consistency runs on a single language model and treats the diversity introduced by stochastic sampling as a substitute for model diversity. Because the method requires no additional training, no external verifier, and no human annotation, it is entirely unsupervised and can be applied off-the-shelf to any language model that supports sampling from its decoder.

Benchmark results across arithmetic and commonsense reasoning

The paper evaluates self-consistency on four language models spanning a wide range of scales: UL2-20B, GPT-3-175B, LaMDA-137B, and PaLM-540B. On all four models, self-consistency improves over standard chain-of-thought prompting across every reasoning task tested. The largest gains appear when self-consistency is applied to PaLM-540B or GPT-3. On GSM8K, a benchmark of grade school math word problems, self-consistency adds 17.9 percentage points of absolute accuracy over standard chain-of-thought prompting. On SVAMP, a math word problem benchmark designed to test robustness to superficial changes in problem phrasing, the gain is 11.0 points. On AQuA, an algebraic question-answering dataset with multiple-choice answers, the gain is 12.2 points. Commonsense reasoning benchmarks show smaller but consistent improvements: StrategyQA gains 6.4 points and ARC-challenge gains 3.9 points. The paper also tests self-consistency on tasks where chain-of-thought prompting does not help or even hurts compared to direct prompting, and shows that self-consistency robustly reduces that gap as well. In comparisons against alternative decoding strategies, self-consistency outperforms sample-and-rank, beam search, and conventional ensemble approaches across all tasks.

• Greedy decoding produces one deterministic chain of thought per prompt; any reasoning error in that chain propagates to the final answer with no way to recover.
• Self-consistency samples multiple reasoning paths with stochastic decoding, then selects the final answer with the most votes across all sampled outputs.
• Correct paths converge on one answer; incorrect paths scatter, so the correct answer accumulates disproportionate votes as sample count grows.
• Gains over chain-of-thought prompting with PaLM-540B or GPT-3: GSM8K +17.9%, SVAMP +11.0%, AQuA +12.2%, StrategyQA +6.4%, ARC-challenge +3.9%.
• Unweighted majority vote performs nearly as well as probability-weighted aggregation and requires no additional computation beyond counting final answers.
• arXiv:2203.11171, submitted March 21, 2022; published at ICLR 2023; authors at Google Research Brain Team.

Applying self-consistency in practice

The primary cost of self-consistency is the number of language model calls per question. Standard chain-of-thought requires one forward pass per input. Self-consistency with 20 to 40 sampled paths requires 20 to 40 forward passes. For tasks where accuracy is critical and latency is acceptable, this tradeoff is generally worthwhile. The paper shows that performance improves consistently as the number of sampled paths increases from 5 to 40, with diminishing returns beyond roughly 20 to 30 paths depending on the task. For latency-sensitive applications, lower sample counts of 5 to 10 still recover most of the accuracy gain at 5 to 10 times the compute cost rather than 40 times. Self-consistency applies directly to any task that produces a discrete final answer parseable from the model's output: numeric answers for math, multiple-choice labels for reasoning tasks, and short string answers for classification. It is less applicable to open-ended generation where there is no single correct answer to vote toward. Practitioners can treat self-consistency as a drop-in replacement for greedy decoding on any chain-of-thought prompt: the prompting format stays identical, and the only change is to sample multiple times and aggregate. Combined with retrieval-augmented generation, self-consistency can serve as a claim verification step, generating multiple independent reasoning chains over a retrieved passage and voting on whether the passage supports a claim. PromptingIndex covers self-consistency alongside chain-of-thought, plan-and-solve, least-to-most, and maieutic prompting as part of its series on prompting methods that move beyond single-pass generation to produce more reliable reasoning from language models.