Auto-CoT: Eliminating Manual Work from Chain-of-Thought Prompting

Chain-of-thought prompting works by showing a model a few examples of step-by-step reasoning before asking it to solve a new problem. The model sees how a similar question was decomposed and follows the same pattern. The results, documented in the original Wei et al. (2022) paper, are dramatic: models that score near chance on multi-step arithmetic and commonsense problems reach much higher accuracy when given reasoning demonstrations. The catch is that those demonstrations have to be written by hand. Someone must sit down, pick representative questions, and compose correct multi-step reasoning chains for each one. On every new task, the process restarts. Auto-CoT, introduced in 'Automatic Chain of Thought Prompting in Large Language Models' (arXiv:2210.03493, October 7, 2022) by Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola at Amazon Web Services, eliminates that manual step entirely. The method constructs demonstrations automatically by clustering questions and generating reasoning chains with the 'Let's think step by step' prompt, then selects one representative from each cluster as a demonstration. Across ten public benchmark reasoning datasets using GPT-3, Auto-CoT consistently matches or exceeds the performance of hand-crafted CoT demonstrations.

The problem with manual demonstrations

Standard few-shot CoT prompting requires two manual decisions for every task. First, which questions to use as demonstrations: they should cover different reasoning patterns and not be too similar to each other. Second, what the step-by-step reasoning chain should look like for each selected question. Both decisions require domain expertise and careful judgment. A poorly chosen set of demonstrations, such as four questions that all share the same problem structure, limits the model's ability to generalize to questions with a different structure. The original Wei et al. (2022) CoT paper uses eight manually written demonstrations per task, selected by the authors. Subsequent work found that the choice of demonstrations matters substantially: wrong demonstrations hurt, and the optimal set varies by task. Auto-CoT reframes the problem. Instead of asking a human to choose good demonstrations, it asks: what properties make a set of demonstrations good? The answer the paper arrives at is diversity. A set of demonstrations that samples broadly from the question space is less likely to miss important reasoning patterns. The Auto-CoT method makes diversity the central criterion and uses question clustering to operationalize it.

Two stages: cluster, then generate

Auto-CoT runs in two stages before any inference happens. In stage one, the method computes sentence embeddings for all questions in the dataset using Sentence-BERT and partitions them into k clusters using k-means clustering, where k is set to the desired number of demonstrations (typically eight, matching the Wei et al. baseline). Each cluster groups questions that are semantically similar. In stage two, the method selects one question from each cluster, choosing the question closest to the cluster centroid to maximize representativeness. It then generates a reasoning chain for that question by appending 'Let's think step by step' and sampling from GPT-3 in zero-shot mode. The generated chain becomes the demonstration for that cluster. Since cluster centroids capture the center of each semantic group, and one demonstration is sampled from each cluster, the resulting demonstration set spans the space of question types without requiring a human to identify what those types are. The final prompt for any new question concatenates all k demonstrations (each consisting of a question plus its auto-generated reasoning chain) followed by the new question.

• Stage 1 (clustering): encode all questions with Sentence-BERT and run k-means to produce k clusters.
• Stage 2 (sampling): for each cluster, pick the centroid question and generate its reasoning chain using Zero-Shot-CoT ('Let's think step by step').
• Heuristics filter out generated chains that are too short (under 60 tokens) or too long (over 5 reasoning steps), keeping simple and likely correct demonstrations.
• The k demonstrations are concatenated to form the few-shot prompt used for all test questions.
• No human writes any reasoning chain; the only manual input is setting k to match the number of desired demonstrations.

Why diversity matters more than correctness

A natural concern with Auto-CoT is that Zero-Shot-CoT generates mistakes. When the model answers 'Let's think step by step' for a tricky arithmetic problem, it sometimes produces an incorrect reasoning chain. Those incorrect chains then appear in the demonstration set. The paper directly addresses this by asking whether diversity or correctness is the more important property. The authors compare Auto-CoT against two ablations: Retrieval-Q-CoT, which selects demonstrations by choosing questions most similar to the test question (maximizing relevance but reducing diversity), and Random-Q-CoT, which samples demonstrations randomly. Both ablations use the same Zero-Shot-CoT chain generation and the same filtering heuristics. Auto-CoT outperforms Retrieval-Q-CoT on most benchmarks. The intuition is that retrieving questions similar to the test question means all demonstrations share a similar structure. When the reasoning pattern for that structure happens to be one that the model handles poorly, every demonstration reinforces the same failure mode. Diverse demonstrations, by contrast, expose the model to a range of structures. Even if a few generated chains contain errors, the diversity effect outweighs the noise. The heuristics that filter out very long or very short generated chains further reduce the probability that any single demonstration is badly wrong.

Benchmark results across ten reasoning datasets

The paper evaluates Auto-CoT against Manual-CoT (the hand-crafted demonstrations from Wei et al. 2022 and follow-up work), Zero-Shot-CoT, and the retrieval and random ablations, using GPT-3 (text-davinci-002) as the backbone model. The ten datasets span three categories. Arithmetic reasoning includes MultiArith, GSM8K, AddSub, AQuA, SingleEq, and SVAMP. Commonsense reasoning includes CommonsenseQA and StrategyQA. Symbolic reasoning includes Letter (concatenate first letters of words) and Coin Flip (track coin flips through a sequence of operations). On arithmetic tasks, Auto-CoT matches Manual-CoT on GSM8K (49.7% versus 46.9% for Manual-CoT in the paper's reported numbers), exceeds it on MultiArith, AddSub, AQuA, SingleEq, and SVAMP. On commonsense and symbolic tasks, Auto-CoT again matches or exceeds Manual-CoT. The overall result is that generating demonstrations automatically with clustering and Zero-Shot-CoT produces a prompt that performs as well as demonstrations written by human experts, with no human time invested in reasoning chain construction.

Practical use: when and how to apply Auto-CoT

Auto-CoT's two-stage design means it requires a pool of unlabeled questions to cluster, which is available any time you have a task dataset without needing answers or annotations. The setup cost is the one-time clustering step plus Zero-Shot-CoT sampling for k questions. Once the demonstrations are generated, they can be reused for all test questions in the same domain without repeating the clustering. The method is most valuable in three situations. First, when you are deploying a reasoning task at scale and writing eight manually crafted demonstrations would require domain expertise you do not have. Second, when you want to adapt CoT prompting to a new subdomain quickly: cluster the new domain's questions, regenerate demonstrations, and the prompt updates to reflect the new structure. Third, when you want to test whether your manually crafted demonstrations are actually diverse: running the clustering step on your existing demonstration pool reveals whether they all fall into the same cluster. If they do, the Auto-CoT framework suggests you should spread your selection across the question space. The method is model-agnostic and has been replicated with models other than GPT-3 in subsequent work. PromptingIndex covers Auto-CoT alongside the original Wei et al. chain-of-thought paper, Kojima et al. Zero-Shot-CoT, and self-consistency, all of which address different parts of the demonstration design problem in few-shot reasoning.

The core contribution of Auto-CoT is not a new model or training procedure. It is a reframing of what makes a good demonstration set and a method for constructing one. The insight that diversity, achievable through clustering, outweighs correctness, hard to guarantee from Zero-Shot-CoT, makes the approach practical. A human reasoning about what makes a good demonstration tends to optimize for correctness of individual examples. Auto-CoT suggests that coverage of the question space is the more important property, and that a few noisy but diverse demonstrations outperform many clean but repetitive ones. That result has implications beyond CoT prompting: any method that selects examples for in-context learning faces the same diversity-versus-correctness tradeoff, and the Auto-CoT findings suggest diversity should be the primary selection criterion in most cases.