Chain-of-Thought Prompting: Make the Model Show Its Work

When you ask an AI for the answer to a hard problem and it gets it wrong, the culprit is usually not the model's knowledge. It is the way you asked. Standard prompts invite the model to jump straight to a conclusion, which forces it to compress all its reasoning into one step. Chain-of-thought prompting fixes this with a single insight: ask the model to explain its thinking first, and the answer gets dramatically better.

What chain-of-thought prompting is

Chain-of-thought (CoT) prompting is a technique where you instruct the model to produce a series of intermediate reasoning steps before giving a final answer. Instead of asking 'What is 17 multiplied by 8, then divided by 4?' you add something like 'Work through this step by step' or 'Show your reasoning.' The model writes out each step of its logic, then lands on a conclusion grounded in that process.

The approach was formalized in a 2022 paper by Jason Wei and colleagues at Google, 'Chain-of-Thought Prompting Elicits Reasoning in Large Language Models' (arXiv:2201.11903, submitted January 28, 2022). The paper showed that providing just eight chain-of-thought examples to a 540-billion-parameter model achieved state-of-the-art accuracy on the GSM8K benchmark of math word problems, outperforming finetuned GPT-3 with a dedicated verifier. That paper has been cited over 32,000 times, a signal of how broadly the technique has spread.

The two flavors: few-shot and zero-shot

There are two main forms of chain-of-thought prompting, and the difference matters for how you use them.

Few-shot CoT provides two to eight worked examples inside the prompt. Each example shows an input, a step-by-step reasoning trace, and a final answer. The model picks up the pattern and applies the same format to new questions. This is the most reliable form because the examples demonstrate exactly the reasoning structure you want.

Zero-shot CoT drops the examples entirely and relies on a single phrase. Kojima and colleagues showed at NeurIPS 2022 that appending 'Let's think step by step' to a prompt is often enough to unlock reasoning. In their experiments, this five-word addition lifted accuracy on the MultiArith benchmark from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% using InstructGPT. The phrase acts as a cue: the model has seen that pattern in training and knows that step-by-step thinking follows.

Why it works

The deeper reason chain-of-thought helps is that it forces the model to allocate more computation to each part of the problem before committing to an answer. In a one-shot answer, the model must hold all the moving pieces in the same forward pass that produces the final token. In a chain-of-thought answer, each intermediate step becomes part of the context that informs the next, giving the model a chance to catch and correct errors along the way.

Wei et al. also observed that chain-of-thought reasoning behaves like an emergent ability: it appears most reliably in large models and becomes less consistent as model size decreases. Instruction tuning has since changed this somewhat, and many smaller fine-tuned models now perform chain-of-thought reliably. But the core finding holds: reasoning traces become more trustworthy as model scale increases.

When chain-of-thought helps most

Chain-of-thought prompting earns its keep on tasks that require multiple steps to reach a correct answer. The clearest wins are:

• Multi-step arithmetic and word problems, where intermediate values matter.
• Logical deduction: if A then B, if B then C, therefore the conclusion follows.
• Commonsense reasoning about cause and effect.
• Planning tasks where order and dependencies matter.
• Code debugging, where explaining the error trace helps isolate the root cause.

It adds less value on tasks where the answer is a direct recall fact or a creative generation. Asking 'What year was the Eiffel Tower built?' with chain-of-thought does not help much because there are no intermediate steps to reason through. Use it selectively, not as a default for every prompt.

How to add it to your prompts today

The zero-shot version takes seconds. After your question or instruction, add one of these phrases:

• 'Let's think step by step.'
• 'Work through this reasoning step by step before giving the final answer.'
• 'Explain your reasoning, then state your conclusion.'

For the few-shot version, write two or three examples where you include the intermediate steps yourself, then add your real question. The examples do not need to come from the same domain as your question, though domain-matched examples tend to produce cleaner reasoning traces.

One useful habit is to ask the model to separate the reasoning from the answer. Ending a prompt with 'First, show your reasoning. Then, on a new line labeled Answer:, give your final response' makes it easy to check the logic before trusting the conclusion.

The limits to know

Chain-of-thought prompting is not error-proof. A model can produce a fluent, confident reasoning trace that leads to a wrong answer. The steps feel like logic but may reflect plausible-sounding patterns from training rather than true deduction. If you rely on chain-of-thought output for high-stakes decisions, treat the reasoning as a signal worth checking, not proof of correctness.

CoT also costs more tokens. Every reasoning step adds to the output length, which means higher latency and cost on commercial APIs. For high-volume applications, weigh the accuracy gain against the token overhead before committing to it everywhere.

Chain-of-thought prompting is one of the highest-leverage techniques in the toolkit: it requires no code, no finetuning, and no special access. A single phrase can turn a model that guesses at a hard problem into one that works through it step by step. Find prompts built on this pattern in the PromptingIndex library, and test which phrasing produces the clearest reasoning for your use case.