Analogical Prompting: How LLMs Self-Generate Their Own Few-Shot Examples

Few-shot chain-of-thought prompting works well, but it carries a hidden cost: someone must write the exemplars. For every new domain or task, a human annotator has to craft question-rationale-answer triples that demonstrate the reasoning process. Analogical prompting, introduced in 'Large Language Models as Analogical Reasoners' (arXiv:2310.01714, Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, and Denny Zhou from Google DeepMind and Stanford University), proposes a different arrangement: ask the model to recall and write its own relevant exemplars before tackling the problem. Published at ICLR 2024, the paper reports that this self-prompting strategy outperforms both 0-shot chain-of-thought and manually written few-shot chain-of-thought across math reasoning (GSM8K and MATH), code generation (Codeforces), and diverse reasoning tasks (BIG-Bench), with an average accuracy gain of +4% over baseline prompting approaches on GPT-3.5, GPT-4, and PaLM2.

The gap between zero-shot and few-shot CoT

Zero-shot CoT is convenient: adding a phrase like 'think step by step' to the prompt requires no annotation and transfers across tasks. But the guidance is generic. It does not teach the model what kind of problem it is facing, what mathematical tools apply, or what a correct solution trajectory looks like. Manual few-shot CoT fills that gap by providing concrete examples, but at a cost. You need annotated examples for each domain, those examples may not match the specific sub-type of problem in the query (a geometry example may not help a probability problem even within the same math benchmark), and retrieving the most relevant exemplars from a corpus requires additional infrastructure. Analogical prompting attempts to capture the specificity benefit of few-shot CoT without the annotation overhead, by having the model supply exemplars that are tailored to the exact problem at hand.

The two-stage prompt structure

The core technique is a two-section prompt given to the model in a single call. The first section instructs the model to recall relevant problems and solutions using a header such as 'Recall relevant problems and solutions:'. The model then generates two or three related problems it knows from training, complete with step-by-step solutions. The second section presents the original problem for the model to solve, now with the self-generated exemplars sitting in context above it. Because the entire sequence, recall and solution, happens in one forward pass, no second API call or retrieval step is required. The paper also tests an extended variant that adds a third instruction, 'Provide a tutorial on key concepts:', prompting the model to write high-level conceptual background before the specific exemplars. This tutorial variant proves especially useful for code generation tasks on Codeforces, where naming the relevant algorithmic paradigm (dynamic programming, graph traversal) before generating examples gives the model a sharper frame for the solution.

• Stage 1, recall: the model generates 2 to 3 related problems and full solutions tailored to the specific problem type.
• Stage 2, solve: the model answers the original problem using the recalled exemplars as in-context guidance.
• Tutorial variant: an optional preliminary section prompts the model to state key concepts or algorithmic knowledge before generating exemplars.
• Single pass: recall, exemplars, and solution are generated end-to-end in one call, with no retrieval system required.
• Problem-specific tailoring: a geometry question gets geometry exemplars; a probability question gets probability exemplars, rather than generic math examples.

Benchmark results on math, code, and reasoning

The paper evaluates analogical prompting against 0-shot CoT and manual few-shot CoT on several base models. On GSM8K (grade-school math word problems), analogical prompting consistently matches or exceeds manual few-shot CoT on GPT-3.5 and GPT-4 without requiring any human-written exemplars. On the harder MATH benchmark, which spans algebra, geometry, number theory, and probability, the self-generated exemplars provide an advantage over 0-shot CoT because they ground the model in the specific sub-domain of each problem. On Codeforces competitive programming problems, the tutorial-plus-exemplars variant outperforms standard CoT approaches, where the model benefits from explicitly naming the relevant algorithm class before producing code. Across the BIG-Bench tasks tested, the average accuracy improvement over 0-shot CoT and few-shot CoT is reported as +4%. The method generalizes across GPT-3.5, GPT-4, and PaLM2, suggesting the underlying mechanism is not model-specific.

Why self-generated exemplars can beat human-written ones

The result that auto-generated exemplars sometimes outperform manually written ones seems counterintuitive. The explanation lies in relevance. A human curating few-shot examples for a math benchmark typically writes one example per broad category, such as a single algebra word problem for the algebra section. When the actual test question involves a specific subcategory, such as quadratic inequalities, the generic algebra example may not activate the right reasoning patterns. The model generating its own exemplars, by contrast, sees the exact problem statement before producing its recall. It can generate an exemplar about quadratic inequalities specifically because it already knows the query is about quadratic inequalities. This per-problem tailoring is the key mechanism the paper identifies as the source of analogical prompting's advantage over fixed, task-level few-shot examples. The paper's ablations confirm that removing the recall step and returning to 0-shot CoT degrades performance, and that replacing the model's self-generated exemplars with random unrelated examples also degrades performance, ruling out a simple 'more tokens in context' explanation.

Practical use and known limits

Analogical prompting is straightforward to apply: the only change to an existing chain-of-thought prompt is adding the recall instruction before the problem. No training, retrieval index, or external data store is required. The method works best on tasks where the model already has relevant knowledge in its weights, which covers most well-studied benchmarks in math, code, and general reasoning. Its limitations follow from the same property. If the model lacks knowledge of the relevant problem type, such as highly specialized or recent domains, the self-generated exemplars may be low-quality or incorrect, and incorrect exemplars can mislead the solution step. The paper notes that for tasks requiring up-to-date factual knowledge, retrieval augmentation remains a stronger choice than analogical prompting. Token cost is also a factor: generating two or three worked examples before answering meaningfully increases prompt length, which has cost and latency implications at scale. As a practical middle ground, the tutorial-only variant (generate conceptual framing without full worked examples) can be cheaper while retaining most of the gain for structured tasks like code generation. The code and evaluation suite are associated with the arXiv submission arXiv:2310.01714. PromptingIndex covers analogical prompting alongside Auto-CoT, self-consistency, step-back prompting, and least-to-most prompting as part of its series on zero-label and automated reasoning techniques.