Self-Ask Prompting: How Asking Follow-Up Questions Closes the Compositionality Gap

There is a surprising failure mode buried inside every large language model. Ask a model for a fact it knows and it answers correctly. Ask it for a second, unrelated fact and it answers correctly again. Now ask a question that requires composing both facts into a single answer and the model fails, even though it demonstrably knows each piece. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis from the University of Washington, MosaicML, Meta AI Research, and the Allen Institute for AI published 'Measuring and Narrowing the Compositionality Gap in Language Models' (arXiv:2210.03350, Findings of EMNLP 2023) to quantify exactly how often this happens and to introduce a prompting technique, self-ask, that substantially reduces it. The results challenge the intuition that scaling models fixes reasoning problems automatically.

The compositionality gap: a persistent 40 percent failure rate

The paper defines the compositionality gap as the fraction of compositional questions a model answers incorrectly, restricted to those questions where the model correctly answers every individual sub-question. If a model knows that Justin Bieber was born in 1994 and knows who won the Masters Tournament in 1994, the compositionality gap measures how often it still fails to answer 'Who won the Masters Tournament the year Justin Bieber was born?' The authors built a dataset called Compositional Celebrities (CC) containing 8,600 two-hop questions to measure this. Questions are generated by combining frequently stated facts in combinations unlikely to have appeared together in pretraining data: birthplace calling codes, birth-year Nobel Prize winners, birth-country capitals, and so on across 17 categories.

The results across the GPT-3 model family are striking. As model size increases, single-hop accuracy improves and multi-hop accuracy also improves, but the compositionality gap stays roughly constant at around 40 percent across all model sizes and training techniques. Bigger models memorize and recall more facts individually but show no corresponding improvement in their ability to compose those facts on demand. The authors also report an estimated compositionality gap of 42.9 percent for ChatGPT and 23.0 percent for GPT-4, though they note the GPT-4 result may be unreliable because CC was released publicly before GPT-4 launched and could have appeared in its training data.

How self-ask works: explicit follow-up questions in the prompt

Self-ask addresses the compositionality gap by restructuring what happens before the model writes its final answer. Instead of asking the model to answer the composite question directly, a self-ask prompt demonstrates a pattern in which the model first identifies the sub-questions it needs to answer, answers each one explicitly, and only then produces the final answer. The format is conversational and structured. For the question 'Who won the Masters Tournament the year Justin Bieber was born?', a self-ask prompt shows the model producing something like: 'Are there any follow-up questions I need to answer before I can answer the main question? Yes. What year was Justin Bieber born? 1994. Who won the Masters Tournament in 1994? Jose Maria Olazabal. So the final answer is: Jose Maria Olazabal.' The critical feature is that the decomposition happens as explicit text inside the model's output, not as invisible internal computation. Making the intermediate steps visible in the output is what separates self-ask from naive prompting and what lets the model use each sub-answer as grounded context before tackling the next step.

Self-ask is a few-shot prompting technique: the prompt contains worked examples that demonstrate the follow-up question format, and the model generalizes it to new questions. The number of sub-questions is not fixed in advance. If a question requires three hops, the model generates three follow-up questions and answers each before writing the final answer. The format is flexible enough to handle questions that require no decomposition at all: for simple one-hop questions, the model answers 'No' when asked if any follow-up questions are needed and proceeds directly to the final answer. This makes self-ask applicable across question types without needing to classify the difficulty of a question in advance.

Results compared to chain-of-thought

The paper compares self-ask against standard few-shot prompting (no reasoning steps) and chain-of-thought prompting (reasoning steps in natural language before the answer) across four datasets: Compositional Celebrities, 2WikiMultiHopQA, Musique, and Bamboogle. Bamboogle is a handcrafted set of 125 two-hop questions written by the authors, each one specifically designed so that an internet search engine cannot answer it directly (because the composite question never appears verbatim online) but where both supporting facts are in Wikipedia. On Bamboogle, which is the cleanest measure of compositional reasoning without retrieval contamination, self-ask outperforms chain-of-thought. The structured format of self-ask forces more deliberate decomposition than chain-of-thought, which can reason in an open-ended way that sometimes skips explicit sub-question identification. On CC and 2WikiMultiHopQA, both elicitive methods substantially outperform naive prompting, narrowing or closing the compositionality gap for a meaningful fraction of questions.

Plugging in a search engine for real-world use

The most practically powerful result in the paper is that self-ask's structured format enables a natural integration with external retrieval. Because the model generates explicit follow-up questions as plain text, those questions can be intercepted before the model answers them and routed to a search engine instead. The search engine result replaces the model's self-generated answer to the sub-question, and the model then reads that retrieved answer as grounded context when generating the next follow-up question or the final answer. The paper demonstrates this using Google search results for the follow-up questions on Bamboogle and shows that search-augmented self-ask further improves accuracy beyond the LM-only version. This is a concrete path to grounded multi-hop reasoning without fine-tuning: standard few-shot prompting establishes the format, and a retrieval step slots into the gaps the format creates. The same structure that makes self-ask effective as a purely generative method also makes it extensible as a retrieval-augmented method.

When to use self-ask and how to apply it

Self-ask is most valuable for questions where the answer depends on composing two or more facts that are unlikely to appear together in training data. Research questions about lesser-known entities, questions that cross domains (sports plus geography, biography plus history), and questions requiring temporal reasoning across multiple steps are all strong candidates. For questions that can be answered in a single lookup, chain-of-thought or direct prompting is sufficient and cheaper. The practical implementation requires only a handful of worked examples in the prompt: the paper uses between two and six demonstrations, each showing the full pattern of follow-up question generation, sub-question answering, and final answer output. No fine-tuning is needed. The authors released their code and data at https://github.com/ofirpress/self-ask, and the prompt format is straightforward enough to adapt from the examples in the paper directly. For teams running RAG pipelines, self-ask provides a structured query decomposition step that can replace ad hoc sub-question generation: the model's follow-up questions become the retrieval queries, and the retrieved passages become the sub-answers the model reads before composing its final response. PromptingIndex covers self-ask alongside chain-of-thought, ReAct, and least-to-most prompting as part of its series on decomposition-based reasoning techniques.