Generated Knowledge Prompting: Ask the Model First, Then Answer

Commonsense reasoning is one of the tasks where language models have historically struggled most. A model that can explain differential calculus may confidently claim that golf balls are heavier than basketballs, or that it is warmer inside a refrigerator than outside. The gap between linguistic fluency and reliable world knowledge has driven years of research into knowledge bases, retrieval systems, and structured external sources. Generated Knowledge Prompting, introduced in 'Generated Knowledge Prompting for Commonsense Reasoning' (arXiv:2110.08387, ACL 2022), proposes a different approach: instead of retrieving facts from a curated database, ask the language model itself to generate relevant knowledge statements before answering, then provide those statements as context for a second inference pass. The method requires no structured knowledge base, no task-specific supervision for knowledge integration, and no additional training. On four commonsense benchmarks, it sets new state-of-the-art results on three of them, improving accuracy by 7 to 10 percentage points in zero-shot settings over strong T5-11B baselines.

The paper and its eight authors

Generated Knowledge Prompting was submitted to arXiv on October 15, 2021 and published at the ACL 2022 main conference. The eight authors are Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. Five of the eight are affiliated with the Paul G. Allen School of Computer Science and Engineering at the University of Washington; all eight have affiliations with the Allen Institute for Artificial Intelligence. Yejin Choi is a MacArthur Fellow known for research in commonsense reasoning and the creator of the ATOMIC and Winogrande datasets. Hannaneh Hajishirzi leads the UW NLP group and contributed to SQuAD 2.0, TriviaQA, and OpenBookQA among other benchmarks. The first author, Jiacheng Liu, went on to work on instruction tuning and long-form generation research. Code for the method is available at github.com/liujch1998/GKP.

The two-step architecture: generate, then integrate

The method's architecture is deliberately simple. A question arrives. In the first step, a knowledge generator language model produces M factual statements related to that question by reading a few-shot prompt containing five demonstration question-knowledge pairs. In the second step, an inference model reads the original question together with each of the M generated knowledge statements, produces a prediction and a confidence score for each, and returns the answer with the highest confidence score. The two models can be the same model or different models. In the paper's experiments, the knowledge generator is GPT-3 (175 billion parameters) or GPT-J (6 billion parameters), while the inference model is T5-11B or UnifiedQA-11B. The two-step separation is important because it lets the knowledge generator focus purely on producing useful facts without simultaneously solving the task, and allows the inference model to reason over grounded context rather than relying entirely on implicit parametric knowledge.

• Step 1: a knowledge generator LLM reads a few-shot prompt and produces M factual statements about the question.
• Step 2: an inference model scores the original question paired with each knowledge statement and picks the highest-confidence answer.
• No structured knowledge base is needed; the LLM acts as the knowledge source.
• The generator and inference model can be different models or the same model.
• The only manual work is writing roughly five demonstration question-knowledge pairs per task.

Knowledge generation: few-shot prompts and the effect of M

Each knowledge generation prompt is constructed as an instruction followed by five demonstrations of question-and-knowledge pairs, then the new question with an empty knowledge slot. The demonstrations are written manually for each task and represent the only task-specific engineering in the method. Once the demonstrations are in place, the knowledge generator can produce new statements for any question in the target domain without further human intervention. The paper evaluates the effect of generating different numbers of statements per question, finding that performance improves consistently as M grows from 1 to 20 on the QASC development set. This result suggests that diversity of generated knowledge matters: multiple independent statements cover different angles of the question, and the confidence-based selection step can identify the most useful one. The generated statements are free-form natural language rather than outputs of pre-defined templates, which distinguishes GKP from earlier methods like self-talk (Shwartz et al., 2020) that require fixed question templates to elicit knowledge.

Benchmark results on four commonsense tasks

The paper evaluates on four commonsense benchmarks. NumerSense (Lin et al., 2020) tests numerical commonsense, asking questions about numeric properties of everyday objects. CommonsenseQA (Talmor et al., 2019) and CommonsenseQA 2.0 (Talmor et al., 2021) test general commonsense. QASC (Khot et al., 2020) tests scientific commonsense, requiring models to compose two facts from a large science corpus. In zero-shot settings with T5-11B as the inference model, GKP achieves the following gains over the vanilla baseline: NumerSense improves from 64.05% to 72.47%, a gain of roughly 8.4 percentage points; CommonsenseQA 2.0 improves from 39.89% to 47.26%, a gain of roughly 7.4 percentage points; QASC improves from 44.89% to 55.00%, a gain of roughly 10 percentage points. On finetuned models, GKP improves NumerSense from 67.5% to 78.0% with T5-11B, and QASC from 76.74% to 80.33% with UnifiedQA-11B, setting new state-of-the-art on both. The paper also compares GKP against retrieval-based knowledge, which queries external corpora for relevant passages. On NumerSense, retrieval-based knowledge improves the T5-11B baseline by 0.18% to 1.02%, while GKP improves it by 7.37% to 8.83%, demonstrating that generated knowledge is substantially more useful than retrieved passages on this task.

What the results say about language models as knowledge sources

The core finding of the paper is not just that GKP works, but what its success implies about how to think about language models. Prior work framed the question as: does a model need external knowledge, or can it rely entirely on what it learned during pretraining? GKP dissolves the dichotomy. The external knowledge comes from a language model rather than a structured database, so there is no retrieval index to build, no entity linking to maintain, and no schema to define. The knowledge generator acts as a flexible, queryable source of world knowledge that can respond to any question phrasing in natural language, not by returning database records. This framing anticipates later work on retrieval-augmented generation, chain-of-thought prompting, and self-consistency, all of which involve using an LLM's own intermediate outputs as reasoning artifacts rather than treating every inference as a single-step black box. Three factors the paper identifies as driving performance are the quality of generated knowledge, the quantity of knowledge statements per question (more tends to be better up to the tested range of 20), and the integration strategy (confidence-based selection outperforms simply concatenating all statements or using a mixture-of-experts approach).

The practical implication for practitioners is straightforward. Before asking a model to answer a difficult factual or commonsense question, prompt it separately to generate relevant facts about the topic, then include those facts in the final answering prompt. The two-step cost is usually modest, and the accuracy gains on knowledge-intensive tasks can be substantial. The technique works even with smaller knowledge generator models: the paper shows that GPT-J at 6 billion parameters can generate useful knowledge statements for the inference step, which means the approach does not require access to the largest available models to show gains. PromptingIndex covers GKP alongside self-consistency, chain-of-thought, and other inference-time prompting techniques. GKP occupies a distinct position among them because it does not ask the model to show reasoning steps or vote across multiple outputs. It asks the model to externalize its knowledge before reasoning, then reasons with that knowledge in hand. The result is a cleaner separation between knowing and reasoning, and a concrete mechanism for giving any inference model access to facts it may otherwise apply inconsistently.