System 2 Attention: Teaching LLMs to Ignore What They Should Not Read

Ask a large language model a factual question and bury an irrelevant opinion inside the same prompt, and the model will often incorporate that opinion into its answer. Ask it which city is the capital of France and include a passage where someone argues Paris is overrated, and the model may hedge its answer in ways that reflect the embedded sentiment rather than the clear fact. This is not a hallucination in the traditional sense. The model knows the right answer. The problem is that transformer soft attention distributes probability mass across the entire input context, including parts that are logically irrelevant to the question. The model attends to everything, and that indiscriminate attention corrupts the output. Jason Weston and Sainbayar Sukhbaatar at Meta AI published 'System 2 Attention (is something you might need too)' (arXiv:2311.11829, November 20, 2023) to address this failure mode directly, with a prompting technique that forces the model to decide what to read before it answers.

The attention problem behind sycophancy

Transformer attention is often called soft attention because it assigns nonzero weight to every token in the context window. Unlike a lookup that retrieves one specific piece of information, soft attention smears probability across the full input. This is largely a feature: it allows models to integrate complex, long-range dependencies in text. But it also means that statistically prominent tokens, such as repeated words or opinionated phrases, receive more weight than their logical relevance warrants. The paper identifies three distinct consequences. First, models are swayed by irrelevant context even when that context has nothing to do with the question, a phenomenon documented in adversarial reading comprehension benchmarks. Second, models exhibit sycophancy: when the input contains an expressed opinion, the model tends to agree with that opinion regardless of its factual accuracy. Third, soft attention interacts badly with position encoding in ways that treat the context partly like a bag of words, making the model sensitive to which topics appear near each other even when their co-occurrence is coincidental. The name System 2 Attention borrows from Daniel Kahneman's dual-process theory of cognition, in which System 1 is fast, automatic, and associative, and System 2 is slow, deliberate, and effortful. Transformer attention functions like System 1: reflexive and broad. The authors' technique imposes a System 2 pass: a deliberate, reasoned choice about what the model should actually read before it generates its answer.

The two-step mechanism

System 2 Attention (S2A) is a two-step prompting procedure. In the first step, the model receives the original input context and is prompted to regenerate it, keeping only the portions that are genuinely relevant to the query and removing everything that could introduce spurious correlations or opinion. This regenerated context is a new, cleaned version of the input. In the second step, the model receives only the regenerated context and answers the original question based on that cleaned input. The original context is discarded entirely; the model never sees the irrelevant or opinionated sections again when it generates its final response. The first step is implemented as a zero-shot instruction to the model: a prompt that asks the model to extract relevant context and separate it from the query. No few-shot demonstrations are required, and no fine-tuning is needed. The technique works with any instruction-tuned LLM that can follow natural language instructions reliably. The computational cost is roughly double that of a single forward pass, since two LLM calls replace one, but the authors note that the two calls can be routed to different models if latency or cost is a concern, using a smaller model for context regeneration and a larger one for final generation.

Benchmark results on three task types

The paper tests S2A on three categories of tasks, each containing a different kind of problematic context. The first is factual question answering using a modified version of TriviaQA where distractor opinions are inserted into the question context. On this task, S2A increases factuality from 62.8% to 80.3% compared to LLaMA-2-70B-chat using standard attention. The second is math word problems from GSM-IC, a variant of the GSM8K grade school math dataset where each problem is augmented with irrelevant in-topic sentences that do not change the correct answer but are thematically related enough to confuse the model. On GSM-IC, S2A improves accuracy from 51.7% to 61.3%. The third is longform argument generation, where the input contains a sentiment indicator that is irrelevant to the actual argument being requested. On this task, S2A increases objectivity by 57.4% and produces outputs that are largely unaffected by the inserted sentiment. Across all three task types, the common thread is that the model's standard attention mechanism fails by over-weighting context that is statistically salient but logically irrelevant, and S2A's explicit regeneration step filters that noise before it can propagate into the final answer.

• Modified TriviaQA with opinion distractors: factuality rises from 62.8% to 80.3% with S2A on LLaMA-2-70B-chat.
• GSM-IC math word problems with irrelevant sentences: accuracy improves from 51.7% to 61.3%.
• Longform generation with distractor sentiment: objectivity increases by 57.4%.
• No fine-tuning required; S2A is implemented entirely through prompting.
• First step can run on a different (smaller) model than the final answer step.

How S2A differs from summarization and chain-of-thought

System 2 Attention may sound like summarization, and the surface operation is similar, but the purpose and mechanism differ in an important way. Summarization compresses information to reduce length. S2A regenerates context with a specific filter: keep what is relevant to this query, remove what is not. The output of the first step is not necessarily shorter than the input; it is more relevant. A long passage of genuinely useful context should survive the regeneration intact. A single irrelevant sentence should be dropped. Chain-of-thought prompting is a closer cousin in spirit: both techniques add an intermediate reasoning step before the final answer, and both improve multi-step reasoning. The difference is what the intermediate step targets. Chain-of-thought asks the model to reason through the problem. S2A asks the model to filter the input. The two techniques are complementary and can be combined: a chain-of-thought prompt can follow an S2A context regeneration step, giving the model both a cleaner input and an explicit reasoning scaffold before it answers. The paper focuses on the regeneration step alone, but the authors note that S2A is a class of techniques and that the first step can be implemented in various ways depending on the task.

Practical applications and when to use S2A

System 2 Attention is most valuable in settings where the input to the model contains information that is correlated with the query topic but not actually relevant to the correct answer. Customer support pipelines where the user's message includes emotional language that should not affect the factual answer are a natural fit. Legal and medical question-answering applications where the input may contain biased framing or advocacy language benefit from the objectivity that context regeneration enforces. Retrieval-augmented generation pipelines that pull multiple documents, some of which are only loosely relevant to the query, can use an S2A-style first pass to filter retrieved context before the final generation step. The technique is also useful as a sycophancy countermeasure in evaluation and critique tasks: when asking a model to assess the quality of an argument, including the argument itself in raw form can bias the model toward agreeing with its conclusions. A context regeneration step that separates the argument's content from its rhetorical framing can produce more objective evaluations. The core implementation is a single zero-shot prompt asking the model to rewrite the context, keeping only what is relevant to the question. No labeled data, no training, and no specialized infrastructure are required beyond the ability to make two sequential calls to an instruction-tuned model. PromptingIndex covers S2A alongside self-refine, chain-of-verification, and reflexion as part of its series on techniques that add deliberate intermediate steps between input and output to improve reliability.