Contrastive Chain-of-Thought: Teaching Models What Not to Do

Chain-of-thought prompting asks a model to show its work, and it works remarkably well. But standard CoT only shows the right way to reason. It never tells the model what kinds of mistakes to avoid. Contrastive Chain-of-Thought, introduced in 'Contrastive Chain-of-Thought Prompting' (arXiv:2311.09277) by Yew Ken Chia, Guizhen Chen, Luu Anh Tuan, Soujanya Poria, and Lidong Bing from DAMO Academy (Alibaba Group), Singapore University of Technology and Design, and Nanyang Technological University, addresses that gap directly. The paper was submitted in November 2023. The central idea is simple: pair every correct few-shot reasoning example with a corresponding flawed one that commits a specific, named error. Experiments using GPT-3.5-Turbo show improvements of 9.8 percentage points on GSM-8K math word problems and 16.0 points on Bamboogle factual multi-hop questions compared to conventional CoT.

A surprising flaw in standard chain-of-thought

Before introducing the contrastive technique, the Chia et al. team had to reckon with a disorienting prior finding. Wang et al. (2023) had shown that replacing a CoT prompt's valid reasoning steps with invalid ones, while keeping the format and structure intact, produced almost the same downstream accuracy. That result implies that LLMs learn the surface pattern of 'show reasoning, then give an answer' far more than they internalize the logical content of the rationales. If the logic in few-shot examples barely matters for standard CoT, why should providing negative examples help? The answer the authors propose is that there is a difference between passively ignoring invalid reasoning and actively being confronted with it. Standard CoT asks the model to observe correct examples and generalize. Contrastive CoT asks the model to distinguish between correct and incorrect reasoning paths in the same prompt, which creates a sharper training signal even within the context window.

The contrastive idea: pair every correct example with a wrong one

The method is a direct extension of few-shot CoT. In a standard CoT prompt, each demonstration consists of a question, a valid chain of reasoning, and the correct answer. Contrastive CoT adds a second block to each demonstration: the same question, but now paired with an explicitly invalid reasoning chain and the wrong answer that follows from it. The model sees both blocks for every few-shot example, learning simultaneously what a correct reasoning path looks like and what a specific type of error looks like. The paper draws an analogy to how humans learn: a student studying worked solutions benefits from seeing not only the right approach but also annotated examples of common mistakes. The invalid demonstrations are not random nonsense. Each one commits a named, representative error type so that the model can observe the contrast clearly, not just guess that something went wrong.

Named error types and automatic negative construction

A practical question arises immediately: where do the negative demonstrations come from? Writing them by hand is feasible but time-consuming. The authors address this with an automatic construction method that transforms existing valid reasoning chains into targeted invalid ones. Their analysis identifies three main categories of reasoning error that appear across arithmetic and factual benchmarks.

• Invalid arithmetic: swap a correct arithmetic operation for a wrong one, such as adding where the problem requires multiplication, or dividing instead of subtracting.
• Calculation error: keep the right operation but introduce a deliberate numerical mistake, such as computing 7 times 8 as 54 instead of 56.
• Invalid logic: change the direction of an inference step so the conclusion contradicts the premises stated earlier in the chain.

The automatic construction pipeline applies these transformations to valid reasoning chains taken from existing annotated datasets. Because the transformation rules are deterministic and scoped to specific token positions in the chain, the method scales without requiring a human to write each negative example from scratch. The resulting negative demonstrations are clearly labeled in the prompt, so the model is never confused about which block is the positive example and which is the foil.

Results across reasoning benchmarks

The paper evaluates contrastive CoT on a range of reasoning tasks: arithmetic word problems from GSM-8K and MultiArith, commonsense reasoning from CommonsenseQA and BoolQ, and factual multi-hop questions from Bamboogle. All main experiments use GPT-3.5-Turbo as the base model. The improvements over conventional CoT are substantial.

• GSM-8K (math word problems): contrastive CoT gains 9.8 percentage points over standard CoT with GPT-3.5-Turbo.
• Bamboogle (factual multi-hop): contrastive CoT gains 16.0 percentage points, the largest single-benchmark improvement reported.
• Compatible with self-consistency: stacking contrastive CoT with majority voting over multiple samples yields further gains beyond either technique alone.
• Error analysis confirms that the gains come from fewer mistakes in intermediate steps, not just different surface formatting.

How to apply contrastive CoT to your own prompts

Applying contrastive CoT in practice requires two things: a set of positive demonstrations and a matching set of negative ones. For the positives, use your existing hand-written or dataset-sourced CoT examples. For the negatives, apply the automatic construction rules above, or write targeted foils by hand if your domain has recognizable error patterns. Label each block clearly in the prompt, for example 'Correct reasoning:' followed by the valid chain, then 'Incorrect reasoning (avoidable mistake):' followed by the flawed chain. Keep the number of demonstrations the same as you would use for standard CoT. The prompt grows roughly twice as long per demonstration, which does consume more tokens, so pair contrastive CoT with models that have at least an 8k context window for tasks with several few-shot examples. For domains where the dominant error type is well-defined, such as unit-conversion mistakes in chemistry or sign errors in calculus, writing precise negative examples by hand tends to outperform auto-generated ones because the foil can target the exact failure mode that actually matters. PromptingIndex covers contrastive CoT alongside chain-of-thought, self-consistency, and step-back prompting as part of its series on reasoning-enhancement techniques.