LIMA: How 1,000 Curated Examples Beat RLHF and Introduced the Superficial Alignment Hypothesis

The dominant view going into 2023 was that aligning a large language model required three stages: supervised fine-tuning on demonstrations, reward modeling from human preference pairs, and reinforcement learning via proximal policy optimization. InstructGPT had established this pipeline in 2022 as the standard recipe for converting raw pretraining capability into a helpful assistant. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, and eleven co-authors at Meta AI and collaborating institutions decided to test how much of that pipeline was actually necessary. Their paper 'LIMA: Less Is More for Alignment,' submitted to arXiv on May 18, 2023 (arXiv:2305.11206) and presented at NeurIPS 2023, showed that a 65B parameter LLaMA model fine-tuned on exactly 1,000 carefully selected prompt-response pairs, with no reward modeling and no RL of any kind, could outperform DaVinci003 (which was trained with RLHF) in 65% of pairwise comparisons judged by humans. The paper introduced the Superficial Alignment Hypothesis to explain the result.

How the 1,000 examples were chosen

The defining feature of the LIMA dataset is not its size but its selection criteria. The authors drew from four sources. Stack Exchange provided high-quality question-answer pairs, filtered to only questions with accepted answers that had accumulated substantial upvotes, ensuring the responses reflected community-validated expertise rather than first-draft replies. WikiHow contributed instructional articles, preprocessed to strip self-referential phrases like 'this article will explain' that would break the illusion of a conversational assistant. A set of hand-selected subreddits contributed posts from communities where thoughtful long-form writing was rewarded by the platform, including communities focused on creative writing, cooking, and general advice. Finally, the authors themselves wrote approximately 250 prompt-response pairs from scratch, focusing on tasks and formats they wanted the model to handle well. Each source required its own preprocessing to remove formatting artifacts. The Stack Exchange examples dropped links to other answers; wikiHow examples stripped cross-references. The shared convention across all sources was that quality was the only filter. The authors explicitly rejected the strategy of scaling to more examples as a substitute for curation.

The training setup: one loss, no reward model

LIMA training used standard supervised fine-tuning with a cross-entropy loss over the response tokens only, not the prompt tokens. The base model was LLaMA-65B, the largest model in the original LLaMA family released by Meta AI in February 2023. The authors added two special tokens to the vocabulary: one to mark the start of a user turn and one to mark the start of an assistant turn. These tokens served as the only structural signal the model received about the conversational format; no further system prompts or formatting instructions were used during training. The full fine-tuning run on 1,000 examples at 65B parameters required modest compute relative to the pretraining cost. No reward model was trained at any stage. No samples were drawn from the live policy during training. No RL hyperparameters were tuned. The entire alignment step was a single supervised learning pass on a static dataset, which the paper compares favorably to RLHF both in engineering complexity and in final output quality.

Human evaluation results: 43% vs. GPT-4, 65% vs. DaVinci003

The primary evaluation was a controlled human study in which annotators compared LIMA outputs to those of other models on a fixed set of test prompts. LIMA responses were rated as equivalent or strictly preferred to GPT-4 responses in 43% of comparisons. Against Bard (Google's conversational AI at the time of submission), the rate was 58%. Against DaVinci003, the version of GPT-3 trained with RLHF that powered early ChatGPT, the rate was 65%. The 65% figure against DaVinci003 is the most pointed result in the paper: LIMA was trained on 1,000 examples with no RLHF, and it outperformed a model that had been aligned using the full RLHF pipeline on a much larger dataset. The authors note that LIMA was not competitive with GPT-4 in absolute terms (43% means LIMA was preferred in fewer than half of comparisons), but the result demonstrated that RLHF was not a prerequisite for producing outputs that humans found useful and well-formatted.

• Zhou et al. (arXiv:2305.11206, submitted May 18, 2023, NeurIPS 2023): 15 authors from Meta AI and collaborating institutions.
• Dataset: 1,000 examples from Stack Exchange (high-upvote accepted answers), wikiHow (instructional articles), curated subreddits, and approximately 250 author-written pairs.
• Training: supervised fine-tuning of LLaMA-65B on response tokens only, no reward model, no PPO, no human preference labels.
• Human evaluation win-or-tie rates: 43% versus GPT-4, 58% versus Bard, 65% versus DaVinci003.
• Scaling ablation: increasing examples from 1,000 to 2,000 did not improve output quality for 7B models; dataset quality dominated dataset size.
• Multi-turn extension: adding 30 multi-turn dialog examples to the 1,000 training examples was sufficient to convert the model to multi-turn conversational behavior.

The Superficial Alignment Hypothesis

The result that 1,000 examples could produce strong alignment led the authors to propose the Superficial Alignment Hypothesis: almost all knowledge that a language model needs to respond helpfully is learned during pretraining, and instruction tuning teaches the model only what subset of that knowledge to deploy and how to present it to a user. The 'superficial' label refers to the alignment step itself, not to the quality of the outputs. The claim is that alignment is a surface-level adaptation over a deep pretraining substrate. The model already knows how to write lists, explain concepts, plan itineraries, and reason through problems because those patterns appeared in pretraining data. Instruction tuning teaches it which register to use when responding to a human, not how to do the underlying task. If this hypothesis is correct, then the primary lever for improving model capability is pretraining quality and scale, and the alignment step should require only enough examples to calibrate the interaction format reliably. The hypothesis has since generated substantial follow-on work testing its boundaries, including papers examining whether it holds at smaller model scales and whether it holds for safety behaviors specifically.

What LIMA changed about alignment research

LIMA's most lasting contribution was reframing the question alignment researchers were asking. Before LIMA, the dominant question was how to make RLHF more efficient: how to reduce the number of human preference labels needed, how to stabilize PPO training, how to prevent reward hacking. LIMA asked whether RLHF was the right abstraction at all, and answered with evidence that the answer was no for a substantial portion of use cases. The paper directly influenced the development of DPO-adjacent methods that treat alignment as a supervised learning problem over a static dataset, confirming that the RLHF pipeline's complexity was partially an artifact of the problem formulation rather than a fundamental requirement. It also established dataset quality as a research variable independent of dataset size, motivating subsequent work on data selection, filtering, and synthetic data generation for alignment. The finding that 30 multi-turn examples were sufficient to unlock dialog behavior became a reference point for practitioners calibrating how many examples of a new capability they needed to include in an instruction dataset. PromptingIndex covers LIMA alongside DPO, RLHF, GRPO, and Constitutional AI in its alignment series on how pretraining capability is converted into the behavior users actually interact with.