Constitutional AI: How Anthropic Trained a Harmless Assistant Without Human Labels for Harm

Training a language model to be harmless typically requires tens of thousands of human-labeled examples of harmful outputs. Crowdworkers read sampled responses, flag the ones that are problematic, and those labels drive a preference model that penalizes harm during reinforcement learning. The process is expensive, slow to iterate, and produces a hidden objective: no one can easily summarize what forty thousand crowd labels actually encode. In December 2022, Anthropic published a paper that cuts this dependence almost entirely. Constitutional AI, or CAI, uses a short list of natural language principles, a model that can critique and revise its own outputs, and AI-generated preference labels in place of human ones. The result, called RL-CAI, is preferred by crowdworkers over models trained with human feedback for harmlessness, and it is virtually never evasive.

The paper and its context

The paper is titled 'Constitutional AI: Harmlessness from AI Feedback' (arXiv:2212.08073v1) and was submitted on December 15, 2022 by Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, and more than thirty co-authors at Anthropic. It builds directly on an earlier Anthropic paper by Bai et al. from 2022 that established the helpful and harmless (HH) RLHF baseline. That baseline had a well-documented problem: training intensively for harmlessness caused models to become evasive. Once a model encountered a sensitive query, it would refuse with phrases like 'I cannot answer that' and often stay stuck in refusal mode for the rest of the conversation. CAI was designed specifically to eliminate that pattern while also reducing the need for human labels identifying harmful content.

The two-phase training process

CAI runs in two sequential phases. The supervised learning (SL) phase begins with a helpful RLHF model that has not been trained for harmlessness. This model generates responses to red-teaming prompts, prompts specifically designed to elicit harmful or toxic outputs. The same model then critiques each response using a principle from the constitution, for example: 'Identify specific ways in which the assistant's last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.' It then revises the response to remove the identified harm. The paper used four revisions per prompt, and found that harmlessness scores improved with each revision. On large models, the critique step did not add noticeable benefit beyond the revision itself, though it improved results on smaller models.

The dataset of revised responses is used to fine-tune the model, producing what the paper calls SL-CAI. For helpfulness prompts, 135,296 crowdworker-collected examples were included to prevent the supervised stage from trading away helpfulness. The reinforcement learning (RL) phase then starts from SL-CAI and trains a preference model using AI-generated labels. The SL-CAI model generates pairs of responses to harmful prompts. A separate model (in the paper, the helpful RLHF policy or a pretrained language model) then evaluates which response is less harmful, guided by constitutional principles. These AI preference labels are used to train a preference model, and that preference model becomes the reward signal for RL training. Because the preference labels come from a model rather than humans, this step is called Reinforcement Learning from AI Feedback, or RLAIF.

What the constitution actually contains

The constitution in the paper is a short list of natural language principles, on the order of ten entries. Many of the principles are general purpose and address harmful content broadly. Others target specific domains. Some are drawn from existing documents: the paper notes that principles can be drawn from sources such as the United Nations Declaration of Human Rights. The deliberate design choice is to keep the list small enough that a human can read and evaluate the full training objective in a few minutes. This contrasts with RLHF datasets, where the effective training objective is distributed across tens of thousands of individual labels that no single person can fully audit. The paper used multiple slightly varied constitutions and found that the number of constitutions did not have a significant effect on harmlessness scores, though more constitutions improved the diversity of revised responses and therefore exploration during the RL phase.

Benchmark results from the paper

The main evaluations used Elo scores derived from crowdworker comparison tests and an absolute harmfulness score rated on a scale from 0 to 4, where higher means more harmful. The crowdworkers were explicitly instructed to prefer non-evasive responses over evasive ones when both options were equally harmless, which directly penalized the refusal behavior that prior HH RLHF models exhibited.

• RL-CAI models trained at the 52B parameter scale are significantly more harmless than both the helpful RLHF baseline and the HH RLHF baseline, as measured by crowdworker Elo scores, while remaining competitive on helpfulness.
• RL-CAI with chain-of-thought reasoning applied during the feedback-generation step is slightly less helpful but slightly more harmless than RL-CAI without chain-of-thought, with chain-of-thought significantly improving performance on the harmlessness evaluation task itself.
• SL-CAI, the model produced by the supervised phase alone before any RL, is already both more helpful and more harmless than the raw pretrained model, and more harmless than the helpful RLHF baseline, though less harmless than HH RLHF.
• RL-CAI is virtually never evasive, engaging with sensitive topics by explaining why it declines rather than refusing with a short dismissal. HH RLHF models, by contrast, frequently produce stuck refusal patterns on red-teaming prompts.
• The AI feedback labels used in RLAIF are reasonably well-calibrated: the paper found that log-probabilities from the feedback model used as soft labels (rather than hard 0/1 labels) improved preference model robustness.

Why this matters for scaling AI supervision

The paper frames CAI under a broader goal that Anthropic calls 'scaling supervision': developing techniques that allow humans to oversee AI systems more efficiently, even as those systems become more capable. The argument is straightforward. Collecting human feedback labels at scale is costly, and the quality of labels degrades as the tasks being evaluated become more subtle or require specialized knowledge. If an AI model can evaluate other AI outputs with reasonable accuracy, the supervision bottleneck shifts from the number of human annotators to the quality of the principles guiding the AI evaluator. The constitution is then the interface through which humans express their values, and the chain-of-thought reasoning used during feedback generation makes the AI's decision process legible rather than opaque.

The CAI approach also reduces the iteration cycle when training objectives need to change. Under RLHF, changing what counts as harmful requires collecting new human labels, a process that takes weeks. Under CAI, updating the constitution is a matter of editing a text file and re-running the feedback-generation step. That flexibility has direct implications for the kind of fine-grained control over model behavior that production AI systems require. PromptingIndex covers alignment and training techniques tested across Claude, ChatGPT, and Gemini, including how RLAIF-trained models respond differently than RLHF-trained ones on adversarial prompts.