Summary
Constitutional AI trains a harmless assistant with no human labels for harmfulness; the only human input on harms is a short list of natural-language principles (a “constitution”). Stage one (SL-CAI): sample responses to red-team prompts from a helpful RLHF model, have the model critique and revise them against randomly drawn principles, then fine-tune on the revisions. Stage two (RL-CAI, or RLAIF): have the model compare pairs of responses against principles, train a preference model on those AI labels (plus human helpfulness labels), and run RL against it. The resulting models are Pareto-better on harmlessness vs helpfulness than human-feedback HH models, and crucially are non-evasive: they engage with harmful requests and explain their objections rather than refusing. Chain-of-thought in both stages improves transparency and performance.
Key points
- Two stages: (1) critique → revision → SFT on revised responses; (2) AI preference labels → preference model → RL (RLAIF). Human oversight enters only through ~10–16 written principles plus helpfulness labels.
- Motivations: scale supervision as models pass human level; eliminate the evasiveness that RLHF-trained HH models learned because crowdworkers rewarded refusals; make governing principles explicit; cut iteration time by not recollecting human labels when the objective changes.
- RL-CAI reaches lower harmfulness at a given helpfulness than HH-RLHF (Elo comparisons from crowdworkers); CoT variants help further.
- Critiques are not strictly needed for the SL stage at 52B scale, but they improve transparency.
- Framed as a step toward automated red-teaming: non-evasive harmlessness makes it possible to red-team intensively without collapsing helpfulness, and online RLAIF can keep the PM on-distribution.
- Authors note principles were chosen ad hoc for research and should be set by broader stakeholders; behaviour beyond HHH is still governed by pretraining generalisation.
Relevance
- Red-Teaming — the response to Red Teaming Language Models to Reduce Harms: red-team prompts are reused as the SL-CAI input, evasiveness is the failure it fixes, and it sets up automated red-teaming with AI feedback. Also the origin of RLAIF as an alternative to the InstructGPT recipe.
- RLHF and Alignment — home concept since 2026-09-10
Notes
(batch ingest; no notes yet)