Summary
Anthropic’s account of manual red-teaming: crowdworkers held open-ended adversarial dialogues with AI assistants to elicit harmful outputs, producing 38,961 attacks that are released as a dataset. Red-teaming was run across 3 model sizes (2.7B, 13B, 52B) and 4 model types: plain LM, HHH-prompted LM, rejection sampling against a harmlessness preference model, and RLHF. Only RLHF models get harder to attack with scale; plain and prompted LMs are flat, and prompting barely helps against adversaries even though it looks good on static evaluations. Rejection sampling is hardest to attack at every size but wins by being evasive. The paper is deliberately transparent about instructions, worker safety, statistics, and uncertainty, and proposes community norms for red-teaming.
Key points
- Scaling: RLHF models become significantly harder to red team as they scale; plain LM, prompted LM and RS show a flat trend. Prompting is not a meaningful adversarial defence.
- Rejection sampling (best of 16 by a harmlessness PM) is the most robust at every scale but achieves it by evasion — harmless without being helpful.
- Dataset: ~39k attacks, an order of magnitude larger than Bot Adversarial Dialogues and against models 10x larger, including RLHF-trained ones; datasheet and pros/cons of release included.
- Success measured two ways: red-teamer self-report and the minimum harmlessness score over the dialogue from a preference model.
- Top attack types (from a tagged 3% sample): discrimination & injustice, hate speech, violence & incitement, non-violent unethical behaviour, bullying & harassment; non-violent unethical requests succeeded most often.
- Gaps the authors flag: no code-generation attacks appeared; roleplay (“4chan mode”) attacks found internally are not in the data; domain-expertise attacks (weapons, chemistry) are hard to grade; bottom-up tagging vs top-down taxonomies is unresolved.
- Manual red teaming is expensive; comparing it with automated LM red teaming (Perez et al.) is named as next work.
Relevance
- Red-Teaming — the primary methods paper: how to run a red team, how to measure attack success, which safety interventions actually resist adversaries, and the open dataset; directly extends the HF blog’s call for collaborative norms.
- InstructGPT is the RLHF baseline this measures; Constitutional AI is the follow-up that tries to remove evasiveness.
- RLHF and Alignment — home concept since 2026-09-10
Notes
(batch ingest; no notes yet)