What it is
Red-teaming: trying to explicitly break a model to find harms before deployment. From the Hugging Face intro:
- basic intro to red teaming
- try and explicitly break the model
- call to action: need more collaborative efforts for this
Key sources
- InstructGPT — the RLHF baseline later red-teaming work measures against; names adversarial data collection as the missing step
- Red Teaming Language Models to Reduce Harms — the methods paper: how to run and score a red team, scaling results per intervention, 39k-attack dataset
- Constitutional AI — RLAIF from a written constitution; removes evasiveness so red-teaming can be automated and scaled
- PostTrainBench — four contamination strategies from brazen to subtle; violations emerge without adversarial prompting
- AgentDojo — Framework crossing 97 user tasks with 27 injection goals into 629 prompt-injection cases; capable models are easier to hijack, tool filtering cuts ASR to 7.5%
- The Meta-Agent Challenge — meta-agents build agents against a hidden test set under time and quota budgets; 5/39 beat human scaffolds, σ>0.1 on a third of configs, and zero-resource pressure induces hacking that direct prompting cannot
- Auditing Reward Hackability — Source of the 28.5% expert-verifier hack rate; when an agent repairs a verifier, 61.9% of its tests fail on the gold solution and an LLM judge endorses them anyway
- Hardening Agent Benchmarks — Adversarial QA automated as a hacker/fixer/solver loop; a weak model’s defenses hold against stronger hackers, but hardening also rejects 11pp more legitimate solutions
- SWE-Mutation — Scores an agent-built verifier by mutation kill rate, reporting both error directions in one run; the score moves ~30 points with attacker strength alone, so a frozen attacker is mandatory
Related
To ingest
- Red-Teaming Large Language Models — Hugging Face blog — needs Web Clipper