What it is
How base models are turned into assistants. InstructGPT set the SFT, reward model, PPO recipe and showed a 1.3B aligned model beating 175B GPT-3 on preference. Anthropic’s red-teaming paper made adversarial data collection a method and found only RLHF gets harder to attack with scale. Constitutional AI replaced human harmlessness labels with AI feedback against written principles, removing evasiveness. These three are the baseline every later post-training and red-teaming paper measures against.
(agent-drafted from the sources’ summaries, not yet reviewed)
Key sources
- InstructGPT — the SFT, reward model, PPO recipe; 1.3B aligned beats 175B GPT-3 on user preference
- Red Teaming Language Models to Reduce Harms — the methods paper: how to run and score a red team, scaling results per intervention, 39k-attack dataset
- Constitutional AI — RLAIF from a written constitution; removes evasiveness so red-teaming can be automated and scaled
Related
To ingest
(none)