Summary
The InstructGPT paper shows that fine-tuning GPT-3 with human feedback aligns it with user intent far better than scaling does. The recipe is three steps: supervised fine-tuning on labeler-written demonstrations, training a reward model on labeler rankings of model outputs, then RL (PPO) against that reward model. Labelers prefer the 1.3B InstructGPT over the 175B GPT-3, truthfulness improves (about half the hallucination rate on closed-domain tasks), toxicity drops modestly, and bias does not change. Regressions on public NLP benchmarks (the “alignment tax”) are mostly removed by mixing pretraining-likelihood updates into PPO (PPO-ptx). The whole alignment run costs under 2% of GPT-3’s pretraining compute.
Key points
- Three-step RLHF recipe: SFT on demonstrations → reward model on pairwise comparisons → PPO against the reward model. ~40 contractors; prompts from the OpenAI API plus labeler-written ones.
- 1.3B InstructGPT outputs are preferred to 175B GPT-3 outputs; 175B InstructGPT is preferred to 175B GPT-3 85 ± 3% of the time (71 ± 4% vs few-shot GPT-3). Alignment beats a 100x size increase on the API prompt distribution.
- Truthfulness: about 2x truthful-and-informative on TruthfulQA; hallucination rate on closed-domain tasks 21% vs 41%.
- Toxicity: ~25% fewer toxic outputs when instructed to be respectful; no improvement on Winogender/CrowS-Pairs bias.
- Alignment tax exists (SQuAD, DROP, HellaSwag, WMT regress under plain PPO) and is largely fixed by PPO-ptx, without hurting preference scores.
- Cost: 175B SFT ≈ 4.9 petaflop/s-days, 175B PPO-ptx ≈ 60, vs 3,640 for GPT-3 pretraining.
- Generalizes to held-out labelers, and to non-English and code prompts it was not supervised on.
- Remaining failures: follows false premises, over-hedges; the authors suggest adversarial data collection (labelers hunting worst-case behaviors) as the fix — the seed of red-teaming as a training signal.
- Public NLP datasets (FLAN, T0) are not representative of how the API is used; models fine-tuned on them lose to InstructGPT on real prompts.
Relevance
- Red-Teaming — the baseline harmlessness pipeline that later red-teaming work (Ganguli et al., Constitutional AI) builds on and measures; also names adversarial data collection as the missing piece.
- Scaling Laws — a counterpoint: on user-facing tasks, alignment fine-tuning buys more than a 100x parameter increase at a tiny fraction of the compute, so compute-optimal scaling is not the whole story for capability as users experience it.
- RLHF and Alignment — home concept since 2026-09-10
Notes
The original note was empty.