Summary
The Chinchilla paper revisits the question of how to split a fixed FLOP budget between parameters and training tokens, using over 400 training runs from 70M to 16B parameters on 5B to 500B tokens. Three independent estimation methods agree that model size and tokens should scale in equal proportion, so contemporary LLMs trained on ~300B tokens (GPT-3, Gopher, MT-NLG) were badly undertrained. To test the prediction they trained Chinchilla, 70B parameters on 1.4T tokens, with Gopher’s compute; it beats Gopher (280B), GPT-3, Jurassic-1 and MT-NLG 530B across the board and reaches 67.5% on MMLU. The result reset the field’s training recipe toward more data per parameter and cheaper inference.
Key points
- Compute-optimal rule: for every doubling of model size, double the training tokens (N_opt ∝ C^0.5, D_opt ∝ C^0.5), versus Kaplan’s 5.5x model / 1.8x data per 10x compute.
- Three methods — fix model size and vary tokens; IsoFLOP profiles; fit a parametric loss L(N,D) = E + A/N^α + B/D^β — all give the same answer.
- Why Kaplan differed: a fixed learning-rate schedule and token count for all runs overestimates loss for short runs; matching the cosine schedule to the token budget fixes it. Kaplan also used few models above 1B.
- Chinchilla: 70B params, 1.4T tokens, same FLOPs as Gopher; outperforms Gopher on nearly every task, 67.5% MMLU (>7 points over Gopher). Smaller model means cheaper fine-tuning and inference.
- Limitations: only two comparable large runs; some concavity in the frontier suggests the optimal size may still be overestimated; all runs are under one epoch.
- The authors argue further scaling now hinges on dataset size and quality, not parameters, and flag train–test overlap and toxicity/privacy of larger web scrapes.
Relevance
- Scaling Laws — the corrective to Scaling Laws for Neural Language Models: same question (how to split compute across N and D), different answer. Together they define the “Chinchilla-optimal” ~20 tokens/parameter heuristic the concept page should carry.
Notes
(batch ingest; no notes yet)