Summary
Kaplan et al. measure how Transformer language-model loss depends on non-embedding parameters (N), dataset size (D), and training compute (C), and find clean power laws in each when the other two are not the bottleneck, spanning more than six orders of magnitude. Architecture details (depth vs width) barely matter; scale does. Larger models are more sample-efficient, so for a fixed compute budget the optimal strategy is to train a very large model on relatively little data and stop well before convergence. The paper turned “make it bigger” into a predictive framework and set the pre-Chinchilla recipe of ~300B tokens for ever-larger models.
Key points
- Performance depends strongly on scale (N, D, C), weakly on shape. Depth/width/heads matter little within a wide range.
- Power laws: L(N) ∝ N^-0.076, L(D) ∝ D^-0.095, L(C_min) ∝ C_min^-0.050, no sign of flattening at the top end studied.
- Overfitting is governed by N^0.74 / D: an 8x bigger model needs only ~5x more data to avoid a penalty.
- Training curves are predictable from their early part, roughly independent of model size.
- Transfer to other text distributions is a constant loss offset; it tracks in-distribution loss.
- Compute-optimal allocation: N ∝ C^0.73, batch ∝ C^0.24, steps ∝ C^0.03 — spend extra compute almost entirely on model size, train big models short of convergence; data needs grow only as D ∝ C^0.27.
- Optimal batch size is a function of loss (via gradient noise scale), ~1–2M tokens for the largest models.
- The authors flag the open question of whether smooth loss improvements hide qualitative capability jumps (“more is different”), and that big models may matter more than big data — the claim Chinchilla later overturns.
Relevance
- Scaling Laws — the founding paper for the concept: defines the variables (N, D, C) and the empirical method, and gives the first compute-optimal allocation rule. Read alongside Training Compute-Optimal Large Language Models, which corrects the N-vs-D split.
Notes
(batch ingest; no notes yet)