What it is
Basic question: for a fixed amount of compute, how do you effectively scale model parameters, architecture, and data?
Variables: model params including embeddings (M), dataset size (D), compute (C).
Key sources
- InstructGPT — alignment fine-tuning beats a 100x size increase on user tasks at <2% of pretraining compute
- Scaling Laws for Neural Language Models — Kaplan power laws in N, D, C; original compute-optimal rule (N ∝ C^0.73) that favours model size over data
- Training Compute-Optimal Large Language Models — Chinchilla; params and tokens should scale 1:1, existing LLMs were undertrained; corrects Kaplan
Related
To ingest
- New Scaling Laws for Large Language Models - LessWrong — covers DeepMind’s paper (not arxiv; needs Web Clipper)