Chinchilla, and why recent models are over-trained on purpose
Build a Small GPT From Scratch · Part 7 of 8
For a fixed budget, should you train a bigger model on less data or a smaller model on more? A famous study gave a clear answer, and then the industry deliberately ignored it. Both choices make sense.
The compute-optimal answer
Hoffmann and colleagues (the “Chinchilla” study, arXiv 2203.15556) trained many models and found that, for a fixed compute budget, the best result comes from growing parameters and training tokens together, at roughly 20 tokens per parameter. Earlier models, guided by Kaplan and colleagues (arXiv 2001.08361), had put most extra compute into model size, which left many of them undertrained for their size.
A formula for the expected loss
The study also fitted a formula for the loss a model should reach: L(N, D) = 1.69 + 406.4 / N0.34 + 410.7 / D0.28. It has three parts: an irreducible floor, a term that shrinks as the model grows and a term that shrinks as data grows. Evaluated for five configurations:
| Configuration | Predicted loss |
|---|---|
| 7B, 140B tokens (20 per parameter) | 2.184 |
| 13B, 260B tokens (20 per parameter) | 2.099 |
| 7B, 1T tokens (143 per parameter) | 2.052 |
| 7B, 2T tokens (286 per parameter) | 2.020 |
| 70B, 1.4T tokens (20 per parameter) | 1.937 |
Why labs over-train on purpose
Look at the orange rows. A 7B model trained on a trillion tokens (about 143 per parameter) is predicted to beat a 13B model trained at the 20-per-parameter optimum: 2.052 against 2.099. It costs roughly twice as much to train (4.2×1022 against 2.0×1022 FLOPs) but is about half the size to run. Training is paid for once. Inference is paid for on every request for the life of the model. Sardana and Frankle (arXiv 2401.00448) formalise this: if you expect heavy use, the best model is smaller and trained for longer than Chinchilla alone says.
That is why recent open models are trained on far more tokens per parameter than 20.
Use it as a guide, not a law
The constants were fitted on one family of models and one data mix. A different tokenizer, architecture or data quality moves them. The formula is good for comparing configurations; it is not a promise about your exact loss.
Takeaways
- For a fixed training budget, growing parameters and tokens together beats growing only the model.
- A good rule of thumb is about 20 tokens per parameter.
- A smaller model trained on much more data can beat a larger compute-optimal one and is cheaper to serve.
- The fitted formula is a guide for comparison, not a precise prediction.
Next in the series (Part 8): Base model or chat model? What your tiny GPT can and cannot do