What are scaling laws in AI?
Scaling laws are empirical findings that a language model's performance improves smoothly and predictably as you increase its size, its training data and the compute used to train it. They let labs forecast how good a model will be before training it, and they explain why AI models have grown so quickly.
Bigger models trained on more data with more compute get better in a way you can predict with a formula. That regularity, more than any single invention, is why billions of dollars flowed into training ever larger language models.
What did the original scaling laws show?
In January 2020, researchers at OpenAI led by Jared Kaplan published "Scaling Laws for Neural Language Models". They trained many models of different sizes and found that the loss, a measure of how well the model predicts the next token, fell as a smooth power law as they increased three things: the number of parameters, the size of the dataset, and the amount of training compute. The trend held across more than seven orders of magnitude. Other details, such as the exact width or depth of the network, mattered much less.
The practical meaning was striking: you could train small, cheap models, fit the curve, and predict the performance of a model a hundred times larger before spending the money.
What is Chinchilla scaling?
Chinchilla scaling is a 2022 correction from DeepMind about how to split a fixed compute budget between model size and training data. Kaplan's results had suggested putting most extra compute into more parameters. DeepMind's Hoffmann and colleagues trained over 400 models and found that parameters and training tokens should grow roughly equally. Their rule of thumb works out to about 20 training tokens per parameter.
To test it, they trained Chinchilla, a 70 billion parameter model on 1.4 trillion tokens, using the same compute as their earlier 280 billion parameter Gopher model. The smaller Chinchilla outperformed Gopher on a wide range of benchmarks. Many earlier large models had been undertrained for their size.
| Finding | Year | Takeaway |
|---|---|---|
| Kaplan et al. scaling laws | 2020 | Loss falls predictably as a power law of parameters, data and compute |
| Chinchilla (Hoffmann et al.) | 2022 | For a fixed budget, scale data and parameters together, about 20 tokens per parameter |
| Overtrained small models | 2023 onward | Training a smaller model on far more data costs more up front but makes every answer cheaper to serve |
| Test-time compute | 2024 onward | Performance also scales with thinking time at answer time, the basis of reasoning models |
Why do labs train past the Chinchilla-optimal point?
Because Chinchilla optimises the cost of training, not the cost of using the model. A model is trained once but may answer billions of requests, and a smaller model is cheaper and faster for every one of them. So labs now often train smaller models on far more tokens than 20 per parameter. Meta reported training its Llama 3 8B model on about 15 trillion tokens, several hundred tokens per parameter, and found it was still improving.
Are there emergent abilities in large models?
This is debated. A 2022 paper by Wei and colleagues described "emergent abilities" that seemed to appear suddenly once models passed a certain size, such as multi-step arithmetic. A 2023 paper by Schaeffer and colleagues argued that many of these jumps come from the metrics used: an all-or-nothing score like exact match turns a smooth underlying improvement into what looks like a sudden leap, while smoother metrics show gradual progress. Both views agree that capability rises with scale; they disagree on whether it arrives in sudden steps.
What do scaling laws mean for AI engineers?
For people building on models rather than training them, scaling laws explain three practical things.
- Do: Model quality keeps improving on a regular cadence, so design systems where swapping in a newer model is easy
- Do: Smaller models are improving fast too, so re-test whether a cheaper model now meets your quality bar
- Do: Extra thinking time is a real lever: a reasoning setting can beat a larger model on hard tasks
- Avoid: Assume the biggest model is always the right choice; for many tasks it is slower and costlier for no gain
- Avoid: Build around a weakness of today's model without an eval to tell you when it disappears
Scaling laws predict the loss on next-token prediction, not success on your specific task. Only your own evals can tell you whether a larger or newer model is worth its price for what you are building.
Frequently asked questions
What is the Chinchilla scaling law?
It is DeepMind's 2022 finding that, for a fixed training budget, model size and training data should be increased in roughly equal proportion, which works out to about 20 training tokens for every parameter.
Are scaling laws still holding?
Pretraining scaling continues to improve models, though gains are increasingly limited by the supply of high-quality data and the cost of compute. Labs now also scale reinforcement learning and test-time reasoning, which have their own scaling trends.
What are emergent abilities in LLMs?
Emergent abilities are capabilities that appear to show up suddenly once a model is large enough. Some researchers argue these jumps are partly an effect of all-or-nothing metrics, and that underlying improvement is smoother.
Does a bigger model always perform better?
On average, across broad benchmarks, yes. On a specific task, a smaller model trained on more data, or a well-prompted mid-sized model, can match or beat a larger one, so test on your own use case.
Last checked for accuracy on . Written by the solidcoder team.