Scaling laws are one of the most critical empirical findings in deep learning. The observation is simple: the training loss decreases predictably as we scale up model size, dataset size, and compute, following a power-law curve. This predictability makes scaling laws highly valuable in practice, allowing researchers to fit on small runs and extrapolate to estimate the requirements for larger models.
The History of Scaling Laws
The predictability of generalization error with scale had already been investigated before scaling laws became a mainstream concept. One of the earliest empirical studies by Hestness et al. explained the relationship between generalization error, model size, and data. Across four different domains, a recurring pattern was observed: generalization error scales as a power law, architecture changes the offset but not the exponent, and the number of model parameters needed to fit a dataset also scales as a power law.
The Kaplan et al. paper popularized the concept of scaling laws in the language modeling community. They found that the cross-entropy test loss scales as a power law with model size, dataset size, and training compute. Their most influential and, in hindsight, most contested conclusion was about compute-optimal allocation. They concluded that model size should grow faster than dataset size.
The Chinchilla Correction
The Chinchilla paper studied this relationship with more careful experimental design and arrived at a different answer. They found that for a compute budget, model size and dataset size should scale equally. This conclusion, that large models at the time were undertrained, was supported by their demonstration that a 70B parameter Chinchilla model trained on 1.4T tokens outperformed a 280B Gopher model trained on 300B tokens.
Data Repetition and Scaling Laws
Classic scaling laws assume effectively infinite data with no repetition. As model size grows, we are running out of high-quality unique tokens, leading to the "data wall" problem. Repetition and multi-epoch training are becoming the norm.
Research by Muennighoff et al. studied how compute should be allocated when training is data-constrained. They found that token value decays with repetition, and their empirical fit suggested allocating more resources to more epochs rather than more model parameters. Later work by Lovelace et al. modeled the interaction between model size and data repetition explicitly, introducing an overfitting penalty term that grows with both the number of epochs and how over-parameterized the model is relative to the unique data available.
The Subtle Art of Fitting
Despite its clean form, scaling law fitting can be surprisingly sensitive to seemingly trivial procedural choices. How you count parameters, how you round precision, and how you sum or average the loss can lead to wild differences in prediction.
The disagreement between Kaplan et al. and Chinchilla is a prime example. A follow-up analysis found that issues like a high loss scale in the minimizer and rounding of coefficients to two digits of precision contributed to the discrepancy. This sensitivity highlights the need for rigor and care when fitting and interpreting scaling laws.