Scaling Laws and Compute-Optimal Training

Scaling laws are the closest thing large-model training has to engineering equations. They let a lab spend a small fraction of its budget on small models and then predict, before committing tens of millions of dollars, how good a large model will be and how to split compute between parameters and data. They also explain several industry patterns you will see constantly: why small open models are trained on far more data than the "optimal" amount, why labs now talk about scaling inference compute, and why high-quality data is becoming the binding constraint.

Intuition: three knobs and a smooth curve

Training a language model has three main resources: the number of parameters $N$, the number of training tokens $D$, and the compute $C$ you can spend, which is tied to the other two by $C \approx 6ND$. Picture the test loss as a landscape over these knobs. The surprising empirical finding is that the landscape is smooth and regular: across many orders of magnitude, loss falls as a power law in each resource, which on log-log axes is a straight line. A straight line can be extrapolated. That is the whole trick: fit lines on cheap experiments, extend them to expensive ones.

A useful analogy is a student preparing for an exam with a fixed number of hours. Studying from a tiny notebook (few parameters) caps what can be remembered no matter how long you study. Owning a vast notebook but only skimming one chapter (too little data) wastes the notebook. The best result for a fixed number of hours balances notebook size and material covered. Scaling laws quantify that balance.

Power laws, formally

A power law has the form $y = a x^{-\alpha}$ with exponent $\alpha > 0$. Taking logs gives $\log y = \log a - \alpha \log x$, a straight line with slope $-\alpha$. A key property: multiplying $x$ by a constant factor $k$ always multiplies $y$ by $k^{-\alpha}$, regardless of where you start. So "10x more compute reduces reducible loss by a fixed fraction" is the signature of power-law scaling.

Kaplan et al. (2020)

Kaplan and colleagues at OpenAI trained many transformer language models and reported that, when the other factors are not a bottleneck, loss follows

$L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}, \qquad L(D) = \left(\frac{D_c}{D}\right)^{\alpha_D}, \qquad L(C_{\min}) = \left(\frac{C_c}{C_{\min}}\right)^{\alpha_C}$

with $N$ counting non-embedding parameters, $D$ tokens, and $C_{\min}$ the compute used when training efficiently. Their fitted exponents were small: $\alpha_N \approx 0.076$, $\alpha_D \approx 0.095$, $\alpha_C \approx 0.050$. The constants $N_c$, $D_c$, $C_c$ depend on the tokenizer and data and have no deep meaning by themselves. They also proposed a joint form,

$L(N, D) = \left[\left(\frac{N_c}{N}\right)^{\alpha_N/\alpha_D} + \frac{D_c}{D}\right]^{\alpha_D},$

and concluded that, for a growing compute budget, model size should grow quickly, roughly $N_{\text{opt}} \propto C^{0.73}$, with data growing more slowly. Other findings held up well: architecture details such as depth versus width matter much less than total parameter count within a broad range, and larger models are more sample-efficient. This paper shaped the 2020 to 2021 era of very large models trained on comparatively few tokens; GPT-3, for example, had 175B parameters trained on about 300B tokens.

Chinchilla: Hoffmann et al. (2022)

DeepMind revisited the question by training over 400 models from 70 million to over 16 billion parameters on 5 to 500 billion tokens. A key methodological difference is that they adjusted the learning-rate schedule to match each run's length; Kaplan's runs used a fixed schedule length, which made shorter training look worse than it is and biased the conclusions toward larger models. Using three separate methods (envelopes of training curves, IsoFLOP profiles, and a parametric fit), they found the compute-optimal model size and data both scale as roughly the square root of compute:

$N_{\text{opt}} \propto C^{a}, \quad D_{\text{opt}} \propto C^{b}, \quad a \approx b \approx 0.5$

(their three approaches gave $a$ = 0.50, 0.49 and 0.46). The parametric approach fits

$\hat{L}(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$

with $E = 1.69$, $A = 406.4$, $B = 410.7$, $\alpha = 0.34$, $\beta = 0.28$. Each term has a clean interpretation: $E$ is the irreducible loss (the entropy of natural text under this tokenizer, which no model can beat), $A/N^\alpha$ is the penalty for a finite model, and $B/D^\beta$ is the penalty for finite data.

The practical upshot is the widely quoted rule of thumb of about 20 training tokens per parameter for compute-optimal training. To test it, they trained Chinchilla: 70B parameters on 1.4 trillion tokens, using the same compute as their earlier Gopher model (280B parameters, 300B tokens). Chinchilla, four times smaller and trained on about four times more data, outperformed Gopher on most evaluations, and being smaller it was also cheaper to serve.

Worked example 1: compute-optimal allocation

Suppose you have $C = 10^{24}$ FLOPs. Using $C = 6ND$ and $D = 20N$:

1. Substitute: $C = 6N(20N) = 120N^2$.
2. Solve: $N = \sqrt{C/120} = \sqrt{10^{24}/120} = \sqrt{8.33\times10^{21}} \approx 9.13\times10^{10}$, about 91B parameters.
3. Data: $D = 20N \approx 1.83\times10^{12}$, about 1.8T tokens.
4. Check: $6 \times 9.13\times10^{10} \times 1.83\times10^{12} \approx 1.0\times10^{24}$. Correct.

Note the square-root behaviour: 100 times more compute gives 10 times more parameters and 10 times more tokens.

Worked example 2: comparing Gopher and Chinchilla with the fitted law

Plug the published sizes into the parametric fit.

Chinchilla, $N = 7\times10^{10}$, $D = 1.4\times10^{12}$:

• Model term: $406.4 / (7\times10^{10})^{0.34} \approx 0.083$.
• Data term: $410.7 / (1.4\times10^{12})^{0.28} \approx 0.163$.
• Predicted loss: $1.69 + 0.083 + 0.163 \approx 1.937$.

Gopher, $N = 2.8\times10^{11}$, $D = 3\times10^{11}$:

• Model term: $406.4 / (2.8\times10^{11})^{0.34} \approx 0.052$.
• Data term: $410.7 / (3\times10^{11})^{0.28} \approx 0.251$.
• Predicted loss: $1.69 + 0.052 + 0.251 \approx 1.993$.

Gopher's huge model term advantage is swamped by its data deficit. The fit predicts Chinchilla wins at roughly equal compute, which is what the evaluations showed. (Later work re-analysing the paper's data argued that the published parametric constants are imprecise; treat them as illustrative and refit on your own setup.)

import numpy as np

E, A, B, alpha, beta = 1.69, 406.4, 410.7, 0.34, 0.28

def loss(N, D):
    return E + A / N**alpha + B / D**beta

def best_split(C, grid=400):
    # Search model sizes on a log grid; tokens follow from C = 6 N D.
    Ns = np.logspace(8, 13, grid)
    Ds = C / (6 * Ns)
    L = loss(Ns, Ds)
    i = np.argmin(L)
    return Ns[i], Ds[i], L[i]

for C in [1e21, 1e22, 1e23, 1e24, 1e25]:
    N, D, L = best_split(C)
    print(f"C={C:.0e}  N*={N:.2e}  D*={D:.2e}  tokens/param={D/N:5.1f}  loss={L:.3f}")

Run this and you will notice two things. First, the optimal tokens-per-parameter ratio drifts upward with compute (from about 49 at $10^{21}$ FLOPs to about 120 at $10^{25}$), because the fitted exponents differ: $N_{\text{opt}} \propto C^{\beta/(\alpha+\beta)} \approx C^{0.45}$. Second, the published parametric constants imply considerably more tokens per parameter than the paper's other two methods, which is exactly the inconsistency that later replication work flagged. The 20:1 figure comes from the IsoFLOP-style analyses and is a convenient summary, not a constant of nature; refit on your own data before relying on any of it.

Over-training for inference efficiency

Chinchilla optimality minimises training compute for a target loss. Real deployments also pay for inference, every token for every user, often for years. A smaller model is cheaper and faster to serve and fits on smaller hardware. Since loss keeps falling (with diminishing returns) as you add data, it is often worth training a smaller model far past 20 tokens per parameter. Meta's Llama 3 8B, for instance, was trained on about 15 trillion tokens, nearly 1,900 tokens per parameter. Sardana and colleagues formalised this trade-off by adding expected inference demand to the cost function: the more tokens a model will serve over its lifetime, the smaller and longer-trained the optimal model becomes.

The cost of over-training is a training-compute premium. From the parametric fit, to reach a given loss with a smaller $N$ you must push $D$ much further, because the data term shrinks only as $D^{-0.28}$. Teams accept this because training is a one-time cost while inference is recurring.

Emergent abilities: a live debate

The case for emergence. Wei et al. (2022) defined an ability as emergent if it is absent in smaller models but present in larger ones, and documented dozens of tasks (multi-step arithmetic, some BIG-Bench tasks, chain-of-thought prompting benefits) where performance sat near chance until a scale threshold and then rose sharply. If true, emergence means capabilities cannot be forecast by extrapolating small-model trends, which matters for both planning and safety.

The case against. Schaeffer, Miranda and Koyejo (2023) argued that much apparent emergence is a property of the metric, not the model. Exact-match accuracy on a five-digit addition is all-or-nothing: if per-token accuracy improves smoothly from 0.5 to 0.9, the probability of getting all five digits right goes from $0.5^5 \approx 0.03$ to $0.9^5 \approx 0.59$, which looks like a sudden jump. Re-scoring with continuous metrics such as token-level log-likelihood or edit distance often reveals smooth, predictable improvement. They also induced "emergence" in vision models simply by choosing discontinuous metrics. They did not claim emergence can never happen.

Where this leaves practitioners. Both observations are useful. Underlying loss improves smoothly, but the task metrics users care about are frequently thresholded, so usefulness can still appear abruptly. Forecast with continuous proxies, track many tasks, and do not assume a capability is absent just because small models fail at it.

Test-time compute scaling

Scaling laws originally concerned training. Since 2024 a second axis has become central: spending more compute at inference. Options include sampling many candidate answers and taking a majority vote (self-consistency), best-of-$n$ selection with a verifier or reward model, tree search, and training models to produce long chains of thought before answering (reasoning models). Snell et al. (2024) studied how to allocate inference compute optimally and found that, on problems of moderate difficulty, adaptively spending test-time compute can outperform a model many times larger, while for the hardest problems more pretraining is still more effective. Reasoning-model reports show accuracy rising roughly log-linearly with the number of thinking tokens on math and coding benchmarks. The practical consequence is a new budgeting question: for a given latency and cost target, is it better to call a larger model once or a smaller model with more thinking? The reasoning-models lesson covers the mechanics.

Data constraints

Chinchilla-style scaling assumes unlimited fresh data. High-quality public text is finite. Villalobos et al. estimated that, at historical growth rates, frontier training runs could exhaust the stock of public human-generated text within roughly the late 2020s, with wide uncertainty. Responses include:

• Repetition. Muennighoff et al. (2023) found that with limited unique data, training for up to about four epochs costs very little compared with fresh data, while many more repetitions bring rapidly diminishing returns. They proposed a data-constrained scaling law that discounts repeated tokens.
• Better filtering, which improves the value per token (see FineWeb-Edu in the pretraining lesson), effectively shifting the scaling curve.
• Synthetic data, generated and verified by models, especially for math and code where correctness can be checked automatically.
• New modalities (images, audio, video) and licensed proprietary data.

Common pitfalls

• Treating 20 tokens/parameter as a law. It depends on data quality, architecture and the fitting method, and is the wrong target if inference cost matters.
• Mixing parameter definitions. Kaplan excluded embedding parameters; Chinchilla counted all. Mixture-of-experts models need the active parameter count for compute and the total count for memory.
• Extrapolating from too narrow a range. Fit over at least two or three orders of magnitude of compute and check residuals.
• Using a fixed learning-rate schedule across run lengths, which biases fits (the Kaplan versus Chinchilla discrepancy).
• Forecasting thresholded metrics directly. Fit loss or log-likelihood, then map to task accuracy.
• Ignoring data repetition. Losses predicted for unique tokens will be optimistic once you start repeating data many times.

In practice

Labs run small "scaling ladders" (for example models from 50M to 2B parameters at several token counts) for every major change of architecture, data mixture or tokenizer, fit $L(N, D)$, and choose the final model size from the fitted curve plus an inference-cost model. Teams training small models for deployment deliberately over-train by one to two orders of magnitude beyond Chinchilla-optimal. When you choose a model to deploy, the scaling view suggests three questions: how many tokens was it trained on relative to its size, how much test-time compute does it use per answer, and is your task's metric smooth or thresholded? For hardware sizing of the models that result, see the site's AI Index methodology.