Scaling laws — if you have 10× more compute, what should grow?

A useful dense-Transformer pretraining approximation is:

where: - \(N\) = parameters - \(D\) = training tokens - \(C\) = training FLOPs

The key relationship is:

Double only model size → ~2× compute. Double only tokens → ~2× compute. Double both → ~4× compute.

The design question¶

With fixed compute, should you use: - a huge model on relatively little data, or - a smaller model trained on much more data?

The Chinchilla result made this trade-off famous.

Gopher: - ~280B parameters - ~300B training tokens

Chinchilla: - ~70B parameters - ~1.4T tokens

Despite being ~4× smaller, Chinchilla performed better on many evaluations at similar compute.

The lesson:

A model can have lots of capacity and still be undertrained.

Rule of thumb — with caveats¶

The original Chinchilla analysis suggested a compute-optimal relation in the neighborhood of:

for its setup.

But that is not a universal law. Modern systems may deliberately train smaller models for more tokens because lifetime inference cost matters.

Training optimum vs serving optimum¶

Imagine: - Model A: 200B parameters - Model B: 50B parameters

If B is trained longer, it may approach A’s capability while being much cheaper to serve billions of times.

So the real objective can become:

rather than minimizing one-time pretraining compute.

Scaling laws¶

Validation loss often changes smoothly with model size, data, and compute, approximately following power-law relationships.

That allows smaller experiments to predict larger runs.

Hidden variable: data quality¶

A trillion good tokens and a trillion duplicated/noisy tokens are not equivalent.

So the simple equation hides another variable:

The best training design balances capacity, quantity of experience, and quality of experience.

Deriving the 6ND intuition¶

A dense Transformer forward pass costs on the rough order of a few multiples of:

because each token interacts with many model parameters.

Training adds backward-pass work, giving the commonly used approximation:

The constant is approximate; architecture and implementation details matter. The important part is the multiplicative scaling.

Why overtraining smaller models can make sense¶

Imagine two models with similar quality:

Model A: 100B parameters
Model B: 30B parameters, trained much longer

If both are served for ten billion requests, Model B’s lower inference cost can dominate the extra one-time training expense.

This is a key shift from thinking purely like a research benchmark to thinking like a deployed-system designer.

Data quality and repeated tokens¶

If the useful unique data supply is exhausted, “more tokens” may mean repeating the same examples.

Repeated training can still help, but the marginal value differs from genuinely new information.

Modern training mixtures therefore involve careful choices about: - deduplication, - source weighting, - synthetic data, - curriculum, - high-quality code/math data.

Scaling laws tell you how much capacity and data to consider. They do not decide which data should fill the budget.

Scaling laws are forecasting tools¶

Suppose you can afford only small experiments:

1B parameters
3B parameters
7B parameters

If validation loss follows a smooth empirical curve, you can fit that curve and estimate the likely payoff from a 30B or 70B run before spending the full budget.

That is enormously valuable when a frontier run may cost millions of dollars.

The same method can forecast the value of additional data.

Compute-optimal does not mean product-optimal¶

A research paper may define “optimal” as:

lowest validation loss for this fixed training FLOP budget.

A product organization may instead care about:

Those objectives can favor a very different model.

This is one reason modern model design cannot be understood from parameter count alone. Training and inference economics are coupled.

Scaling-law curves have diminishing returns¶

Power laws mean performance often improves smoothly but with diminishing returns.

If loss behaves approximately like:

then doubling \(N\) does not halve the remaining loss. It removes only a fraction determined by the exponent \(\alpha\).

That is why frontier gains become so expensive: each additional increment of quality may require a multiplicative increase in compute.

Scaling laws are therefore both encouraging and sobering. They say progress is predictable, but they also quantify the rapidly rising price of pushing further along the same curve.