The gradient says which way to move; the learning rate says how far.
A rate that is too low wastes compute. Too high can overshoot or destabilize the model.
Warmup¶
Suppose peak learning rate is:
Instead of starting there, training might ramp:
Why?
Early optimizer statistics are poorly established and internal activations can change rapidly. Warmup prevents aggressive early updates.
Decay¶
Later, the model has useful representations worth preserving.
A gradually decreasing learning rate allows increasingly fine adjustments.
A common shape is cosine decay:
You need not memorize the equation; remember the shape: warm up, spend a long period learning productively, then refine.
What does “converged” mean?¶
Not zero loss.
Language contains genuine uncertainty. For:
Bharath walked into the restaurant and ordered...
many continuations are plausible.
Validation loss matters because the aim is generalization, not perfect memorization.
Perplexity¶
Loss 2.3:
Loss 2.2:
A small loss change can correspond to a meaningful reduction in next-token uncertainty.
This smooth predictability leads naturally to scaling laws.
Why the schedule is tied to the total training budget¶
A learning-rate schedule usually assumes some expected number of training steps or tokens.
If training is planned for 2 trillion tokens, the point at which the learning rate begins to fall is part of the experiment design.
Extending training far beyond the planned schedule with an already tiny learning rate may waste compute; restarting at a high rate can disturb learned structure.
Warmup and scale¶
At the beginning of training: - representations are unstable, - optimizer moments are immature, - gradients may have unusual variance.
Warmup gives the system time to establish reasonable numerical scales before taking large steps.
Why a small loss improvement can matter¶
Cross-entropy is averaged across enormous numbers of predictions.
If a model reduces loss by only 0.01, that tiny average difference is repeated across virtually every token prediction it makes.
At internet scale, seemingly small aggregate improvements can correspond to substantial capability changes.
Validation loss and overfitting¶
Training loss almost always falls if the model continues fitting its data.
Validation loss answers a different question:
Did the model learn reusable structure, or merely become better at this particular training corpus?
That distinction becomes increasingly important when training data are repeated or synthetic.
Token-based schedules¶
At very large scale, scheduling by tokens processed can be more meaningful than scheduling by raw optimizer step count.
If batch size changes during a run, 1000 steps before and after the change may represent different amounts of data.
A token-based progress variable keeps the schedule tied to actual training exposure.
Why late learning rates are small¶
Near the end of a run, the model already represents a huge number of useful regularities.
A large update intended to improve one narrow behavior can unintentionally disturb many others.
Smaller late-stage steps make training behave more like careful refinement.
That idea reappears in post-training, where KL penalties and small policy updates protect the valuable structure inherited from pretraining.
So learning-rate decay is not merely an optimizer trick; it reflects an increasingly conservative attitude toward a model whose parameters now encode a great deal of useful computation.
Warmup also protects distributed optimization¶
At the start of training, a rare large gradient on one worker gets averaged into the global update. With a large learning rate, that early noisy event can affect billions of weights before the model has established stable representations.
Warmup reduces the consequences of these early high-variance updates.
Later, when the model has processed enormous amounts of data, the optimizer statistics and gradient directions are generally better behaved, so larger steps are safer—until decay begins to prioritize refinement.