Batches and gradient accumulation — what is a training step, really?

A gradient from one sentence is a noisy opinion about how the model should change. A gradient averaged over millions of tokens is a much more reliable vote.

Suppose three examples produce gradients:

\[ g_1,g_2,g_3 \]

A batch update can use:

\[ g=\frac{g_1+g_2+g_3}{3} \]

then:

\[ W_{t+1}=W_t-\eta g \]

For LLMs, it is often more useful to think in tokens per optimizer update than examples per batch.

Concrete scale¶

Suppose: - sequence length = 4096 tokens - 8 sequences/GPU - 256 data-parallel GPUs

Per GPU:

\[ 8\times4096=32768 \]

tokens.

Globally:

\[ 32768\times256=8,388,608 \]

tokens per update.

So “step 83,421” could correspond to hundreds of billions of token exposures.

Why not make batches enormous?¶

Larger batches reduce gradient noise, but beyond some point returns diminish and optimization dynamics change.

Batch size is a real hyperparameter, not merely “as large as fits.”

Gradient accumulation¶

If 64 sequences do not fit on one GPU but 8 do, process eight microbatches of 8 without updating weights:

8 seq → forward/backward → gradients
8 seq → forward/backward → add
...
8 times
        ↓
optimizer update

This approximates the gradient of a much larger batch.

A useful formula:

\[ \boxed{ \text{tokens/update} = S\times b\times D\times A } \]

where: - \(S\) = sequence length - \(b\) = sequences/GPU/microbatch - \(D\) = data-parallel replicas - \(A\) = accumulation steps

Example:

\[ 4096\times4\times128\times8 = 16,777,216 \]

tokens/update.

Learning rate interaction¶

The learning rate controls how aggressively the model responds to the averaged gradient.

Large batches generally produce lower-variance gradients and often support different learning-rate choices than small batches.

Training systems therefore tune batch size, learning rate, warmup, and decay together.

Why token count is the better unit¶

Two “examples” can have wildly different lengths. A 100-token message and an 8,000-token document contribute very different amounts of next-token prediction work.

Hence training budgets are often discussed in terms of total tokens:

\[ D = \text{number of training tokens} \]

and batch size as global tokens per update.

Gradient accumulation does not make memory free¶

Accumulation lets you simulate a larger effective batch without storing all examples simultaneously, but each microbatch still needs a full forward/backward pass.

So increasing accumulation: - reduces peak activation memory for a target batch size, - increases the elapsed work before each optimizer update.

The gradients must also be scaled correctly so that eight accumulated microbatches behave like the intended average rather than an eight-times-larger update.

Synchronization cadence¶

In distributed data parallel training, workers eventually have to combine gradients.

Communication may happen after each microbatch or be delayed/overlapped depending on implementation.

Large systems aggressively overlap: - gradient computation, - reduce-scatter/all-reduce, - loading the next work.

The goal is to hide network time behind GPU arithmetic whenever possible.

Statistical intuition¶

A small batch is like surveying ten voters: responsive, but noisy.

A huge batch is like surveying a million: stable, but expensive and potentially less informative per additional sample.

Optimization works best at an appropriate noise scale, not at “maximum batch at any cost.”

Packing sequences efficiently¶

Real training datasets contain documents of different lengths.

If every short document were padded to the maximum context length, GPUs would waste huge amounts of compute predicting padding.

Training pipelines therefore pack multiple documents into fixed-size token sequences where possible.

The attention mask ensures unrelated documents do not improperly attend across boundaries.

This is another example of a broader rule:

GPU efficiency depends on turning irregular real data into large regular tensor operations.

Batch composition can affect learning¶

A batch is not only a memory/systems concept. Which examples occur together influences the instantaneous gradient estimate.

Training data mixtures may intentionally control proportions of: - web text, - books, - code, - math, - multilingual data, - synthetic data.

So the effective gradient is shaped by both total batch size and the data mixture inside it.

When people say a model was “trained on 10T tokens,” the ordering and weighting of those tokens can matter substantially.

Effective batch size across the whole cluster¶

When people report a batch size such as “4 million tokens,” that is usually the global batch after combining all data-parallel workers and accumulation steps.

A single GPU may see only a tiny slice of that batch at once.

This distinction matters when reproducing training: local microbatch size determines activation memory, while global batch size influences optimization statistics.

Two runs with the same global batch but different numbers of GPUs can therefore have similar optimization behavior yet very different systems behavior.