Why LLM decoding is often memory-bandwidth bound

We have now built up three pieces of the Transformer’s internal computation:

the residual stream as a shared scratchpad,
attention as communication between token positions,
MLPs as large banks of conditional feature transformations.

Today we will connect that architecture to a systems fact that explains a surprising amount about LLM inference:

Generating one token at a time is often limited less by how fast the GPU can multiply numbers than by how fast it can move model weights and KV-cache data through memory.

This is the idea behind saying that LLM decoding is often memory-bandwidth bound.

The key concept is arithmetic intensity:

\[ \boxed{ \text{arithmetic intensity} = \frac{\text{amount of computation}} {\text{amount of data moved}} } \]

If an operation performs lots of math on each byte it loads, GPUs are excellent at it.

If it loads enormous amounts of data but performs relatively little math with each byte, the GPU’s arithmetic units may spend much of their time waiting for memory.

That distinction helps explain why prefill and decode behave so differently.

Start with a simple matrix multiplication¶

Suppose a Transformer layer contains a weight matrix:

\[ W\in\mathbb R^{4096\times4096}. \]

That matrix contains:

\[ 4096^2=16{,}777{,}216 \]

parameters.

If stored in BF16, each parameter takes 2 bytes, so the matrix occupies roughly:

\[ 16.8\text{M}\times2\approx33.6\text{ MB}. \]

Now consider multiplying one token vector by this matrix:

\[ y=xW. \]

The input is:

\[ x\in\mathbb R^{1\times4096}. \]

The operation performs roughly:

\[ 2\times4096\times4096\approx33.6\text{ million FLOPs}. \]

Why the factor of 2? Each matrix element participates in approximately one multiply and one add.

So we have roughly:

weight data read:    33.6 MB
math performed:      33.6 MFLOPs

Ignoring smaller reads and writes, the arithmetic intensity is about:

\[ \frac{33.6\text{ MFLOPs}}{33.6\text{ MB}}\approx1\text{ FLOP/byte}. \]

That is very low. The GPU must stream a huge matrix from memory just to process one token.

Now process 1,000 tokens at once¶

During prefill, the model does not necessarily process one token independently.

Suppose the prompt contains:

\[ T=1000 \]

tokens.

Instead of:

\[ [1,4096]\times[4096,4096], \]

we can compute:

\[ [1000,4096]\times[4096,4096]. \]

The same 33.6 MB weight matrix can now participate in computation for 1,000 token rows.

The FLOPs become roughly:

\[ 2\times1000\times4096\times4096\approx33.6\text{ GFLOPs}. \]

But the weight matrix is still only:

\[ 33.6\text{ MB}. \]

So, looking only at the dominant weight traffic, the arithmetic intensity becomes roughly:

\[ \frac{33.6\text{ GFLOPs}}{33.6\text{ MB}}\approx1000\text{ FLOPs/byte}. \]

Same weights. Much more computation per byte loaded.

That is exactly what GPUs like.

The restaurant-kitchen mental model¶

Imagine a restaurant kitchen with an enormous pantry.

Fetching ingredients from the pantry is slow. Cooking with ingredients already on the counter is fast.

Decode¶

One customer orders one dish. The chef walks through the pantry gathering ingredients:

fetch ingredients
cook one dish
fetch ingredients
cook one dish
...

A lot of time is spent moving ingredients.

Prefill¶

One thousand customers order the same style of dish. The chef brings a large batch of ingredients to the counter once and uses them repeatedly:

fetch ingredients
cook 1,000 dishes

Now the expensive ingredient movement is amortized across much more cooking.

GPU memory is the pantry. Tensor cores are the cooks. Model weights are the ingredients.

Why decode loses this reuse¶

Autoregressive generation has a dependency:

\[ x_{t+1}\text{ cannot be computed until }x_t\text{ has been sampled}. \]

For:

The capital of France is Paris

the model must first generate Paris before it knows the next input token.

So decode proceeds:

token t
   ↓
run model
   ↓
sample token t+1
   ↓
run model
   ↓
sample token t+2

We cannot simply compute 1,000 future generated tokens in one giant matrix multiplication because those tokens do not exist yet.

That destroys much of the batch dimension that makes matrix multiplication efficient.

But batching users restores some reuse¶

Suppose one request is decoding. A projection might look like:

\[ [1,4096]\times[4096,4096]. \]

Now suppose 128 users are decoding simultaneously. The server can batch their current tokens:

\[ [128,4096]\times[4096,4096]. \]

The same weights can now be reused across 128 requests.

This is why batching is enormously important for inference economics.

A GPU serving one interactive user may have poor hardware utilization. The same GPU serving many users simultaneously can amortize weight reads across the batch.

Conceptually:

one request:
load W → use once

128 requests:
load W → use across 128 token vectors

This is also why inference systems use techniques such as continuous batching. They try to keep forming useful batches even though requests arrive and finish at different times.

A rough bandwidth calculation¶

Imagine a model has:

\[ 70\text{ billion parameters}. \]

At 2 bytes per parameter:

\[ 70\text{B}\times2=140\text{ GB}. \]

As a deliberately simplified lower-bound thought experiment, suppose generating one token required streaming essentially all 140 GB of weights once from HBM.

If the hardware provided:

\[ 3.0\text{ TB/s} \]

of usable memory bandwidth, the absolute bandwidth-only time would be:

\[ \frac{140\text{ GB}}{3000\text{ GB/s}}\approx0.0467\text{ s}. \]

That corresponds to only:

\[ \frac{1}{0.0467}\approx21.4 \]

tokens per second.

This is not a prediction of a real system. Real inference depends on tensor parallelism, quantization, batching, caching, kernel efficiency, communication overhead, architecture, memory hierarchy, speculative decoding, and many other details.

But the calculation reveals something important:

Even a GPU capable of astonishing amounts of arithmetic cannot outrun the requirement to move the model’s data.

Quantization now has a second meaning¶

We often think of quantization as: make the model smaller so it fits in memory.

But if decode is bandwidth bound, quantization can also directly improve speed.

Suppose weights move from BF16:

\[ 2\text{ bytes/parameter} \]

to INT8:

\[ 1\text{ byte/parameter}. \]

Our hypothetical 70B model falls from:

\[ 140\text{ GB} \]

to roughly:

\[ 70\text{ GB}. \]

If bandwidth were the only bottleneck, the weight-streaming time could approximately halve.

At 4 bits:

\[ 0.5\text{ bytes/parameter}, \]

the theoretical traffic falls again.

Of course, lower precision may require dequantization or specialized kernels, and not every part of a model is quantized identically.

But this explains why quantization can improve both capacity and throughput.

Attention introduces another bandwidth problem: the KV cache¶

Weights are not the only thing decode must read.

Recall the KV cache.

For every previous token, each attention layer stores key and value vectors.

At decode step \(t\), the new query attends to previous keys:

\[ q_tK_{1:t}^T \]

and uses the resulting attention weights to combine previous values:

\[ \operatorname{softmax}\left(\frac{q_tK_{1:t}^T}{\sqrt{d_k}}\right)V_{1:t}. \]

So as context grows:

t = 100      → read roughly 100 cached positions
t = 100,000  → read roughly 100,000 cached positions

The model weights stay the same size. The KV cache does not.

This means long-context decoding introduces growing memory traffic.

Why GQA helps¶

Earlier we discussed Grouped Query Attention.

Suppose ordinary multi-head attention uses:

\[ H_q=32,\qquad H_{kv}=32 \]

while GQA uses:

\[ H_q=32,\qquad H_{kv}=8. \]

Several query heads now share the same K/V heads.

The KV cache can be roughly four times smaller for that part of the architecture:

\[ \frac{32}{8}=4. \]

That matters not merely because memory capacity is precious. It also reduces the amount of KV data that must be moved during decode.

So GQA is partly a memory-bandwidth optimization.

This gives us two bandwidth terms¶

For decode, a useful simplified picture is:

\[ \boxed{ \text{memory traffic} \approx \text{weight traffic} + \text{KV-cache traffic} } \]

Weight traffic is largely determined by model size.

KV traffic grows with:

\[ \text{batch size}\times\text{context length}\times\text{layers}\times\text{KV-head dimensions}. \]

For short contexts, weight movement can dominate. For sufficiently long contexts or large batches, KV-cache movement becomes increasingly important.

The roofline idea¶

Computer architects often visualize this using the roofline model.

Imagine a graph:

performance
FLOPs/s
   ^
   |
   |                 ───────────── compute ceiling
   |               /
   |             /
   |           /
   |         /
   |       /
   |     /
   |   /
   | /
   +---------------------------------> arithmetic intensity

At low arithmetic intensity, performance rises roughly with:

\[ \text{memory bandwidth}\times\text{arithmetic intensity}. \]

The operation is memory bound.

Eventually the operation reaches the GPU’s maximum arithmetic throughput. Beyond that point, additional arithmetic intensity does not help much.

The operation is compute bound.

So approximately:

\[ \text{attainable FLOPs/s} = \min\left( \text{peak compute}, \text{bandwidth}\times\text{arithmetic intensity} \right). \]

This equation is a powerful way to reason about AI hardware.

Prefill moves right; decode sits farther left¶

Now put LLM inference onto that picture.

Large prefill matrix multiplications have substantial reuse:

\[ \text{high arithmetic intensity}. \]

They move toward the right side of the roofline graph and can make good use of tensor-core compute.

Single-token decode has much less reuse:

\[ \text{low arithmetic intensity}. \]

It lives farther left and is often constrained by bandwidth.

So the same Transformer running on the same GPU can operate in two very different regimes:

PREFILL
many prompt tokens at once
large matrix multiplications
high weight reuse
higher arithmetic intensity
often more compute-heavy

DECODE
one new token per sequence
small matrix dimension on token axis
low weight reuse
plus KV-cache reads
often more bandwidth-heavy

Why “FLOPs” alone can mislead¶

Suppose GPU A advertises:

\[ 2\times \]

the tensor-core FLOPs of GPU B.

You might expect:

\[ 2\times \]

the LLM token-generation rate.

But if decode is limited by HBM bandwidth and GPU A has only:

\[ 1.2\times \]

the memory bandwidth, token generation may improve much less than \(2\times\).

Conversely, a hardware design with exceptional memory bandwidth can perform surprisingly well on decode even if its headline FLOPs are less impressive.

This is why evaluating AI accelerators using only TFLOPs is dangerous.

You also care about:

HBM capacity
HBM bandwidth
interconnect bandwidth
memory hierarchy
supported precisions
kernel efficiency

The workload determines which resource matters.

Why speculative decoding can help¶

Autoregressive dependency is the fundamental reason decode has such a small effective batch along the time dimension.

Speculative decoding attacks exactly that problem.

A smaller draft model proposes several tokens:

Paris is the capital ...

The large model then verifies several proposed positions in parallel.

Instead of invoking the large model for exactly one useful token every time, we try to turn some decode work into a larger verification operation.

Conceptually:

ordinary decode:

large model → 1 token
large model → 1 token
large model → 1 token
large model → 1 token

versus:

speculative:

small model → proposes 4 tokens
large model → verifies 4 positions together

If several draft tokens are accepted, the expensive model produces multiple output tokens per weight-loading pass.

In other words, speculative decoding tries to increase effective arithmetic intensity and amortize the expensive large-model work.

Why inference optimization is really resource optimization¶

This gives us a better framework for understanding many techniques we have discussed.

Quantization¶

Reduces:

\[ \text{bytes moved per parameter}. \]

Batching¶

Increases:

\[ \text{work done per weight load}. \]

GQA / MQA¶

Reduces:

\[ \text{KV-cache bytes}. \]

FlashAttention¶

Reduces unnecessary movement between levels of the memory hierarchy during attention.

Speculative decoding¶

Attempts to get:

\[ \text{multiple generated tokens} \]

from each expensive large-model verification pass.

Tensor parallelism¶

Spreads both computation and model data across accelerators, but introduces communication costs.

These techniques may look unrelated at first.

Many are really attempts to manipulate one of three quantities:

\[ \boxed{ \text{compute}, \quad \text{memory traffic}, \quad \text{parallelism} } \]

The deeper mental model: the Transformer as a factory¶

Think of a GPU as a gigantic factory.

It has an almost absurd number of workers:

tensor cores
CUDA cores
specialized matrix units

But the workers need raw materials.

Those materials arrive from HBM.

During prefill, a truck arrives carrying a weight matrix and thousands of tokens can make use of that delivery. The workers stay busy.

During low-batch decode, the truck arrives carrying a huge weight matrix and perhaps only one token needs it. Then another giant matrix must arrive. Then another.

The factory may have plenty of workers standing ready, but the limiting factor is the loading dock.

That gives us today's central idea:

\[ \boxed{\text{fast arithmetic does not guarantee fast inference}} \]

What matters is whether the workload can feed that arithmetic efficiently.

And that is why the question:

“How many FLOPs does this GPU have?”

is usually incomplete.

For LLM inference, a better set of questions is:

How much data must move?
How often can it be reused?
How large is the batch?
How long is the context?
What precision are the weights?
How large is the KV cache?
How much memory bandwidth is available?

Once you start thinking this way, many seemingly arbitrary choices in modern LLM architecture and serving systems become connected.

They are all trying, in one way or another, to keep an enormous mathematical machine from waiting for its data.