Yesterday we explored disaggregated prefill and decode: separate GPU pools can serve prompt processing and token generation because the two phases stress hardware differently.
Today we'll put numbers behind that claim using the roofline model, a simple way to ask whether a workload is limited by arithmetic or by moving data.
The intuition is:
A GPU can have enormous computational power and still spend most of its time waiting for numbers to arrive from memory.
That distinction explains why batching, KV-cache quantization, and speculative decoding can produce large speedups without making the model perform fewer mathematical operations.
Two ceilings on performance¶
Suppose a computation requires \(F\) floating-point operations and moves \(M\) bytes from a particular level of memory. A device offers peak compute throughput \(P\) FLOP/s and memory bandwidth \(W\) bytes/s.
Its execution time cannot be lower than either of these bounds:
Equivalently, achieved arithmetic throughput is bounded by
Here \(I\) is arithmetic intensity: how much useful math we do for each byte moved. The crossover point is
Below \(I^*\), the memory-bandwidth ceiling is lower. Above it, the compute ceiling is lower. This is a lower-bound model, not a latency predictor: kernel launch overhead, communication, cache behavior, occupancy, and imperfect hardware utilization also matter.
Why decoding one token can be memory-bound¶
For a dense Transformer with \(N\) parameters, a rough forward-pass estimate is
FLOPs per generated token, excluding the context-dependent attention work. Each multiply-add counts as roughly two FLOPs.
Take an illustrative 7-billion-parameter model stored in BF16:
Generating one token requires roughly
If the model weights must be streamed once from high-bandwidth memory for that token, the arithmetic intensity of the weight computation is only
That's surprisingly low. A GPU may have hundreds of trillions of FLOPs per second available, but one request cannot use them efficiently if it must repeatedly read gigabytes of weights for relatively little computation.
Worked example: the same model at different batch sizes¶
Imagine a GPU with 300 TFLOP/s of usable peak BF16 compute and 3 TB/s of peak memory bandwidth. These are illustrative hardware ceilings, not measurements of a specific product.
Its crossover intensity is
For a single token, the idealized lower bounds are:
The memory bound is about 100 times larger than the compute bound. A faster arithmetic unit alone would barely help this simplified case.
Now batch 32 independent decoding requests together. Each request still needs about 14 GFLOPs, but a matrix multiplication can reuse the same model weights across 32 tokens:
The idealized bounds become approximately 1.49 ms for computation and 4.67 ms for weight reads. Total work increased 32-fold, yet the weight-read floor did not. That is why batching can greatly improve tokens per second, even though an individual request may wait longer for its turn.
At batch size 128, the weight-only intensity approaches 128 FLOPs/byte, above our hypothetical crossover. But that does not prove real decode is compute-bound: the KV cache, activations, and other memory traffic have not yet been counted.
The KV cache changes the calculation¶
From Day 65, a BF16 KV cache with 32 layers, 8 KV heads, head dimension 128, and an 8,192-token context occupies 1 GiB per request.
A conventional full-attention decode step may need to read approximately that much historical KV data for each request. At batch size 32, that is roughly 32 GiB of KV reads, in addition to about 14 GB of shared weight reads:
At 3 TB/s, the combined traffic alone gives an idealized floor near
This is not a full performance estimate: attention adds FLOPs, and optimized kernels may reuse or avoid some memory traffic. But it reveals the important shift: as context and batch size grow, KV traffic can dominate even when model-weight reuse is excellent.
It also explains why grouped-query attention (GQA) and KV-cache quantization matter. They reduce the bytes of attention history that decoding must retain and potentially read.
Why prefill looks different¶
During prefill, thousands of prompt tokens are known at once. The GPU can apply each weight matrix to a large matrix of token activations instead of to a single vector.
Ignoring attention's additional cost, a 4,096-token prefill for the same 7B model performs approximately
If weights are read once and reused across those positions, the weight-only arithmetic intensity is roughly 4,096 FLOPs/byte—far above our 100-FLOP/byte crossover. Prefill can therefore make much better use of compute hardware.
There are important caveats: causal attention introduces context-length-dependent work, and intermediate activations, attention kernels, and memory limits complicate the picture. Still, the basic contrast is robust: large matrix-matrix operations in prefill versus smaller, often memory-hungry operations in decode.
How this connects to the last four lessons¶
Technique Resource pressure it addresses
------------------------ -----------------------------------------
KV-cache quantization Fewer bytes of attention history
Continuous batching More reuse of weights across requests
Speculative decoding More output per expensive target pass
Prefill/decode separation Different hardware and schedules per phase
Roofline analysis Explains when each technique can helpThe roofline model also warns against simplistic benchmarks. A change that doubles single-request tokens/sec may not double throughput at large batches. A quantized kernel might save memory bandwidth but spend extra time unpacking values. A larger batch may improve total throughput while worsening time to first token or time between tokens.
Mental model: a factory with fast machines and slow deliveries¶
Imagine a factory with a machine that can assemble 300 parts per second, but a delivery conveyor that supplies only 3 boxes per second.
If each box lets the machine assemble just one part, the machine sits idle. If the factory can reuse the contents of a box to assemble 100 parts, the machine finally becomes the bottleneck.
Batching lets many orders reuse one delivery of model weights. KV quantization makes each delivery of attention history smaller. Prefill resembles a long production run with extensive reuse; single-token decode resembles restarting production for one custom item.
The factory's fastest machine does not determine its output. The bottleneck does.
The deeper lesson¶
When evaluating an inference optimization, first ask which resource is limiting the workload at the actual batch size and context length.
The practical question is not simply, “How many FLOPs does this GPU have?” It is: How many FLOPs can the workload perform for every byte it must move—and how does that ratio change as we batch requests or lengthen their contexts?