Prefill vs. decode — why LLM inference is really two different workloads

Over the last several lessons, we have learned how large models are spread across GPUs: tensor parallelism, pipeline parallelism, context parallelism, and expert parallelism.

Now let us return to inference.

A surprisingly useful fact about serving LLMs is that a single request contains two computational phases that behave very differently:

During prefill, the model reads the prompt.

During decode, it generates new tokens one at a time.

The same Transformer weights are used in both phases, but the hardware bottlenecks can be almost opposite.

Understanding this distinction explains a great deal about modern inference systems: KV caches, continuous batching, time-to-first-token, tokens-per-second, and even why some serving architectures separate prefill GPUs from decode GPUs.

Start with a simple request¶

Suppose you send an LLM a 4,000-token prompt and ask it to generate 500 tokens.

Conceptually:

4,000 prompt tokens
        ↓
      PREFILL
        ↓
 first output token
        ↓
      DECODE
        ↓
 token 2
        ↓
 token 3
        ↓
 ...
        ↓
 token 500

The first 4,000 tokens already exist.

The model can process them together.

The 500 generated tokens do not exist yet.

Token 2 depends on token 1.

Token 3 depends on token 2.

So generation is inherently sequential:

That single dependency changes the shape of the computation dramatically.

Prefill is a large parallel computation¶

During prefill, suppose the prompt contains:

tokens.

At a Transformer layer, the hidden states have shape roughly:

The QKV projections can process all 4,000 token vectors together:

These are large matrix multiplications.

GPUs love large matrix multiplications.

A simplified GEMM looks like:

When \(T\) is large, there is a great deal of arithmetic available for each time the model weights are fetched from high-bandwidth memory.

This often makes prefill relatively compute-bound.

Decode shrinks one dimension to 1¶

Now suppose the prompt has been processed and we want to generate the next token.

The model receives essentially one new token position:

The same projection now looks like:

Compare:

prefill:

[4000 × d] @ [d × d]

decode:

[1 × d] @ [d × d]

The weight matrix is the same.

But during prefill, thousands of token rows reuse it.

During single-sequence decode, one token uses it.

That difference is enormous.

The warehouse mental model¶

Imagine the model weights are a giant warehouse containing every component needed to build a product.

Prefill is like receiving an order for 4,000 products at once.

You walk through the warehouse once and repeatedly use each shelf:

load shelf A → use for 4,000 items
load shelf B → use for 4,000 items
load shelf C → use for 4,000 items

Decode for one request is like receiving one product order at a time:

load shelf A → use once
load shelf B → use once
load shelf C → use once

Then another token arrives and you walk through the warehouse again.

The arithmetic is no longer the dominant problem.

Moving the warehouse contents past the compute units becomes the problem.

That is why decode is often memory-bandwidth-bound.

Arithmetic intensity¶

We can make this intuition more precise with arithmetic intensity:

High arithmetic intensity means we perform lots of computation for each byte loaded.

Low arithmetic intensity means we move lots of data for relatively little computation.

Consider a weight matrix:

During decode, one token computes:

The matrix \(W\) contains:

weights.

We must access roughly all of them, but each weight participates in only a small amount of arithmetic for that one token.

During prefill with \(T\) tokens:

the same weights can contribute to computations for \(T\) token rows.

Very roughly:

So increasing batch or token count increases weight reuse and pushes the workload toward the GPU's compute limits.

A concrete example¶

Suppose a model has 70 billion parameters stored in BF16.

The weights alone occupy roughly:

bytes:

Imagine, just for intuition, that generating one token required streaming essentially all 140 GB of weights through the hardware once.

If the aggregate memory bandwidth available to the model were:

then the absolute bandwidth-only lower bound would be:

That corresponds to:

Real systems are more complicated: weights may be sharded across GPUs, kernels overlap operations, caches matter, MoE models activate only some parameters, and not every byte behaves like this simplified calculation.

But the mental model is valuable:

During low-batch decode, tokens per second can be constrained by how quickly the system can stream model state through memory.

Batching changes decode¶

Now suppose instead of decoding one request, we decode 128 requests simultaneously.

The matrix multiply becomes roughly:

The same weights are now reused across 128 token vectors.

That increases arithmetic intensity.

So:

This is one of the fundamental reasons high-throughput LLM serving systems batch requests together.

But there is a catch.

Users arrive at different times and have prompts of different lengths.

We cannot simply wait until 128 perfectly aligned requests appear.

This leads to continuous batching.

Continuous batching¶

Imagine three requests:

request A → currently generating token 51
request B → currently generating token 8
request C → just finished prefill

At the next decode step, the serving system can combine their current token positions:

A51
B8
C1
 ↓
one batched model step

Then perhaps request B finishes.

A new request D arrives.

The next batch becomes:

A52
C2
D1
 ↓
one batched model step

The batch is continuously changing.

Hence:

rather than fixed traditional batches.

This lets the server keep the GPU busy while still accepting and completing requests dynamically.

The KV cache makes decode possible¶

Why do we not recompute the entire prompt every time we generate a new token?

Because we store the keys and values from previous positions.

For each attention layer, prefill computes:

and:

These are stored in the KV cache.

When generating token \(T+1\), we compute only:

The new query attends to cached keys:

and cached values:

So decode does not redo all previous Transformer computation.

Instead:

prefill
→ build KV cache

decode step 1
→ read old KV cache
→ append new K,V

decode step 2
→ read larger KV cache
→ append new K,V

decode step 3
→ read still larger KV cache
→ append new K,V

This is why KV-cache management is central to inference serving.

Decode has two major memory streams¶

For every generated token, the system may need to move substantial amounts of:

model weights
+
KV-cache data

The relative importance depends on model size, batch size, context length, attention architecture, quantization, and parallelism.

For short contexts and low batch sizes, weight bandwidth may dominate.

For very long contexts, KV-cache traffic can become increasingly important.

This gives decode an interesting scaling property.

As context length grows:

the amount of historical K/V state that attention must inspect also grows.

So later tokens can be more expensive than earlier ones.

Prefill and decode optimize different user-visible metrics¶

Suppose a user sends a prompt at time:

The first token appears at:

Then subsequent tokens arrive:

Two important metrics are:

Time to first token¶

This includes prompt processing.

Long prompts increase prefill work and therefore often increase TTFT.

Inter-token latency¶

For generated tokens:

This reflects decode speed.

A model can therefore have:

fast prefill + slow decode

or:

slow prefill + fast decode

and users will experience them differently.

For an interactive chatbot, both matter.

Why tokens per second can be misleading¶

Suppose system A has:

TTFT = 0.3 s
decode = 30 tokens/s

and system B has:

TTFT = 2.0 s
decode = 60 tokens/s

For a 20-token answer:

System A takes approximately:

System B takes:

System B advertises twice the decode throughput, but system A feels much faster for this short interaction.

For a 2,000-token generation, the balance changes dramatically.

So inference performance cannot be summarized by one number.

You need to know:

prompt length
output length
TTFT
inter-token latency
throughput
concurrency

Prefill and decode compete for the same GPU¶

Now we reach an important serving problem.

Suppose a GPU is happily decoding 100 interactive users.

Then a new request arrives with a 100,000-token prompt.

Its prefill is a huge compute-heavy operation.

If the server runs that prefill as one monolithic job, existing decode requests may wait.

Their inter-token latency spikes.

From the new user's perspective, this is fine—they need their prompt processed.

From the existing users' perspective, the chatbot suddenly freezes.

So modern schedulers often chunk prefill.

Instead of processing:

100,000 prompt tokens

in one giant operation, the server might process:

chunk 1
decode work
chunk 2
decode work
chunk 3
decode work
...

This sacrifices some scheduling simplicity to protect latency.

Disaggregated prefill and decode¶

Once we recognize that prefill and decode are different workloads, a natural idea appears:

Why must they run on the same GPUs at all?

A serving system can conceptually use:

PREFILL GPUs
optimized for:
- large GEMMs
- prompt throughput
- compute utilization

        ↓ transfer KV cache

DECODE GPUs
optimized for:
- memory bandwidth
- continuous batching
- low inter-token latency

This is called prefill-decode disaggregation.

The difficult part is the arrow.

The prefill workers create a potentially large KV cache.

The decode workers need it.

So the system must transfer:

across machines quickly enough that the separation is worthwhile.

Once again, an optimization creates a communication boundary.

The recurring distributed-systems pattern¶

This should now feel very familiar.

Tensor parallelism:

split matrix computation
→ communicate tensor fragments

Pipeline parallelism:

split layers
→ communicate activations

Context parallelism:

split sequence
→ communicate K/V blocks

Expert parallelism:

split experts
→ communicate tokens

Prefill/decode disaggregation:

split inference phases
→ communicate KV cache

There is a general rule hiding underneath all of these:

The systems question is always whether the savings are larger than the cost of moving that thing.

The deeper mental model: one Transformer, two machines¶

Mathematically, prefill and decode are executions of the same model.

But from the hardware's perspective, they can look like two different applications.

Prefill resembles:

large matrices
high parallelism
high arithmetic intensity
compute-heavy

Decode resembles:

small token dimension
sequential dependency
lower arithmetic intensity
bandwidth-sensitive

This is why the phrase "LLM inference performance" is too broad.

There is no single inference workload.

There are at least two.

And their optimization goals can conflict.

Connect this back to training¶

Training usually processes many tokens simultaneously.

For a training batch:

the effective matrix dimension:

can be very large.

That gives training GEMMs substantial arithmetic intensity.

So, in this particular respect, prefill looks more like training than decode does.

A useful spectrum is:

training
   ↓
huge number of token rows

prefill
   ↓
many token rows

batched decode
   ↓
some token rows

single-request decode
   ↓
one token row

As we move downward, weight reuse decreases and memory bandwidth becomes increasingly important.

That connects the training systems we have studied to inference systems surprisingly cleanly.

The key lesson¶

When you send a prompt to an LLM, do not imagine one uniform computation.

Imagine two machines hidden behind the API:

Machine 1: the reader

takes thousands of known tokens
processes them in parallel
builds the KV cache
wants lots of compute

             ↓

Machine 2: the writer

produces one unknown token
adds it to the context
produces the next
wants fast memory access

They happen to use the same Transformer weights.

But their performance characteristics are fundamentally different.

That gives us today's main principle:

And once you understand that distinction, many modern inference techniques stop looking like isolated tricks.

Continuous batching, KV caching, chunked prefill, quantization, speculative decoding, and prefill-decode disaggregation are all attempts to make one of these two very different machines run more efficiently.