Prefill vs. Decode — why LLM inference has two very different phases

Last time we got to the KV cache: during generation, the model saves the keys and values for previous tokens so it doesn't have to recompute them.

That leads to a surprisingly important fact about LLM inference:

Reading your prompt and writing the answer are fundamentally different computational workloads.

They're called prefill and decode.

Suppose you send:

[5,000-token prompt]

Explain the key risks in this contract.

and the model generates a 500-token answer.

Inference looks roughly like:

          PREFILL                     DECODE

5,000 prompt tokens        token 5001
        │                      ↓
        ▼                   token 5002
process them                 ↓
in parallel                token 5003
        │                      ↓
        ▼                      ...
build KV cache             token 5500

Prefill: reading¶

During prefill, the model processes all 5,000 input tokens.

Transformers are extremely good at this because GPUs can process many token computations simultaneously.

Very roughly:

is multiplied by enormous model weight matrices.

For example:

where \(X\) contains representations for thousands of tokens and \(W\) might contain billions of parameters.

GPUs love large matrix multiplications like this.

The hardware can keep thousands of arithmetic units busy.

So prefill tends to be compute-bound: you're limited substantially by how quickly the GPU can perform arithmetic.

This determines a metric you've probably experienced directly:

TTFT — Time To First Token.

A huge prompt often means a longer wait before the answer begins.

Decode: writing¶

Now suppose the model has generated:

The

To generate the next token, it must run another forward pass:

The primary

Then another:

The primary risk

Then another:

The primary risk is

Generation is inherently sequential because token \(n+1\) depends on token \(n\).

You can't fully compute token 100 before deciding what token 99 is.

This is one reason LLMs don't instantly generate entire paragraphs.

But something even more interesting happens at the hardware level.

During decode, we're doing roughly:

one token
   ×
huge model weights

rather than:

thousands of tokens
   ×
huge model weights

The arithmetic workload per weight loaded is therefore much lower.

The library analogy¶

Imagine a model has hundreds of gigabytes of weights.

During prefill, it's like:

Fetch a giant reference book from the shelf and use each page to answer 1,000 questions at once.

Fetching the book is expensive, but you're getting lots of useful computation from it.

During decode:

Fetch the same giant reference book and use it to answer one question.

Then repeat.

This makes decode frequently memory-bandwidth-bound rather than compute-bound.

The GPU may be capable of performing arithmetic faster than you can feed the model weights to its compute units.

That distinction is enormously important.

A simplified numerical example¶

Imagine a hypothetical model whose weights occupy:

and a GPU with memory bandwidth:

If generating one token required reading all 100 GB of weights once, the theoretical minimum time would be:

So approximately:

Notice something surprising.

We haven't talked about GPU FLOPS at all.

A GPU with twice the arithmetic performance might barely improve this workload if memory bandwidth remains the bottleneck.

This helps explain why inference hardware discussions obsess not only about FLOPS, but also about:

• HBM capacity
• HBM bandwidth
• quantization
• batching
• KV-cache size

They directly affect how efficiently the model can be served.

But batching changes everything¶

Suppose instead of generating one user's next token, the GPU generates the next token for 100 users simultaneously.

Conceptually:

User A ── next token ─┐
User B ── next token ─┤
User C ── next token ─┤
...                   ├─ GPU
User Z ── next token ─┘

Now we load the model weights and use them for many token calculations.

Arithmetic intensity rises dramatically.

This is why batching is one of the central economics tricks of LLM inference.

A provider would rather do:

than perform 100 independent passes through \(W\).

The cost of loading those enormous weights gets amortized across users.

This creates a business trade-off¶

Imagine an inference server with requests arriving continuously.

If it immediately processes every request:

low latency, poor batching

If it waits briefly:

request
request
request
request
request
      ↓
    batch
      ↓
     GPU

it can achieve:

higher throughput, lower cost per token

but potentially worse user latency.

Inference systems therefore continuously balance:

This is one reason serving an LLM efficiently is much more complicated than simply "put the model on a GPU."

Systems dynamically combine requests, manage KV caches, schedule prefills and decodes, and move requests through batches as sequences finish.

Now quantization makes more intuitive sense¶

Suppose we store model weights in 16-bit numbers.

A 70-billion-parameter model requires roughly:

just for weights.

If we can represent them effectively with 4 bits:

Ignoring some practical overhead, that's roughly 4× less weight memory.

That gives two potential wins.

First, the model fits into much less GPU memory.

Second—and crucially for decode—we have much less data to move from memory.

So quantization isn't merely:

"How do we fit the model onto the GPU?"

It's also potentially:

"How do we move the model through memory fast enough to generate tokens?"

That's why low-precision formats such as FP8, FP4 and various integer representations have become so important for inference.

One more subtle consequence¶

There are now two major latency numbers worth distinguishing.

Time to First Token

Dominated heavily by prompt processing/prefill.

And:

Inter-Token Latency

How long you wait between generated tokens.

Dominated heavily by decode.

You've experienced both.

When ChatGPT sits for several seconds before producing anything:

████████████████ prompt
                 ↑
                TTFT

When text starts appearing but slowly:

The ▌ model ▌ generates ▌ tokens ▌ slowly
    ↑       ↑          ↑
       inter-token latency

Those are different performance problems.

The mental model to keep¶

Think of an LLM as a chef.

Prefill: 100 ingredients arrive simultaneously. The chef examines and prepares all of them together.

This is highly parallel work.

Decode: the chef must now produce a 20-course tasting menu, but each course depends on how the previous course turned out.

course 1
   ↓
course 2
   ↓
course 3
   ↓
...

That's sequential.

So:

Once you understand this distinction, a lot of otherwise mysterious LLM infrastructure starts making sense: continuous batching, quantization, speculative decoding, prefix caching, disaggregated prefill/decode, and specialized inference chips are largely attempts to exploit the very different characteristics of these two phases.