Disaggregated inference — why prefill and decode may belong on different GPUs

Over the last several lessons, we have treated LLM inference as two different workloads hiding behind one API call.

Prefill takes a whole prompt and processes many tokens in parallel. Decode generates one new token per active sequence at a time. We then saw continuous batching try to mix those workloads efficiently on the same serving hardware.

Today we will push that idea one step further:

\[ \boxed{\text{what if prefill and decode should not run on the same GPUs at all?}} \]

This is the idea behind disaggregated inference.

Instead of sending a request to one worker that performs both phases, the serving system can send the prompt to a prefill worker, transfer the resulting KV cache, and then hand generation to a separate decode worker.

The intuition is simple:

If two phases want different things from the hardware, separating them may let us optimize each one independently.

The catch is equally important:

The handoff itself can be expensive.

Understanding that tradeoff ties together almost everything we have learned about modern LLM serving.

Start with the ordinary colocated design¶

A simple inference worker does both phases:

request arrives
      ↓
   PREFILL
      ↓
build KV cache
      ↓
    DECODE
      ↓
stream output tokens

The same GPUs hold the model weights and execute both workloads.

This is attractive because the KV cache never needs to leave the worker.

After prefill computes:

\[ K_{1:T},V_{1:T}, \]

decode simply continues using those tensors in local GPU memory.

But the GPU is being asked to alternate between two very different jobs.

Prefill and decode stress hardware differently¶

Recall our rough matrix-multiplication picture.

During prefill with \(T\) prompt tokens:

\[ [T\times d][d\times d]. \]

There are many token vectors available at once.

The GPU can reuse weights across them and keep its arithmetic units busy.

Prefill therefore tends to have relatively high arithmetic intensity:

\[ I= \frac{\text{FLOPs}}{\text{bytes moved}}. \]

Decode for one sequence looks more like:

\[ [1\times d][d\times d]. \]

There is much less work available per weight load.

At modest batch sizes, decode is often much more sensitive to memory bandwidth.

So, very roughly:

prefill:
lots of parallel arithmetic
→ often compute-oriented

decode:
small amount of work per step
→ often bandwidth-oriented

These are not absolute laws. Model architecture, batch size, context length, kernels, and hardware all matter.

But the distinction is strong enough to influence serving architecture.

The interference problem¶

Suppose a GPU is currently decoding 200 interactive requests.

Every few milliseconds, those users expect another token.

Now a new request arrives with a 100,000-token prompt.

If the same worker begins a huge prefill operation, the decode workload may have to share compute, memory bandwidth, and scheduling time with it.

Conceptually:

before:

GPU:
[decode][decode][decode][decode][decode]

large prompt arrives:

GPU:
[decode][PREFILL PREFILL PREFILL][decode]...

The large prefill can disturb the rhythm of token generation.

Continuous batching and chunked prefill help by interleaving the work more carefully.

Disaggregation asks a more radical question:

Why make them compete in the first place?

Split the serving path¶

A disaggregated design might look like:

                  ┌────────────────┐
prompt ──────────→│ prefill worker │
                  └───────┬────────┘
                          │
                          │ KV cache
                          ▼
                  ┌────────────────┐
                  │ decode worker  │
                  └───────┬────────┘
                          │
                          ▼
                    output tokens

The prefill worker processes the prompt and creates the KV cache.

Then the system transfers the relevant KV state to a decode worker.

The decode worker continues autoregressive generation from that state.

Mathematically, the model has not changed.

We have changed the placement of the computation.

The factory mental model¶

Imagine manufacturing a car.

One stage uses giant stamping presses to shape body panels.

Another stage performs delicate final assembly.

You could build one factory where every production line contains both kinds of machinery.

Or you could specialize:

Plant P:
high-throughput stamping

Plant D:
carefully paced assembly

Each plant can now be designed and scheduled for its own workload.

But there is a new cost:

The half-finished car must travel from Plant P to Plant D.

Disaggregated inference has exactly the same tension.

Specialization helps.

Transportation hurts.

The "half-finished car" is the KV cache.

The KV cache is the handoff state¶

Why can decode start on another worker at all?

Because prefill has already computed the state needed to continue the causal sequence.

For every layer, decode needs the cached keys and values from the prompt.

Very roughly:

\[ \text{prefill output state} = \{K_\ell,V_\ell\}_{\ell=1}^{L}. \]

If those tensors are transferred correctly, the decode worker does not need to recompute the entire prompt.

It can begin with the next token.

So the serving boundary becomes:

tokens
  ↓
prefill
  ↓
KV state
  ↓
network transfer
  ↓
decode
  ↓
tokens

This makes the KV cache not merely a local optimization, but a distributed-systems object.

How large is the handoff?¶

For a simplified Transformer, suppose the KV cache stores keys and values for:

\(L\) layers,
\(T\) cached tokens,
\(N_{\text{KV}}\) key/value heads,
head dimension \(d_h\),
\(b\) bytes per stored element.

Then approximate KV size is:

\[ M_{\text{KV}} \approx 2LTN_{\text{KV}}d_hb. \]

The factor of 2 is for keys and values.

Consider a simplified example:

\[ L=80, \qquad T=16{,}000, \qquad N_{\text{KV}}=8, \qquad d_h=128, \qquad b=2. \]

Then:

\[ M_{\text{KV}} = 2\times80\times16{,}000\times8\times128\times2. \]

That is:

\[ 5{,}242{,}880{,}000\text{ bytes} \]

or roughly:

\[ 5.2\text{ GB}. \]

So the handoff can involve gigabytes of state for a single long request.

This is why disaggregation is not as simple as calling a different server after prefill.

Transfer time creates a break-even condition¶

Suppose transferring the KV cache takes:

\[ T_{\text{transfer}}. \]

Separating prefill and decode is worthwhile only if the benefits of specialization outweigh that transfer and coordination cost.

A simplified latency model is:

\[ T_{\text{total}} = T_{\text{queue,p}} + T_{\text{prefill}} + T_{\text{transfer}} + T_{\text{queue,d}} + T_{\text{decode}}. \]

A colocated system avoids most of:

\[ T_{\text{transfer}}. \]

But its prefill and decode times may be worse because the phases interfere or because the hardware mix is not ideal.

So disaggregation wins when something like:

\[ \text{specialization benefit} > \text{KV transfer + extra scheduling overhead}. \]

There is no universal answer.

It depends on the workload and infrastructure.

Network bandwidth suddenly matters to token generation¶

Suppose our KV cache is:

\[ 5.2\text{ GB}. \]

If effective transfer bandwidth is:

\[ 100\text{ GB/s}, \]

the raw transfer-time lower bound is approximately:

\[ \frac{5.2}{100} = 0.052\text{ s} = 52\text{ ms}. \]

At:

\[ 25\text{ GB/s}, \]

it becomes roughly:

\[ 208\text{ ms}. \]

Real transfers include additional overheads, and effective bandwidth may differ from link headline rates.

But the lesson is clear:

\[ \boxed{\text{the interconnect can become part of time to first token}} \]

Once inference is disaggregated, networking is directly on the critical path between prompt processing and generation.

Why GQA helps twice¶

Earlier we learned about grouped-query attention, or GQA.

With ordinary multi-head attention, there may be many K/V heads.

GQA lets multiple query heads share fewer K/V heads:

\[ N_{\text{KV}} < N_Q. \]

We usually motivate this as a way to shrink the KV cache during decode.

Now look again at:

\[ M_{\text{KV}} \approx 2LTN_{\text{KV}}d_hb. \]

Reducing \(N_{\text{KV}}\) also reduces the amount of state that must cross the network during a prefill-to-decode handoff.

So GQA can improve:

decode memory footprint
KV-cache bandwidth
disaggregated transfer size

One architectural decision helps several systems bottlenecks at once.

KV quantization helps too¶

Day 48 gave us another lever: reduce \(b\), the bytes per KV element.

If we safely quantize the transferred cache from 16-bit to 8-bit storage, then approximately:

\[ b:2\rightarrow1 \]

and the transfer volume roughly halves.

Our simplified 5.2 GB cache becomes about:

\[ 2.6\text{ GB}. \]

Again, there may be conversion overhead and quality considerations.

But this shows how recent lessons compose:

\[ \text{GQA} \rightarrow \text{fewer KV heads} \]

\[ \text{KV quantization} \rightarrow \text{fewer bytes per KV value} \]

\[ \text{both} \rightarrow \text{cheaper disaggregation}. \]

The model architecture and the cluster architecture are coupled.

Why separate pools can improve scheduling¶

Suppose we have 16 identical GPU servers.

A colocated design might let every server perform both phases:

worker 1: prefill + decode
worker 2: prefill + decode
...
worker 16: prefill + decode

A disaggregated design could instead create:

prefill pool:
workers 1–6

decode pool:
workers 7–16

Now the system can scale the pools according to demand.

If prompts become much longer but outputs stay short, we may need more prefill capacity.

If prompts stay short but users request long generations, we may need more decode capacity.

Conceptually:

\[ \text{prefill demand} \propto \text{input-token workload} \]

while:

\[ \text{decode demand} \propto \text{output-token workload and active sequences}. \]

Separating the pools lets us provision those resources independently.

This can reduce head-of-line interference¶

Imagine two requests:

A:
100,000-token prompt
20-token answer

B:
500-token prompt
2,000-token answer

A is prefill-heavy.

B is decode-heavy.

On one colocated worker, A's huge prompt and B's long generation compete for the same resources.

With disaggregation:

A's expensive prefill
→ prefill pool

B's long generation
→ decode pool

The workloads still consume cluster capacity, but they no longer have to interfere on the same worker in the same way.

This can make latency more predictable.

And predictable latency matters greatly for interactive systems.

Different phases can prefer different batch shapes¶

A prefill server wants many prompt tokens available for large matrix multiplications.

It might happily process:

request A: 8,000 prompt tokens
request B: 4,000 prompt tokens
request C: 12,000 prompt tokens

subject to its token budget.

A decode server wants a large number of active sequences:

A → 1 next token
B → 1 next token
C → 1 next token
...
Z → 1 next token

So the natural batching dimensions differ.

Prefill batches across:

\[ \text{many prompt positions}. \]

Decode batches across:

\[ \text{many active sequences}. \]

Separate pools let each scheduler optimize for its own batch geometry.

The hardware does not even have to be identical¶

Once the phases are separate, we can ask a more interesting question:

Should they use the same accelerator type?

Perhaps prefill benefits most from extremely high matrix-compute throughput.

Perhaps decode benefits more from abundant memory capacity and bandwidth.

Then a heterogeneous cluster could conceptually use:

prefill pool:
compute-dense accelerators

decode pool:
memory-bandwidth / memory-capacity optimized accelerators

Whether this is economical depends on actual hardware, kernels, model formats, networking, and fleet complexity.

But disaggregation creates the option.

A colocated worker forces one machine configuration to compromise between both phases.

Prefix caching changes the equation¶

Now connect to Day 50.

Suppose a 20,000-token prompt contains a 16,000-token cached prefix.

The prefill worker may need to compute only the uncached suffix.

Even better, if the required prefix KV blocks already exist where the decode worker can access them, the system may avoid transferring or reconstructing some state.

Conceptually, the ideal becomes:

cached prefix KV ───────────────┐
                               ▼
new suffix → prefill → new KV → decode

This suggests a broader architecture in which KV state has a lifecycle independent of any single request:

compute
cache
share
transfer
evict
reuse

That begins to resemble a distributed memory system.

PagedAttention gives the state manageable units¶

A 5 GB monolithic KV tensor is awkward to move and share.

PagedAttention already divides KV memory into blocks.

That means a system can reason about:

which blocks already exist
which blocks are shared
which blocks are new
which blocks must move
which blocks can be reclaimed

Instead of treating the entire context as one indivisible object.

Again, the virtual-memory abstraction becomes useful beyond its original purpose.

Paging first solved fragmentation.

Then it helped prefix sharing.

Now it also gives us a natural unit for distributed KV management.

Continuous batching still matters inside each pool¶

Disaggregation does not replace continuous batching.

It creates two scheduling domains.

The prefill pool can continuously admit prompt chunks.

The decode pool can continuously add and remove active sequences.

Conceptually:

PREFILL SCHEDULER
─────────────────
prompt A chunk
prompt B chunk
prompt C chunk

        │ KV handoff
        ▼

DECODE SCHEDULER
────────────────
A B D E F G H ...

So instead of one scheduler balancing incompatible workloads, we can have specialized schedulers connected by a state-transfer boundary.

The hard part becomes orchestration¶

Disaggregation creates new failure modes and decisions.

The router must answer questions such as:

Which prefill worker should process this prompt?

Which decode worker has enough KV memory?

Which decode worker is least loaded?

Where is the cached prefix?

Can the destination receive the KV state fast enough?

What if the chosen decode worker fills up before transfer completes?

A locally simple optimization can therefore make the global system more complicated.

This is common in distributed systems.

We remove one kind of contention by introducing coordination.

A useful break-even mental model¶

Imagine two workshops.

Workshop P can do prefill in:

\[ 80\text{ ms} \]

and Workshop D can perform the relevant decode workload in:

\[ 500\text{ ms}. \]

If colocating both phases causes interference that makes them take:

\[ 120\text{ ms} \]

and:

\[ 650\text{ ms}, \]

then separation saves:

\[ (120+650)-(80+500) = 190\text{ ms} \]

of compute time.

If KV transfer plus extra queueing costs only:

\[ 60\text{ ms}, \]

disaggregation is attractive.

If the handoff costs:

\[ 300\text{ ms}, \]

it is not.

The numbers are illustrative, but the decision rule is powerful:

\[ \boxed{ \text{separate phases only when specialization saves more than separation costs} } \]

This is the same logic behind many distributed architectures.

The deeper lesson: "the model" is becoming a pipeline of services¶

At first, it is natural to picture an LLM as one giant function:

\[ y=f_\theta(x). \]

At serving scale, the physical execution can look much more like:

request router
      ↓
prefix-cache lookup
      ↓
prefill scheduler
      ↓
prefill GPUs
      ↓
KV transfer
      ↓
decode scheduler
      ↓
decode GPUs
      ↓
streaming response

The mathematical model may still be one Transformer.

The serving system is not.

It decomposes the model's execution according to the resource behavior of each phase.

That is a major conceptual shift.

The key mental model¶

Think of inference as a relay race.

In a colocated system, one runner must perform both legs:

runner:
sprint uphill
then run a long flat section

The two legs reward different abilities.

Disaggregated inference uses specialists:

prefill specialist
      │
      │ baton = KV cache
      ▼
decode specialist

The specialists may each be faster at their own leg.

But the baton handoff takes time.

A bad handoff can erase the advantage.

That gives us today's main principle:

\[ \boxed{ \text{prefill and decode are different enough that the best place to run them may also be different} } \]

And the deeper connection to the previous lessons is this:

Quantization reduces the size of the state.

PagedAttention divides that state into manageable blocks.

Prefix caching lets state survive and be reused.

Continuous batching schedules work around that state.

Disaggregated inference asks where the state — and therefore the computation — should move next.

Once LLM serving reaches this point, inference optimization is no longer only about making matrix multiplication faster.

It is about designing a distributed computer around the shape of autoregressive computation.