Disaggregated prefill and decode — why one GPU pool may not fit both jobs

Yesterday we explored speculative decoding: a cheap draft proposes several tokens, and an expensive target verifies them together. That optimization exploits a fundamental asymmetry in Transformer inference.

The same asymmetry motivates today's serving architecture: disaggregated prefill and decode.

The intuition is:

Reading a long prompt and generating its continuation are different computational jobs. Why insist that the same GPU handle both?

Instead of assigning one worker to a request from beginning to end, a serving system can use one pool of GPUs for prefill and another for decode, transferring the request's KV cache between them.

Two phases, two performance profiles¶

Recall the autoregressive inference loop:

Prefill: process the input prompt, build its KV cache, and compute the distribution for the next token.
Decode: repeatedly generate the next token, reusing the cached keys and values.

A long prefill involves substantial matrix multiplication over many prompt positions. It often achieves high arithmetic intensity: many operations are performed per byte fetched from memory. The decode phase typically handles far fewer new positions per step, while repeatedly reading model weights and historical KV states.

This leads to a useful—but not universal—rule of thumb:

\[ \boxed{\text{prefill is often compute-heavy; decode is often memory-bandwidth- or latency-sensitive}} \]

Batch size, context length, attention implementation, and hardware can change which resource dominates. Still, the two phases frequently benefit from different scheduling policies and GPU allocations.

Why mixing them can hurt¶

Consider a server handling two requests:

Request A: 12,000-token prompt, waiting for prefill
Request B:  2,000-token prompt, already decoding

If A's prefill occupies the GPU for a long uninterrupted interval, B's next token may be delayed. That matters because users notice pauses between streamed tokens, not just total completion time.

Day 66's chunked prefill helps by dividing A's prompt processing into smaller schedulable pieces. Disaggregation goes further: it places prefill and decode on different workers, so a burst of long prompts need not directly interrupt decode on the same GPU.

A simplified architecture looks like this:

incoming request
       |
       v
  prefill pool ---- build prompt KV cache ----+
       |                                      |
       |           transfer KV cache          |
       +--------------------------------------v
                                         decode pool
                                              |
                                       stream output tokens

The transfer is essential. The decode worker needs the exact attention history computed during prefill; it cannot continue from just the prompt's last hidden state.

The hidden price: moving the KV cache¶

From Day 65, an approximate BF16 KV-cache size is

\[ M_{KV}=2LTH_{kv}d_hb, \]

where \(L\) is layer count, \(T\) is prompt length, \(H_{kv}\) is the number of KV heads, \(d_h\) is head dimension, and \(b\) is bytes per element. The factor of two is for keys and values.

Worked example. Take a model with

\[ L=32,\quad H_{kv}=8,\quad d_h=128,\quad T=4096,\quad b=2. \]

Then

\[ M_{KV}=2(32)(4096)(8)(128)(2) =536{,}870{,}912\text{ bytes}=512\text{ MiB}. \]

Suppose the prefill and decode workers communicate over a 200 Gb/s link. Its theoretical maximum is about 25 GB/s. Ignoring protocol overhead and contention, transferring this cache takes at least

\[ t_{\mathrm{transfer}}\geq \frac{536{,}870{,}912}{25\times10^9} \approx 21.5\text{ ms}. \]

Actual transfer time can be higher because of effective bandwidth, synchronization, serialization, memory copies, and competing traffic. With longer prompts, larger KV dimensions, or weaker interconnects, this cost grows.

Notice the connection to KV-cache quantization: compressing KV states can reduce transfer bytes as well as GPU memory usage. But both sides must agree on the representation, and quantization or decompression introduces its own work.

Does disaggregation actually reduce latency?¶

Not necessarily. Imagine the following illustrative timings for one request:

Prefill computation: 120 ms.
First decode step: 20 ms.
KV transfer between pools: 22 ms.

With no queueing, a colocated worker might deliver the first token in approximately

\[ TTFT_{\mathrm{colocated}}=120+20=140\text{ ms}. \]

If the disaggregated system performs the first decode step after transferring the cache, its time is approximately

\[ TTFT_{\mathrm{split}}=120+22+20=162\text{ ms}. \]

The split is slower for this isolated request. We added communication without removing any computation.

Now suppose a mixed workload causes 150 ms of queueing on the colocated worker, while dedicated prefill and decode workers have negligible queues. The comparison becomes roughly 290 ms versus 162 ms. Disaggregation wins because it reduces interference and waiting, not because moving KV states is inherently fast.

This simplified example assumes the same compute times and one particular first-token handoff design. Real systems may sample the first token on the prefill worker, overlap communication with computation, or use different batching policies.

A useful break-even approximation is

\[ \boxed{\text{queueing and interference saved} >\text{transfer and coordination overhead added}.} \]

That inequality captures latency, but throughput and hardware cost also matter.

Why separate pools can improve throughput¶

A colocated worker must accommodate both large prefill jobs and many tiny decode steps. A disaggregated service can independently tune:

Prefill capacity: how many prompt tokens per second the prefill pool processes.
Decode capacity: how many output tokens per second the decode pool sustains, subject to latency targets.
Routing and admission: which requests enter each pool and when a KV handoff occurs.

This separation can make performance more predictable under mixed traffic. It can also make scaling more precise: if prompt ingestion is the bottleneck, add prefill capacity; if long generations dominate, add decode capacity.

But dedicated pools can strand resources when demand shifts. If nearly everyone is decoding, idle prefill GPUs may not be available for decode. Both pools generally need model weights, and the service now depends on reliable, high-bandwidth KV transfers. Better isolation is not automatically better utilization or lower cost.

How this fits with our earlier lessons¶

These techniques operate at different layers of the serving stack:

PagedAttention       → allocate KV memory flexibly
Prefix caching       → reuse common prompt KV states
KV quantization      → reduce KV bytes stored or transferred
Continuous batching  → schedule active requests efficiently
Speculative decoding → advance several tokens per target pass
P/D disaggregation   → place prefill and decode on separate workers

They can reinforce one another, but integration is nontrivial. A paged KV cache may consist of many physical blocks that must be transferred and mapped correctly. Prefix-cache hits may avoid some prefill work, changing which pool is busy. Speculative verification can resemble a short prefill-like computation, complicating a rigid separation between the two phases.

Mental model: a restaurant with two stations¶

Imagine a restaurant where one chef both prepares ingredients for new orders and plates dishes for customers already eating a multi-course meal.

A huge new order can interrupt plating, so diners experience long pauses between courses. One solution is chunking: prepare ingredients in smaller bursts. Another is disaggregation: a prep kitchen handles incoming orders while a service station plates the next courses.

The second design reduces interference—but only if ingredients can be delivered quickly. A slow hallway between kitchens, an idle prep team, or a shortage of plating staff can erase the benefit.

KV transfer is that hallway. The right architecture depends on the flow of work, not merely the speed of either station.

The deeper lesson¶

Inference optimization is no longer only about faster kernels or smaller models. It is also about placing different phases of a request on the hardware best suited to them.

\[ \boxed{\text{the fastest individual GPU path need not be the best end-to-end serving system}} \]

Disaggregated prefill and decode may increase isolated-request overhead while improving service-wide latency consistency, throughput, or capacity under mixed workloads. The deciding measurements are time to first token (TTFT), time between output tokens, throughput, GPU utilization, and total cost—not just model FLOPs.

The practical question is: When does the value of isolating two very different workloads exceed the cost of moving their shared state?