Disaggregated prefill and decode — why one LLM request needs two kinds of GPU work

Yesterday we explored speculative decoding, which tries to make the sequential part of inference faster by verifying several candidate tokens together.

Today we zoom out from one request to an entire inference service. A surprisingly important design choice is whether the same GPUs should process incoming prompts and generate outgoing tokens.

The intuition:

Prefill and decode use the same model, but they behave like two different workloads. Separating them can make both easier to optimize.

This is called prefill–decode disaggregation (often abbreviated P/D disaggregation).

One request, two phases¶

Suppose a user submits an 8,192-token document and asks for a 256-token summary.

The model does not immediately start producing the summary. First, it processes the prompt and constructs the attention KV cache for its tokens. Then it generates the answer one token at a time.

8,192 prompt tokens
        │
        ▼
     PREFILL
  process known tokens
  construct KV cache
        │
        ▼
      DECODE
  generate token 1
  generate token 2
       ...
  generate token 256

Both phases run Transformer layers, but their performance characteristics differ sharply.

Prefill is often compute-intensive¶

During prefill, all prompt tokens are known. The GPU can execute matrix multiplications over a large token dimension:

\[ XW,\qquad X\in\mathbb{R}^{T\times d}. \]

With thousands of rows in \(X\), the hardware can reuse model weights across many tokens. Large matrix operations tend to utilize GPU arithmetic units well.

Prefill also performs attention over the prompt. For ordinary full attention, its arithmetic grows roughly as:

\[ O(T^2d) \]

for sequence length \(T\) and hidden dimension \(d\), ignoring architectural constants. Memory-efficient attention avoids storing the full quadratic score matrix, but not the underlying attention arithmetic.

So a long prompt can create a large, compute-heavy burst of work.

Decode is often bandwidth- and latency-sensitive¶

During autoregressive decode, each sequence usually contributes just one new token per step.

prefill matrix shape: [8192 tokens] × weights

decode matrix shape:  [1 token]    × weights

For a small serving batch, each decode step does relatively little arithmetic compared with the amount of model-weight data that must be accessed. It can therefore be limited by memory bandwidth rather than peak FLOPs.

Decode also repeatedly reads the growing KV cache, and each next token depends on the previous token.

This makes inter-token latency important: users notice pauses between streamed tokens, even if the GPU is theoretically capable of enormous throughput.

The distinction is not absolute. Large decode batches can become compute-heavy, and very long-context attention can make decode expensive in different ways. But the two phases frequently have different optimal batch sizes and scheduling policies.

Why mixing them creates interference¶

Imagine a GPU serving ten conversations, each streaming one token every 25 milliseconds.

Then a new request arrives with a very long prompt. Its prefill requires a substantial chunk of GPU time.

If the scheduler executes that prefill as one large, uninterrupted job, the ongoing conversations may stall:

time ───────────────────────────────────→

existing requests:
decode  decode  decode   [waiting]   decode

new request:
                      LONG PREFILL

The scheduler can instead break prefill into chunks and interleave them with decode. Chunked prefill is a powerful alternative, and a well-designed unified scheduler can work very well.

But there is still a resource-allocation tension: a GPU configuration optimized for large prefill batches may not be the best one for low-latency decode.

The disaggregated architecture¶

P/D disaggregation assigns the phases to different worker pools:

incoming request
       │
       ▼
PREFILL GPU POOL
- process prompt
- construct KV cache
       │
       │ transfer KV cache
       ▼
DECODE GPU POOL
- generate tokens
- stream response

The prefill workers can focus on large, throughput-oriented operations. The decode workers can focus on steady token generation and predictable latency.

This resembles the topology-aware parallelism lesson: we are mapping different communication and compute patterns onto different parts of the cluster.

But separating phases introduces a new problem: moving the KV cache.

The KV cache is the handoff artifact¶

Recall that each attention layer stores keys and values for previous tokens.

For a simplified model with \(L\) layers, \(H_{kv}\) KV heads, head dimension \(d_h\), context length \(T\), and \(b\) bytes per element, cache size for one sequence is approximately:

\[ M_{KV}=2LT H_{kv}d_hb. \]

The factor of 2 represents keys and values.

Consider this illustrative configuration:

\[ L=32,\quad H_{kv}=8,\quad d_h=128, \]

\[ T=8192,\quad b=2\text{ bytes (BF16)}. \]

Then:

\[ M_{KV}=2(32)(8192)(8)(128)(2) =1{,}073{,}741{,}824\text{ bytes} \]

or 1 GiB of KV cache for one prompt.

That is the state the decode worker needs before it can continue the request. The example assumes all layers' KV tensors are transferred and ignores metadata, padding, and implementation-specific layouts.

A worked transfer-cost example¶

Suppose the effective end-to-end bandwidth between the two pools is 50 GB/s.

Ignoring startup latency and contention, moving 1 GiB takes approximately:

\[ t_{transfer}\approx\frac{1.074\text{ GB}}{50\text{ GB/s}} \approx21.5\text{ ms}. \]

That may be acceptable if separating the workloads saves hundreds of milliseconds of queueing or improves steady-state throughput.

But if the transfer crosses a congested network and takes 150 ms, the handoff could dominate a short request's time to first token.

Disaggregation is therefore not automatically faster:

\[ \boxed{\text{benefit from specialization}>\text{handoff and coordination overhead}} \]

must hold for the relevant workload.

Notice the dependence on prompt length. Doubling \(T\) roughly doubles the KV bytes transferred, even when memory-efficient attention is used.

Three metrics tell different stories¶

Inference systems often track:

TTFT (time to first token): how long a user waits before the response starts.
ITL (inter-token latency): the delay between successive generated tokens.
Throughput: how many useful tokens or requests the service completes per second.

Disaggregation might improve ITL by protecting decode workers from prefill bursts, while making TTFT worse because of KV transfer.

Or it might improve total throughput by allowing each pool to use better batch sizes.

There is no single scalar called “inference performance.” The right architecture depends on which metric matters and on the arrival pattern of requests.

The scheduler now manages two queues¶

With separate pools, the system needs to coordinate more than GPU kernels:

prefill queue → prefill workers
                    │
                    ▼
              KV transfer
                    │
                    ▼
decode queue  → decode workers

If prefill capacity is too high relative to decode capacity, completed prompts accumulate waiting for decode slots. Their KV caches occupy memory somewhere while they wait.

If decode capacity is too high, decode GPUs sit idle while prefill workers struggle to admit requests.

A rough capacity-planning model uses request arrival rate \(\lambda\), mean prefill service demand \(P\), and mean decode service demand \(D\):

\[ \text{prefill work arriving per second}=\lambda P, \]

\[ \text{decode work arriving per second}=\lambda D. \]

These quantities must be compared with the effective capacities of their respective pools. Real systems also need to account for batching, variable prompt lengths, output lengths, and GPU sharing.

The important point is that the ratio of prefill to decode resources should follow the workload, not an arbitrary fixed split.

A concrete mental model: a restaurant kitchen¶

Think of prefill as preparing ingredients for an order and decode as plating and serving courses one at a time.

A shared kitchen can do both jobs. But a sudden catering order requiring extensive prep might disrupt cooks who are plating dishes for seated guests.

A specialized setup has a prep kitchen and a service kitchen:

prep kitchen → prepared ingredients → service kitchen

The service kitchen gets a steadier workflow. The prep kitchen can handle big batches efficiently.

But transporting prepared ingredients takes time, and the service kitchen must have room to receive them. If the two kitchens are far apart, specialization may not pay off.

For an LLM service:

prep kitchen       = prefill workers
prepared ingredients = KV cache
service kitchen    = decode workers
transport          = KV-cache transfer

The goal is not simply to make each kitchen fast. It is to keep the whole restaurant fast without leaving diners waiting between courses.

The deeper lesson¶

Speculative decoding changed how we execute the sequential part of inference. P/D disaggregation changes where different phases of inference execute.

Both are responses to the same underlying observation: a Transformer forward pass is not one uniform workload.

The model may be mathematically identical across phases, but its computational shape, memory behavior, batching opportunities, and latency constraints change dramatically.

Today's main principle is:

\[ \boxed{\text{optimize inference by separating workloads when their hardware needs conflict}} \]

The caveat is equally important: every separation introduces a boundary. For P/D disaggregation, that boundary carries potentially gigabytes of KV state. The engineering problem is to gain more from specialization than we lose to moving state and coordinating queues.