Disaggregated inference — why reading and writing may belong on different GPUs

Prefill and decode have very different hardware profiles.

• Prefill: large parallel matrix operations, often compute-heavy.
• Decode: sequential next-token work, often memory-bandwidth-heavy and latency-sensitive.

If the same GPU pool handles both, large prefills can interfere with streaming decodes.

Traditional mixed worker¶

One GPU might be simultaneously handling: - long new prompts - existing users waiting for next tokens

This creates scheduling conflict.

Split the jobs¶

prompt
  ↓
PREFILL GPUs
  ↓
KV cache transfer
  ↓
DECODE GPUs
  ↓
tokens

Prefill workers build the initial cache; decode workers specialize in continuous generation.

The handoff cost¶

Using: - 32 layers - 8 KV heads - head dimension 128 - BF16 - 32K-token prompt

KV bytes are roughly:

≈ 4.2 GB.

At 100 GB/s, transferring 4.2 GB has a lower bound around 42 ms.

So disaggregation is a real trade:

Why bother?¶

Workloads vary.

A legal-analysis request may have an 80K prompt and 300-token answer: prefill dominates.

A writing request may have a 2K prompt and 15K answer: decode dominates.

Separate pools let capacity scale independently.

They also let operators optimize TTFT and inter-token latency separately.

Combined serving architecture¶

Disaggregation can combine with: - prefix caching, - GQA, - quantization, - continuous batching, - speculative decoding.

What appears externally as one generate() call may be a distributed pipeline.

Why prefill and decode interfere¶

A long prefill wants to occupy large matrix kernels for substantial chunks of time.

A streaming decode user wants a tiny next-token computation every few tens of milliseconds.

Putting both on the same GPU creates a scheduling problem similar to mixing giant batch jobs with interactive requests on one CPU.

TTFT and ITL as separate service-level objectives¶

Disaggregation lets an operator reason separately about:

and:

A service may decide that first-token latency can vary more than token-stream smoothness, or vice versa.

KV movement is the tax¶

The prefill worker has created valuable intermediate state. Sending it to decode workers is effectively transferring “prepared context.”

Longer prompts and more KV heads increase that tax.

High-speed interconnect and cache compression therefore become part of the design.

A broader systems pattern¶

This is an example of specializing hardware pools for phases of one model computation.

The model appears monolithic mathematically, but the serving system can decompose it according to workload characteristics.

Disaggregation resembles a producer-consumer pipeline¶

Prefill workers produce KV state.

Decode workers consume KV state while extending it token by token.

If prefill production outruns decode capacity, requests queue before decode.

If decode is plentiful but prefill is overloaded, GPUs sit waiting for new prepared sequences.

So the cluster needs queueing and autoscaling policies for both phases.

Hardware specialization¶

The two pools need not even use identical accelerator configurations.

A prefill pool might value peak matrix FLOPs.

A decode pool might prioritize HBM capacity/bandwidth and efficient small-batch kernels.

This possibility is conceptually important: a single model architecture can map onto heterogeneous serving hardware.

The optimal datacenter is therefore not necessarily “a giant homogeneous pool of the fastest GPU.”

Failure domains and scheduling¶

Disaggregation also changes reliability. If a prefill worker fails before handing off its cache, the prompt may need to be recomputed. If a decode worker fails after receiving the cache, the system may need another copy or a way to restore state.

Large serving systems therefore think about KV state as a valuable distributed object: - where it lives, - how long it lives, - whether it is replicated, - how it is routed.

The simple mathematical model generate(prompt) hides a lifecycle-management problem for gigabytes of temporary state.