Yesterday we looked at KV-cache quantization: reducing the bytes occupied by each request's attention history. Together with PagedAttention and prefix caching, it helps an inference server fit more requests in GPU memory.
But having room for more requests is not the same as serving them efficiently.
Today's question is:
When requests arrive at different times, with different prompt lengths and output lengths, who gets the GPU next?
That is the job of the inference scheduler. Two important techniques are continuous batching and chunked prefill.
The intuition:
Why ordinary batching is awkward for generation¶
In traditional static batching, the server collects requests, runs them together, and waits for the batch to finish before assembling another one.
This works well when every example takes roughly the same amount of computation. Autoregressive generation is different: one request might produce 12 tokens, another 800.
Imagine a batch with three requests:
Request A: 12 output tokens [############]
Request B: 80 output tokens [################################...]
Request C: 5 output tokens [#####]If the runtime treats the batch as one indivisible job, short requests can finish early while their slots remain unavailable to new arrivals. The GPU may continue carrying inactive or padded work, or new requests may simply wait.
Continuous batching (also called in-flight batching) changes the unit of scheduling. Rather than fixing membership for the lifetime of a request, the server can reconsider the active batch at generation-step boundaries.
step 1: [A B C]
step 2: [A B C]
step 3: [A B C]
...
C finishes → [A B D] (D joins)
...
A finishes → [E B D] (E joins)The important property is not that the GPU executes one token at a time globally. It can generate one next token for each of many active sequences in a batched forward pass.
The scheduler has two kinds of work¶
Recall Day 62's distinction:
- Prefill processes prompt tokens and creates their KV states. With many prompt tokens available at once, it can use large, compute-efficient matrix operations.
- Decode advances active requests by one generated token per step, reading their existing KV caches. At small batch sizes and long contexts, memory traffic often dominates.
A new request needs prefill before its first output token. Existing requests need decode steps to keep streaming.
This creates a conflict. If the GPU spends a long uninterrupted interval prefilling a giant prompt, already-streaming users may see an uncomfortable pause.
Three useful service metrics expose that tension:
Time to first token (TTFT) measures responsiveness when a request begins. Time per output token (TPOT) captures how smoothly its answer streams. Aggregate throughput measures how much work the service completes.
A policy that improves one can worsen another.
Chunked prefill: split the long job¶
Instead of processing a long prompt as one monolithic prefill, a scheduler can divide it into chunks and interleave them with decode work.
Without chunking:
[ decode ][ long prefill ][ decode ][ decode ]
With chunking:
[ decode + prefill chunk ][ decode + prefill chunk ]
[ decode + prefill chunk ][ decode + prefill chunk ]The diagram represents scheduling, not necessarily concurrent kernel execution. A real engine may combine work into one model execution, run separate kernels, or use specialized scheduling paths.
For causal attention, a later prompt chunk can use the KV states computed for earlier chunks. The model still attends to the same earlier positions; the computation is merely partitioned differently.
Chunking does not mean discarding earlier prompt tokens or shortening the context. It changes when their KV states are produced.
Worked example: a long prompt interrupts four streams¶
Suppose four requests are already generating answers. For a deliberately simplified single-GPU model, assume:
- One batched decode step for the four requests takes 8 ms.
- A newly arriving 2,048-token prompt takes 80 ms to prefill in one uninterrupted pass.
- Splitting that prompt into eight 256-token chunks takes 10 ms per chunk, excluding the interleaved decode work.
Without chunked prefill, the new prompt occupies the GPU for 80 ms. If a decode step was due immediately, the existing streams may wait about:
between completed decode steps, rather than their usual 8 ms. The new request gets through prefill quickly, but the four ongoing streams stall.
With chunked prefill, the scheduler alternates one 10 ms prefill chunk with one 8 ms decode step:
P1 D P2 D P3 D P4 D P5 D P6 D P7 D P8 D
10 8 10 8 10 8 10 8 10 8 10 8 10 8 10 8 msDuring this interval, an existing stream sees roughly 18 ms between decode completions, rather than one approximately 88 ms gap. Its cadence is slower than the original 8 ms, but far more regular.
The cost is that the new prompt finishes prefill only after roughly:
Its TTFT can therefore be worse than with uninterrupted prefill. These numbers are illustrative: real chunk runtimes are not generally linear in chunk size, and kernel overlap, batching, KV transfers, and queueing complicate the comparison.
The lesson is not “chunking is always faster.” It is that chunking gives the scheduler finer control over who waits.
Token budgets and admission control¶
A practical scheduler often limits how much work can enter a scheduling iteration. A simplified token budget might look like:
where \(P\) is the number of prompt tokens scheduled for prefill, \(D\) is the number of decode-token positions scheduled, and \(B\) is the iteration's token budget.
If \(B=256\) and four existing sequences each need one decode token, a naive accounting rule leaves:
prompt tokens for new prefill work in that iteration.
This is a scheduling constraint, not an equation for GPU time: one prefill token and one decode token can have very different compute, memory, and latency costs. Real runtimes also enforce limits on active sequences, available KV blocks, and other resources.
When KV memory is full, a scheduler may delay admission, preempt a request, evict cache blocks, or recompute some work later. That connects directly to Days 63–65: prefix caching, paging, and KV quantization change what the scheduler can afford to admit.
The hidden tradeoff: fairness versus efficiency¶
Larger batches usually improve GPU utilization, up to a point. But a policy optimized only for total tokens per second may make individual users wait too long.
Likewise, always prioritizing decode can keep current streams smooth while starving new requests of prefill capacity. Always prioritizing prefill can improve new-request TTFT while disrupting existing streams.
The right policy depends on the service's workload and objectives: interactive chat, batch document processing, coding agents, and latency-sensitive voice systems may choose different balances.
Continuous batching is the mechanism that lets requests enter and leave dynamically. Chunked prefill is one tool for balancing the two phases. Neither guarantees lower latency on every request.
Mental model: an airport with a shared runway¶
Think of the GPU as a runway.
A decode step is a small plane that needs a short, repeated runway slot to keep its journey moving. A long prefill is a large aircraft operation that could occupy the runway for an extended period.
Static batching is like closing the schedule until every plane in the current group has completed its journey. Continuous batching lets the tower assign slots as planes arrive and depart.
Chunked prefill is like breaking a long runway operation into stages, allowing smaller planes through between stages. The long operation may finish later, but everyone else avoids one enormous delay.
The deeper lesson¶
We often describe inference performance in terms of model architecture, FLOPs, memory bandwidth, and KV-cache size. Yet the order in which the runtime schedules requests can be just as important to the experience users observe.
The practical question is not merely “How fast can this model generate tokens?” It is:
How does the system allocate scarce GPU time and KV memory among many requests while meeting TTFT, streaming-latency, and throughput goals?
That is why an inference engine is more than a fast implementation of Transformer layers. It is also a real-time resource manager.