Prefix caching — when an LLM can skip reading the same prompt twice

Yesterday we explored prefill–decode disaggregation: separate GPU pools can specialize in processing prompts and generating tokens, but the handoff requires moving the attention KV cache.

Today we ask a related question:

If a model has already processed the beginning of a prompt, why calculate that beginning again for the next request?

This is the idea behind prefix caching (sometimes called automatic prefix caching): reuse the key/value states of an identical prompt prefix rather than running prefill on those tokens again.

The intuition is simple:

But the details reveal an interesting interaction between Transformer causality, memory management, scheduling, and multi-tenant serving.

Why prefixes are reusable at all¶

Recall causal self-attention. At position \(t\), a token can attend to positions \(1,\ldots,t\), but not to future positions.

For a sequence of tokens:

A B C D E F

when the model processes A B C, its hidden states and KV states do not depend on D E F.

That means these two requests share exactly the same computations for the first three tokens, assuming the same model, positions, attention rules, and other relevant execution settings:

Request 1: A B C D E F
Request 2: A B C X Y Z
           └───┘
        shared prefix

The model must compute different states after the split, but it does not need to recompute the shared prefix.

This follows from the causal attention mask—not from a heuristic guess about semantic similarity.

What is actually cached?¶

An attention layer forms keys and values from token representations:

During autoregressive inference, the model stores the keys and values for previous positions so later tokens can attend to them.

A prefix cache keeps those per-layer KV tensors available for reuse across requests. It is not merely caching the text or the final answer.

Conceptually:

First request:

shared system prompt ──prefill──> KV blocks ──> decode
                                 │
                                 ▼
                           prefix cache

Second request:

same system prompt ──cache hit──> reuse KV blocks
new user message  ──prefill────> compute only new suffix

The second request still needs computation for its unique suffix and generated output. Prefix caching removes repeated prefill, not the need to decode.

Why exact tokens matter more than similar meaning¶

Consider two system prompts:

You are a helpful assistant. Answer concisely.

and:

You are a helpful assistant. Answer briefly.

A person might treat these as equivalent. The model cannot reuse the entire first prompt's KV states for the second because their token sequences diverge.

Even one changed token can alter the hidden states of every subsequent position, since those positions attend to earlier tokens.

So the safe reuse rule is approximately:

In practice, the cache key must also account for model/version identity and anything else that changes the computation, such as relevant attention configuration or adapter weights.

For position-dependent attention schemes such as RoPE, matching the tokens but placing them at different logical positions is generally not sufficient.

The cache usually operates in blocks¶

A serving engine rarely wants to manage millions of individual-token cache entries.

Instead, it can divide KV storage into fixed-size blocks, perhaps 16 tokens per block:

Block 0      Block 1      Block 2
[ 16 tokens ][ 16 tokens ][ 16 tokens ]

The engine can hash a block's token IDs together with its preceding prefix identity and other required metadata. That creates a chain: a block is reusable only in the context of the same preceding tokens.

A simplified cache key is:

Real implementations include additional metadata and must handle hash collisions safely.

Why include \(h_{i-1}\)? Because the same 16-token string can have different KV states when preceded by different text.

This makes the cache prefix-aware, not just a dictionary of repeated text fragments.

A worked example: a repeated 4,096-token system prompt¶

Suppose a service uses a 4,096-token instruction and tool-definition prefix for every request. Each user then contributes 2,048 unique prompt tokens.

Without prefix caching, each request prefills:

After the first request warms the cache, later requests may need to prefill only:

The fraction of prompt positions whose prefill computation is avoided is:

For 100 requests, ignoring eviction and assuming one warm-up request, the total number of token positions passed through fresh prefill changes from:

to:

That avoids fresh prefill for:

Caveat: this is not a prediction of a 66.7% latency or cost reduction. Prefill time is not always linear in token count, especially because attention over the suffix still reads the cached prefix. Queueing, batching, memory traffic, and cache lookup overhead also matter.

How much GPU memory does the shared prefix occupy?¶

Use the simplified KV-cache formula from Day 62:

For 32 layers, eight KV heads, head dimension 128, BF16 values (two bytes each), and a 4,096-token prefix:

That is 512 MiB of KV state.

If 100 requests would otherwise store independent copies of that same prefix, the duplicated prefix states alone would occupy about:

A system that supports safe sharing can potentially keep one physical set of prefix blocks and let many requests reference it, while each request maintains its own unique suffix.

Actual savings depend on the KV layout, whether blocks are shared across requests, and when cached blocks are retained or evicted.

Prefix caching is a memory-management problem¶

A cache hit saves computation, but cached KV blocks occupy valuable accelerator memory.

That memory could instead hold:

more active requests
longer contexts
larger decode batches
other reusable prefixes

A serving engine therefore needs an eviction policy. Frequently reused prefixes are valuable; old prefixes that are unlikely to recur may not justify their memory footprint.

This creates a familiar cache tradeoff:

It also means request hit rate can be misleading. Reusing 4,000 tokens on one request saves much more prefill work than reusing 20 tokens on another.

A useful metric is token-weighted reuse:

Even that metric does not directly equal GPU time saved, but it is often more informative than counting cache-hit requests.

How this connects to disaggregated prefill and decode¶

Yesterday's architecture looked like:

prefill GPU ──transfer KV──> decode GPU

Now imagine that the decode GPU already has the shared prefix KV blocks from a previous request.

The system might need to transfer only newly computed suffix blocks—or arrange for a worker with the needed cached blocks to handle the request.

But if the cached prefix lives on another machine, reuse may require a remote lookup and a large transfer. Sometimes recomputing is cheaper than moving the cache, especially on a slow or congested network.

So scheduling becomes cache-aware:

Which worker has the prefix?
How many bytes must move?
How much prefill computation would reuse save?
Will moving the request worsen queueing?

The best answer depends on the cluster topology and current load. A cache hit is not automatically a system-level win.

Two important correctness and security caveats¶

First, reuse must preserve the model's computation. A different model revision, tokenizer, adapter, attention mask, or incompatible positional context can invalidate a cached prefix. Minor floating-point differences can also arise from changing execution paths, even when the mathematical computation is equivalent.

Second, a shared cache is part of the serving system's trust boundary. Cross-user reuse must not expose one tenant's private prompt or KV state to another tenant. Engines need correct isolation, authorization, cache-key design, and memory lifecycle management. Public shared instructions and private conversation prefixes should not be treated interchangeably.

These constraints matter just as much as the performance benefit.

A concrete mental model: a photocopy master¶

Imagine a company preparing 100 employee handbooks.

Every handbook begins with the same 200-page policy section, followed by a personalized 20-page section.

The wasteful approach is to typeset the 200 common pages again for every employee.

The smarter approach is to prepare one reusable master of those pages and produce only the personalized section each time.

common policy pages ──> reusable master
                              │
                              ├──> employee A's unique pages
                              ├──> employee B's unique pages
                              └──> employee C's unique pages

But if one word in the common policy changes, the master is no longer valid from that point onward. And keeping thousands of masters in storage is expensive.

For an LLM:

common pages       = shared token prefix
reusable master    = cached KV blocks
personalized pages = unique prompt suffix
storage space      = GPU memory

The key is that the model reuses intermediate computation, not an already-written answer.

The deeper lesson¶

Speculative decoding saves time by predicting future tokens and verifying them. Disaggregated prefill/decode saves time by specializing hardware. Prefix caching saves time by recognizing that some past computation does not need to happen again.

These techniques optimize different sources of waste:

speculative decoding  → repeated sequential target-model steps
P/D disaggregation    → interference between different workloads
prefix caching        → repeated computation of identical context

They can complement one another, but each adds runtime state and scheduling complexity.

Today's main principle is:

As inference services mature, optimizing the Transformer kernels is only part of the story. Knowing what has already been computed, where it lives, and whether it can be reused safely becomes a first-class systems problem.