Prefix caching — why repeated prompts need not be prefilling from scratch

Suppose a coding assistant prompt contains:

8K system instructions
30K repository context
10K conversation
2K new user request

Total: 50K tokens.

After prefill, the model has K/V state for all 50K tokens at every layer.

If the next request begins with the same first 48K tokens, the server can reuse the KV state for that prefix and prefill only the new suffix.

Why exact prefix?¶

Suppose one prompt begins:

The cat sat on the mat...

and another:

The dog sat on the mat...

Even the representation of later identical text can differ because preceding context changed.

So arbitrary repeated substrings are not generally reusable. Shared prefixes are.

What is cached?¶

Not final logits.

The useful reusable state is K/V for the prefix at every layer.

Layer 1: K₁...K₄₀₀₀₀, V₁...V₄₀₀₀₀
Layer 2: ...
...

Cache management¶

A production server cannot keep every prefix forever.

It needs policies for: - identifying matching prefixes, - deciding what to retain, - evicting old entries, - allocating KV memory.

Block/page-based KV memory¶

Instead of requiring one contiguous region per sequence, KV state can be stored in pages/blocks and logically linked.

This reduces fragmentation and makes sharing common prefix blocks practical.

Why prompt layout matters¶

Bad layout:

Current time: 08:01
[40K stable instructions]

Next request:

Current time: 08:02
[40K stable instructions]

The prefix diverges immediately.

Better:

[40K stable instructions]
Current time: 08:02

Now most of the prompt is cacheable.

Numerical intuition¶

1000 requests, each: - 20K identical prefix - 1K unique suffix

Without reuse:

token positions prefilling.

With prefix reuse:

roughly 20× fewer token positions in this simplified count.

Prefix caching versus RAG¶

These solve different problems.

Prefix caching says:

I have already computed this exact prompt prefix before; reuse the neural state.

RAG says:

Search an external corpus for text that seems relevant, then compute its neural state now.

A system can use both: retrieve a stable document collection, arrange common material early in the prompt, and reuse its computed prefix across many queries.

Tree-shaped reuse¶

If many conversations branch from a common history, the cache can conceptually form a tree:

system + tools
      |
 common conversation
   /       \
branch A   branch B

The shared trunk can point to the same KV blocks.

Cache identity is stricter than textual appearance¶

Changes in tokenization, positions, model version, attention configuration, or preceding context can invalidate reuse.

Operational systems therefore hash or otherwise identify exact reusable blocks.

Economic significance¶

For workloads with very long repeated instructions, prefix caching can remove much of TTFT and prefill cost.

It effectively converts some inference computation into a reusable stored artifact.

That is why prompt organization becomes part of serving-system design.

Prefix caching changes prompt-engineering economics¶

Historically prompt design was discussed mainly in terms of model behavior.

At large serving scale, ordering can also change compute cost.

Two prompts with identical semantic content may have very different cacheability depending on where dynamic information appears.

For example:

[stable 50K-token tool/reference prefix]
[user-specific 500 tokens]

is much friendlier to reuse than:

[user id + timestamp]
[stable 50K-token tool/reference prefix]

because prefix identity breaks at the first changed token.

This means an inference platform may expose metrics such as cache-hit rate or reused-prefix tokens.

Prompt structure becomes part of performance engineering, just as SQL query structure can affect database execution.

Prefix caching does not remove decode cost¶

Reusing 50K tokens of prefill can dramatically reduce time-to-first-token, but once generation starts the model still has to attend against the resulting 50K-token cache.

So a cache hit saves:

not:

The long KV cache still occupies memory, and decode attention still scans historical keys.

This distinction matters when evaluating long-context workloads: prefix caching can make repeated prompts much cheaper, but it does not turn a 100K-token conversation into a 1K-token conversation.