KV-cache quantization — making long-context inference fit in memory

Yesterday we explored PagedAttention: the inference runtime stores a request's KV cache in reusable blocks instead of reserving one large contiguous region. Before that, we saw how prefix caching can share KV blocks across requests.

Both techniques use memory more efficiently. But neither changes how many bytes a key or value actually occupies.

Today's question is:

What if we could store the KV cache using fewer bits, without significantly changing the model's predictions?

That is KV-cache quantization. Its central idea is:

\[ \boxed{\text{keep the attention history, but represent each stored number more compactly}} \]

This is related to weight quantization, but it solves a different problem. Model weights are largely fixed during inference; the KV cache grows with every token and every active request.

Why the KV cache grows so quickly¶

At each Transformer layer, attention uses stored keys and values from previous positions:

\[ \operatorname{Attention}(q_t,K_{1:t},V_{1:t}) =\operatorname{softmax}\left(\frac{q_tK_{1:t}^{\top}}{\sqrt{d_h}}\right)V_{1:t}. \]

The cache contains a key vector and a value vector for each relevant token, layer, and KV head.

A useful approximate memory formula is:

\[ M_{KV}=2LT H_{kv}d_hb, \]

where:

\(L\): number of layers
\(T\): cached token positions
\(H_{kv}\): number of key/value heads
\(d_h\): dimension of each head
\(b\): bytes per stored element

The factor of two accounts for keys and values.

The first four factors describe how many numbers we store. Quantization attacks the last factor: bytes per number.

From BF16 to INT8, INT4, or even fewer bits¶

BF16 uses 16 bits, or two bytes, per value. An 8-bit representation uses one byte. A packed 4-bit representation uses half a byte.

Ignoring metadata and implementation overhead, the storage ratios are:

\[ \begin{aligned} \text{INT8 versus BF16:}&\quad 2\times\text{ smaller},\\ \text{INT4 versus BF16:}&\quad 4\times\text{ smaller},\\ \text{INT2 versus BF16:}&\quad 8\times\text{ smaller}. \end{aligned} \]

But quantization is not simply chopping bits off a floating-point number. We need a mapping from real values to a small set of representable integers.

For a simple symmetric quantizer with \(b_q\) signed bits, one possible scheme is:

\[ s=\frac{\max_i |x_i|}{2^{b_q-1}-1}, \]

\[ q_i=\operatorname{clip}\left(\operatorname{round}\left(\frac{x_i}{s}\right),-Q,Q\right), \qquad Q=2^{b_q-1}-1. \]

To use the stored value in attention, the kernel approximately reconstructs it:

\[ \hat{x}_i=sq_i. \]

The difference \(\hat{x}_i-x_i\) is quantization error. Other implementations use asymmetric zero-points, different integer ranges, or nonuniform schemes; the principle is the same.

Why scales and groups matter¶

Suppose a KV vector contains both very small values and a few large outliers:

0.03   -0.07    0.12    0.09    6.4

If one scale must cover the entire vector, the large value forces a coarse quantization step. The small values may all round to zero.

Instead, an engine can quantize groups of elements separately, with one scale per group:

group A: small values → small scale
group B: large values → larger scale

Smaller groups can reduce error, but each group needs metadata. There is no free compression: scales, optional zero-points, packing, and alignment all consume space.

The grouping direction also matters. Keys may contain persistent outlier channels across token positions, while values may have different distributions across tokens. Some KV quantization methods therefore use different grouping strategies for keys and values rather than treating them identically.

Worked example: an 8,192-token context¶

Use the same illustrative model as Days 62–64:

\[ L=32,\quad H_{kv}=8,\quad d_h=128,\quad T=8192. \]

With BF16 (\(b=2\) bytes), the KV cache is:

\[ M_{KV}=2(32)(8192)(8)(128)(2) =1{,}073{,}741{,}824\text{ bytes} =1\text{ GiB}. \]

Ideal packed 4-bit storage would reduce this to:

\[ \frac{1\text{ GiB}}{4}=256\text{ MiB}. \]

Now add a simple, realistic source of overhead: one two-byte FP16 scale for every 64 quantized elements, with no zero-point.

Average storage per element becomes:

\[ \frac{4}{8}+\frac{2}{64} =0.53125\text{ bytes}. \]

The resulting size is approximately:

\[ 1{,}024\text{ MiB}\times\frac{0.53125}{2} =272\text{ MiB}. \]

So this example reduces KV storage from 1,024 MiB to about 272 MiB, a 3.76× reduction, before padding, residual buffers, and other runtime overhead.

For 100 independent 8,192-token contexts, the corresponding aggregate would fall from roughly 100 GiB to 26.6 GiB. That does not mean one GPU can necessarily serve all 100 requests: model weights, temporary activations, and scheduling headroom still require memory.

Why key errors and value errors are different¶

Look again at attention:

\[ \underbrace{\operatorname{softmax}\left(\frac{qK^\top}{\sqrt{d_h}}\right)}_{\text{attention weights}} \underbrace{V}_{\text{information combined}}. \]

Quantizing keys perturbs the scores that determine where the model attends. Quantizing values perturbs the information that is combined after the attention weights are chosen.

Imagine two candidate positions have attention logits:

position A: 4.0
position B: 3.8

A small key-quantization error could change them to:

position A: 3.8
position B: 4.0

The preferred position flips. Softmax is smooth, so this does not imply a catastrophic change every time, but it illustrates why key errors can be especially consequential.

Value errors affect the weighted output more directly. Neither kind is automatically harmless; their impact depends on the model, layer, head, context, and task.

Why compression may also improve speed¶

During decode, attention repeatedly reads historical keys and values. With long contexts, moving KV bytes from GPU memory can become expensive.

If quantization reduces those bytes by nearly 4×, the attention kernel may need much less memory bandwidth.

But it must also unpack and dequantize the values, or perform specialized low-precision operations. Those extra instructions cost time.

So the real performance comparison is:

\[ \boxed{\text{saved memory traffic}\quad\text{versus}\quad\text{quantization and dequantization overhead}} \]

At long context lengths and small decode batches, memory savings can be particularly attractive. At short contexts, conversion overhead may outweigh the bandwidth benefit. Quantization can increase concurrency even when it barely improves single-request latency.

Why some systems keep a full-precision window¶

One practical design keeps recently generated KV states in BF16 and quantizes older states:

old context                         newest tokens
[quantized][quantized][quantized]  [BF16 residual]

This can avoid repeatedly quantizing an incompletely filled group and may preserve precision for recent positions. When the residual buffer fills, its contents can be quantized and moved into the compressed cache.

It is a tradeoff: larger full-precision windows use more memory but may improve quality or simplify updates. The best policy is workload- and model-dependent.

How this connects to paging and prefix caching¶

The three techniques address different kinds of waste:

PagedAttention    → allocate KV memory in flexible blocks
prefix caching    → reuse already computed KV blocks
KV quantization   → store each KV block in fewer bytes

They can be combined. A serving engine might share a prefix, store its blocks in quantized form, and use block tables to reference those physical blocks from multiple requests.

But the details matter: a cached block must be interpreted using the correct quantization format and scales, and its sharing rules must preserve tenant isolation and model compatibility. Compression does not remove those correctness requirements.

A concrete mental model: compressing a warehouse¶

Imagine a warehouse that stores one box of documents for every conversation token.

Paging replaces fixed private warehouse aisles with movable standardized crates. Prefix caching lets multiple customers reference the same crate of common documents. Quantization scans each document into a more compact format so the crate itself takes less space.

Each technique saves something different:

paging       = less stranded floor space
prefix reuse = fewer duplicate crates
quantization = smaller contents per crate

The catch is that compressed documents take effort to decode, and fine print may become harder to read. More compression is not always better.

The deeper lesson¶

Weight quantization asks how cheaply we can store and execute the model's parameters. KV-cache quantization asks how cheaply we can retain the model's growing working memory.

That distinction becomes increasingly important as context windows and serving concurrency grow. A model can fit comfortably in GPU memory while its active requests' KV caches do not.

Today's main principle is:

\[ \boxed{\text{in long-context serving, memory precision is a runtime design choice—not just a model-weight choice}} \]

The practical goal is not the lowest possible bit width. It is the best combination of quality, memory capacity, latency, and throughput for the actual workload. A four-bit KV cache that slightly slows a single request may still be valuable if it lets the service handle several times as many concurrent requests without running out of memory.