Quantization — why fewer bits can make LLM inference dramatically faster

Over the last several lessons, we have been building up the systems view of LLM inference. Yesterday’s speculative decoding lesson showed how we can get more useful tokens out of each expensive target-model pass. Today we attack a different bottleneck: how much data must move for every pass.

The key idea is:

If a weight uses 16 bits instead of 32, we can move twice as many weights with the same memory bandwidth. If it uses 8 bits, four times as many. If it uses 4 bits, eight times as many.

This is the basic motivation for quantization.

But quantization is more subtle than simply chopping off bits. The model was trained with real-valued weights, and replacing those values with a small set of discrete numbers introduces error.

The engineering question is therefore:

How few bits can we use before the approximation noticeably damages the model?

Start with ordinary floating-point weights¶

Suppose a tiny weight vector contains:

In BF16 or FP16, each value occupies 16 bits.

A 70-billion-parameter model therefore requires roughly:

bytes:

just for its weights.

During low-batch decode, those weights must repeatedly feed the matrix multiplications that generate each new token.

The bandwidth mental model is roughly:

HBM
 ↓
model weights
 ↓
GPU compute units
 ↓
one new token

If weight movement is the bottleneck, reducing the number of bytes per weight can directly improve speed.

Quantization replaces many possible values with fewer possible values¶

An FP16 number can represent a very large set of values.

An integer with \(b\) bits has only:

possible bit patterns.

For INT8:

For INT4:

So a 4-bit representation cannot independently encode every possible real-valued weight.

Instead, we map real values onto a small set of representable levels.

A simple uniform quantizer can be written as:

where \(s\) is a scale factor.

To approximately reconstruct the weight:

The original weight \(w\) is replaced during computation by the approximation \(\hat{w}\).

The difference:

is the quantization error.

A concrete example¶

Suppose we choose:

For:

we compute:

The reconstructed value is:

So the error is:

Similarly:

The model no longer uses the exact original weights.

It uses nearby representable values.

The central question is whether millions or billions of these small perturbations materially change the model's behavior.

Surprisingly often, modest quantization works extremely well.

The ruler mental model¶

Imagine measuring lengths.

With an infinitely precise ruler, you could record:

6.327184 cm

With a ruler marked every millimeter, you might record:

6.3 cm

With a very coarse ruler, perhaps:

6 cm

Quantization is choosing the ruler.

More bits mean finer tick marks.

Fewer bits mean coarser tick marks.

The goal is not to preserve every weight exactly.

The goal is to preserve enough numerical information that the neural network's overall computation remains useful.

Why one global scale is usually a bad idea¶

Suppose a tensor contains:

One weight is a huge outlier.

If we choose a single quantization scale large enough to represent \(12.0\), the small values may all collapse toward the same few quantized levels.

Conceptually:

small weights:
-0.02  0.03  -0.01  0.04

outlier:
12.0

The outlier determines the ruler, making the ruler far too coarse for everything else.

This is why practical quantization usually uses groups.

Group-wise quantization¶

Suppose a weight matrix contains millions of values.

Rather than using one scale for the entire matrix, divide it into groups.

For example:

weights 0–127       → scale s0
weights 128–255     → scale s1
weights 256–383     → scale s2
...

Each group gets its own scale:

Then a weight in group \(g\) is reconstructed as:

Smaller groups generally allow the quantizer to fit local weight distributions better.

But scales themselves require storage and processing.

So there is another tradeoff:

A group size such as 32, 64, or 128 weights is common in many practical schemes, though the best choice depends on the model and kernel.

The memory savings are enormous¶

Return to our 70B model.

At 16 bits per parameter:

At 8 bits:

At 4 bits, ignoring scales and metadata:

So approximately:

BF16 → 140 GB
INT8 →  70 GB
INT4 →  35 GB

This can completely change deployment economics.

A model that previously required multiple GPUs may fit on fewer GPUs.

And even if we keep the same number of GPUs, less data must move through memory during decode.

Quantization can improve speed for two different reasons¶

The obvious benefit is capacity:

But the second benefit is often more important for serving:

Suppose our simplified 70B model must stream its weights once per low-batch decode step.

At BF16:

At idealized 4-bit storage:

If memory bandwidth is the limiting resource, that is potentially a very large speed advantage.

This connects directly to our prefill-vs-decode lesson:

But 4-bit storage does not mean every operation is 4-bit arithmetic¶

This distinction is important.

A system might store weights in 4-bit form but unpack or dequantize them into a higher-precision representation during matrix multiplication.

Conceptually:

4-bit weights in HBM
        ↓
load compact representation
        ↓
apply scale / dequantize
        ↓
multiply with activations
        ↓
accumulate in higher precision

For example, weights may be 4-bit while activations or accumulators use FP16, BF16, FP32, or another format.

So when someone says a model is "4-bit," ask:

4-bit what?

weights?
activations?
KV cache?
matrix multiply inputs?
accumulators?

These are different design choices.

Weight-only quantization¶

For inference, a common strategy is:

This is called weight-only quantization.

Why is it attractive?

Weights are static.

Once the model is loaded, their values do not depend on the user's prompt.

We can analyze them ahead of time and choose good quantization parameters.

Activations are dynamic.

Their distributions change from token to token and prompt to prompt.

That makes aggressive activation quantization harder.

So weight-only quantization gives much of the memory-bandwidth benefit while avoiding some of the trickiest numerical problems.

Weight-and-activation quantization¶

If both weights and activations are quantized, we might see notation such as:

meaning approximately:

8-bit weights
8-bit activations

Likewise, a system might use lower-bit weights with higher-bit activations.

Quantizing activations can reduce memory traffic and allow specialized low-precision tensor-core operations.

But activation distributions can contain difficult outliers.

This is one reason practical low-precision inference is not simply "convert every tensor to INT4."

Different tensors tolerate different precision.

Why outliers matter so much¶

Imagine most activation values lie between:

but a few values reach:

If one shared scale must cover the range:

then most quantization levels are wasted representing regions where almost no values occur.

The dense cluster near zero gets poor resolution.

Modern quantization methods therefore often do some combination of:

use per-channel or per-group scales
identify outlier channels
keep sensitive values at higher precision
rescale weights and activations
use non-uniform number formats

The basic goal is always the same:

Spend numerical precision where the model actually needs it.

Not all weights are equally sensitive¶

Suppose changing one weight by \(0.1\) barely changes the model's output.

Changing another by \(0.1\) causes a large change.

Those weights should not necessarily be treated identically.

This motivates calibration methods.

We can run representative data through the model and observe which weight or activation errors matter most.

A quantizer can then choose scales or transformations that minimize something closer to actual model error rather than merely raw weight reconstruction error.

This leads to an important distinction.

Minimizing:

is not always the same as minimizing degradation in:

The network cares about how quantization changes its computation, not how aesthetically close the stored numbers look.

Post-training quantization¶

One approach is to train the model normally, then quantize it afterward.

This is post-training quantization, or PTQ.

Conceptually:

train BF16 model
      ↓
finished checkpoint
      ↓
analyze weights / calibration data
      ↓
quantize
      ↓
serve lower-precision model

This is attractive because we do not need to repeat the enormously expensive original training run.

Many inference quantization methods therefore focus on preserving model quality after training is already complete.

Quantization-aware training¶

Another approach exposes the model to quantization effects during training.

This is quantization-aware training, or QAT.

Conceptually:

forward pass:
simulate low-precision weights/activations

backward pass:
update trainable higher-precision parameters

The model learns parameters that remain effective despite the eventual low-precision representation.

QAT can often recover quality at aggressive bit widths, but it adds training complexity and cost.

So the broad tradeoff is:

PTQ
→ cheap and convenient
→ harder at extreme precision

QAT
→ model adapts to quantization
→ requires additional training

Quantization during training is a different problem¶

So far we have focused mostly on inference.

Training needs more numerical care because we are not only performing forward matrix multiplies.

We also compute:

activations
gradients
optimizer statistics
weight updates

Some of these values can have very different dynamic ranges.

If a tiny gradient is rounded to zero, learning may stop for that parameter.

If a large value overflows, training can become unstable.

That is why training has historically used techniques such as mixed precision: perform expensive operations in lower precision while preserving sensitive state or accumulation in higher precision.

The objective is not simply:

It is:

Floating point versus integer quantization¶

"8-bit" does not uniquely specify a number format.

INT8 and FP8 both use 8 bits, but distribute their representable values very differently.

An integer quantizer with a scale has evenly spaced levels:

... -3s, -2s, -s, 0, s, 2s, 3s ...

Floating point uses exponent bits, giving it a wider dynamic range but non-uniform spacing.

That can be valuable when tensor values span several orders of magnitude.

So bit width alone is not enough to understand numerical behavior.

We also need to know the format.

Why 4-bit formats can be non-uniform¶

Suppose neural-network weights approximately cluster near zero.

Using 16 evenly spaced levels may not be ideal.

We might prefer more representable values near regions where weights occur frequently.

A non-uniform 4-bit codebook can map each 4-bit pattern to a selected real value:

0000 → -1.00
0001 → -0.70
0010 → -0.52
...
0111 → -0.08
1000 →  0.00
1001 →  0.08
...
1111 →  1.00

The exact values depend on the quantization scheme.

The deeper idea is that:

Choosing those levels intelligently can reduce error.

The KV cache can be quantized too¶

From our prefill/decode lesson, remember that decode moves not only model weights but also KV-cache data.

KV-cache memory scales roughly with:

For long contexts and many concurrent users, the KV cache can consume enormous amounts of GPU memory.

So we can also quantize:

If a KV cache stored in 16-bit values is reduced to 8 bits, its raw storage roughly halves.

That can allow:

more concurrent requests
longer contexts
larger batches

which may indirectly improve throughput even further.

But again, quantizing K/V changes the attention computation, so quality must be checked.

Quantization and speculative decoding attack different bottlenecks¶

Yesterday's speculative decoding tried to get more useful tokens from each expensive target-model invocation.

Quantization tries to make each invocation cheaper.

They are complementary.

Conceptually:

speculative decoding
→ fewer expensive sequential target passes

quantization
→ less data moved during each pass

A production inference stack can use both.

This is a recurring theme: LLM serving performance usually comes from stacking several orthogonal optimizations rather than discovering one magical trick.

A useful roofline intuition¶

Hardware performance can be thought of as limited by two ceilings:

and:

A workload with low arithmetic intensity hits the bandwidth ceiling first.

A workload with high arithmetic intensity eventually hits the compute ceiling.

Recall:

Quantization reduces the denominator.

If we move half as many bytes while performing similar useful computation, arithmetic intensity rises.

So quantization can shift decode away from the bandwidth-limited region and closer to the compute-limited region.

This is why quantization is not merely a compression technique.

It changes where the workload sits relative to the hardware's limits.

The deeper mental model: precision is a resource¶

When first learning neural networks, we often treat numbers as exact mathematical objects:

Computers never actually store arbitrary real numbers.

They allocate a finite number of bits.

So precision is a resource just like:

GPU memory
memory bandwidth
FLOPs
network bandwidth

Using more precision spends more bits to describe each number.

Sometimes those extra bits materially improve the model.

Sometimes they carry information the model does not need.

Quantization asks:

Which bits are actually earning their keep?

That gives us today's main principle:

The remarkable fact is not that reducing precision introduces error.

Of course it does.

The remarkable fact is how much numerical precision large neural networks can often lose while preserving most of their useful behavior.

And for autoregressive inference, where decode repeatedly streams enormous parameter tensors through memory, every unnecessary bit is not merely stored.

It may be moved again for every generated token.