Quantization stores model numbers with fewer bits.
Suppose a learned weight is:
The model may behave almost identically if that becomes 0.137 or 0.14.
Across billions of weights, removing unnecessary precision can save enormous memory and bandwidth.
A simple 4-bit example¶
Suppose:
Use only 16 possible codes because:
Choose scale:
Quantize:
For \(w=0.63\):
Reconstruct:
Error:
For \(0.27\):
Groupwise scaling¶
One scale for an entire model is poor because different regions have different numeric ranges.
So weights are often quantized in groups, for example:
weights 1–128 → scale A
weights 129–256 → scale B
...Smaller groups improve fidelity at the cost of metadata.
Why inference benefits¶
70B parameters:
BF16:
Idealized 8-bit:
Idealized 4-bit:
Decode is often bandwidth-bound, so moving 35 GB rather than 140 GB can radically change throughput.
But compute support matters¶
Low-bit storage is useful only if kernels/hardware can efficiently dequantize or directly multiply low-precision values.
The stack must align:
Why training is harder¶
If representable values near a weight are only:
an optimizer update:
may round straight back to 0.625 and vanish.
Hence mixed precision and quantization-aware methods.
The broad principle:
Quantization as noise¶
A useful mathematical view is:
where \(\epsilon\) is quantization error.
The goal is to choose scales, grouping, and representations so the network’s computation is insensitive to that error.
Some weights are much more sensitive than others, particularly outliers. Good quantization methods often devote more precision or special treatment to these difficult values.
Weight-only versus activation quantization¶
Weight-only quantization shrinks persistent model storage and weight bandwidth while leaving activations at higher precision.
Quantizing activations can unlock more acceleration, but activation ranges vary dynamically with inputs, making the problem harder.
KV caches can also be quantized, which is particularly attractive for long-context serving.
Why 4-bit is not literally “every number has only 16 global choices”¶
Practical systems use many local scales and sometimes zero-points. Each small block may have its own mapping between low-bit codes and real values.
That lets 4 bits work surprisingly well despite the apparent coarseness.
Accuracy-throughput trade-off¶
A lower-bit model is valuable only if quality remains acceptable.
Thus deployment is an empirical decision:
For some models/tasks, 4-bit is excellent. For others, 8-bit or higher precision is justified.
Quantization and matrix multiplication¶
Suppose the original operation is:
With quantized weights we store integer-like codes \(Q\) and scale metadata \(s\), conceptually:
A kernel can either reconstruct values on the fly or use hardware instructions that directly operate on supported low-precision formats.
The best implementations fuse these steps so temporary full-precision copies of the weights are never written back to HBM.
Otherwise much of the bandwidth advantage would disappear.
KV-cache quantization¶
Long-context serving may use enormous KV caches even after weight quantization.
Quantizing cached K/V values attacks a different memory term:
Dropping cache precision from 16 to 8 bits roughly halves that component.
Again the engineering question is not simply “can we compress it?” but “does attention remain accurate enough after compression?”
Calibration and quantization¶
Quantization parameters are often chosen from representative calibration data.
The goal is to observe realistic activation or weight ranges and select scales that minimize harmful error.
Poor calibration can waste much of the available numeric range or clip important outliers.
This is a reminder that low-bit inference is not simply a file-conversion step. Good quantization is a small model-optimization problem in its own right, involving the distribution of values the network actually uses.