Quantization

Quantization is the reason a model that needs 140 GB in its original form can run on a desktop computer. It is also the source of most of the cryptic suffixes, such as Q4_K_M, AWQ or FP8, that you see on model downloads.

What quantization does

Every weight in a model is a number stored with some precision, the number of bits used to represent it. Models are usually trained in 16-bit formats. Quantization means storing those weights with fewer bits, for example 8 or 4, after (or sometimes during) training.

An analogy: a photo saved as a high-quality image versus a compressed one. The compressed file is a fraction of the size and usually looks the same at a glance, but push the compression too far and you start to see blocky artefacts. Quantization is lossy compression for model weights.

Fewer bits help in two ways:

• Memory: a 4-bit model is roughly a quarter to a third the size of the 16-bit original, depending on overhead.
• Speed: generating text is limited mainly by how fast weights can be read from memory, so smaller weights mean more tokens per second.

How it works, in brief

You cannot simply chop bits off a number. Instead, weights are split into small blocks (often 16 to 256 weights). For each block the method stores one scale, a shared multiplier, and then stores each weight as a small integer or tiny floating-point value relative to that scale. When the model runs, the values are multiplied back by the scale. Because each block gets its own scale, a few unusually large weights do not ruin precision for the rest.

The scales cost extra bits, which is why "4-bit" formats typically average 4.25 to about 4.8 bits per weight.

The common formats

What the names mean:

• FP16 and BF16 are 16-bit floating-point formats. BF16 ("brain float") keeps the same range as 32-bit numbers with less precision, which makes training more stable.
• FP8 is an 8-bit floating-point format supported natively by recent data-centre GPUs. INT8 stores 8-bit integers plus scales.
• GGUF is the file format used by llama.cpp and the tools built on it. Its quant names read as: Q plus the bit count, K for the "k-quant" block scheme, and S, M or L for small, medium or large mixes. A Q4_K_M file stores most weights at about 4.5 bits but keeps some sensitive tensors at higher precision, averaging about 4.8 bits. Types beginning with IQ use an importance matrix, collected by running sample text through the model, to decide where precision matters most; they are the usual choice below 4 bits.
• GPTQ and AWQ are post-training methods popular for GPU serving. GPTQ adjusts the remaining weights to compensate for rounding errors, layer by layer. AWQ (activation-aware weight quantization) observes which weights handle the largest signals on sample data and protects that small fraction; its authors report that protecting about 1% of salient weights greatly reduces error.
• MXFP4 and NVFP4 are 4-bit floating-point formats designed for hardware support. MXFP4 is an open "microscaling" standard that groups 32 values under one power-of-two scale. NVFP4, from NVIDIA, groups 16 values under a more precise 8-bit scale, about 4.5 bits per value in total, trading a little size for accuracy.

What you give up

Quality loss grows as bits shrink, but not evenly. As rough guidance, not a guarantee:

• 8-bit is generally very close to the original.
• 5 and 6-bit are usually close enough that most users cannot tell.
• 4-bit is the common sweet spot: a noticeable size saving with a small, task-dependent quality loss.
• Below 4 bits, errors grow quickly, and smaller models suffer more than large ones.

Averages can hide problems. A 2025 study of low-bit post-training quantization (AWQ, GPTQ and SmoothQuant) on small models (0.5B to 7B) found mathematical reasoning could degrade sharply, because an early slip in a multi-step solution carries through to the answer. Coding, maths, long-context recall and non-English text are worth checking specifically.

Quantization-aware training

Most quantization happens after training (post-training quantization). In quantization-aware training (QAT), the model is trained or fine-tuned while simulating low-precision arithmetic, so it learns weights that survive rounding.

Worked example: Google released QAT versions of Gemma 3. The 27B model needs 54 GB for its weights in BF16 and 14.1 GB in int4, and Google reported that, compared with ordinary 4-bit (Q4_0) quantization, QAT cut the rise in perplexity by 54% on its llama.cpp test. (Perplexity is a measure of how surprised a model is by text; lower is better.) Similarly, gpt-oss-120b was post-trained with its expert weights in MXFP4, so that 4-bit version is the intended form, not a later compromise.

How to pick a quant

1. Start from memory. Estimate file size as parameters × bits ÷ 8, plus 5%, then add the KV cache and about 1.5 GB of overhead. Example: an 8B model at Q4_K_M is about 5 GB; at Q8_0 about 9 GB; at BF16 about 16–17 GB.
2. Prefer the highest precision that fits with room for context. Leave space for the KV cache.
3. A bigger model at 4-bit often beats a smaller model at 8-bit for the same memory, but very low-bit versions of large models can lose that advantage. Test both if you can.
4. Use the official quant when one exists. QAT or vendor-made releases are usually better than ad-hoc conversions.
5. Match your runtime. GGUF for llama.cpp-based tools, AWQ, GPTQ, FP8 or NVFP4 for GPU serving stacks, and MXFP4 where the model ships in it.

Going deeper

Uniform quantization: scale and zero point

Almost every integer scheme is built on the affine (asymmetric) map between a real value $x$ and a $b$-bit integer $q$:

where $s > 0$ is the scale (the step between representable values), $z$ is the integer zero point (the integer that represents real 0), and $\hat{x}$ is the dequantized value. To cover a range $[x_{\min}, x_{\max}]$:

Symmetric quantization, common for weights because they are roughly centred on zero, fixes $z = 0$ and uses a signed range: $s = \max|x| / (2^{b-1} - 1)$ and $q \in [-(2^{b-1}-1),\, 2^{b-1}-1]$, so 4-bit uses the integers $-7$ to $7$.

Worked example (asymmetric, 8-bit). For the range $[-0.8, 2.4]$: $s = 3.2 / 255 \approx 0.012549$ and $z = \operatorname{round}(63.75) = 64$. To store $x = 1.0$: $1.0 / s \approx 79.69$, rounds to 80, plus 64 gives $q = 144$. Dequantizing: $0.012549 \times (144 - 64) \approx 1.0039$, an error of 0.0039, within the guaranteed bound $s/2 \approx 0.0063$.

How big is the error?

For values inside the range, rounding error is at most $s/2$. If values are spread smoothly across many steps, the error behaves like a uniform random variable on $[-s/2, s/2]$, whose mean squared error is

Each extra bit roughly halves $s$, cutting mean squared error by about 4, or about 6 dB of signal-to-quantization-noise ratio (SQNR) per bit. Two consequences matter in practice:

1. Outliers are expensive. In symmetric quantization $s$ is set by $\max|x|$, so one large weight inflates the step for its whole block.
2. Low bit widths fall off a cliff. Going from 4 to 3 bits changes the symmetric step from $\max|x|/7$ to $\max|x|/3$, so error variance grows by $(7/3)^2 \approx 5.4$ times, not 2.

Worked example (the outlier problem). Quantize the block $(0.12, -0.50, 0.33, 0.07, -0.21, 1.90, 0.02, -0.15)$ to 4-bit symmetric. Here $s = 1.90 / 7 \approx 0.271$, and the integers are $(0, -2, 1, 0, -1, 7, 0, -1)$. Three of the eight weights round to exactly 0 and two more to a single step, and the mean squared error of the seven ordinary weights is about $6.2 \times 10^{-3}$. If the outlier 1.90 is stored separately at higher precision, the remaining block has $s = 0.50/7 \approx 0.071$ and the error on those seven weights drops to about $2.5 \times 10^{-4}$, about 25 times smaller. Isolating outliers in higher precision is the idea behind LLM.int8() (which does it for the few activation feature dimensions with extreme values) and behind formats that keep sensitive tensors at more bits. Small blocks, per-channel scales and outlier handling are all ways of keeping $s$ small.

The bit budget of block formats

With block size $B$ and $m$ bits of metadata per block, the average cost is $b + m/B$ bits per weight. A 4-bit block of 32 with a 16-bit scale costs $4 + 16/32 = 4.5$ bits (llama.cpp's Q4_0 layout); adding a 16-bit minimum for asymmetric blocks gives 5.0 (Q4_1). MXFP4 uses an 8-bit power-of-two scale per 32 values: $4 + 8/32 = 4.25$. NVFP4 uses an 8-bit scale per 16 values: $4 + 8/16 = 4.5$.

Smarter rounding: GPTQ in one equation

Round-to-nearest treats each weight alone. GPTQ instead minimises each layer's output error on calibration inputs $X$:

It processes the weight matrix one column at a time (in the same order for every row) and, after rounding weight $w_q$ in a row, adjusts the not-yet-quantized weights $F$ to compensate, using the inverse of the Hessian $H = 2XX^\top$:

Intuitively, if rounding pushed one weight up, correlated weights are nudged down so the layer's output barely changes. AWQ takes a different route: it multiplies the weights of input channels that carry large activations by a per-channel factor (dividing the activations by the same factor, so the product is unchanged) before quantizing, which makes rounding error on those important weights relatively smaller.

Code: block size versus error

import numpy as np

def quantize_blockwise(w, bits=4, block=32):
    """Symmetric round-to-nearest quantization with one scale per block."""
    qmax = 2 ** (bits - 1) - 1                       # 7 for 4-bit
    wb = w.reshape(-1, block)
    scale = np.abs(wb).max(axis=1, keepdims=True) / qmax
    scale[scale == 0] = 1.0                          # avoid divide-by-zero
    q = np.clip(np.round(wb / scale), -qmax, qmax).astype(np.int8)
    return q, scale

def dequantize(q, scale, shape):
    return (q * scale).reshape(shape)

rng = np.random.default_rng(0)
w = rng.standard_normal(4096 * 256) * 0.02           # weight-like values
w[rng.choice(w.size, 200, replace=False)] *= 25      # a few large outliers

for bits in (8, 4, 3):
    for block in (32, 4096):
        q, s = quantize_blockwise(w, bits, block)
        err = w - dequantize(q, s, w.shape)
        sqnr = 10 * np.log10(np.mean(w**2) / np.mean(err**2))
        bpw = bits + 16 / block                      # fp16 scale per block
        print(f"{bits}-bit, block {block:4d}: {bpw:.3f} bits/weight, "
              f"SQNR {sqnr:5.1f} dB")

On Gaussian weights with a few injected outliers, the 4-bit SQNR is about 19.6 dB with blocks of 32 but only about 5 dB with one scale per 4,096 weights, for a saving of under half a bit per weight. That is why every practical low-bit format uses small blocks.

Common pitfalls

• Judging by perplexity alone. Average perplexity can barely move while maths, code or long-context recall degrade; test the tasks you care about.
• Calibrating on the wrong data. GPTQ, AWQ and importance matrices learn from sample text; English-only calibration can hurt other languages and code.
• Comparing "4-bit" files without checking bits per weight. Q4_0, Q4_K_M, AWQ and NVFP4 differ in real size and quality.
• Quantizing everything uniformly. Embeddings, output heads and some attention tensors are more sensitive; good recipes keep them at higher precision.
• Expecting speed without kernel support. Weight-only quantization speeds decode mainly when the runtime has fused low-bit kernels for your hardware.

In practice

• Treat Q4_K_M (or the equivalent 4-bit format) as a sensible default for local use, and step up to 5, 6 or 8-bit if you have memory to spare.
• Check a quantized model on the tasks you care about, especially maths, code and long documents.
• When comparing hosted providers serving the same open model, ask what precision they use; it can explain quality differences.