Quantization explained: fp32, fp16, int8 and 4-bit models

Why size matters

Every parameter of a model is a number, and every number takes memory. A model's size in memory is roughly the number of parameters multiplied by the bytes per parameter.

def weights_gb(params_billions: float, bits: int) -> float:
    return params_billions * 1e9 * bits / 8 / 1e9


for bits in (32, 16, 8, 4):
    print(f"7B model at {bits:>2}-bit: {weights_gb(7, bits):5.1f} GB")

A model that needs 28 GB can't run on a laptop, a phone or a cheap GPU. The same model at 4-bit fits in about 3.5 GB, which is why you can now run capable models on local machines and edge devices.

How numbers are stored

Computers store decimal numbers as floating-point values: a sign, an exponent and a mantissa (the significant digits). More bits means a wider range and more precision.

Format	Bits	Notes
fp32	32	Full precision, the training reference
fp16	16	Half precision, smaller range, can overflow
bf16	16	Same range as fp32 with less precision, standard for training and serving on modern GPUs
int8	8	Integers from -128 to 127, plus a scale to map them back to real values
4-bit (int4, NF4)	4	16 possible values per weight, plus scales per group of weights

Going from 32-bit to 16-bit floating point barely changes model quality, because neural networks tolerate small rounding errors. Going below 16 bits means you can no longer store the floating-point values directly, so you store small integers plus scaling factors, and that is where the real trade-offs begin. See floating-point precision.

How quantization works

The simplest version, symmetric quantization, maps a group of weights to integers using one scale:

1. Find the largest absolute value in the group.
2. Divide it by the largest integer you can store (127 for int8) to get the scale.
3. Divide every weight by the scale and round to the nearest integer.
4. To use the weights, multiply the integers by the scale again.

def quantize(weights: list[float], bits: int = 8) -> tuple[list[int], float]:
    qmax = 2 ** (bits - 1) - 1
    scale = max(abs(w) for w in weights) / qmax
    return [round(w / scale) for w in weights], scale


def dequantize(q: list[int], scale: float) -> list[float]:
    return [v * scale for v in q]


weights = [0.42, -1.37, 0.05, 0.91, -0.66, 1.12, -0.08, 0.3]
for bits in (8, 4):
    q, scale = quantize(weights, bits)
    restored = dequantize(q, scale)
    error = max(abs(a - b) for a, b in zip(weights, restored))
    print(f"{bits}-bit: {q}  max error {error:.4f}")

With 8 bits, the restored weights are very close to the originals. With 4 bits, there are only 15 usable levels in this scheme, so the error grows. Real methods reduce that error with small groups of weights per scale, smarter rounding, and by protecting the weights that matter most.

Methods and formats you'll meet

• Post-training quantization (PTQ): quantize a trained model without retraining. Most of what you use is this.
• GPTQ and AWQ: PTQ methods for GPUs that use a small calibration dataset to decide how to quantize with the least damage. AWQ protects the weights connected to the most important activations.
• GGUF: the file format used by llama.cpp and Ollama, with quantization levels like Q8_0, Q5_K_M and Q4_K_M. Q4_K_M is a common default for laptops.
• bitsandbytes: on-the-fly 8-bit and 4-bit (NF4) quantization in Hugging Face Transformers, used for QLoRA training.
• FP8: an 8-bit floating-point format supported by recent NVIDIA GPUs, used for fast serving with very small quality loss.
• Quantization-aware training (QAT): simulate quantization during training so the model learns to tolerate it. More work, better results at low bit widths.

What you trade

Precision	Bytes per weight	7B model (weights only)	Typical use
fp32	4	~28 GB	Training reference
fp16 or bf16	2	~14 GB	Standard serving
int8 or FP8	1	~7 GB	Cheaper serving
4-bit (GPTQ, AWQ, GGUF Q4)	0.5	~3.5 GB	Laptops, edge, single small GPU

Quality loss depends on:

• The model: larger models usually tolerate quantization better than small ones.
• The task: reasoning, maths and code can degrade more than chat or classification.
• The method: a good 4-bit method can be close to 16-bit, a naive one can be much worse.

Speed doesn't always improve. Lower precision moves less data, which helps the memory-bound decode step, but if the hardware doesn't support the format natively, converting values back adds work.

Also remember the KV cache, which grows with context length and batch size and can take as much memory as the weights. Some servers quantize the KV cache too. See inference optimization.

How to choose

1. Decide where the model must run and how much memory you have.
2. Pick the highest precision that fits, with room for the KV cache.
3. Run your eval set on the 16-bit model and on the quantized versions.
4. Compare pass rate, latency, throughput and memory.
5. Ship the smallest version whose pass rate you can accept.

From the floor

In the AI data structures session, I showed the numbers this way: a 7 billion parameter model in 32-bit floating point takes about 28 GB, and quantized to 8-bit integers it takes about 7 GB, a 75% reduction, which is why models can now run on local machines and edge devices.

Subnibble, a library I'm building as a learning project for Buildership, explores quantization below 4 bits.

Related

• Floating-point precision
• Run LLMs locally
• Inference optimization
• LoRA and QLoRA

Sources

• GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, Frantar and others (2022)
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, Lin and others (2023)
• Quantization, Hugging Face Transformers documentation
• llama.cpp