Floating point precision and memory in AI models
Model weights are floating-point numbers, and each one takes 4 bytes at fp32, 2 at fp16 or bf16, 1 at int8 and half a byte at 4-bit. Weight memory is parameters times bytes per parameter, so a 7 billion parameter model needs about 28 GB at fp32, about 14 GB at fp16 and about 7 GB at int8, before the KV cache and activations. Fewer bits mean less memory and faster serving, with some loss of precision, and that's the cost model behind most decisions about where a model can run.
Why decimals
1.4021 carries more information than 1. A model's knowledge lives in billions of these decimal numbers, its parameters, and the precision of each one is part of what the model knows. Reduce the precision, and you lose a little of that information, in exchange for memory and speed.
How floating-point numbers are stored
A floating-point number has three parts: a sign bit, an exponent (the scale), and a mantissa (the significant digits).
| Format | Bits | Exponent bits | Mantissa bits | Notes |
|---|---|---|---|---|
| fp32 | 32 | 8 | 23 | Full precision |
| fp16 | 16 | 5 | 10 | Smaller range, can overflow |
| bf16 | 16 | 8 | 7 | Same range as fp32, less precision, standard for training |
| int8 | 8 | none | none | Integers from -128 to 127, plus a scale factor |
| int4 or NF4 | 4 | none | none | 16 levels, plus scale factors per group |
Integer formats can't store decimals directly, so quantized models store small integers plus scale factors that map them back to real values. See quantization.
Model memory
def weight_memory_gb(params_billion: float, bits: int) -> float:
return params_billion * 1e9 * bits / 8 / 1e9
for name, params in [("8B", 8), ("13B", 13), ("70B", 70)]:
sizes = ", ".join(f"{bits}-bit {weight_memory_gb(params, bits):.1f} GB" for bits in (32, 16, 8, 4))
print(f"{name}: {sizes}")
Add memory for the KV cache, which grows with context length and concurrent users, and for activations and the serving engine. A 13 billion parameter model needs about 26 GB at 16-bit for weights alone, so it won't load on a 16 GB GPU without quantization, while at 4-bit (about 6.5 GB) it fits with room for the cache.
Floating-point math isn't exact
Floating-point arithmetic rounds, so the order of operations matters.
a = (0.1 + 0.2) + 0.3
b = 0.1 + (0.2 + 0.3)
print(a, b, a == b)
GPUs add numbers in different orders depending on how work is split, so the same input can give slightly different results, which is part of why LLM outputs vary even at temperature 0.
Precision and accuracy
- fp32 to fp16 or bf16: almost no quality loss for inference, and half the memory.
- 16-bit to 8-bit: small loss with good methods.
- 8-bit to 4-bit: more loss, depending on the method, the model and the task, so test on your own evals.
Check your understanding
Compute the weight memory for three models you might deploy at 16-bit and 4-bit, and compare with your GPU's memory. Then add a KV cache estimate from the KV cache page.
From the floor
In the AI data structures lecture, I showed the numbers this way: a 7 billion parameter model in 32-bit floating point is about 28 GB, at 16-bit you cut the memory in half with very little loss, and at 8-bit integers it's about 7 GB, a 75% reduction, which is why models can now run on local machines and edge devices, as long as you keep in mind whether your priority is a faster model or a more accurate one.