KV cache: why LLM memory grows with context length
When an LLM generates a token, attention needs the keys and values of every earlier token, and the KV cache stores them so they're computed once instead of at every step. Its size is 2 × layers × KV heads × head dimension × bytes × tokens, so it grows linearly with context length and with the number of concurrent requests. On a busy server, the KV cache, not the weights, is usually what limits how many users fit on a GPU.
Why the cache exists
Generation is sequential: to produce token 501, the model attends to tokens 1 to 500. Without a cache, it would recompute keys and values for all 500 at every step, so the total work grows with the square of the length. With a cache, each step computes keys and values only for the new token and reads the rest from memory.
That's a trade of compute for memory, and on a GPU, reading memory is the bottleneck during generation. That's why generation is memory-bound and why the cost model in this roadmap counts bytes.
Sizing it
def kv_cache_gb(layers: int, kv_heads: int, head_dim: int, tokens: int, bytes_per: int = 2) -> float:
return 2 * layers * kv_heads * head_dim * bytes_per * tokens / 1e9
per_token_mb = kv_cache_gb(32, 8, 128, 1) * 1000
print(f"per token: {per_token_mb:.3f} MB")
for context in (4_000, 32_000, 128_000):
print(f"{context:>7} tokens: {kv_cache_gb(32, 8, 128, context):.2f} GB per request")
users, context = 20, 8_000
print(f"{users} users at {context} tokens: {users * kv_cache_gb(32, 8, 128, context):.1f} GB")
The numbers are for a model shaped like Llama 3 8B (32 layers, 8 KV heads, head dimension 128) at fp16. A 128,000-token context needs about 16.8 GB of cache for one request, more than the 16 GB of weights.
Ways to shrink it
| Technique | What changes |
|---|---|
| Grouped-query attention (GQA) | Fewer KV heads than query heads, set by the model's architecture |
| KV cache quantization | 8-bit or lower cache values |
| Shorter context | Send less: retrieval and summaries instead of everything |
| Prefix sharing | Requests with the same system prompt share cached blocks |
| Paging | Allocate in small blocks instead of reserving the maximum |
Paging: virtual memory for the cache
Early serving systems reserved cache memory for the maximum possible length of every request, and most of it sat unused. PagedAttention, from vLLM, splits the cache into fixed-size blocks and keeps a block table per request, the same idea as virtual memory pages in an operating system. Blocks are allocated as the sequence grows, freed when it ends, and shared between requests with the same prefix.
The paper reports that this reduced memory waste enough to serve 2 to 4 times the throughput of earlier systems at the same latency, on their benchmarks.
Prefix caching
When many requests start with the same system prompt or document, the cache for that prefix can be computed once and reused. Serving engines do this with a structure similar to a trie over token blocks, and API providers expose it as prompt caching, with lower prices for cached input tokens. Put the stable part of your prompt first, so the prefix matches.
Check your understanding
Compute how many concurrent 8,000-token requests fit on a 24 GB GPU running an 8B model at 4-bit weights.
From the floor
In the AI data structures lecture, the KV cache was one of the stages after similarity search, and in the questions at the end, when a builder asked about long context windows, I said the 1 million token context window is largely a marketing claim, that in my experience hallucination increases after about 200,000 tokens, and that I clear the context between working sessions instead of letting it fill up. That's my experience, not a benchmark, and the practical rule is to plan for an effective context smaller than the advertised one, which the KV cache numbers above also argue for.