KV Cache Memory Calculator
The memory a transformer’s key–value cache takes for a context length and batch — from layers, KV heads, head size and precision — and how many tokens fit in the memory left after the weights.
During generation the model keeps the keys and values of every previous token in every layer so it need not recompute them: 2 (K and V) × layers × KV heads × head dimension × bytes per element for each token.
How the kv cache memory calculator works
During generation the model keeps the keys and values of every previous token in every layer so it need not recompute them: 2 (K and V) × layers × KV heads × head dimension × bytes per element for each token. At fp16 a 32-layer model with 8 KV heads of 128 uses 128 KB a token — a gigabyte for 8,000 tokens per request. Grouped-query attention (fewer KV heads than query heads) and 8-bit caches are how long contexts are made affordable.
Formula: bytes per token = 2 × layers × KV heads × head dim × bytes per element; total = per token × tokens × batch
Worked examples
| Inputs | KV cache for the batch | Note |
|---|---|---|
| 32 layers, 8 KV heads of 128, fp16, 8k × 8 | 8.59 GB | 131 KB/token; 8.6 GB |
| No GQA: 32 KV heads | 34.36 GB | 34.4 GB — over budget |
| int8 cache, 128k context, one request | 8.59 GB | 8.6 GB |
FAQFrequently asked questions
Why does the KV cache matter more than the weights?
Weights are paid once per GPU; the cache is paid per token per request, and long contexts with many concurrent users make it the larger of the two. It is what limits how many users a serving GPU can hold.
What is grouped-query attention?
Sharing each key–value head among several query heads: a model with 32 query heads and 8 KV heads has a cache a quarter the size of one with 32 of each, at little cost in quality. Most recent models use it.
Does quantising the cache hurt?
Eight-bit KV caches are close to lossless for most models and halve the memory; four-bit caches show some degradation on long-context tasks. Serving frameworks expose it as an option.
How does paged attention help?
It allocates the cache in blocks on demand instead of reserving the maximum context per request, so memory is used for tokens that exist rather than tokens that might. The per-token figure is the same; the waste disappears.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- National AI Centre — Australia's national AI body
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.