GPU Memory Requirement Calculator
How much GPU memory a model needs — weights, KV cache and overhead — and whether it fits the card you have.
Weights are parameters times bytes per parameter: two bytes at 16-bit, one at 8-bit, half at 4-bit.
How the gpu memory requirement calculator works
Weights are parameters times bytes per parameter: two bytes at 16-bit, one at 8-bit, half at 4-bit. A 70-billion-parameter model is 140 GB at fp16 and 35 GB at 4-bit, which is the difference between four cards and one.
The part people forget is the KV cache, which grows with batch size and context length and can exceed the weights on long-context serving. Loading is not the same problem as serving, and this shows both.
Formula: memory = params × bytes + KV cache + overhead
Worked examples
| Inputs | Total memory needed | Note |
|---|---|---|
| A 70B model at fp16 | 326.8 GB | needs several 80 GB cards |
| The same at 4-bit | 82.45 GB | fits one card |
| Long context | 1.53 TB | the KV cache dominates |
FAQFrequently asked questions
How much VRAM does a model need?
Two bytes per parameter at fp16, so a 7B model is 14 GB and a 70B is 140 GB — before the KV cache and overhead.
What is the KV cache?
The stored keys and values for every token in the context, per layer. It grows with both context length and concurrent requests, and on long context it can exceed the weights.
Does quantisation hurt quality?
8-bit is usually indistinguishable. 4-bit costs a little accuracy on hard tasks and buys a fourfold memory reduction, which is often the right trade.
Why does batch size matter so much?
Because each concurrent request carries its own KV cache. Doubling concurrency doubles that part of the memory.
What is the overhead?
Activations, workspace and the framework itself. Ten per cent of the weights plus a gigabyte is a reasonable planning allowance.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- National AI Centre — Australia's national AI body
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.