Quantized Model Size Calculator
How big a model is at each weight precision — 32, 16, 8 and 4 bits — whether it fits the memory of an edge device with room for activations, and the rough inference rate the device's compute allows.
A parameter at 16 bits is two bytes, so a 7-billion-parameter model is 14 GB — far beyond a phone.
How the quantized model size calculator works
A parameter at 16 bits is two bytes, so a 7-billion-parameter model is 14 GB — far beyond a phone. At 4 bits it is 3.5 GB plus a small overhead for the scale factors, and fits. Quantization is the difference between a model that runs on the device and one that does not.
The device also needs working memory for activations and the KV cache, and its compute sets a ceiling on tokens per second: each token needs about two operations per parameter.
Formula: size = N × bits / 8 × (1 + overhead); tokens/s ≈ TOPS × 10¹² × utilisation / (2N)
Worked examples
| Inputs | Model size at this precision (GB) | Note |
|---|---|---|
| A 3B model at 4-bit on a 6 GB phone | 1.59 | fits with room |
| A 7B model at 8-bit | 7.42 | does not fit |
| A tiny model at 16-bit | 1 | fits, fast |
FAQFrequently asked questions
What is quantization?
Storing weights in fewer bits — 8 or 4 instead of 16 — with a scale factor per group to recover the range. It cuts memory and bandwidth by two to four times with a small accuracy cost.
How much accuracy is lost?
Little at 8-bit; usually a point or two of benchmark score at 4-bit for models above a few billion parameters. Below 4 bits the loss grows quickly.
What is the overhead?
The scale and zero-point stored for each group of weights — a few per cent at 4-bit with 32-weight groups, less with larger groups.
Why does the device need working memory too?
Activations, the attention cache for the context so far, and the runtime itself. A gigabyte is a reasonable allowance for a small model and a modest context.
What limits speed on a device?
Memory bandwidth as often as compute: every token reads every weight. The compute-bound estimate here is an upper bound; bandwidth can halve it.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- NIST AI Risk Management Framework — the US federal AI framework
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.