LLM Parameter Count Calculator
How many parameters a decoder-only transformer has from its depth, hidden size, feed-forward width and vocabulary — and the memory those weights take at 16-, 8- and 4-bit precision.
Each transformer block has four attention projections of d × d and a feed-forward pair of d × f and f × d; with the usual f = 4d that is 12 d² per layer.
How the llm parameter count calculator works
Each transformer block has four attention projections of d × d and a feed-forward pair of d × f and f × d; with the usual f = 4d that is 12 d² per layer. Add the token embedding of vocabulary × d (and the same again if the output head is not tied to it). Layer norms and biases add a rounding error. A 32-layer, 4,096-wide model with a 32k vocabulary comes to about 6.6 billion parameters — the shape of a “7B” model.
Formula: per layer ≈ 4d² + 2 d f; total ≈ L (4d² + 2df) + V d (× 2 if untied)
Worked examples
| Inputs | Parameters | Note |
|---|---|---|
| 32 layers, d 4,096, 32k vocab, tied | 6.57 billion | 6.57 billion |
| GPT-2 small: 12 layers, d 768, 50k vocab | 123.53 million | 124 million |
| 80 layers, d 8,192, untied 128k vocab | 66.52 billion | 66.5 billion |
FAQFrequently asked questions
Why does the count come out under the advertised size?
Marketing rounds up, and architectures vary: gated feed-forward layers (SwiGLU) use three matrices with f ≈ 2.7d, grouped-query attention shrinks the K and V projections, and some models untie the output head. Enter your model’s real f and tying to get closer.
How much memory do I need to run it?
The weights row is the floor; inference also needs the KV cache (see that calculator) and working buffers, typically another 10–30%. Training needs several times more for optimiser states and gradients.
Does the parameter count predict quality?
Roughly, at the same training data — scaling laws tie loss to parameters and tokens together. A smaller model trained on far more tokens often beats a larger one trained on fewer.
What about mixture-of-experts models?
They have many more total parameters than they use per token: count the experts’ feed-forward layers into the total, but only the active experts into the per-token compute.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- AI Safety Institute — the UK government AI body
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.