Compute-Optimal Model Size Calculator
For a given training compute budget, the model size and dataset size that make best use of it under the Chinchilla scaling rule — about twenty tokens per parameter — and what a chosen model size would need instead.
Training compute is roughly six FLOPs per parameter per token: C = 6ND.
How the compute-optimal model size calculator works
Training compute is roughly six FLOPs per parameter per token: C = 6ND. The Chinchilla result is that for a fixed C, loss is lowest when tokens are about twenty times parameters. Substituting D = 20N gives N = √(C / 120): the compute-optimal model size grows with the square root of the budget.
The budget itself comes from the cluster: GPUs times their sustained throughput times the days you have. Model FLOPs utilization — the share of peak the training loop actually achieves — is usually 30 to 50%.
Formula: C = GPUs × FLOP/s × MFU × seconds; N_opt = √(C / 120); D_opt = 20 N_opt
Worked examples
| Inputs | Compute-optimal model size (parameters) | Note |
|---|---|---|
| 256 H100-class GPUs for a month | 46.77 billion parameters | about a 20B model |
| A small cluster | 3.99 billion parameters | a few hundred million parameters |
| Over-training a small model | 14.79 billion parameters | fewer parameters, far more tokens |
FAQFrequently asked questions
What is the Chinchilla rule?
The 2022 finding that for a fixed compute budget, loss is minimized with roughly twenty training tokens per parameter. Earlier models were far larger and under-trained for their budgets.
Why six FLOPs per parameter per token?
Two for the forward pass, four for the backward, per parameter per token, as a rule of thumb. Attention over long contexts adds more; this is the standard estimate.
What is model FLOPs utilization?
The share of the hardware's peak that the training loop sustains, after communication, memory stalls and pipeline bubbles. Thirty to fifty per cent is typical; above sixty is excellent.
Should I always train compute-optimal?
Only if training cost is all that matters. A smaller model trained on far more tokens costs more to train but less to serve, which is why production models are often trained well past twenty tokens per parameter.
What if I already know the model size?
Enter it: the page gives the tokens that size wants at your ratio, the compute that takes, and whether it fits inside the cluster-days you have.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- NIST AI Risk Management Framework — the US federal AI framework
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.