Model Training Cost Calculator
What a training run costs — from parameters and tokens to GPU hours, wall-clock time and money.
The compute for a transformer training run is about 6 × parameters × tokens floating-point operations.
How the model training cost calculator works
The compute for a transformer training run is about 6 × parameters × tokens floating-point operations. Divide by the hardware's effective throughput and you have GPU-seconds.
The word doing the work is "effective". A card's peak FLOPS is a marketing number; real training runs achieve 35 to 55% of it, and using peak rather than achieved throughput is the single biggest source of optimistic training estimates.
Formula: FLOPs ≈ 6 × params × tokens; hours = FLOPs / (GPUs × achieved FLOPS)
Worked examples
| Inputs | Training cost | Note |
|---|---|---|
| A 7B model on 140B tokens | 10,321.87 | 20 tokens per parameter |
| Optimistic throughput | 4,128.75 | less than half the cost — and unachievable |
| A heavily trained small model | 73,727.67 | over-trained by design |
FAQFrequently asked questions
How is training compute estimated?
About six floating-point operations per parameter per token — two for the forward pass and four for the backward pass.
What utilization should I assume?
Thirty-five to fifty-five per cent of peak for a well-tuned large run. Using peak makes the estimate roughly half of reality.
What is compute-optimal?
The Chinchilla work put it near twenty tokens per parameter for a fixed compute budget. Models meant to be served cheaply are deliberately trained far past that.
Does this include failed runs?
No, and it should. Real projects spend substantially on runs that diverge, restart or get thrown away.
What about data preparation?
Not counted here. Cleaning, deduplicating and tokenizing a large corpus is significant compute in its own right.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- NIST AI Risk Management Framework — the US federal AI framework
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.