Inference Throughput Calculator
How fast a model serves — tokens per second, time to first token, total latency, and what a served request costs on your own hardware.
Generation is memory-bandwidth bound, not compute bound: each token requires reading the whole model from memory, so tokens per second is roughly bandwidth divided by model size.
How the inference throughput calculator works
Generation is memory-bandwidth bound, not compute bound: each token requires reading the whole model from memory, so tokens per second is roughly bandwidth divided by model size. That is why a bigger model is slower even on the same card, and why quantisation speeds generation up as well as shrinking it.
Batching is the lever. One request wastes most of the bandwidth; batching amortises the same read across many requests, which multiplies throughput at the cost of a little latency per request.
Formula: tokens/s ≈ bandwidth / model bytes × batch efficiency
Worked examples
| Inputs | Tokens per second (total) | Note |
|---|---|---|
| A 13B model at fp16 | 1,360.1 | batching multiplies throughput |
| No batching | 128.8 | the same card, a fraction of the output |
| 4-bit weights | 5,440.4 | four times the token rate |
FAQFrequently asked questions
Why is generation memory-bound?
Because producing each token reads every weight in the model. The arithmetic is trivial next to the bandwidth needed to fetch the weights.
How much does batching help?
A great deal — the same weight read serves every request in the batch. Returns diminish, but going from 1 to 16 typically multiplies total throughput several times over.
Does batching make responses slower?
Slightly, per request. Total throughput rises far more than individual latency does, which is the right trade for a busy service.
Why does quantisation speed things up?
Because there are fewer bytes to read per token. 4-bit weights are a quarter the bandwidth of fp16, and generation scales almost directly with that.
What about the prompt?
Prompt processing is compute-bound and parallel, so it is fast — it is generation, one token at a time, that dominates latency.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- AI Safety Institute — the UK government AI body
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.