LLM Response Time Calculator
How long a language-model response takes from the user's point of view — time to first token plus tokens at the generation rate — and how streaming, a shorter answer or a faster model change what the user waits for.
A response has two phases: the prefill, where the prompt is processed and the first token appears, and the decode, where tokens stream out one at a time.
How the llm response time calculator works
A response has two phases: the prefill, where the prompt is processed and the first token appears, and the decode, where tokens stream out one at a time. Total time is the first-token latency plus output tokens over the generation rate. With streaming the user starts reading at the first token and the wait that matters is the time to first token; without it, they wait for everything.
The lever most people miss is the length of the answer. Halving the output halves the decode time; a faster model saves less.
Formula: total = TTFT + output tokens / tokens per second; perceived (streaming) = TTFT, if reading keeps pace
Worked examples
| Inputs | Total response time | Note |
|---|---|---|
| A 400-token streamed answer | 10.68 s | feels quick |
| The same without streaming | 10.68 s | a ten-second wait |
| A slow model | 100.68 s | reading overtakes generation |
FAQFrequently asked questions
What is time to first token?
The delay before the first output token appears — the prompt being processed plus queueing and network. It is what a streaming user experiences as the wait.
Why does streaming help?
Because reading is slower than generation for most models, so the user reads while the rest arrives. The ten-second answer becomes a sub-second wait.
What generation rate is typical?
Twenty to a hundred tokens a second for hosted models, depending on size and load. Reading is about five to six tokens a second, so anything above ten keeps pace.
What is the cheapest way to feel faster?
A shorter answer. Decode time is proportional to output tokens, and users rarely want the last third of a long response.
Does the prompt length matter?
Through time to first token — prefill grows with prompt tokens. A long context or a big document ahead of the question shows up there, not in the generation rate.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- NIST AI Risk Management Framework — the US federal AI framework
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.