Batching Latency–Throughput Calculator
What happens to throughput and latency when a serving system batches requests — from the single-request time, how much each extra request adds, the arrival rate and the maximum wait to fill a batch.
A GPU does a batch of eight in much less than eight times the time of one, because the weights are read once and the arithmetic units are better used; a marginal factor of 0.
How the batching latency–throughput calculator works
A GPU does a batch of eight in much less than eight times the time of one, because the weights are read once and the arithmetic units are better used; a marginal factor of 0.3 means each extra request adds 30% of a single request’s time. Throughput rises with batch size, but a request now waits for the batch to fill — (B − 1) ÷ arrival rate, capped by a timeout — and then for the longer batch to run. The calculator shows both sides so you can pick the batch that meets a latency target.
Formula: batch time = t₁ × (1 + (B − 1) × k); throughput = B ÷ batch time; latency ≈ min(fill time, timeout) + batch time
Worked examples
| Inputs | Throughput at this batch size | Note |
|---|---|---|
| 40 ms alone, k 0.3, batch 8, 100 req/s | 64.5 req/s | 65 req/s — overloaded at 100/s |
| Batch 32 at the same rate | 77.7 req/s | 82 req/s — still short |
| Cheap batching: k 0.1, batch 16, 200 req/s | 160 req/s | 160 req/s at about 137 ms |
FAQFrequently asked questions
What is a typical marginal factor?
For transformer inference on a GPU, 0.1–0.3 at small batches — the work is memory-bound and extra requests are nearly free — rizing toward 1 once the arithmetic units saturate. Measure it: time batch 1 and batch 8 and solve for k.
Why does latency rize with batch size?
Two ways: a request waits for the batch to fill, and the batch takes longer to run than a single request. At low arrival rates the fill wait dominates and batching hurts; at high rates the batch fills instantly and only the run time matters.
What is the timeout for?
To bound the wait when traffic is light: the server runs whatever it has after the timeout rather than waiting for a full batch. It trades a little throughput for predictable latency.
What is continuous batching?
For token generation, adding new requests to a batch as others finish instead of waiting for whole batches — it keeps the GPU full without the fill wait. The trade-off here still applies per decoding step.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- NIST AI Risk Management Framework — the US federal AI framework
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.