Edge Inference Speed Calculator
How fast a model runs on an edge device — inferences per second and milliseconds each — from the model’s operations per inference, the device’s rated throughput and a realistic utilization, with the energy per inference.
A device rated at N TOPS can, at best, do N trillion operations a second; a model that needs G billion operations per inference then runs at N × 1,000 ÷ G inferences a second.
How the edge inference speed calculator works
A device rated at N TOPS can, at best, do N trillion operations a second; a model that needs G billion operations per inference then runs at N × 1,000 ÷ G inferences a second. Real utilization is far below peak — 20–40% is typical, less for memory-bound models — so the calculator divides by an efficiency you choose. Divide the device’s power by the rate for the energy each inference costs, the number that sets battery life.
Formula: inferences/s = device TOPS × 1,000 × utilisation ÷ GOPs per inference; energy = power ÷ rate
Worked examples
| Inputs | Inferences per second | Note |
|---|---|---|
| ResNet-50 (8 GOPs) on a 4 TOPS NPU at 30% | 150 | 150 fps, 33 mJ each |
| YOLOv8n (8.7 GOPs) on a 1 TOPS chip at 25% | 28.7 | 28.7 fps — just short of 30 |
| MobileNetV2 on a phone at 20% | 666.7 | 667 fps |
FAQFrequently asked questions
Why is utilization so low?
Peak TOPS assumes the accelerator’s multiply units are all busy; real layers are limited by memory bandwidth, unsupported operations falling back to the CPU, and small batches. 20–40% is good on an NPU; a CPU running an unoptimized model may reach 5%.
TOPS at which precision?
Vendors quote peak at int8 (or int4); running fp16 typically halves it and fp32 quarters it. Match the TOPS to the precision your model actually runs at after quantization.
How do I find a model’s GOPs?
Model cards and papers list MACs or FLOPs per inference at a given input size (1 MAC = 2 operations). Halving the input resolution roughly quarters the count for convolutional models.
What about battery life?
Energy per inference times inferences per day is the daily budget; the per-watt-hour row gives inferences per watt-hour, so a 10 Wh battery running only the model supports that many times ten.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- NIST AI Risk Management Framework — the US federal AI framework
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.