Accuracy Confidence Interval Calculator
A confidence interval around a measured accuracy from the number of test items — the Wilson score interval — and the test-set size needed to pin accuracy to a chosen margin.
An accuracy of 85% on 200 items could easily be 80% or 89% on another 200: the Wilson interval gives the range consistent with the count, and it behaves properly near 0% and 100% where the simple ± formula fails.
How the accuracy confidence interval calculator works
An accuracy of 85% on 200 items could easily be 80% or 89% on another 200: the Wilson interval gives the range consistent with the count, and it behaves properly near 0% and 100% where the simple ± formula fails. The sample-size row inverts it: to know accuracy within ±2 points at 95% confidence you need about 1,200 items near 85%, and 2,400 near 50%. Two models whose intervals overlap heavily are not distinguished by the benchmark.
Formula: Wilson: (p + z²/2n ± z√(p(1−p)/n + z²/4n²)) ÷ (1 + z²/n); n ≈ z² p(1−p) ÷ margin²
Worked examples
| Inputs | Accuracy | Note |
|---|---|---|
| 170 of 200 at 95% | 85 % | 85%; 79.4–89.4% |
| 1,700 of 2,000 | 85 % | 85%; 83.4–86.5% |
| A perfect small set: 20 of 20 | 100 % | 100%, but the interval starts at 83.9% |
FAQFrequently asked questions
Why not the simple ± 1.96√(p(1−p)/n)?
That Wald interval collapses to zero width at 0% or 100% and can run outside 0–100%. Wilson’s formula pulls the centre toward 50% by a small amount and stays sensible for small samples and extreme scores.
My model is 2 points ahead — is it better?
Only if the gap is larger than the intervals suggest is chance. The last row gives the smallest difference the test-set size can resolve; benchmarks with a few hundred items cannot separate models within 3–5 points.
Does this apply to other metrics?
To any per-item pass/fail rate — accuracy, exact match, pass@1. F1 and BLEU are not binomial; bootstrap their intervals instead.
What about test-set contamination?
An interval says nothing about bias: if the model saw the test items in training, the accuracy is inflated however narrow the interval. Statistics quantify noise, not leakage.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- National AI Centre — Australia's national AI body
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.