A/B Test Significance Calculator
Whether the difference between two variants is real or noise — with the p-value and the confidence interval.
Two variants almost always differ.
How the a/b test significance calculator works
Two variants almost always differ. The question is whether the difference would survive running the test again, and that depends on the sample size as much as the gap.
This uses a two-proportion z-test. A p-value below 0.05 is the conventional threshold for calling a result significant, meaning a difference this large would appear by chance less than one time in twenty if the variants were truly identical.
Two warnings the arithmetic cannot give you. Stopping a test early because it looks significant inflates false positives badly — the p-value assumes you fixed the sample size in advance. And statistical significance is not business significance: a real 0.1% lift may not be worth the engineering.
Formula: z = (p₁ − p₂) / √(p̄(1−p̄)(1/n₁ + 1/n₂))
Worked examples
| Inputs | Verdict | Note |
|---|---|---|
| 250/5000 against 310/5000 | B wins — strong evidence (p = 0.00907) | B wins, p ≈ 0.011 |
| A small sample | no significant difference — this is what noise looks like | not significant |
| No difference | no significant difference — this is what noise looks like | noise |
FAQFrequently asked questions
How do I know if my A/B test is significant?
A p-value below 0.05 is the usual threshold — a difference that large would occur by chance less than one time in twenty if the variants were identical.
Can I stop the test as soon as it looks significant?
No. That inflates false positives substantially, because the p-value assumes the sample size was fixed in advance.
What sample size do I need?
It depends on the effect you want to detect. Small lifts need very large samples — often tens of thousands per variant for a 10% relative change on a 2% base rate.
What does the confidence interval tell me?
The range of true differences consistent with your data. If it spans zero, no effect is ruled out.
Is a significant result always worth shipping?
No. Statistical significance says the effect is probably real, not that it is large enough to matter.
Where these figures come from
- Interactive Advertising Bureau — measurement guidelines — the impression and viewability definitions CPM depends on
- Australian Competition and Consumer Commission — the Australian consumer law regulator
Last checked: September 2026. These are standard industry definitions; where platforms disagree, the page says so.