A/B Test Calculator
Whether variant B really beat variant A — the two-proportion test on conversion rates, with the lift, its confidence interval, and an honest note on peeking.
Two conversion rates are compared with a two-proportion z-test: the difference divided by the standard error of that difference under a pooled rate.
How the a/b test calculator works
Two conversion rates are compared with a two-proportion z-test: the difference divided by the standard error of that difference under a pooled rate. The p-value is the chance of a gap this large if the two variants were identical.
The statistics are the easy part. The hard part is that stopping the moment the result looks significant inflates the false-positive rate several times over — so the sample size must be fixed before the test starts.
Formula: z = (p₁ − p₂) / √(p̄(1−p̄)(1/n₁ + 1/n₂))
Worked examples
| Inputs | p-value (two-tailed) | Note |
|---|---|---|
| 5% against 6% on 5,000 each | 0.02829487 | p = 0.031 — B wins |
| Identical rates | 1 | p = 1 |
| The same lift on 500 visitors | 0.48796934 | nowhere near significant |
FAQFrequently asked questions
How long should I run an A/B test?
To a sample size fixed in advance, and for whole weeks — weekday and weekend traffic behave differently.
Why is peeking a problem?
Because checking repeatedly and stopping at the first significant result can push the false-positive rate from 5% to over 25%. If you must monitor continuously, use a sequential testing method designed for it.
What is a realistic lift?
Most real tests move conversion by a few percent relative. Detecting a 5% relative lift on a 5% base rate takes tens of thousands of visitors per variant.
Should I report relative or absolute lift?
Both. "20% lift" and "one percentage point" describe the same result and give very different impressions.
What about testing more than two variants?
Each extra comparison adds its own false-positive chance. Correct for multiplicity, or the winner is often just the luckiest arm.
Where these figures come from
- NIST Digital Library of Mathematical Functions — reference definitions for elementary and special functions
- Wolfram MathWorld — definitions and formulas for every topic on this page
- NIST/SEMATECH e-Handbook of Statistical Methods — the statistical formulas (mean, variance, z, confidence intervals, sample size)
- Australian Curriculum (ACARA) — Mathematics — the terms and methods taught in Australian schools
Last checked: September 2026. Formulas are fixed by mathematics and do not change with tax years or regulations.