A/B Test Sample Size & Significance Calculator

Plan an experiment and read its result. Two-tailed z-test for proportions with pooled variance. Type a baseline rate and a minimum detectable effect, see the per-arm sample size and a calendar duration. After the test, paste the two counts and see the p-value, the 95% CI on the difference, and a verdict at your chosen α. Live, in-browser, no upload.

Mode

Plan a test

Pick a baseline rate, the smallest relative lift you want to detect, the confidence level, and the power. The page returns the per-arm sample size and a calendar duration for your daily traffic.

Per-arm sample size — each arm (control + variant)
Total traffic — 2× per-arm
Detected effect (absolute) — |p2 − p1|
Detected effect (relative) — p2 / p1 − 1
Days to run — at your daily traffic
Weeks · months — approximate

Sample size by lift

How the per-arm sample size changes with lift size at your baseline rate and chosen confidence. Useful for "is this test even worth running" decisions.

How this works — formulas, assumptions, caveats

The sample-size formula is the canonical two-tailed z-test for proportions with pooled variance:

n = (z1−α/2 + z1−β)2 · (p1(1−p1) + p2(1−p2)) ÷ (p1 − p2)2

where n is the per-arm sample size and the z values come from the standard-normal inverse CDF. For the most common pairings (95% / 80%, 95% / 90%, 99% / 80%) the values are pinned to a preset table so the result is stable across rebuilds. Non-preset pairs (e.g. 93% / 87%) use the canonical Wichura AS241 algorithm provided by the bell-curve library.

Post-hoc significance

Once the test has run, paste the per-arm visitors and conversions. The page computes the pooled proportion p̂, the standard error under the null, the z-statistic, and the two-tailed p-value. The 95% CI on the difference uses the unpooled SE for a more honest interval.

z = (p2 − p1) / √(p̂(1−p̂)(1/n1 + 1/n2))

Assumptions

  • Independence — observations across and within arms are independent. A user counted in two arms breaks this.
  • Randomised assignment — each visitor is randomly assigned to control or variant.
  • Stable baseline — the conversion rate isn't drifting during the test. Cherry-pick a window with stable traffic.
  • Binary outcome — the formula is for proportions (yes/no). For continuous metrics, the t-test version uses σ instead of the binomial variance.

Why duration is a separate step

The sample size tells you how many observations you need. The duration tells you how long to run the test, which depends on your daily traffic. A test that needs 50,000 visitors takes one day at 50k/day and four months at 400/day. Plan both before you start.

Common pitfalls

  • Peeking — checking the p-value every day and stopping when it's < 0.05 inflates the false-positive rate. Commit to the planned duration before peeking.
  • Stopping early on a "winner" — the planned duration is the floor. Short tests have wide CIs and unstable estimates.
  • Multiple variants without Bonferroni — 3 variants at α = 0.05 inflate the family-wise error rate to ~14%. Use Bonferroni (α / k) or a sequential test.
  • Ignoring novelty & primacy effects — the first week of a test may show a transient lift that fades. A 7-day minimum is a common rule of thumb.

Pure client-side math. The page never sends your numbers anywhere. The standard-normal inverse CDF is provided by the bell-curve library (Wichura AS241, max error < 1.15e-9).