Dashboard

A/B test calculator

A/B test calculator

Enter visitors and conversions for two versions of a page to see whether the difference is real or noise, or plan the sample size before you run the test.

  • Free
  • No signup
  • Runs in your browser
A · control
B · variant

Result

B wins

A difference this large would show up by chance less than 5% of the time if the two versions were really the same.

Change B vs A
+37.6%
4% → 5.50%
p-value
0.014
significant below 0.05
95% range for B − A
+0.30% to +2.71%
in percentage points

Decide the size first and read the result once, at the end: checking every day and stopping at the first “win” finds wins that are not there. Two-sided z-test on two proportions; the range uses the normal approximation.

Test your page on real visitors

A free launch on LarkLaunch sends a week of makers and early adopters to your site: real traffic to learn from, not a guess.

Launch your product

How to use the A/B test calculator

  1. Planning a test: enter today’s conversion rate and the smallest improvement worth finding. The calculator gives the visitors each version needs, and the days it takes at your traffic.
  2. Checking a result: enter visitors and conversions for both versions once the test has reached that size.
  3. Read the verdict, the p-value and the range for the difference, and copy the result into your notes.

An example

Version A converted 96 of 2,400 visitors (4.0%) and version B 131 of 2,380 (5.5%). That is a 38% relative lift with a p-value of 0.014, so at 95% confidence the difference is unlikely to be chance. The 95% range for the gain runs from about 0.3 to 2.7 percentage points: real, but possibly much smaller than it first looks.

Planning is the step most small sites skip. At a 4% conversion rate, finding a 20% relative improvement takes about 10,300 visitors per version; a 50% improvement takes about 1,900. With little traffic, test bold changes.

The method

A two-sided z-test on two proportions, with the confidence range from the normal approximation, and the standard sample size formula for two proportions. These assume you fix the sample size first and look once. Checking every day and stopping at the first significant result makes false wins much more likely. Running whole weeks evens out weekday and weekend visitors.

Testing headlines? The guide to writing a launch listing has variants worth testing.

Frequently asked questions

What does statistically significant mean here?

That a difference as large as the one you saw would rarely happen by chance if the two versions really converted the same. At 95% confidence, "rarely" means less than 5% of the time. It does not tell you the change is large or worth keeping.

Can I stop the test as soon as it is significant?

No. Checking repeatedly and stopping at the first significant result makes false wins far more likely. Decide the sample size first, run the test to that size, then read the result once.

My site has little traffic. Can I still A/B test?

Only big differences. The planner shows how many visitors a test needs; if that takes months, test bolder changes, or talk to users instead of splitting the traffic.